But the assessed value will still increase as the buying frenzy increases and soon your taxes will take more than you can afford.

The rich folks have money. They will spend it on things that will make them more money. Once they reach critical mass, you can't stop them.