This topic caught my eye immediately since my hand is in several of these aspects of the LEGO after market since I primarily buy and sell common used sets, but every now and then opportunity comes up to sell highly collectible sets.
The biggest window of opportunity has probably already passed since there are certain collectible themes like UCS, Landmarks, Minifigures and Modulars that are most likely going to be a continuous theme, and the early sets were retired much more quietly. People get hooked on the newer, more readily available sets and want to go back and complete their "collection." I don't think there is anything necessarily wrong with that, but the Green Grocer buyers certainly have an upper hand since I can't imagine nearly as many units were sold of Green Grocer as the Pet Shop, I imagine. As more and more people have gotten into these themes, LEGO has made new ones more available so the reselling market has dried up a bit. To really profit from speculation you have to guess which new theme will catch on and have sets retired the quickest. If you picked Pharoah's Quest, good luck to you; it's just not a lasting theme, even though the sets are good.
It does seem like you could have a much safer investment in other things than LEGO: a fire could easily wipe out your investment and how are you going to explain that to your insurance company? The space required would be crazy, and space doesn't come free either. You wouldn't want to leave them in sunshine or damp conditions. It's just not a practical long term financial investment. I am not aware of many opportunities to flip within a year or two.
There are certainly bubbles of demand that do occur like The Republic Gunship; for a while set 7676-1 was selling very well for me, but then the price dropped significantly off when the new version of the gunship was released. Just imagine what might happen if LEGO released a better interior-ed version of Green Grocer or Corner Cafe?
The price of what you pay now, I don't think has anything to do with speculation, to be honest. LEGO is a very popular children's toy with millions of buyers that far out weigh the speculators.
As far as retiring sets, they are just keeping in step with the toy industry, in general. The toy industry moves very quickly; and they are trying to stay ahead of any potential competitors.