That's a big worry :/
In Firefox I always have Flash set to "Ask To Activate", which is one of the options in the Add-ons preferences (in Firefox anyway). That way you can always choose which sites get to access and load it, and at what times. It's especially good for sites that have run-away levels of Flash advertising (I'm looking at you, Wowhead!). I don't resent sites funding through advertising, but there are limits to what how much intrusive crap I'll put up with
Anyway, setting to to "Ask" may be a decent compromise, as you can control it to only things you really trust.