No matter how smart or curious you are, someone out there has you beat by miles.

using the nest to find out when you are home and on vacation and then you got the usual of using the Nest to break into your WiFi.

How can you tell? Roll your own firewall and log every packet that goes in and out of the house. Using deep packet inspection, you can see exactly what every device is doing. Theoretically, Pull up to a house in a van marked with the local cable company logos, splice a sniffer into the wire (hell just ask to enter and 'check the equipment') and now you can monitor internet activity. If we were running a game on a high value individual, within 30 days we have that person's schedule, where they web surf (even if the data is encrypted the metadata has significant worth cue the NSA) when they are home, when they sleep, what sorts of devices they own (ie are they worth conning/robbing?) and can probably get their circle of friends and figure out their net worth based on what websites they go to.

I am conscious that it is an internet appliance, constantly hooked up to the Web, with more telemetry than I'm using (or allowed to use). I also know that the configuration utility presented to me cares precious little about security.

Is everything above possible? Right now, hell yes it is. Thanks to Snowden we know the NSA, FBI and UK governments do this to people they want to monitor. Our main advantage is that we are nobodies and below the radar. I'm not going to do anything more disruptive to the government other than write some campaign contribution checks and bitch about my tax bill this year. I'm much more worried about some Central European Bitcoin gangs who can sit in a room and figure this stuff out because stealing $4000 feeds and shelters them for three months. The risk/reward dynamic gets set all out of whack when you know the local PD don't care about "internet" crime overseas.

For the record, I do not have any internet connected "smart" devices in the house other than the computers, cell phones, tablet et al I use to work and read. And with those devices alone the above scenario would be to my detriment, no smart devices needed. My lack of ownership of these devices is not due to paranoia but to my being a cheap bastard who can live with a $20 thermostat I got off Amazon when I got the house, and the fire alarms are all standard 10 year dumb devices because they are cheap, subsidized by the local fire department, and they work. If i had kids, I can see that changing.

Is this something that 'normal' people should worry about? Worry, probably not. Freak out over? Definitely not. Have in the back of their heads? Absolutely. But I'm starting to see that there is good money to be made in helping the normals secure their stuff against the bad guys.