As a very experienced player in Dota 2 my advice to you would be to make good use of the mute button that appears near the names of players on the scoreboard. As for the penalty for leaving unfinished games, I know that the mode "Limited Heroes" has no leaving penalty, it's mainly for practice as you play heroes you are unfamiliar with but it's downside is that players will leave if they aren't playing well and it can ruin your experience of the game. The only alternative to this would be custom maps which are really fun and due to their multiplayer nature, most don't require teammates and hence players disconnecting will have little effect on your player experience. Good luck and have fun!
It's actually been a pretty positive experience with Dota 2. I've met maybe two people who have mics and rarely did they speak at all. I am aware of the mute feature which I will use if it ever gets to that point. I'm actually really excited for the Beta because of the custom maps. Warcraft 3 was a lot of fun but my parents wouldn't let me play it online when I was younger. Now that I'm on my own, it's great to see the community has sorta moved there and I hope my other friends who don't like dota that much might give it a shot.