This seems a timeless conundrum. More often than not, I consider that the 'lesser of two evils' to be the pragmatic move. Reading history, I've come to the conclusion that no matter how well-intentioned, any minority party will most likely screw up terribly if they are quickly given power. It's very easy to hate on politicians, and I don't think that most people are honest with themselves about all that goes into governing, especially in a representative/democratic society. My view is that voting in the direction that you want to see your State move is usually more likely to enfranchise the viewpoints and policies that your favored (but unelectable) candidate has, rather than going with the protest vote. In the US we can look back and discuss how many times the Democrats or Republicans have taken power, however, what those parties represent has not been consistent over that period of time. Thus, as much as I see my vote going for a particular candidate, I also see it as a vote for a cultural shift.
I guess the biggest problem with protest voting is it doesn't say much other than "I don't like any of these guys". Its demands are little more than "give me other options". So, as a strategy for policy change, it seems ineffective. Perhaps if united with a large-scale civil movement, it may work, but not by itself. Wikipedia has some historical examples of major voter abstention.