I think a big difference here is that with other species, our manipulation of their genetics doesn't change very much.

Perhaps this is more of a misconception since we don't really feel the impact of modifying other species? I would argue that genetically modifying other species can have a bigger impact than our own. For the simple fact that other species (excepting pets) are generally integrated in an ecosystem, so changes made to them will inevitably affect the whole ecosystem web with consequences that we can't really predict. Also they will self-replicate and eventually extinguish the native species, whipping out millions of years of evolution of unique genetic material.

On the other hand, we humans have managed mastery above nature and have distanced ourselves from it enough where changes to ourselves shouldn't affect the rest of nature. That's my perspective.

We should have had this debate when we started genetically modifying plant crops in the early 90's. Now the GM ball is rolling it'll be much harder to stop it, if possible at all.