Having flown in an air stack many times with Predators in pattern with me, I fear a pred flying absent-mindedly through my air space far more than the Taliban actually shooting worth a damn. The mid-air collision system would probably go off once a month as a pred tried to kill us, not maliciously, but because the guy flying it in Creech had spilled coffee on the keyboard (at least that's what we said).
There's a lot of factors going on to make that job shitty. I think the biggest one is that you have guys and ladies who grew up wanting nothing more in the world than to be a fighter pilot. They go through all the training, they graduate high in their class, and the Air Force says, "Yeah, we don't need fighter pilots to soar with the eagles and chase down Migs. We need a drone pilot because only officers have kill authority in the Air Force from aircraft. So you're a drone pilot. You'll never fly again.