I wonder how low of a ping would be required to have it still be immersive. I know that something like 50ms is playable online, but it might be a lot less for real time things, in order not to be disorienting. I've found a site that says people can tell if something is off by 1/300^th of a second, which is about 91.44ms. However, that's just being able to tell if a light flashed for 1/300^th of a second, and they were trained pilots. I'm getting <1 ms pings (Sorry for lack of screenshot) from the router from my laptop, and I'm on WiFi (Granted, I'm in the same room as it and everyone's asleep), but I've seen people get up to 4ms. The only thing that might really affect that is the processing, but it would be no different than having the device do it already. I don't know, someone smarter than me would need to look at the actual numbers, but I think it would work.
its when you move your head and the rendered stuff lags behind it will wobble. Even a tiny amount of latency would be easily visible. It's a bit like how looking at a fluorescent light directly, you can't see it's 60Hz flicker, but if you move your eyes left to right quickly, you can discern it. John Carmack has studied it extensively and I would consider him to be the expert these days on that stuff as he works for oculus.