I've been watching plenty of Markiplier's Let's Plays of horror games lately, and at some point I caught myself thinking that those aren't really scary. Yes, they have jumpscares, yes, they have the darkness, and typical horror music and so on, but they aren't scary. They don't terrify, don't make me grab to my seat and look around cautiously.
My reaction may be dulled by the fact that I'm watching someone else play the game and audiocomment over it (in case of Markiplier, with a small webcam segment as well). Even then, watching Birgirpall play through Alien: Isolation while exchanging jokes with Banzaii made me engaged into the game process even though I wasn't in any part involved in it. The game made me care about it even though I was behind two screens, and through that care, I was able to feel the horror it was supposed to represent.
I have a few ideas on the matter and I wonder what everyone else have uncovered so far.
First of all, it's the feeling of helplessness even while being in control of your character and their actions. In Alien: Isolation, you're hunted by a giant alien who's so much more powerful than you and can tear you apart if you make the wrong move. As you move through the game, you find tools to either distract the alien or even fight it enough to make it go away for a time. Then, the androids at the station get unconditionally aggressive towards the character and will hunt you down unless you escape or kill them, while displaying no emotions whatsoever, which leads me to another point.
The atmosphere. It's very hard to nail, and I'm not certain what exactly that makes it truly mortifying. Apparently, the darkness plays a lot into it, which is most of the horror games are set either at night or underground/inside. What I believe to be true is the game world's lack of care for the character: the fewer helpful tools and bonuses there are, the more helpless you feel, the more scared you feel. So, scarcity? Does darkness matter that much? Can you make a broad-daylight horror that would terrify the player?
Sound isolation can be important. Absolute silence terrifies more than sad violins or the constant noises in the corner, for then you have no idea what's coming. I've never seen a game that utilised that.
Jumpscares are oddly satisfying to experience, but are they important to the horror? Many games do good with it (the most recent example I've seen being Terrorift, here played by Markiplier using Oculus Rift), just as many simply place a thing that's not terrifying at all in front of you.
How about subtlety? It's not as simple to implement, but if done correctly, it can add the subconscious "Something's not right here" feeling while preserving the overall unbroken tension, to break it harder later.
The absolutely most terrifying game I've ever played is Kuon for the PS2. I'm getting chills up my spine and on my head just thinking about this fucked up masterpiece. You can either play as one of two sisters. One's a melee character, aka you get a letter opener (yes, literal letter opener) to fight with. The other is a magic user. You get the world's worst fireball, or some other craptastic spell. You're investigating some Japanese manor where things aren't going so well for anyone. There are gaki, Gollum-esque demons that comprise the majority of your enemies at the start, gooey, pink masses of bones and organs, and, of course, the typical long-black-hair-over-face ghost girl. She lives in a trunk or something.
I was probably 14 or 15 when I played this game. Found it in the corner of an EB Games and picked it up immediately. I'd just run through a few Resident Evils and Silent Hills, but this was a whole new level of insanity. The atmosphere around this mansion just screams at you to either run away or become an arsonist, but neither of these are options, sadly. Pretty much everything will kick your ass. There's about no sound except for your tiny feet against the wooden floor. The whole game makes you feel helpless, at least up until the point where I quit and didn't look back.
The first boss of the game is this guy who has had his arms and legs broken. He crawls around on the ceiling and shit looking like a spider. Oh, his throat is also slit and his head is rotated around 180 degrees. After dying to him a couple times, I bailed on the game.
I think you're right on the atmosphere. It really does have to play into our primal fears. Darkness, isolation, unknown and/or unassailable predators. Less is more, as well. Less light, weapons, friends, information, sound, direction. Obviously, you can't just put a person in a pitch-black room and say "here you go, time to be scared!" The game has to be engaging enough to make the player want to continue forward. I think that the original Resident Evil did pretty well with that. Ammo was fairly limited, your team was separated and dying, this giant mansion had traps galore that you'd trigger accidentally, plus zombies and friends. The notes gave you glimpses into the backstory, just enough information for you to form a picture of what may have happened. There wasn't a big, glowing "go here" arrow, just what you could deduce from your surroundings.
The only way I can imagine a broad-daylight horror game working is if you, the character, knew something was wrong, but nobody else believed you. Could be that he's having a break with reality. I don't know how the whole story would work beyond that, though.