I think the fact you are asking this question shows some key weaknesses in your story/plot/method.

The method is a distraction. It's not the scary part.

What is authentically scary is why someone does something.

Have your killer kill his victims by pulling their hangnails until they die. That'll squick people out, but the scary part is that he does it because he has a compulsion to pick at things. He picks at the gum stuck under the railings in the subway station. He picks at the edge of peeling stickers. He has a picking compulsion, and THAT is why he picks their hangnails until they die.

Because here's the thing: You are asking the question like a man would. "What could he DO to someone?" That's scary for a man, but emotional torture is far more scary to a woman.

Take the movie Misery, for example. The guys I know completely lose their shit when she breaks his ankles with the sledgehammer. The women I know cringe, but then get on with it.

Women process pain differently than men, and they spend a lot more time in the mental space around a situation, considering all the angles and impulses and consequences, and torture themselves over the mental aspects, much more than the physical aspects of torture.

So. Back to the hangnails.

If the dude has a picking fetish, and only tortures the women to death by picking their hangnails because he has this compulsion - it has nothing to do with the woman at all - then it is FAR more scarier. Because you have just removed the woman's humanity. She isn't a person at all, any more. She is just an object he picks at, like gum.

THAT is some fucked up shit right there.

So anyway, in asking your question I think you have highlighted the core problems with your story more eloquently than any type of invented ridiculous torture can fix.

(Footnote: That's why I don't write stuff like this. It is too easy, and there is too much of this humans-being-shitty-to-each-other crap our there already. It's easy. It's done. There isn't really any originality in the genre any more, other than just ratcheting up the "Eeeewwww" factor until people walk out. No offense intended. Solely my opinion, expressed on absolutely zero knowledge of you, your writing, or anything. So take this all with a big old 5lb block of salt! Welcome to hubski!)

This question concerns me slightly.

And I'm not a woman, but I can tell you this is super subjective and you really should be able to better know the answer for your book characters than any real woman could tell you what is or isn't horrifying to them.