No, I was told by someone else as confirmation after I had figured it out. He will find out one way or another. And it will break him. He will probably be able to get himself back together again, but there's a strong possibility you won't be there to see that.
I'm happy for you that you don't feel guilty, that must make it a lot easier for you to look him in the eye. But the fact it's just casual sex and not an emotional relationship will make absolutely no difference to him.
You aren't just destroying your marriage, you're destroying his social circle too. He won't know who knew, who helped cover it up, and worse, who he can trust. He may never recover. I have one friend from before my ex-wife's infidelity. One. Sure I see most of the others every so often. I can smile and joke with them. But I no longer see them as friends and do not trust them.
Just because you get a kick out of something and think you can cover it up; it doesn't mean it's okay to do it.