I feel this in my bones. As my interests get more niche, I feel like I'm slightly (and quite unintentionally) pulling away from some of my friends. I bonded with the people I'm closest with partly through our shared interests (then music, anime, video games, tabletop games). Now that I don't watch shows or play games much at all without them, I feel like I have less to talk about with them. They will always be irreplaceable, but the slight distance is saddening.
I am making new friends with my newer interests, but as you allude to, the process is difficult and takes a lot of time. I wish you the best of luck.