It's a book series about a race (or group, or caste, or aristocracy) of privileged people (wizards) who have power (magic) over a race/group/caste/proletariat of the underprivileged (muggles).

More than that, it's a book series about a group of active people who want to use their power and a group of passive people who will allow that group of active people to do pretty much whatever the fuck they want so long as it doesn't disturb the status quo. The only real villain in Harry Potter - the only faction with zero sympathetic characters, the only group whose motives are never made anything other than execrable, are the press. And yeah - the British press are execrable but fuckin' hell woman I like how the stand you took was against the tabloids who were fucking with your life.

The point where you start to run into problems is when you start naturalizing this hierarchy.

The fundamental issue Rowling faced was the tension between the wizarding world and the muggles couldn't be maintained through anything other than fascism. If wizards are nice and muggles are nice and the only thing keeping them apart is the MInistry of Magic, the MInistry of Magic has to be dicks. And if the goal of your series is to preserve the hegemony of the Ministry of Magic, you get to be the nazis.

It's aristocratic, colonialist, racist, Tory bullshit: a white savior complex in kid-friendly form.

And more than that, you only get to be the savior if you follow the rules to the letter.