There's always an underlying model; the only question is whether or not the analysts know what that model is. If you don't understand why your results are what they are, your analysis has no predictive power!
This is what gets me with people who believe that AI will lead to scientific results beyond the ability of humans to comprehend. What's the point? Humans have a hard enough time listening to other humans, even if their arguments are well-reasoned and easy to understand! How are we going to get society to blindly follow what machines say?
I think that's the easier bit. Everything computers do is magic to people who don't understand computers, and everything that predictive modelling and big data do is magic to people who don't understand both computers and math. People have historically listened to magic that wasn't real at all, such as astrology, once it had enough cultural inertia to not be new and weird. And although big data is definitely still in the "new and weird" category, it might not always be that way.
I'm more concerned that we will believe the machines, even though we don't understand why they're saying what they're saying. Let's say an HR department at a large company starts using a predictive model to predict who will be a successful employee at their company (e.g. low likelihood of being terminated or quitting in <2 years). They feel that have to do something like this to reduce numbers, even if they don't wholly trust the magic, since they get so many applicants and don't have that many openings. After they start using the model, it appears to be working (the two-year attrition rate goes down), so they keep using. Word gets around, and all other companies in their sector start using it, with similar results. This effectively partitions the workforce into two groups: the small subset of "hireables" who can be hired at any of the companies that use this model, and move freely between them, and the large group of "unhireables" who cannot find a job anywhere and remain unemployed permanently.
Unfortunately, it's entirely plausible that a large majority of the "unhireables" (let's say even 80% of them) also would not have left their job in less than two years and are perfectly capable workers, they just share some qualities with the 20% of "unhireables" that actually shouldn't have been hired. As a result, these perfectly fine workers who would have been able to find work can never find employment, for no other reason than "the algorithm says so." Although there may be ways to flag this sort of error, most companies who don't understand the model they're using aren't likely to do that, especially if the model continues to have the desired result. As a result, we end up with a permanent population of people who don't know why they can't find a job, even when some of their friends have no trouble getting work, which certainly doesn't sound healthy for anyone.