This is interesting. What do we do in cases when we don't know the selection process? Is there an assumption-less way to proceed? Thorny. And it's going to bother me now...

We guess. Or use domain specific knowledge, if available. Imagine playing 3-card monte with your friend (who you know has no history of doing card tricks, possibly has learned something knew in the last week but nothing major) versus somebody on the street. You'd assign different probabilities of yourself winning in each case, right?

Say I flip a coin once and you have to guess the face. You ask if the coin is fair. I answer "Unknown". One could assume that it's probably fair, and if it's unfair it's equally likely to be unfair in either direction, in which case it's 0.5 each (for the first flip only).

b_b mentioned Bayesian inference, that's a way to include prior knowledge. But of course people with different prior assumptions will get different answers. So it goes.

My gut says that many there aren't always clear ways to mathematize statistics problems (even in the case where we can count all possible outcomes, perhaps counter-intuitively). In every day life, we sue Bayesian inference to make qualitative or semi-quantitative judgments about what is likely, and the "hard numbers", so to speak, are irrelevant. I think in the fake coin problem, we run into this difficulty. The fake or non-fake coin has already been selected, so there isn't *really* a 17/18 chance that heads will be tossed. There's really a 100% or a 50% chance heads will be tossed, depending on the condition of coin selection. Since there's only a 12.5% chance that we'll toss three heads consecutively with a fair coin (on an independent trial that consists of three coin flips), most of us would infer that we have selected the double head coin and assume that heads will be tossed indefinitely unless or until we are proved wrong empirically. With each successive heads, we become more convinced of our assumption, even though it is entirely possible (though unlikely) that 5 or 6 heads in a row could happen with a fair coin. Conditional probabilities don't add or multiply in the same way that independent trials do, so often we are left with assumptions, qualitative judgments and previous experience to guide us. I think this is probably unsatisfactory on a mathematical level, but it is very helpful on a behavioral level.

I think your gut is right that there are not always clear ways to mathematize statistics problems, which is what accounts for debates about these kind of problems!

I do think the calculation for a 17/18 chance in this problem is both satisfying and satisfactory. It is spot-on for gambling, the fair odds for making a bet, and it reflects mathematically what you say of becoming more convinced of our assumption.

It can be very strange how knowledge changes probabilities... and it is not always clear how to recalculate based on the value of that knowledge!