Here's a related problem that helps to understand the coin problem: I have two children. The youngest is a boy. What is the probability both are boys? The answer is 1/2. Now compare to this: I have two children. One of them is a boy. What is the probability that both are boys? It is not 1/2, as it is not specified which child is a boy in the initial information. To solve this, we consider all possibilities for two children: BB, BG, BG and GG. Since we are told one child is a boy, we eliminate GG as a possibility. There are then 3 equally likely possibilities: BB, BG and GB. The chance that both are boys are 1/3, and the chance that one is a boy and one is girl is 2/3. With the coin problem, each head flipped adds some information and the probability changes in an equally weird kind of way...
This is sort of right but there are some underlying assumptions that it glosses over. Situation (a): We take a set of parents which all have 2 children. Select the group in which the youngest is a boy. What fraction of that group has two boys? 1/2, as stated in your first situation. Situation (b): Take a set of parents which all have 2 children. Select the group in which at least 1 is a boy and send the others home. What fraction of that group has two boys? 1/3, as stated in your second situation. Situation (c): Take a set of parents which all have 2 children. Select a group in which the youngest is a boy and send the others home. Tell the stranger "one of them is a boy". What fraction of that group two boys? The GGs and BGs left, the BBs and GBs stayed. So it's 1/2, same as situation (a). Situation (d): Take a set of of parents which all have 2 children. Have them pick one of their kids at random. If the kid they picked is a girl, send them home. Keep the remainder and tell the stranger of these parents children, "one of them is a boy" (which is true). What fraction of that group has two boys? 1/2, same as (a). The GGs all left, the BBs all stayed, half the BG and half the GB stayed, the pool will be equally divided between 2 boys and 1boy/1girl. In situations (c) and (d) we generated the samples in different ways, but phrased the statement in accordance with situation (b). Misleading maybe, but not technically wrong. The manner in which the children were selected guaranteed that at least 1 would be a boy. The normal intuition (at least my intuition) in the "one of them is a boy" statement is that the parent picked a one of their children at random and then stated its sex (first part of situation (d)). Were that the case, the probability of the other child being a boy would be 1/2. What this comes down to is the sample generating process matters, and the language describing a problem contains clues but they may not be fully fleshed out. This is known as the "Boy or Girl Paradox", see wikipedia for more details.
We guess. Or use domain specific knowledge, if available. Imagine playing 3-card monte with your friend (who you know has no history of doing card tricks, possibly has learned something knew in the last week but nothing major) versus somebody on the street. You'd assign different probabilities of yourself winning in each case, right? Say I flip a coin once and you have to guess the face. You ask if the coin is fair. I answer "Unknown". One could assume that it's probably fair, and if it's unfair it's equally likely to be unfair in either direction, in which case it's 0.5 each (for the first flip only). b_b mentioned Bayesian inference, that's a way to include prior knowledge. But of course people with different prior assumptions will get different answers. So it goes.
My gut says that many there aren't always clear ways to mathematize statistics problems (even in the case where we can count all possible outcomes, perhaps counter-intuitively). In every day life, we sue Bayesian inference to make qualitative or semi-quantitative judgments about what is likely, and the "hard numbers", so to speak, are irrelevant. I think in the fake coin problem, we run into this difficulty. The fake or non-fake coin has already been selected, so there isn't really a 17/18 chance that heads will be tossed. There's really a 100% or a 50% chance heads will be tossed, depending on the condition of coin selection. Since there's only a 12.5% chance that we'll toss three heads consecutively with a fair coin (on an independent trial that consists of three coin flips), most of us would infer that we have selected the double head coin and assume that heads will be tossed indefinitely unless or until we are proved wrong empirically. With each successive heads, we become more convinced of our assumption, even though it is entirely possible (though unlikely) that 5 or 6 heads in a row could happen with a fair coin. Conditional probabilities don't add or multiply in the same way that independent trials do, so often we are left with assumptions, qualitative judgments and previous experience to guide us. I think this is probably unsatisfactory on a mathematical level, but it is very helpful on a behavioral level.
I think your gut is right that there are not always clear ways to mathematize statistics problems, which is what accounts for debates about these kind of problems! I do think the calculation for a 17/18 chance in this problem is both satisfying and satisfactory. It is spot-on for gambling, the fair odds for making a bet, and it reflects mathematically what you say of becoming more convinced of our assumption. It can be very strange how knowledge changes probabilities... and it is not always clear how to recalculate based on the value of that knowledge!