This is true, but for colloquialisms to be understood and taken in, context is necessary. Again, if languages are very similar to one's L1, colloquialisms will more likely make sense to the learner. Take English and French for example. For an English learner of French, the phrase "cherchez la femme" might be familiar, but translating it (look for the woman) into their L1 is likely to convey a similar meaning in context.

If we take an English learner of Vietnamese (which I speak a bit of and have trouble with) then the idioms and colloquialisms will seem very strange unless or until one has sufficient context for them and a certain amount of accepting those idiomatic quirks as "just the way the language is." For example, a common phrase "không sao đâu" literally translates to "no stars where" but colloquially, it means "never mind".

It's hard to become a fluent speaker of languages that are vastly different from one's own using only immersion or only classes. What the literature suggests (which I support, due to my own experiences in teaching) is that learners need high context environments, dealing with realia (real examples of the languages as native speakers use it) supported by exercises designed to break down reaction time and to eliminate word for word translation from L1 to L2. Learners are also helped to acquire language by other learners rather than being taught to as is found in traditional teacher-centric classrooms as it allows greater opportunity for self-assessment, monitoring and comparison of language use.