Baker began using heroin in the 1950s, resulting in an addiction that lasted the remainder of his life. At times, Baker pawned his instruments for money to maintain his drug habit. In the early 1960s, he served more than a year in prison in Italy on drug charges; he was later expelled from both West Germany and the UK for drug-related offenses. Baker was eventually deported from West Germany to the United States after running afoul of the law there a second time. He settled in Milpitas in northern California, where he played in San Jose and San Francisco between short jail terms served for prescription fraud.

In 1968, Baker was savagely beaten (allegedly while attempting to buy drugs) after a gig in The Trident restaurant in Sausalito, California sustaining severe cuts on the lips and broken front teeth, which ruined his embouchure. He stated in the film Let's Get Lost that an acquaintance attempted to rob him one night but backed off, only to return the next night with a group of several men who chased him. He entered a car and became surrounded. Instead of rescuing him, the people inside the car pushed him back out onto the street, where the chase by his attackers continued, and subsequently he was beaten to the point that his teeth, never in good condition to begin with, were knocked out, leaving him without the ability to play his horn. He took odd jobs, among them pumping gas. Meanwhile he was fitted for dentures and worked on his embouchure. Three months later he got a gig in New York.

After developing a new embouchure resulting from dentures, Baker returned to the straight-ahead jazz that began his career. He relocated to New York City and began performing and recording again, including with guitarist Jim Hall. Later in the 1970s, Baker returned to Europe, where he was assisted by his friend Diane Vavra, who took care of his personal needs and otherwise helped him during his recording and performance dates.

  A kiss is still a kiss

  A sigh is still (just) a sigh

  The fundamental things apply

  As time goes by