I think the difficulty a non-native English speaker might have reading an English-language poem would echo the difficulties any reader would encounter, when they delve into creative writing (& yes especially poems) in languages that aren't their inundated born-and-bred own.
That is to say I'd think the inherent difficulty there would center more around said poem's potential use of slang, idioms, bastardized idioms, mixed metaphors and one-off one-word half-references, than word-by-misspelled-or-poorly-grammared-word comprehension. I think poems especially offer great avenues for playing with words, language, sound and sense. They also beg to fill up with echoes of other rhymes, lines, poems, lyrics and evocative works.
So it's hard to learn all the fables & folk songs of a different culture which speaks a different language; it's hard to know their pop music and puzzle out whether a turn of phrase is a bastardized idiom or an original, surreal or imagistic or otherwise unexpected use of words language invention.
It would be hard to read a poem in a language you didn't grow up with because poems are more like quilts than they're like comforters - sure both of them make warm blankets but one's made out of blocks and patterns within patterns and fabrics from the clothes you grew up wearing out but couldn't part with.
The other one's like two colors or has the same generic printed pattern of lines or polka dots which Target and a $50 price limit combine to guarantee.
Both will keep you warm but one you'll throw out after it gets ragged and starts pooping increasingly-gross-graying cotton batting from its broken corners or side seams, approx 3-5 years after purchase.
Sure it might be dramatic but basically what I'm saying is, it ain't the text that I expect would throw you. It's all the murky subtext you can't look up on genius or urban dictionary.