I'm not saying that Feynman's lectures are easy. I'm saying that IMO they are a lot easier than TAOCP. Possible bias notwithstanding, Feynman uses less advanced mathematics (seriously), so there's your barrier to entry difference right here. Conceptually they are probably around the same difficulty, but Feynman's presentation is – again, IMO – much more approachable.
Also, TAOCP uses some oddball defined-in-the-book variant of assembler for all code examples, which isn't the friendliest language out there. Getting through 200+ lines of asm code is most definitely one of those 'acquired taste' things.