Testes size indicates that we are between the chimpanzee and gorilla - not closer to the gorilla. In fact, in terms of testes size to body size ratio we are closer to the chimp; gorillas have almost no testicles. I agree with you that sperm competition was reduced in our lineage - however, understanding the degree to which it was reduced is difficult. To me, our testicle size indicates that we evolved in a system where a minority of men had the ability to practice polygyny, with other males defaulting to monogamy (with infidelity) and/or serial monogamy.
If the ratio of the testicle size to the size of the female (the most thoughtful metric as sexual dimorphism can be the cause of small equipment) is used we are much closer to Orangutans than anything else. (pan and gorilla are at freakish extremes) And as everyone knows they are bigamists (and in some instances rapists) human female fidelity and male promiscuity is the received wisdom. But if I remember right american male cheating is about 30% and female 15% and non- paternity is about 10% (I don't have the study in front of me.) Non-paternity in "mates for life species" tends to be around 30%. So humans are near saints when it comes to things sexual.