There is some traffic on tracks, actually. Acela trains have priority over all other trains and therefore if an Acela and a different train, like a regular Amtrak, are both approaching, the Amtrak will have to slow down on the rails and stop until the Acela pulls in, releases & loads passengers, etc. I don't know if that excuses a full 2.5 hour long wait, and the fact that Amtrak is notoriously unreliable and often late makes me very angry, but I know this is one factor for why trains are delayed.
I have had to ride on trains quite a lot. I have seen them delayed to pull into stations quite frequently. I have also been on a train or two that had to stop on the rails and restart the engine because they were having technical issues. I suspect these are among the most common "delay factors" that we experience. Weather and car crashes, debris on the rails, etc, can all slow trains down. However I too struggle with the idea that these factors could conceivably delay a train for so long. I find Amtrak very frustrating, and I agree - I never count on the train being on time.