Im{Aging} edit: that is to say, our perception of the "passage" of time may be a unique condition imposed on consciousness. But now we're metaphysical, and there's not really any way to debate such things. Need maths.
Can't find the earlier comment, but there was an idea that even if it's possible to move back and forth in time, or stay in one place, we can only ever be aware of moving in the direction of increasing entropy, and we call that direction "forward." To be aware of the passage of time, you need to form memories. Forming memories requires work, in the thermodynamic sense. Work that creates memories increases entropy, so memories will only exist at a time with more universal entropy than existed when the remembered events occurred. Something like that, maybe you can sprinkle some math in.