Physical distance means nothing on the internet.

That's true to an extent but you are forgetting that nothing in this world is 100% efficient. Electrical resistance will always be a factor in a latency test. Even fibre-optics isn't 100% efficient or infact anywhere near it. It just happens to be alot better than a copper pair.