It can be a problem with your ISP (If you are not using the same ISP for example).
Another idea that I have is that your MTU is not optimized.
MTU = Maximum Transfer Unit. which is the maximum size of a data packet that windows can send.
I had a situation that the MTU wasn't in its best and it slowed down my internet.
Search for some internet tweaking tools that optimize the MTU, and see if they manage to solve the problem.
