Problem at your ISP? Did you try disable your antivirus and firewall? Also make a traserout to the server, then you maybe see where the problem are. If you have high ping to one connection on the way, then the problem can be there.