This is a drivers problem that I've had myself with some network cards, the problem is solved by making sure the network card has its own IRQ thats not shared with anything else.
The documentation usually mentions this.