What kind of internet connection do you have?
If it's DSL or cable, do you have a router?
Does this happen no matter what the second network is, or is it with any network?
How about the time of day? Might you be hitting a time when your sevice provider might be doing updates/upgrades?

Get back to me with those answers, and I might be able to be of more assistance.