If a connection has been made to that port recently and you then close the listening connection, Windows places that address/port in a TIME_WAIT state, which means that it is still busy and cannot be used until Windows makes it available again. You can learn more about why Windows behaves this way by searcing for TIME_WAIT through Google.
