you need to use /server irc.server.address [port] [password] (so you're doing it right in that case)
When I do this, I get a "connection refused" message.
I just found out that a friend has the same problem using this command, so it would point to a problem rather on the server's end.