The latter. The "bad guy" could still obtain the text on the channel by eavesdropping on the connection of the one person that does not have a secure connection. The fact that your connection is secure, does not prevent that.
As they say, security is only as good as its weakest link.
Note that your whole connection to the server is using either SSL or plaintext - there is no such thing as using SSL for only some channels. I believe that some networks that support SSL, also feature a channel flag that requires all people on the channel to be using SSL, though.
