I have no experience with twitch, but this might actually be intentional: with channels that large, the joins/parts/quits would create an incredible amount of useless noise, and for that reason, some IRC servers support an "auditorium mode" (or something similar) which hides channel users from each other.
