Only thing I can think of is that your script is trying to get the number of people in the channel, before your /join completes.
The following code, should solve the problem.
on *:invite:#:{ .join # }
on *:join:#:{
if $nick(#,0) > 5 { .notice $nick Hi $nick $+ ! }
else .notice $nick Error, less than 5 people in the room | .part #
}