The ! prefix is reserved (a special symbol that makes the event trigger for everybody except you), so I don't think you'll be able to use it for that purpose. So, you need an /if statement that checks if you are !'ed:
on !*:join:#: if $nick(#,$me,[color:green]![/color]) { do your stuff }
Yes, you can use mode symbols too with $nick()