Two problems. First, your brackets are messed up some there. And second, putting the IF directly in the timer will check the status immediately rather than when the timer triggers. You'd need to prevent the evaluation.
.timer 1 5 if ($($nick isreg $chan,0)) { mode $chan +h $nick }
*** That *might* work. Normally, you can prevent the pre-evaluation this way, but I've never tried with an isreg type of check and can't test it right now to find out. Try it out and see. Have someone join and then you op them immediately (or voice) and see if it triggers. Then try without opping/voicing them and see if it triggers. If it doesn't work, use the alias or someone else can show the right way to handle this.