Yep, looks like a bug. It also happens if you use if instead of while.
You get an error because mIRC evaluates %a before it's passed to /inc, which results in an invalid /inc 1.
This works:
//var %a = 1 | while (%a != 5) !inc % [color:blue]$+ a[/color]
