You can't return a &bvar this way, here $hex2bvar will return "&bvar", [...]
The way mIRC handles binary variables is very primitive. All binary variables are globally defined and persist until the entire script finishes processing. (In other words, even though the binary variable was first created from inside the $hex2bvar call, the variable will still exist and can be accessed after returning to the alias that called $hex2bvar.
Since the variable is globally defined, returning the NAME of the binary variable is fine. The /sockwrite command will be able to read the contents of that binary variable.
By the way, if you are making heavy use of binary variables you may want to be careful about overlapping binary variable names since they are defined globally!