it think this explains the problem:
//echo -a $regsubex(a,/(.)/,$(\1,0))
returns $1.
$regsubex(12345,/(\d)/g,X $regsubex(\1,/\d/,x) \1) = X x 1XXXX
will only work once because the 2nd regsubex tokenizes again leading to $0 being 0 and therefor \1 ($1) doesnt excist anymore when it does the substitution for 2.
//echo -a $regsubex(12345,/(\d)/g,X $regsubex(\1,/\d/,x) $(\0))
X x 5X 0X 0X 0X 0