It needs square braces, to override the order of evaluation. %flood_ is likely $null, so you end up with several strings joined together into a string that doesn't begin with a percent, followed by a 1. You would need to do something like:
//inc -eu $+ $iif(%d,100,10) % $+ flood_ [ $+ [ $me ] $+ _ $+ [ $me ] $+ _spam ] 1
When you do square-brace evaluation, it helps to check the variables tab of alt-r to see exactly which string is being created, as it's possible to have a variable named %flood_$chan_$nick_spam instead of evaluating the channel name and nick before using it.
If you use hash tables, you can avoid the need to use square braces entirely
//hinc -mu $+ $iif(%d,$v1,10) tablename flood_ $+ $chan $+ _ $+ $nick $+ _spam 1
//echo -a $hget(tablename,flood_ $+ $chan $+ _ $+ $nick $+ _spam)
Hash tables look intimidating to use at first, but they're actually quite simple.
https://en.wikichip.org/wiki/mirc/hash_tables
