Saying someone is 'big enough and ugly enough' to defend themselves is far from an insult and is more a term of endearment where I come from!Can I assume you are Australian then? Because here the term "big and ugly enough" is infact used 'lightly' to describe someone as tough enough to go their own way, though we also have a habit of minding our own business.
