I thought this was the case for hats that could be considered part of an outfit, but not all hats. For example, if I'm cross dressing, and wearing a "boy hat" then I should be expected to follow boy rules. (No baseball hats inside. No rain hats inside. Floofy easter hats may be kept on.)