You know, around here we call guys who are allergic to vaginas "gay".
I suspect it's far more likely his way of putting some distance between him and at least some of those 26 million crazy, rabid TwiTards who would love nothing more than to satisfy their own "flesh and blood" desires with the poor guy. I mean, just consider things from his point of view. Every time I've seen him make an appearance on TV, he just looks so hesitant and apprehensive, probably expecting some 14-year old or looney tune housewife to randomly jump on stage to touch him, say that she's his biggest fan, gush about his movies, or harrass him in some other way. Can you blame him for perhaps putting up a fake facade to take the wind out of the sails out of even some of those freaks? I sure don't.
Also, having had some experience with British humor, I think it could've also been meant in jest, maybe even as a sly acknowledgement that yes, he
does actually like it!