My formative Dr Who period was in the late 70's to early 80's, with the ABC repeating Jon Pertwee and Tom Baker stories until the new seasons were delivered. With that foundation in place, I watched as it went through Peter Davison, Colin Baker, and Sylvester McCoy.
Tom Baker will remain my favourite for being showing the Doctor as a manic genius with a penchant for flippancy in the face of danger, and utter weirdness. Jon Pertwee was great fun, and the stories were magnificent, but his character didn't grab me as much as Tom's.
The Peter Davison era had some amazing stories, and I consider them to be a high point in the original run of Dr Who, but I must admit that the Doctor himself seemed a little bland. Didn't really like the Colin Baker set, and the Sylvester McCoy set became more and more woeful.
When I heard that McCoy used to drop ferrets down his pants as part of an act, it didn't bode well. Not to mention the bizarre and unexplained stories like The Greatest Show In The Galaxy, or the blatant rip off of the 'they're on top of us!' radar scene straight out of Aliens. Gah.
I have yet to see any of the new series since McCoy! The Paul McGann movie, well, it was kind of okay and I certainly don't count it as canon.
For me, it's Tom Baker. He is the (fourth) man.