As far as I can tell, I was drawn to Physics out of sheer masochism. In highschool I excelled in history and English without putting any effort in, but had to work on my math and physics to do well. This trend only intensified in university. I found quantum mechanics to be the most challenging sub-field of physics (except, perhaps, solid state), and guess what I wound up doing in grad school!
Maybe there just aren't enough masochists willing to embrace the pain out there.
I'm currently doing a double major in Philosophy and Mathematics part-time while working a full-time job and caring for a house and 5-month old daughter with my wife. I think one of the things that initially drew me to those subjects and keeps me plugging away is that I find both of them confounding, infuriating, confusing, and hard as shit. I have a couple tech-related college diplomas that I really didn't have too work too hard to get, but in 8 to 10 years when I'm being presented with my degree, I'm going to really feel like I accomplished something.
Maybe there just aren't enough masochists willing to embrace the pain out there.