I think that your point about learning being easier when you've got a problem is probably true. But as someone without such a problem immediately at hand, I could really use a list of interesting problems to give me a starting point. And, even if I had a problem I wanted to solve, I expect I might not even know where to begin on it (so then I'd have two problems, as they say). So in practice I'm not sure what to do with your advice.
Write simulators that use the mathematics you want to learn. One way or another you will learn the mathematics if you do this. A game engine with bone animation is great for really getting familiar with 3D vector mathematics. Electrical circuits simulator and a Physics engine are great follow ups. Fluid dynamics is another one. A symbolic math engine will get you even more familiar with lots of other pure mathematics.
Honestly speaking I've only built the circuits simulator, partially built the symbolic math engine and currently working on the physics engine. I have learnt a great deal though.
