How long does it take to mis-align itself like that? Is it almost instantaneous once you start the engine or does it take a little while?

Is it an OEM belt?

Did you replace the tensioner and the water pump? It's possible that the tensioner bracket may be bent or the tensioner pulley may be loose, which could be creating just enough misalignment for the belt to start walking.. I'm not sure what engine we're talking about here, so if you have additional idlers, etc., they could be a problem also. usually what happens is that if the bearing has too much play in it, it will cause the pulley to tilt and make the belt walk off; the same thing can happen with a frozen tensioner bearing because of belt drag.
Same thing with a overly worn water pump bearing, although by that point everything is usually loose enough that the seals will wear prematurely so you might see the pump leak.

Other than a bad tensioner or a poor quality timing belt, I can't think of anything else that would make it walk like that unless there was something really wrong with one of the other pulleys or there was a spacing problem, like the wrong aftermarket cam gears for that engine, etc.

It could be something as simple as having the belt too tight, which can flex the spokes of the cam gears or flex the water pump housing such that the pulleys will tilt (same concept as a loose tensioner bearing but for different reasons).

Basically your problem shouldn't be happening if everything is in good working order and adjusted correctly, so take the belt off and take a good hard look at everything to see if it's too loose, too tight, etc.