Lifehacker has some useful advice about Staying on a Horse. I like the tip about not gripping because I hate it when I read books and I find a statement about the character holding on by gripping the legs. Hap cured me of gripping when I was startled by bolting forward whenever I would do it.
The last tip is especially good: I have never fallen off a horse while following it.
The first four pieces of advice are contingent on you ignoring the fifth bit.