I would not think cat litter would be good. Most commercial litters contain all sorts of deodorizers. You can buy simple baked clay with no additives, but in my experience using this in potting soil formulas, it will compact into a solid ball in short time, no (or very little) water flow. I agree, just small gravel.
If you do not have a clean source of gravel, you can buy granite chips from any animal feed store (It is used for chicken scratch). It comes in several sizes.