I'd throw a layer of filter fabric down just as cheap insurance that the fine soil you're backfilling with over the gravel doesn't leach down over time and plug the gravel or worse, the drains.
Site drainage, IMO is one of those thing you can never "over do." The cost/hassle to fix a drainage issue down the road is very prohibitive. (Think tearing out your drive pad or a deck or landscaping just to prepare to make a mess re-excavating.)
If you have ground water I'd consider putting drain board on the foundation walls down to just below the top of the gravel as well. If no ground water, no problem, but the key is to allow ANY water that is going to contact your foundation (especially block) the quickest ride possible back away from the house.
'06 X2 MCX
"I understand why some people may not want to do this the way I have recommended but I can't understand the death grip some people have on a toilet plunger with a hose fitting." -JimN