Well, all telly canon indicates that Time Lords are not ''born'' as adults (I think Three was the first Doctor to mention his childhood), so there's no reason why you can't reconcile that with LOOMS. The estate could have been one of those ''hunting estates'' that the aristocracy tends to possess. Somewhere you went on holidays and things like that.