Need a bulk url crawler - search 2 links deep for broken/unregistered links

My goal is to find unregistered domains to buy, they must have high authority backlinks, so I will be entering massive amounts of high authority urls such as 40 million urls into this bot to search out links on each of those urls or within 2 pages or so. Similar programs are like xenu and screaming frog but they "run out of memory" because they don't auto save and delete the displayed info. So theyre useless because of that. 1. Your variable multi-threaded bot will be run from my windows desktop most likely which has 4-8gb ram, unless you have a better idea. It will crawl each website and find external "broken links", these are links that have no host, so most likely they would be available for registration. 2. The bot would require a variable 1-1000 levels deep option, to crawl real far into the links on the sites. 3. The bot would need to save on the fly so that I could pull a list of urls anytime I want while keeping it running. I don't want it "running out of memory" either, like similar xenu and screaming frog programs. 4. The bot would then need a simple domain availability checker, will prob use a namecheap api unless you have a better suggestion. Or maybe an additional bot since this may cause the bot to lag etc. Then be able to save to notepad or excel for extraction and copying. 5. The biggest issue with this crawler bot crawling urls is the lag associated. It needs to be consistent and as fast as possible. No dying, freezing, lagging :) 6. The bot doesn't have to be super fancy looking, but of course an interface is always nice. Continued support, and maintenance is always appreciated as well. Example of similar bots are here, but they "run out of memory" and or freeze, lag: xenu screaming frog