What I need is a tool for collecting URLs of all websites. I want it to collect all URLs on the web, it might be something like billions.. so I need to know if this is possible to be done using PHP. Then I will use some PHP commands to download these pages.. I'm making a new Search Engine and other services.. so I want to enter the market with a big database.. So again, a PHP application that give me a result of all the URLs found on the web.. and then it has the option of saving the results as CSV in addition to mySQL database. The results will be "URL (link), Keywords and Title". Sure I want a good keyword maker, and there is no problem if you use any opensource ones which are available online. With documentation about how it works (I know it is a simple command using PHP.. but I need to know everything about how it works)
1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done.
2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables):
a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment.
b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request.
3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement).
On any platform that supports PHP.