We are looking for an experienced coder to create a web site link checker and crawler.
This program should be able to do the following:
1. Crawl all pages on a site
2. Collect the title tag and meta description on each page
3. Keep track of how many outgoing links each page has, and how many incoming links each page has
4. Keep record of all internal anchor text links (text and where that text links to). For example "home builder san diego" links to [[url removed, login to view]]. This should be a separte report that shows ONLY anchor text links (not image or other links)
This program should be able to export this data to an excel spreadsheet.
It should have multiple threads to make it fast.
For a good comparison program you can look at the website below (remove the C's). It does everything we want except for displaying the meta description, and anchor text links. It does a lot of things that we don't need as well. We only need pages, not images or anything else.
[url removed, login to view]**lCiCnCkC**.html