We need to retrieve a lenders and their company information. I would like a couple of bots written to retrieve the data from websites. The sites we would like to pull this data from are:
**Site 1**
[login to view URL]
- For each Link the says "Lender", only in the U.S.
- Get the list of Lenders.
- retrieve all of the information, including the following:
- Company name
- Address
- All phone numbers (Phone, Fax, Mobile)
- Email (retrieve from contact us link)
- Website (retrieve from contact us link)
- Do a duplicate check to make sure it is not a duplicate entry.
- Store in an Excel spreadsheet (or CSV file), along with the state that was used to retrieve it.
**Site 2**
[login to view URL]
- From the section titled "Mortgage Broker Directory:" near the bottom of the page
- For Each State
- Get the list of brokers.
- Retrieve all of the information, including the following:
- Broker name
- Address (City, State)
- Do a duplicate check to make sure it is not a duplicate entry.
- Website
**Then, using the website link, crawl the site up to 2 levels deep, and try to retrieve**
- All phone numbers (Phone, Fax, Mobile)
- Email
- Store in an Excel spreadsheet (or CSV file), along with the state that was used to retrieve it.