Crawler to Automatically Collect Data from Websites

I need an experienced developer that can write an application that has crawler type of functionality that will do the following things:

1. Have the crawler go to the specified website every day at the time that I specify – login with the username & password that I will also specify (I should be able to type in this information into the application’s interface).

2. Then the crawler should download the daily five .csv files that contains various user type of information into a folder that the application should automatically create on the local hard drive.

3. Once these files are downloaded then your code should parse these files and extract the user’s first name, last name, and email address. Keep in mind that each one of these .csv files could have its own unique format when it comes to storing this information.

4. And once your application has parsed these files and retrieved the user’s first name, last name and email address then your application should automatically insert this data into a database that I will provide for you.

This is the main functionality of this application – but, in addition to all of the functionality that I outlined above, your application should also do the following:

1. On the day of the month that I specify – please have your application login to the same website as specified above and download a monthly .csv file that your code should parse and extract all first name, last name and email address data and automatically insert into the same database & table as above. In other words – this functionality is exactly the same as above, except that it will only happen once every month.

2. Your application should also go to another website and automatically log in with the username and password that I will provide you with. Then your application should download the daily .txt file (only 1 file per day) and extract the first name, last name, and email address data from the .txt file and insert this data into the same database (different table).

3. On the day of the month that I specify – please have your application login to this second website and download a monthly .txt file where your code should parse and extract all first name, last name, and email address and automatically insert into the same database as specified earlier.

4. Then I want this application to automatically launch a webpage and allow the code to run within the background so that I don’t see it on my computer screen. Now, this webpage has already been coded – so you won’t have to do any of that. All your application needs to do is to automatically launch and allow this page to execute until I stop it. Of course the application’s interface should allow me to enter this page’s URL that should be automatically executed in the background.

Now, that concludes the functionality description for this application – and now, here are some of the features that I want for this application.

1. I want this application to be written as a client server application that can be minimized to the system tray where it could automatically run in the background without any intervention from me. In other words, I want everything to be fully automated and if there’s any kind of problem then I want to be notified.

2. I want this application to give me the ability to type in all of the settings that I’ve talked above. In other words – I want to be able to specify the hard drive location where all of the files will be saved, the username and password for both of these websites, the day of the month for both monthly downloads, and I want to be able to type in the connection string that this application uses to connect to my database.

3. I want this application to never export the same data files to the database. In other words once a day’s data has been exported to the database then it should keep track so that it doesn’t do it again. In addition, I do want some type of reporting so I can make sure that everything is working properly.

4. I want to have the final word on how this application looks. In other words – in the beginning you can make the application look in anyway that you like, but at some point I will provide you with a design that you should implement for the application.

5. This application must be completed within 3-4 business days – I’m on a very strict and tight schedule and I urgently need this application to be completed as soon as possible. And, once the first four items have been completed then and only then I will fund escrow. In others I want to make sure that you can do this type of work and once I start seeing some results then I’ll gladly fund escrow while you complete this application. But, if you can’t complete the first four items that I’ve listed above then I reserve the right to look and hire another developer that can do the job.

6. In addition, I want a developer that I could come back to and re-hire to add other websites with their corresponding downloads and importing of data as needed. In other words – if you do a good job, you’ll get additional work from me...

So, if you’re an experienced developer that can do this application as specified above then please place your bid. In addition, please be sure to tell me what kind of previous experience you have that qualifies you to do this project. Here’s the criteria that I will be judging all bidders on:

1. Your Bid Amount

2. Time of Delivery

3. Experience

Thank for taking the time to read this and good luck to you…

Gabe U.

Habilidades: .NET

Veja mais: automatically collect data websites, collect data website automatically, collect data websites automatically, collect data web sites, collect data websites, download data website automatically, automatically collect data website, automatically collect website data, collect data automatically websites, collect information websites automatically, collecting data websites, collect data website, working from, working for first data, work from here, where to hire application developer, where is escrow, what should i need to add design my webpage, what should i do to design website, what's escrow, what's an escrow, what is escrow reporting, what do i need to do as a developer, websites net, websites design download

Acerca do Empregador:
( 92 comentários ) Folsom, United States

ID do Projeto: #155141

Concedido a:


i can do it. please check pm

$300 USD em 4 dias
(9 Comentários)

6 freelancers estão ofertando em média $275 para esse trabalho


please check pmb if you want good quality work.

$300 USD in 7 dias
(20 Comentários)

Check PM please.

$300 USD in 15 dias
(12 Comentários)

We are expert in .net,we will do it with satisfactory manner.

$150 USD in 20 dias
(12 Comentários)

Hi please see pm

$300 USD in 14 dias
(0 Comentários)

I will prove to be your best choice when it comes to designing crawlers. I have gone though everyline of your project description and have understood it well. We can well achieve your task, however, the budget ra Mais

$300 USD in 6 dias
(0 Comentários)