Overall description: (see attachment for more detail)
I am going to build a system to collect some data from websites. I would like to use AWS, open source frameworks for this purpose.
- Graduate the university of Information technology.
- Already learn the can do a separate python code to extract a specific website in python, save the result to text files.
- Doing web crawling on AWS, using framework, storing result in NoSQL database is totally new to me.
I would like to have an expert to:
Guide me to do the thing onetime, so that I can develop the detail (such as add more urls, writing more code for new format of new urls, adding more fields to database).
All the steps are started from standard material, so that I can follow to build the system by myself after I understand the mechanism.
Do not need to explain me the concepts, I can Google to study if I do not understand. I just need the steps to understand the foundation.
7 freelancers are bidding on average $453 for this job
I graduated from Carnegie Mellon University with a master degree. I have lots of industry experience in big data area. I worked at IBM, Twitter before. CMU is the top 1 University in Computer science!