Creation of Low Latency Cassandra Economic Database with Automatic Updates

Hi there! I have 130+ spreadhseets, which have multiple sheets, and some additional data, all in Excel that I would like to be:

1) transformed into a low latency database such as Cassandra. Latency is very very important.

2) I would like the database to be updated automatically, using APIs. I will provide the links.

3) I would also like some webscraping for at least 2 web pages, if not more. The webpages are updated monthly. Also 50 PDFs scraping. I can provide the links. The PDFs are also updated monthly.

4) All the data is either with me or publicly available online from multiple websites.

5) The database architecture should be scalable. So that I can add more data sources in the future.

6) I plan to connect the database to Python and run econometric/datascience/machine learning models on it. So I need that functionality

7) The database has to be extremely low latency as I will also be connecting my stock broker's API to it. It should be able to capture and store live prices of some currencies, stocks and commodities that are in my watchlist. I am open to alternative solutions for this.

8) The data is of different time horizons, so ONE KEY DELIVERABLE would be to have the functionality to roll up the data in different time horizons, such as monthly quarterly and annually, both through linear interpolation downwards and summing up upwards. I will build the front end myself in Tableau.

9) The database and models will run on a purchased machine/server and not on the cloud or online.

10) I will also be connecting Twitter API to build NLP (natural language processing) models, using text data. So the database has to be able to process and store that for the models I build. I want the functionality to pick and choose what is stored and for how long, including updates.

Please let me know of any credible references that I can check. I really want someone who knows what they are doing and want to get this right.

The support from you might become ongoing after a few months of initial deliverable, and there is a chance of regular work as I scale bigger and bigger into the future. This project can generate long term monthly income. I am looking for someone whom I can work with long term, as a team.

Habilidades: Captura de dados na web, Python, Cassandra, Programação de Banco de Dados, Database Design

Veja mais: aws cassandra, cassandra vs dynamodb, cassandra architecture, cassandra download, cassandra database, cassandra database tutorial, cassandra vs mongodb, install cassandra on ec2, script start automatic updates, database automatic client follow letters fax, scrolling news page automatic updates, profile box automatic updates facebook, magento automatic updates, sql database automatic backup, java low latency volume, low latency programming, low latency server, low latency server server communication, low latency architect, designing low latency web server

Acerca do Empregador:
( 0 comentários ) Hatfield, United Arab Emirates

ID do Projeto: #25719571

9 freelancers are bidding on average $306 for this job


Hello, Thanks for inviting me to your project. I think Cassandra is not good for this project and ElasticSearch seems a better solution. Especially because you need to group by the data with year, date, ... and also y Mais

$255 USD in 7 dias
(152 Comentários)

python master here. i can deliver this in a few days. i have delivered many projects in python. let me know once back so that we can talk more.

$250 USD in 7 dias
(46 Comentários)

Hello sir. As a python developer, I'm glad to see your project. If you check my profile, you can see I have much experience of python. In fact python is my first language and I have finished many projects with python s Mais

$500 USD in 7 dias
(9 Comentários)

@@@ Scraping Expert @@@ Hi Mujtaba, I read your job detail carefully and I can satisfy your requirements. I am a Web & Data Scraping Expert who have career for 6 years over. I am very happy to bid on your job. I have a Mais

$400 USD in 3 dias
(10 Comentários)

Hi, I have 8 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with ha Mais

$35 USD em 1 dia
(2 Comentários)

Hi, Hope you are doing good. I have gone through your requirement and I do have skill set you are looking for. I am certified advanced RPA and QA Automation Professional. I have 2+ year of experience with Automation Mais

$357 USD in 5 dias
(2 Comentários)

Hi I read your project carefully. Im an expert in ecxel VBA, Python, etc. I'm ready to start right now. I will make formula as per your requirement with good and professional quality without any error. Please contact w Mais

$255 USD in 7 dias
(1 Comentário)

We are a team of python expert and developers. We are dedicated to helping you with any web/App development as well as python projects. We will be honored to work with you on this project. Kindly message us to discuss Mais

$450 USD in 3 dias
(0 Comentários)

HI Sir, I have read your project and eager to work with you. I have relevant skill set and experience and can start working from today. What I will bring ? I have 10+ years of experience and have the skills you wan Mais

$255 USD in 7 dias
(0 Comentários)