Em Andamento

ETL and Cron Job for Custom Data Feed

I have a custom data feed (CDF) that gets posted to a secure FTP site every hour by my technology partner. Each file has approximately 2 million rows and 30 headers.

1)Read and understand the CDF file and its columns. CDF documentation will be provided.

2)Create a MySQL database schema

3)Create a table RM_CDF_EVENT_LOG. This table should have all the columns defined in the CDF. We need to add couple of extra columns called FILENAME, MODIFIED_ON (DEFAULT SYSDATE) to the table

4)Partition the table by DATETIME column in CDF and FILENAME

5)Create a script that would do the following

a) Download the file from a FTP server

b) Run GPG on the file to decrypt it (can stubbed for now)

c) Run unzip/untar on the file to extract the file

d) Create a new partition in the tbale for the DATETIME and FILENAME

e) Break apart column data (separated by spaces) into separate rows

f) Insert rows from the file into the table

g) Set this script to be executed as a cron job to be executed every 30 minutes

6)Script can be written in any language (Java, PHP, C, etc)

7)If possible we need to use Apache Hadoop to break apart the rows that are delimited by spaces inside a column. (This is not a requirement but is considered a bonus if you can do it)

Habilidades: Apache, Programação C#, Java, MySQL, PHP

Ver mais: etl file feed, cron job etl, cron etl, etl cron job, data feed etl, cron database schema, cron mysql insert, etl cron job custom data feed, unzip ftp, technology job, java 1.5 documentation, java 1.4 documentation, ftp unzip, every job, etl job, c language job, cron data feed, server job, job requirement, hadoop job, mysql/java-mysql-job, java job, hour job, hadoop, gpg

Acerca do Empregador:
( 8 comentários ) San Francisco, United States

ID do Projeto: #595273

Premiar a:

gord

Please check PMB.

$750 USD em 5 dias
(4 Avaliações)
5.8

17 freelancers estão ofertando em média $475 para este trabalho

omsoftware

Hello We Understand the project and we can create a Data Feed Reader and Update the Database as cron Job written in PHP or ASP.NET Please check PMB fore more details Thanks Raj

$450 USD in 5 dias
(82 Comentários)
8.2
A2Design

We're interested in this project. Take a look on our portfolio at: [url removed, login to view]

$750 USD in 16 dias
(30 Comentários)
7.7
sunztech

Please see PMB.

$750 USD in 10 dias
(31 Comentários)
7.3
AndrwProjects

Hi, interested to take the job.

$399 USD in 7 dias
(175 Comentários)
6.3
interpb

Hello, I am interested in this project Thanks

$280 USD in 8 dias
(53 Comentários)
6.1
MAST3R

You have PM

$450 USD in 10 dias
(10 Comentários)
5.5
Ivan83

Hi, Please check PM. Thanks.

$450 USD in 10 dias
(26 Comentários)
5.3
savtargm

Please check PM. Thanks.

$900 USD in 20 dias
(1 Comentário)
4.1
amsak

Pls see PM

$250 USD in 14 dias
(7 Comentários)
3.6
waqasshami

Hi ropak I am ETL Developer. Please see PM for more details. Thanks

$400 USD in 7 dias
(3 Comentários)
2.1
suryavikas

Hi, I have vast experience in ETL, and have done lot of similar projects in my company. I have the knowledge and the skill set to finish the project on time and without bugs. Idea is to use an Open source ETL Mais

$500 USD in 10 dias
(1 Comentário)
1.6
ephraiminjamuri

I am a c#, SQL and scripting programming. I wrote this kind of applications. only thing different is the number of rows.

$500 USD in 14 dias
(0 Comentários)
0.0
frosters

Can do your task using perl and some unix tools (cron, shell). It looks not so difficult :)

$250 USD in 3 dias
(0 Comentários)
0.0
DISoln

Please check your inbox for my Proposal

$500 USD in 14 dias
(0 Comentários)
2.4
sailu145cw

Hi! I have gone through your requirement and i am glad that i can accomplish this task, i would be more interested to speak to you on IM. Pls give us an opportunity to work with you.

$250 USD in 2 dias
(0 Comentários)
0.0
toronto0013

Hi We have done a similar job. See pm Thank you Project manger Mais

$250 USD in 15 dias
(0 Comentários)
0.0