Senior Data Engineer

Senior Data Engineer

Technical Skills

Languages – Python, SQL, Java, HCL, HTML/CSS/Javascript, Bash

Database Technology – Spark, SybaseIQ, DB/2, Snowflake, Redshift, Hive, Presto, Oracle PL/SQL

Tools – AWS, Terraform, Kubernetes, Docker, Jupyter, Intellij, vim, Git, SVN, Apache, nginx, Splunk, SSH

· Primarily should have worked on the Data Lake, a petabyte-scale Data Warehouse built for Goldman Sachs’ unique requirements. The lake is used across hundreds of teams for many time-sensitive critical applications.

· Derived a variety of SLOs and health indicators for the lake. Successfully optimized the lake, bringing ingestion time down under 15 minutes for more than 90% of users.

· Designed an event-driven near real-time SLO monitor for the lake that processes millions of events a minute.

· Crafted terraform AWS configurations from scratch to deploy key lake components to the cloud.

· Developed and maintained a Jupyter notebook ecosystem on Kubernetes to support the SRE team.

· Wrote Jupyter notebooks to analyze telemetry metrics, develop insights, and establish SLOs. Notebooks typically pulled in data using SQL or Pyspark, and further processed in Pandas. Visualizations were done using matplotlib.

· Designed an automation framework for Jupyter notebooks to schedule, cache, serve, and email them to clients.

· Implemented and maintained Prometheus metrics for high-level monitoring of the lake. These metrics are pushed to Grafana for visualization and Pagerduty for alerting.

· Developed on Facebook’s Hadoop system through Hive and Presto, using Facebook’s internal ETL framework.

· Maintained solutions with third parties for ad data ingestion and delivery, including coordination of data definitions and validation checks during ETL process.

· Created APIs using hack (PHP) for upload endpoints.

· Developed dashboards for sales lift data normalized across third parties using Tableau and internal tools.

· Maintained ETL processes to solve bugs, data quality issues, CPU and space optimization, and adding columns to tables, which were mainly core ad metrics data sets that had a wide impact across the company.

· Developed Facebook status tables which was a dataset that exceeded 150TB and over 1.2 trillion rows, from Facebook’s graph structure and curated into an easily digestible hive table, used by research teams for insights, sentiment analysis, and machine learning applications.

Habilidades: Python, ETL, SQL, AWS Lambda

Veja mais: html css javascript sql engineer programming karachi, php java script html css, java jsp html css, html css javascript python django, electrical engineer data entry operator clerk data entry mis executive auditor lead engineer technical lead project lead job in, freelance software engineer python java php javascript html5 css3 android, php programmer senior technical skills, java,j2ee,html,css,, java html css javascript, senior data engineer resume, senior data engineer job description, rf engineer technical skills, java swing html css, senior data engineer salary, getting data from text file and display it in html table javascript, qa engineer technical skills, senior data engineer interview questions, data engineer technical interview questions, network engineer technical skills, senior android developer technical skills

Acerca do Empregador:
( 0 comentários ) Mysore, India

ID do Projeto: #31572045

3 freelancers estão ofertando em média ₹102778 nesse trabalho


Hi, We at Tecogno Solutions are a team of Passionate Data Science and Full Stack professionals having more than five years of combined experience in multiple areas including Backend, Frontend, Machine learning (ML), C Mais

₹150000 INR in 31 dias
(2 Comentários)

Hi. I am a data scientist. I am very familiar to Deep learning apis such as Tensorflow and fastai, mxnet. I have a good hands on working with Advanced R and Python and BI tools and technologies, AI, Big Data. I have Mais

₹83333 INR in 30 dias
(4 Comentários)

Hi There, *I am available to start immediately.* I have read your post requirement for the " · Developed on Facebook’s Hadoop system through Hive and Presto, using Facebook’s internal ETL framework. ". Now I have some Mais

₹75000 INR in 5 dias
(0 Comentários)