Hadoop Jobs

Por favor, Cadastre-se ou Faça Login para ver os detalhes.

For the next four weeks I need a seasoned DevOps professional who can step in as the acting lead for my small engineering team. Your focus will be threefold: setting up a robust Jenkins-based CI/CD pipeline, automating our AWS infrastructure (we run our workloads in Kubernetes), and putting solid monitoring and logging in place so the team has clear operational visibility. You will work directly with three developers and a QA engineer, guiding best practices while getting hands-on with the code, Terraform/CloudFormation templates, Helm charts, and our existing Jenkins files. I expect daily Slack or Teams touch-points and a concise written summary of progress at the end of each week. Deliverables • Fully functioning Jenkins multibranch pipeline that builds, tests, and deploys to our ...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

The goal is to design a future-proof relational backbone for a multi-channel retail platform, built natively on PostgreSQL. I already run an operational system, and its live transactional records will have to be migrated into the new structure without downtime. Everything else—users, inventory, orders—will start fresh. Core design expectations The operational schema should be fully normalised to 3NF and optimised from day one for high concurrency. Think table partitioning (range and list), well-chosen GIN and B-Tree indexes, plus JSONB columns where semi-structured flexibility makes sense. Triggers, constraints and stored procedures must enforce business logic consistently, so the application tier can stay lightweight. Analytics layer Alongside the OLTP schema I need a se...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

I have a Hadoop cluster holding several large data sets, and I need a seasoned PySpark developer who also writes rock-solid SQL. The immediate aim is to connect to the cluster (YARN/HDFS with Hive metastore), develop or refine PySpark jobs, optimise the accompanying SQL, and make sure everything runs smoothly end-to-end. You’ll receive access to a staging namespace plus a sample of the data. Once the logic checks out we’ll promote the code to the full environment. Deliverables • A clean, well-commented PySpark notebook or .py job that executes successfully on the cluster • The corresponding SQL script or view definitions ready for Hive or spark-sql • A concise README detailing execution steps, parameters, and expected outputs Acceptance criteria &bul...

I need an experienced Python engineer who works confidently with AWS Glue to build and manage a small suite of data-integration jobs for a Hyderabad-based project. The core of the work is to design and automate Glue ETL pipelines that pull data from our production databases, catalog it accurately, and transform it into analytics-ready tables. Here is what I expect from the engagement: • Develop, test, and deploy Glue ETL jobs in Python. • Populate and maintain the Glue Data Catalog so new tables are discoverable and properly version-tracked. • Implement efficient transformation logic that cleans, enriches, and partitions data for downstream reporting. • Optimise job performance and cost by selecting the right worker types, job parameters, and database connectio...

Hiring: DevOps Developers DevOps Engineer Requirements: -Expertise in CentOS, Ubuntu, Debian -Experience with HBase, Hadoop, Storm, ArangoDB, Prometheus -Strong benchmarking and system scalability testing skills Responsibilities: -Configure and optimize high-availability servers and databases -Implement monitoring and performance evaluation tools -Review and enhance DevOps workflows Long-term opportunities available!

- **Core Architecture:** Spring Cloud + Kafka + Hadoop + Python Automation，This project requires a certain level of technical expertise.