
Closed
Posted
Paid on delivery
I’m looking for hands-on help building and optimising data-engineering workflows inside Databricks. My immediate focus is on ingesting data that arrives in two ways—batch loads from relational databases and real-time streams—then transforming, testing and storing it so downstream analytics teams can trust the output. Here is what I need from you: • Design and implement robust ingestion pipelines that pull incremental data from the source databases and process the streaming feeds with minimal latency. • Develop scalable transformation logic inside notebooks (PySpark / SQL) and package reusable jobs so they can be scheduled and monitored through Databricks Workflows. • Advise on cluster configuration, performance tuning and cost-efficient autoscaling as the data volumes grow. • Document the solution clearly so future engineers can onboard quickly, and leave me with repeatable deployment steps (ideally via Databricks repos or CI/CD). If you have a proven track record delivering similar data-engineering projects in Databricks and you’re comfortable juggling both relational and streaming inputs, I’d love to work together. Please share a brief note about your relevant experience and any links to comparable work you can show.
Project ID: 40753801
18 proposals
Remote project
Active 2 hours ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
18 freelancers are bidding on average ₹20,696 INR for this job

Hi there, I am writing to express my strong interest in helping you build and optimize your data engineering workflows inside Databricks. With extensive experience in Python, PySpark, and advanced SQL, I specialize in designing scalable architectures that handle both batch loads and real time streams seamlessly. In my recent projects, I orchestrated a complete data pipeline for a microfinance client using Azure Data Factory and Databricks. I designed robust incremental ingestion pipelines to pull from relational databases and applied rigorous transformation and testing logic inside Databricks notebooks. I packaged these transformations into reusable jobs monitored through Databricks Workflows, ensuring the downstream reporting layer was perfectly accurate and trustworthy. Beyond writing the transformation logic, I am highly comfortable advising on cluster configuration, performance tuning, and setting up cost efficient autoscaling to manage growing data volumes. I also prioritize clear documentation and repeatable deployment steps utilizing Databricks Repos and CI/CD pipelines so your future engineers can onboard quickly and safely. I would welcome the opportunity to discuss your specific ingestion challenges and how we can build a highly performant Databricks environment together. Best regards, Morris Mwaura
₹37,500 INR in 7 days
5.2
5.2

Hi there, I’m Malix Azis. I can build reliable Databricks pipelines for both incremental database loads and real-time streams, using PySpark and SQL. I’ll add testing, transformations, Databricks Workflows, performance tuning, monitoring, and clear deployment documentation. Do you already have the source databases and streaming platform selected? I have strong experience with Databricks, PySpark, SQL, ETL, data warehousing, and performance optimisation. I can provide perfect results and start immediately after we finalise the plan. Let’s discuss the details via chat. Best regards, Malix
₹25,000 INR in 7 days
2.9
2.9

Hi There, Hope you are well. I have experience designing and implementing ingestion pipelines in Databricks for both batch and real-time data. I can help develop scalable transformation logic, advise on cluster configuration, and document the solution for easy onboarding. I look forward to the opportunity to collaborate on this project. Thanks in advance for the opportunity to working with you. Best Regards, Serazum
₹32,000 INR in 1 day
2.1
2.1

As a data engineer with over six years of experience, I can confidently handle your Databricks project. Throughout my career, I've worked on numerous projects just like yours, involving the intricacies of real-time and batch data processing, transformation, and reliable storage - all within the Databricks ecosystem. Having a strong background in SQL and PySpark, I am well-versed in manipulating large datasets for reliable downstream analytics outputs. Moreover, I bring a distinct set of skills to this project that will greatly benefit your team. My expertise in optimizing cluster configurations, performance advancements and cost-efficient autoscaling will help ensure that you achieve the most out of your Databricks infrastructure as your data volumes continue to grow. Additionally, my meticulous documentation practice will aid future data engineers who might onboard the project. In conclusion, my consistent adherence to the 'extraordinary chemistry in long-term relationships' will help propel us towards success. Trust me with your project and let's build something great together in Databricks!
₹35,000 INR in 15 days
2.3
2.3

I will focus on building robust and scalable data engineering workflows within your Databricks environment that efficiently handle both incremental batch ingestion and real-time streaming from relational databases, delivering high-quality data to analytics teams. My approach: I will develop ingestion, transformation, and data quality validation pipelines using PySpark, Spark SQL, and—where required—Structured Streaming and Delta Lake. I will configure reusable jobs, scheduling, monitoring, and failure recovery within Databricks Workflows. I will balance performance and cloud costs through cluster sizing, autoscaling, partitioning, and query optimization. I will also implement deployment processes based on Databricks Repos/CI-CD and provide clear technical documentation. My questions: 1. What are the source databases and streaming feeds—e.g., PostgreSQL, SQL Server, Kafka, or others? 2. What are the data volumes, ingestion frequency, and expected latency? 3. Are an existing Databricks workspace and Delta Lake architecture already in place? 4. Is there an existing pipeline that needs optimization, or should everything be built from scratch? I will design practical solutions for your analytics workflows by prioritizing reliability, data quality, maintainability, and cost-efficient execution.
₹12,500 INR in 7 days
2.1
2.1

I have developed parsers and scrapers with anti-bot protection bypass and integration via REST API. I have built data processing pipelines using FastAPI and automated CI/CD for a team. How do you plan to ensure monitoring and fault tolerance for streaming data?
₹17,352 INR in 14 days
1.3
1.3

I'd deliver the incremental loads and streaming job as PySpark and SQL notebooks scheduled in Workflows, with deployment steps via Databricks repos. Autoscaling caps matter early with streaming. I can deliver it in 8 days, and it includes up to 2 rounds of adjustments. Which database engine and streaming source are you using, and is workspace access provided? You can see my work here: https://www.freelancer.com/u/Deived3339
₹17,000 INR in 8 days
0.0
0.0

Your focus on building robust ingestion pipelines and minimizing latency is crucial for reliable analytics. I can design and implement these pipelines, leveraging PySpark and SQL in Databricks to ensure efficiency. My approach includes establishing clear documentation and CI/CD processes for smooth onboarding. What specific databases are you using for batch loads? This will help tailor the ingestion strategy to your needs. I look forward to discussing how we can optimize these workflows together. Best Regards, Charl
₹28,150 INR in 7 days
0.0
0.0

Hello, I understand your need for hands-on support in building and optimizing data-engineering workflows within Databricks. Ensuring reliable data ingestion and transformation is crucial for your analytics teams. With experience in designing robust ingestion pipelines that effectively handle both batch loads and real-time streams, I can help you implement scalable transformation logic using PySpark and SQL. My approach focuses on minimizing latency and ensuring that your data is processed efficiently. Additionally, I prioritize clear documentation and repeatable deployment steps, which will aid future engineers in onboarding and managing the workflows. I’d love to discuss how I can help you achieve your goals. Let’s set up a free consultation to explore this further. With regards, Maanie
₹12,500 INR in 7 days
0.0
0.0

The decision that saves the most pain here is building it on Delta Lake properly from day one: Auto Loader or Structured Streaming with checkpoints for the real-time feeds, and incremental batch pulls from the databases landing in a bronze layer, then MERGE into silver so late or duplicate records update rows instead of piling up. With checkpoints and MERGE, any job can be re-run after a failure without double-counting, which is what makes downstream teams trust the numbers. For trust specifically: data quality expectations at the silver step (nulls, ranges, referential checks) that quarantine bad rows instead of failing silently, plus row-count reconciliation against the source. PySpark/SQL notebooks packaged as Databricks Workflows with retries and alerts, job clusters sized per workload with autoscaling and auto-termination to keep costs down, and docs plus a repeatable deployment (Asset Bundles). What are the source databases, and what does the stream come from: Kafka, Event Hubs, or Kinesis?
₹15,000 INR in 7 days
0.0
0.0

Hi, Dejan here, from Croatia. I can help you build and optimise data-engineering workflows in Databricks. I will design robust ingestion pipelines for batch and streaming data, develop scalable transformations, and set up scheduled jobs with clear monitoring. I’m a senior data engineer with 8+ years of experience in PySpark, SQL, ETL, and Databricks. My plan: 1) Set up incremental batch ingestion from your relational databases. 2) Build real-time streaming pipelines with low latency. 3) Develop transformation logic in PySpark/SQL notebooks and package reusable jobs. 4) Configure clusters for performance, autoscaling, and cost efficiency. 5) Document the solution and set up repeatable deployment via Databricks repos or CI/CD. I’ve delivered similar Databricks projects for batch + streaming workloads and can share examples in chat. I can start right away. Happy to discuss details and your data sources. Looking forward to hearing from you, Dejan
₹25,000 INR in 5 days
0.0
0.0

I understand the importance of building efficient data-engineering workflows in Databricks, especially when it comes to ingesting both batch and streaming data. My experience in full-stack development and data engineering aligns perfectly with your needs. I've designed and implemented robust ingestion pipelines that handle incremental data from relational databases while ensuring minimal latency for real-time feeds. I specialize in PySpark and SQL for scalable transformation logic and have packaged reusable jobs that integrate seamlessly with Databricks Workflows. Additionally, I can provide insights on cluster configuration and performance tuning to ensure cost-effective autoscaling as your data volumes increase. I prioritize clear documentation, making it easier for future engineers to onboard and follow deployment steps. I’m ready to discuss how I can contribute to your project and help your analytics teams trust the data outputs. Let’s connect!
₹12,500 INR in 7 days
0.0
0.0

Hi, I'll build ingestion pipelines that handle both your batch loads from relational databases and real-time streams, then transform and store that data so your analytics teams can trust it. Since you need CDC-style incremental loads alongside low-latency streaming, here's my approach: 1. Set up incremental batch ingestion with watermarking/CDC logic per source table 2. Build Structured Streaming jobs for the real-time feeds, with checkpointing for fault tolerance 3. Write PySpark/SQL transformation logic with data quality checks, packaged as reusable Databricks Workflows jobs 4. Tune cluster/autoscaling settings for cost efficiency, then document everything and set up Repos-based deployment In 7 days you'll receive: working ingestion pipelines (batch + streaming), transformation notebooks, scheduled Workflows jobs, a cluster-config/tuning writeup, and a README covering architecture and deployment steps via Databricks Repos. To scope cluster sizing correctly - which relational database engine and streaming source (Kafka, Event Hubs, Kinesis) are you working with, and is Databricks hosted on AWS, Azure, or GCP?
₹16,245 INR in 7 days
0.0
0.0

Pathapadu, India
Member since Oct 13, 2020
$30-250 USD
₹12500-37500 INR
₹600-610 INR
$8-15 USD / hour
₹600-1500 INR
₹1250-2500 INR / hour
₹12500-37500 INR
£20-250 GBP
$10-30 USD
₹750-1250 INR / hour
€250-750 EUR
₹100-400 INR / hour
$250-750 USD
$250-750 USD
$250-750 USD
min $50 USD / hour
$30-250 USD
$30-250 USD
$8-15 USD / hour
₹1500-12500 INR