PySpark Jobs

Role Summary We are seeking an experienced Databricks Consultant with hands on expertise in implementing Databricks AI/BI Genie to support a short-term engagement. The consultant will guide us through real world deployment, configuration, optimization, and knowledge transfer for Genie within an Azure Databricks Lakehouse environment. Key Responsibilities • Implement and configure Databricks AI/BI Genie within Azure Databricks and Unity Catalog. • Set up Genie Spaces, connect curated datasets, define metadata, refine prompts, and optimize NLQ performance. • Advise on best practices for data preparation, governance, and security needed for Genie adoption. • Build and validate use cases using conversational BI, insights generation, and AI-driven analytics. • Provide ...

I have a Hadoop cluster holding several large data sets, and I need a seasoned PySpark developer who also writes rock-solid SQL. The immediate aim is to connect to the cluster (YARN/HDFS with Hive metastore), develop or refine PySpark jobs, optimise the accompanying SQL, and make sure everything runs smoothly end-to-end. You’ll receive access to a staging namespace plus a sample of the data. Once the logic checks out we’ll promote the code to the full environment. Deliverables • A clean, well-commented PySpark notebook or .py job that executes successfully on the cluster • The corresponding SQL script or view definitions ready for Hive or spark-sql • A concise README detailing execution steps, parameters, and expected outputs Acceptance criteria &bul...

I run a data platform built on Spark and Python, and I need an experienced engineer who can sit with me in Pune three to four times a week (hardly 2 hours in a day) to keep it running smoothly. Most of the immediate work revolves around tracking down bugs in existing PySpark jobs, but the role naturally extends to writing fresh code when gaps appear, tightening our data-pipeline orchestration, and mapping end-to-end data lineage so every downstream consumer stays confident in their numbers. Typical day-to-day work you will tackle: • Debug production PySpark jobs and accompanying Python utilities • Refactor or rewrite modules where quick fixes will not suffice • Optimise and monitor pipeline schedules (Palantir Foundry) • Document lineage and hand off clear, repro...