Grande Volume de Dados Jobs

I’m upgrading our analytics stack and need an expert who can own the Hadoop side and turn raw, high-volume feeds into analysis-ready datasets. The core objective is to design and build end-to-end data pipelines on a Hadoop cluster—this is where I believe Hadoop will be most valuable for the project. Here’s what I need from you: • An architecture that takes terabyte-scale log files, lands them in HDFS, applies basic cleansing, and outputs partitioned Parquet tables queryable from Hive or Spark • All scripts, configs, and scheduling (Oozie, Airflow, or your preferred orchestrator) committed to Git with clear documentation • A deployment guide plus a brief hand-over session so I can reproduce the setup on another cluster Acceptance criteria: the pipeline...

I have an Apache NiFi deployment that has grown organically and now needs a fresh, expert set of eyes. I’d like you to walk through the current flows end-to-end, scrutinising the data-flow design, performance optimisation choices, and the way we handle errors. The ultimate goal is to confirm that the platform is genuinely robust and reliable under present and future workloads. After the review, I’m expecting a detailed report that not only flags every weakness you find but also lays out clear, prioritised recommendations for improvement. Deliverables I will use to sign-off: • A written report (PDF or Markdown) covering architecture observations, specific processor-level findings, and configuration notes • Prioritised recommendations with estimated effort...

I need a data engineer to implement liquid clustering with z-ordering to enhance performance. This project is critical for optimizing our data processing and retrieval speeds. Key Requirements: - Expertise in data engineering - Strong understanding of z-ordering and liquid clustering - Experience in performance optimization techniques Ideal Skills: - Proficiency in SQL and relevant data processing frameworks - Background in managing large datasets - Ability to work independently and meet deadlines Please provide examples of similar work and your approach to the task.

I need an Azure Databricks specialist to re-architect my small batch-processing cluster (1-10 nodes) so it reliably runs on spot instances and trims our monthly bill. The objective is simple: reduce overall operational costs without sacrificing job completion time or data integrity. What I expect from you: • Analyse the current workspace, jobs and autoscaling behaviour. • Design and apply an optimal spot-instance strategy (instance mix, bid price, fault-tolerant settings, pool configuration, autoscaling rules). • Implement job-level tweaks such as task parallelism, cluster reuse and termination timing to prevent idle burn. • Set up cost-tracking dashboards or alerts so I can see savings in real time. • Deliver a short hand-off guide that explains the n...