Senior Data Engineer (Brazil)

@vishal1145

Your Spark jobs will bottleneck if you're not partitioning by the right keys and colocating shuffle operations. I've debugged pipelines where a single skewed partition caused 2-hour delays because the engineer didn't profile executor memory before deploying to production. Before architecting your multi-cloud pipeline, I need clarity on two things: What's your current data volume growth rate month-over-month, and are you replicating data across AWS, GCP, and Azure or routing workloads based on cost optimization? The answer changes whether we build a unified lakehouse or federated query layer. Here's the execution plan: - SPARK OPTIMIZATION: Implement dynamic partition pruning and broadcast joins to cut processing time by 50-60%. I'll profile your DAGs with Spark UI to eliminate stage skew and tune executor configs for your specific workload patterns. - MULTI-CLOUD ORCHESTRATION: Build Airflow DAGs with cross-cloud sensors that handle GCS-to-S3 transfers, BigQuery-to-Redshift syncs, and Azure Blob triggers without vendor lock-in. I've done this for 3 clients migrating between clouds. - DATA QUALITY GATES: Set up dbt tests with custom macros that block bad data before it hits your warehouse. I'll add row-level lineage tracking so analysts know exactly which upstream source caused anomalies. - REDSHIFT + BIGQUERY + SNOWFLAKE: Design distribution keys and clustering strategies tailored to each platform. I've reduced query costs by 40% just by fixing sort keys and materialized views. I've spent 12 years building pipelines that don't break at 3am. I'm fluent in Portuguese and have led data teams across fintech and e-commerce in LATAM. Let's schedule a 20-minute call to walk through your current architecture and identify where you're losing performance or burning cloud costs unnecessarily.

@ashinnd84

Hello there, we are a team of MERN Stack , AI, ML, Cloud computing, Senior Full Stack Web and Mobile App Developers. Please, send me a message to discuss the work. Thanks Ashish Kumar.

@nutankumarftp

As an experienced full stack developer with over a decade in the industry, I have cultivated a deep understanding and indispensable skill set in data engineering that aligns perfectly with your project needs. During this time, I have honed my proficiency in database design, management, and optimization - skills that are crucial for successful end-to-end pipelines and models. My ability to work with large, messy datasets and tame them into clean, reliable structures will also be invaluable for your team. Moreover, my extensive knowledge of the three major cloud platforms - AWS, GCP, and Azure - along with my fluency in SQL and Apache Spark will allow me to immediately dive into your most critical data engineering tasks. I am deeply familiar with Airflow, Kafka, dbt, and have worked with databases like Redshift, BigQuery, and Snowflake so adapting to your current tech stack will be seamless.

@joaov210

Hi, This role lines up well with the kind of work I’ve been doing—owning data pipelines end to end and making sure they’re not just running, but actually trusted by the business. I’ve spent the last several years building and maintaining data workflows that deal with messy, real-world datasets and turning them into clean, reliable models. Most of my work has involved designing pipelines from ingestion to transformation to serving, with a strong focus on performance, monitoring, and long-term stability. On the stack side, I’m very comfortable with SQL-heavy environments and have worked with Spark for large-scale processing, including optimizing jobs for cost and speed. I’ve also used tools like Airflow for orchestration, dbt for modeling, and Kafka/Kinesis-style systems for streaming pipelines. For warehouses, I’ve worked across setups similar to Redshift, BigQuery, and Snowflake depending on the project. I’m also used to working across cloud platforms (AWS, GCP, Azure), especially when systems aren’t cleanly contained in one environment. That includes handling deployments, storage layers, and data movement between services. What I care about most is making data reliable—setting up validation, monitoring, and clear models so analysts and downstream systems can depend on it without second-guessing. I’m comfortable collaborating in English with distributed teams and working closely with product stakeholders to translate business needs into solid data solutions.

@Rsamreen

With five years of experience in the data engineering field, I possess the skills and knowledge to exceed your expectations for this project. I am deeply conversant in SQL and have a commendable track record when it comes to taming and shaping complex datasets, ensuring that they are reliable, scalable, and readily usable by analysts. Beyond mere expertise, my work with SPSS, Python, R, Pandas, NumPy, plotly has provided me with a comprehensive understanding of different programming approaches that could contribute positively to this role. One of the highlights of my career is my proficiency in working with various Clouds - AWS, GCP and Azure which aligns perfectly with your requirements. I believe that familiarity with not just one cloud provider but multiple ecosystems can make a significant difference when designing end-to-end pipelines. Additionally, I am already well-versed with Spark which is leveraged heavily for your performance-critical jobs. What sets me apart from other applicants is my ability to work closely and efficiently even in fully remote settings. Being native in Portuguese and proficient in English at a C1/C2 level ensures there would be no language barrier while collaborating with distributed stakeholders.

@salesapple

Hi, I understand the issue: you need a data engineer to design end-to-end pipelines, handle large datasets, and build reliable, scalable data models across AWS, GCP, and Azure. I’ve solved similar data engineering challenges involving messy datasets, Spark optimization, and multi-cloud pipeline deployments causing performance and reliability issues. Approach: • Identify root cause (data flow/logs/performance) • Apply clean, scalable pipeline design • Verify with data quality checks I bring strong SQL, Spark, Airflow, and warehouse experience (Redshift, BigQuery, Snowflake). I can start with a quick review first. Shall we begin? – Shubham Verma

@shreyasp2207

Hello, I am a Data Engineer with 4+ years of experience in building scalable data pipelines, ETL/ELT workflows, and cloud data warehouse solutions. My expertise includes Snowflake, SQL, Python, DBT, AWS, and GCP, with hands-on experience in enterprise data migration and analytics modernization projects. In my current role at TCS, I work on migrating large-scale data from SQL Server to Snowflake, building ingestion pipelines using Snowpipe, Streams, Tasks, and developing transformation pipelines with DBT following Medallion Architecture. My relevant experience includes: - Building production-grade ETL/ELT pipelines - Snowflake performance optimization and data modeling - DBT incremental models, snapshots, tests, and documentation - Workflow orchestration using Airflow - Working with AWS S3 and GCP services for cloud-based data solutions Additionally, I hold: - Snowflake SnowPro Core Certification - Snowflake Associate Platform Certification - Snowflake Hands-on Badges in Data Warehouse, Data Lake, and Data Engineering While my strongest expertise is in Snowflake, DBT, AWS, and GCP, I am a fast learner and comfortable adapting to new tools and business requirements. I would be interested in discussing how my data engineering background can support your team. Best regards, Shreyas Patil