Kafka data ingestion trabalhos

Continuar projeto de construção de nossa infraestrutura em kubernetes, com apache kafka, para filas, em ambiente de produção no google e homologação no Hostgator, migrar banco de dados Mongodb, para google, fazendo backup internamente no GCP e fora do google para termos redundância para um plano de contingencia, criar CI/CD para automatizar os processos de deploys.

oi Renato tenho projeto para Kafka teria interesse?

Oi Robson teria interesse em trabalhar num projeto de Kafka?

...uma aplicação simples IoT para aprendizado e que fará parte de minha pesquisa. <br />Atualmente, a aplicação trabalha com alto fluxo de dados e processamento em tempo real. Para isso, minhas premissas são: Java, Apache Storm, Apache Kafka, Apache Zookeeper e um gerador de JSON que irá alimentar o Kafka (tenho tentado utilizar este: ).<br /><br />O fluxo da aplicação é simples, porém as dificuldade que estou tendo é em fazer estes frameworks conversarem. É obrigatório que eu faça em Java e utilize Apache Storm, Kafka, Zookeeper e um gerador de Json. Futuramente irei converter para um hardware real, por isso os dados precisam vir de um g...

...seeking a skilled developer to set up and program a framework using Apache Kafka for distributed messaging. The project aims to establish an efficient and scalable messaging system that can handle high-volume message processing and filtering. Project Overview: The developer will be responsible for setting up Apache Kafka on a Windows environment and developing a master script and worker scripts to facilitate distributed messaging. The key requirement is to implement a mechanism for bulk-reviewing messages in the queues every 10 minutes to filter out unnecessary messages and optimize system performance. Responsibilities: Apache Kafka Setup: Install and configure Apache Kafka on a Windows machine. Set up the necessary Kafka topics, partitions, and brok...

My project requires an experienced Java SpringBoot developer with a strong understanding of Kafka for asynchronous communication. The ideal candidate should also have: - Proficiency in building and managing REST APIs, focusing on CRUD operations - Ability to integrate with external APIs and service data repository calls - Expertise in crafting and handling SQL queries, preparing for moderate traffic and frequent daily queries - Competence in setting up batch schedulers - Proficiency in sending email notifications This project offers an excellent opportunity to apply and enhance your Java SpringBoot and Kafka skills.

Looking for a skilled developer to assist with configuring Apache Kafka for a distributed messaging system. I have running zookeeper, kafka-server and kafka-connect. the project requires: 1) configure kafka source connector for receiving a Json and store it on a topic. 2) configure a kafka sink connector for storing the json data into a PostgreSQL table 3) guidance for installing the plugins 4) setup the connectors over kafka-api the kafka server will be running on ubunut 24.x without a docker Ideal Skills and Experience: - Familiarity with Apache Kafka platform - Experience in distributed messaging - Strong problem-solving skills - Attention to detail We are eager to work with someone who is able to provide solutions ...

...CERTIFICATIONS: Google Advanced Data Analytics Professional Certificate | Google - View Credential Google Cybersecurity Professional Certificate | Google - View Credential Machine Learning Engineering for Production (MLOps) Specialization | - View Credential Deep Learning Specialization | - View Credential SKILLS Programming: Python, JavaScript, RUST, Golang, C, C++ Frontend Development: HTML, CSS, JavaScript, ReactJs Responsive design, Cross-browser compatibility Web performance optimization Backend Development: , Flask, RESTful API Microservices, Websockets, OAuth Nginx server management, Kubernetes, Kafka CI/CD integration Databases: RDBMS (MySQL, PostgreSQL) NoSQL (MongoDB) Query optimization, ORM/ODM (SQLAlchemy, Mongoose) Data warehousing, Distributed

I'm in search of a seasoned DevOps consultant with strong expertise in Kafka and Akka. The role centers around the design, implementation, and monitoring of our cloud-based infrastructure on AWS. Strong scala programming skills and experience with Scala frameworks such as Akka. Efficient in designing and developing REST API’s. Experience with SQL and NoSQL databases. Proficiency in software design patterns and principles. Experience with version control tools such as Git. Good understanding of Agile methodologies and a collaborative mindset. Excellent problem- solving and analytical skills. Experience in unit testing (eg. JUnit/ Mockito). Experience in cloud- based environments such as AWS or Azure. Experience with Kafka and Elasticsearch. Experience : 5+ years ...

I'm in need of a proficient Node.js developer for a project that involves Kafka. Here's an overview of what I'm looking for: The ideal candidate should have experience with: - Node.js - Kafka - Building, implementing, and debugging Kafka workflows Please get in touch if you can meet these requirements.

I need an expert to conduct a local PrivateGPT installation on my Linux server with a pgvector database configuration. This setup aims to ingest various document and data file formats to enable a local Retrieval-Augmented Generation (RAG) Question-Answering system. Here are some details: - **Objective**: Incorporating GPT technology to generate a question-answering system. - **Document/Data Types**: JSON, CSV, PDF, XLSX, DOCX, TXT files - **Skills Required**: - Proficiency in GPT - Experience with Linux server configuration - Familiarity with document and data ingestion into pgvector database. You should be able to ensure full functionality and provide guidance or documentation on how to use and maintain the system upon project completion.

...intermediate level big data engineers who can tackle essential tasks within the domain. The ideal candidate for this project should be able to: - Perform data analysis: This includes understanding, interpreting, and presenting data to inform decision-making. - Manage data integration: The candidate should be capable of combining data from various sources to create a cohesive dataset. - Develop data visualization: The engineers will need to create meaningful and understandable visual representations of the analyzed data. The necessary skills for this role are: - Proficient in Python: The candidate should be well-versed in Python programming language to work with our system. - Knowledge of Hadoop: Experience with Hadoop is a must for thi...

Project Description: We are looking for a skilled Java developer with experience in Apache Flink (version 1.19) to create a data processing job that reads from parquet files located on a Network File System (NFS) and sends selected data to a Kafka topic. Core Responsibilities: - Develop a Flink job using Java that can read parquet files from NFS. - Implement functionality to allow the selection of specific columns from the parquet files to be processed. - Ensure the selected data is published to a specified Kafka topic efficiently. - Manage dependencies and libraries required to execute the job without compatibility issues.

I'm seeking an expert who can help me integrate Kafka and UiPath for the purpose of automating specific business processes that involve financial transactions. Key Responsibilities: - Develop a seamless integration between Kafka and UiPath, using code stage in C# or .Net Skills and Experience: - Proficiency in Kafka and UiPath - Proven track record in automating - Strong understanding of data streaming and business automation - Ability to provide clear documentation for the implemented solution Please bid if you have relevant experience in this domain and can deliver a robust and efficient solution for our needs.

Deployable data engineering Project Idea: Real-time Sentiment Analysis for an Online Store 1. Overview: - Develop a web application for an online store where users can submit reviews for products. - Utilize Apache Kafka for real-time event streaming. - Implement Apache Airflow for workflow orchestration. - Use Apache Spark for data processing and sentiment analysis. - Store processed data in Cassandra DB for recording activities. - Store analyzed data in Amazon S3. 2. Workflow: - User submits a review through the web UI. - The review data is streamed to Kafka topics. - Airflow schedules tasks to ingest data from Kafka, process it using Spark, and store it in Cassandra. - Spark performs sentiment...

I'm seeking a proficient developer to build a comprehensive demo or starter code integrating Snowflake and Kafka using AWS cloud. The ideal candidate should have deep understanding of concepts such as: - Data ingestion - Data transformation - Data storage and retrieval The code should ensure: - Optimal performance - Scalability - Security The code can be written in Python or Java. In-depth knowledge of these languages, as well as experience with Snowflake and Kafka integration, is a necessity for a successful bid on this project.

Je suis en train de développer un mini projet qui intègre Jupyter et Kafka avec Spark Streaming pour le traitement des données, et utilise Cassandra pour le stockage. Pour la visualisation et la gestion des alertes, j'emploie Grafana. Le projet inclut également une fonctionnalité spécifique pour afficher la température de différentes régions, gérée de manière aléatoire. Je suis à la recherche d'une personne capable de prendre en charge le développement de ce projet. Je suis disposé à rémunérer pour ce service. Si vous avez l'expertise requise et êtes intéressé par cette opportunité, n'hésitez pas &a...

I'm using Kafka 2.6.0 with a sink connector to Postgres on Confluent Cloud. The issue I'm facing is due to extensive modifications to the default configurations for the sink connector. Key Points: - The primary issue I'm encountering is related to the configuration of the sink connector. - The connector is not functioning correctly, and data is not arriving in Postgres as expected. - I have made significant modifications to the default configurations for the sink connector. I need an expert who is proficient in Kafka, Confluent Cloud, and sink connector configuration to help debug and rectify the issue. You should be able to: - Analyze and troubleshoot the sink connector configuration to identify the root cause of the problem. - Make necessary amend...

...service that can help in calculating features for real-time streaming of stock tick data. This application needs to be proficient in: - Maintaining the full lifecycle of stocks for specific time periods like pre, trading, and closing per day. - Customizable day feature(calculate in startOfDay) and tick feature (calculate when tick comes in) real-time calculation. - High performance for calculating simultaneously. - An API that can serve all features for a specific stock for the current day for training input - Support for customizable trading logic triggering based on market snapshot features every N seconds. - Monitoring progress status (per day, and days) - support backtesting using mongodb data or kafka - better to be linux workable after simple switch since no r...

...storage services for data persistence and retrieval. - Implement best practices for error handling, retry mechanisms, caching, monitoring, and logging. - Ensure the application adheres to security best practices and website terms of service. - Collaborate with our development team to provide knowledge transfer and documentation. Requirements: - Strong experience in designing and architecting scalable web scraping solutions. - Proficiency in Node.js and familiarity with the NestJS framework. - Expertise in cloud technologies and services (AWS, Google Cloud, or Azure). - Knowledge of serverless architectures, containerization, and orchestration technologies like Kubernetes or Docker. - Experience with message queues and event-driven architectures (e.g., RabbitMQ, Apache Kafka...

...(Contract) Technical: Master's degree with 10 – 12 years related experience, of which at least 7 + years in development, 4+ years of product engineering experience Experience in Analytics and Business Intelligence is a plus Strong design and coding experience Expertise with the Go Lang platform or higher Experience with integration, data handling technologies and other open source frameworks such as Spring, Kafka, Elasticsearch, Solr etc Experience in NLP, Data Science and Text analysis is a plus. Good Knowledge for Python and GoLang. Experience in building event driven cloud native micro services architectures Experience in building ELK stack Hands on experience in Mongo, Redis, Cassendra, Neo4j, Postgres is a plus Experience with massive-sc...

Need assistance with the development of a well-structured, efficient data pipeline. This system is expected to integrate a range of data sources, including APIs, DB views and SFTP, and prepare them for backend ingestion in CSV format. Key Responsibilities: - Source data from APIs, DB views, and SFTP - Transform ingested data into CSV format Ideal Skills and Experience: - Expertise in creating and managing data pipelines - Proficiency in various data integration methods, specifically APIs, DB views, and SFTP - Excellent knowledge of data transformation and loading processes Please note: While optional transformations like data filtering, aggregation, and mapping are not explicitly required, the ideal candidate should be able ...

As the professional handling this project, you'll engage with big data exceeding 10GB. Proficiency in Python, Java, and Pyspark are vital for success as we demand expertise in: - Data ingestion and extraction: The role involves managing complex datasets and running ETL operations. - Data transformation and cleaning: You'll also need to audit the data for quality and cleanse it for accuracy, ensuring integrity throughout the system. - Handling Streaming pipelines and Delta Live Tables: Mastery of these could be game-changing in our pipelines, facilitating the real-time analysis of data.

...in full-stack technical development encompassing Java, Cloud, Web, UI, and mobile technologies, as well as exposure to Domain-Driven Design, Event-Based Architecture, API, and microservices architecture. The ideal candidate should possess a strong enthusiasm for emerging technologies and demonstrate a proficient understanding of various technologies spanning Cloud, UI, Application, Security, and Data architecture domains. Roles and Responsibilities:- • Your role will involve joining Fiserv's Global Issuer organization's Architecture team to lead the development of new cloud-native applications. Relevant experience in digital solutions within the payment solutions domain will be highly valued. • As the Lead Engineer, you will be expected to comprehend a w...

Looking for crash course on "Apache Kafka" Tutor

As a client, I'm seeking a Python developer who that can help me trasforming data. The primary source of the data will be company-related information. Ideal Skills and Experience: - Proficiency in Python - Extensive experience in data analysis, particularly with telecomdata - Familiarity with APIs for data collection - Knowledge of telecom - Ability to understand Python application and python scripts -Know about Airflow , Spark, Kafka, Transaction Mysql Key Requirements: - Develop a Python program that can efficiently trasform non useful to useful data. - Strong understanding of telecom data -Spark sessions, ETL ,Kafka, Mysql

...protocol of a list of sites to augment the LLM. Key Criteria for Success: Robust technical execution that meets core requirements Modular, extensible architecture for flexibility over time Quality, well-tested code delivered on time Clear, prompt communication throughout project Key Components: Text/file ingestion and vectorization Semantic similarity search in Milvus/SPTAG Local post-hoc model explanations Production-ready Flask/FastAPI UI Documentation and usage guides Document Ingestion (spaCy, Beautiful Soup, PyMuPDF): Scrape/ingest docs, extract text, handle formats Vectorization (Sentence/Doc2Vec): Convert text to distributed representations Vector Database (Milvus, SPTAG): Store/index vectors at scale for retrieval Semantic Search (FAISS): Leverage ANN for related...

...of a Full-Stack developer well-versed in Apache Kafka for configuration, message serialization, deserialization, event streaming, and processing. Additionally, you will be responsible for developing Spring Boot microservices that encompass user authentication, authorization, database integration, data modeling, and REST API development. Key responsibilities include: - Kafka setup and configuration - Kafka message serialization and deserialization - Kafka event streaming and processing - Designing and implementing user authentication and authorization within Spring Boot - Integrating databases and modeling data effectively - Developing REST APIs with Spring Boot The ideal candidate for this project should have extensive experience with Kafk...

Need a apache kafka tutor for self learning

I need a Node.js developer with expertise in transitioning a fully operating monolithic application to a microservices architecture using Kafka, Redis Server, and Openshift. A successful migration will preserve current functionalities including: - User profile management - Transactions processing - Content management The deadline is paramount, thus a professional with great time management skills is ideal. Prior experience in similar projects is highly desirable. Your expertise will ensure a seamless transition with zero interference to the current users.

I am seeking a technically adept developer skilled in Java, Typescript, and Kafka for the purpose of integrating multiple technologies. This project is primarily focused on setting up a microservice architecture to efficiently process transactional data. Your responsibilities will include: - Developing robust Kafka producers and consumers to enable efficient data flow - Utilising your Java and Typescript expertise to build functional microservices Strong experience in Java, Typescript, Kafka, and microservice development is vital. An understanding of transactional data processing would be beneficial. Keen attention to detail, problem-solving skills, and a drive for tech integration efficiency are qualities sought for this project.

...Conduct a comprehensive assessment of Luxetrip's business requirements, target market, and competitive landscape to inform the platform architecture and roadmap 2. Define the key components, modules, and interfaces of Luxetrip's AI-powered travel platform, including recommendation engines, pricing algorithms, and personalization frameworks 3. Design and develop the platform's data infrastructure, including data ingestion, storage, processing, and analytics capabilities 4. Architect and implement the platform's microservices-based backend, ensuring scalability, reliability, and security 5. Collaborate with the front-end development team to define and implement the platform's user experience and interface, optimizing for conversion and eng...

I am looking for a skilled data engineer to assist with a project encompassing data ingestion and storage, transformation and processing, as well as ensuring adherence to data governance and compliance standards. Key Responsibilities: - Implementing real-time data streaming and batch data processing for the ingestion and storage of data. - Leveraging cloud-based data storage solutions, particularly within the Azure framework, to ensure scalability and reliability. Data governance and data protection Desired Outcomes: - Data transformation and processing should focus on efficient and effective data cleaning and normalization. - Further, data aggregation and summarization should help in derivi...

I am in need of a data engineer with strong design architecture skills to create a databrics UI UX prototype for the primary purpose of processing flat and text files. Steps - create a generic pipeline that can process data files from multiple formats and frequency upto 1 tb/day. Create the design in databrics UI. There should be multiple approaches/architectures to design the requirements. This project specifically targets data engineers, requiring the design to enhance how they interact with data and improve efficiency in their daily tasks. Your role will encompass: - Designing an interface that simplifies the data ingestion process - Crafting intuitive tools for effective data transformation - Integrating seamless data integr...

Por favor, Cadastre-se ou Faça Login para ver os detalhes.

...stakeholders to gather and understand visualization and analytics requirements for the platform. • Design a comprehensive cloud-native architecture for the visualization and analytics platform, encompassing data ingestion, processing, storage, visualization, and analytics components. • Leverage your expertise to recommend open-source technologies, off-the-shelf software tools, or a combination of both to implement the chosen architecture. • Translate the overall architecture into a specific AWS cloud implementation plan. o Select appropriate AWS services for each component of the architecture (e.g., data storage, compute, analytics services, visualization tools). • Consider cost optimization strategies for the AWS cloud implementation • Iden...

I'm searching for a PySpark expert who can provide assistance on optimizing and debugging current PySpark scripts. I am specifically focused on PySpark, so expertise in this area is crucial for...assistance on optimizing and debugging current PySpark scripts. I am specifically focused on PySpark, so expertise in this area is crucial for the successful completion of this project. Key Responsibilities: - Optimizing PySpark scripts to improve efficiency - Debugging current PySpark scripts to resolve existing issues Ideal Candidate: - Proficient with PySpark - Experience in big data management, data ingestion, processing, analysis, visualization, and reporting - Strong problem-solving skills to identify and resolve issues effectively - Knowledgeable in perfo...

We are looking for a developer for a LONG TERM cooperation for a developer to participate in the product development PLEASE EXPLAIN IN YOUR ANSWER WHY YOU WOULD LIKE THIS OPPORTUNITY AND WHY YOU ARE A GOOD CANDIDATE FOR IT We would like to have excellent skills in - Angular (version 11+) - Java springboot - Ionic Mobile Apps - MySQL - Kafka Other good to have skillls - Understanding of e-commerce catalog & order data model - Payment provider integration - Bluetooth & POS printer connectivity - Java FX for desktop application - nice to have No agencies, please! Ideally, we would like a developer that would be able to dedicate 20 - 40 h per week, so it is ok to have another job too but 8h/day can be considered Must have - Excellent English / communication skills - ...

...updated by reading data from a TXT file. The process must be automated via a script, filling in blank cells, distributed across all the sheets in the document. Key requirements include: 1. The ability to read and accurately interpret all fields from the TXT file. 2. Updated data should be posted to all sheets within the excel document, in the required cells. 3. In the case of data discrepancies within the TXT file, enable an automated resolution by defaulting to a specific value. Leveraging your expertise in data extraction, manipulation, and Excel automation will be instrumental for this project. Ideally, you are proficient in scripts suitable for handling such tasks, debatably the likes of Python or VBA, contingent on your recommendation. Familiarity ...

...specializes in Kafka and Genesys system integration. The following is the tasks and main aim for this project: i) Deploy (Apache) Kafka on existing YTLC CES server and ensure it is compatible with the required Co-Browse version ii) Integrate Genesys Co browse to Kafka for it to send the Co-Browse Reporting Event to Kafka iii) Integrate Infomart to Kafka for it to extract data from the Kafka data stream and transforms it into the COBROWSE_FACTtable iv) To ensure / enable RAA to summarize and organizes the Info Martdata in ways that enable GI2 to extract meaning. v) To support required Custom Report template based on available reporting data field and ensure the required Co-browse reporting data appear in the GI2 Custo...

...specializes in Kafka and Genesys system integration. The following is the tasks and main aim for this project: i) Deploy (Apache) Kafka on existing YTLC CES server and ensure it is compatible with the required Co-Browse version ii) Integrate Genesys Co browse to Kafka for it to send the Co-Browse Reporting Event to Kafka iii) Integrate Infomart to Kafka for it to extract data from the Kafka data stream and transforms it into the COBROWSE_FACTtable iv) To ensure / enable RAA to summarize and organizes the Info Martdata in ways that enable GI2 to extract meaning. v) To support required Custom Report template based on available reporting data field and ensure the required Co-browse reporting data appear in the GI2 Custo...

I am looking for a skilled developer experienced in Spring Boot, Kubernetes, Dockers, Kafka, and AWS deployment to build a microservices app. This project requires: - User Authentication: The app should be able to handle user signup, login, password recovery, and security checks. - Data Processing: Facilitating data collection, organization, processing, and efficient management. - Third Party Service Integration: Seamless interoperability with other external applications or systems for enhanced functionality. The app is geared towards the general public and business-to-consumer (B2C). It is therefore essential that the app is user-friendly and aligns to the best industry practices for B2C applications. The ideal freelancer should have a proven track record in de...

We require a highly skilled Kafka expert with deep expertise on real production environments. We want to have a high available solution which is also fail tolerant to destination processing systems Strong expertise on Kafka and how to read data from one system, parse this data reliably with options to debug later and finally post data to the destination system What is not required: - HA solutions of kafka - high load - failsave Budget? will not be disclosed, place your best bid to get considered What is next? We will share you a NDA and afterwards a paid test task. Payment? - you estimate in a WBS (optimistic, expected, pessimistic, where optimistic < expected < pessimistic) after getting the task - we discuss about cle...

...only a few hours!! I'm seeking a Python developer with experience in data analysis and visualization to accomplish a beginner-level project. The primary feature will be the utilization of business data for analysis and processing. Key attributes of this project include: - **Skillsets Required** - Python programming, specifically for data analysis. - Experience with libraries such as Pandas, NumPy, Matplotlib and Seaborn. - Comprehension of business and financial data. - **Project Outcomes** - Ingestion and processing of hotel data. - Generation of interactive visual graphs from processed data. Please note, the ultimate expectation is to deliver a Python solution that can analyze, process, and transform data i...

For this task, I require a skilled developer with solid background in Docker and Kafka. The task at hand is to work on an existing Docker Compose script for a Kafka application centered around authorization. Key Requirements: - Incorporating SASL/SCRAM authentication mechanism. - Focusing on the SCRAM-SHA-256 version for Kafka authentication. A deep understanding of the listed technologies and profound experience in adding security features to Kafka applications will be a significant advantage. Secure, reliable streaming is the goal. Therefore, the freelancer needs to be knowledgeable about how SASL/SCRAM authentication enhances Kafka application's security. Only bid if you are highly confident you can deliver my exact requirement.

*Title: Freelance Data Engineer* *Description:* We are seeking a talented freelance data engineer to join our team on a project basis. The ideal candidate will have a strong background in data engineering, with expertise in designing, implementing, and maintaining data pipelines and infrastructure. You will work closely with our data scientists and analysts to ensure the smooth flow of data from various sources to our data warehouse, and to support the development of analytics and machine learning solutions. This is a remote position with flexible hours. *Responsibilities:* - Design, build, and maintain scalable and efficient data pipelines to collect, process, and store large volumes of data from diverse sources. - Collabora...

I need a Airflow expert to help redesign my data flow and put in place a solution that is scalable and can account for future needs. Our basic flow is making API calls to third-party solutions and getting data for various customers. This requires using the right API keys for each customer. The data will then be put in a Kafka queue for further processing. I am looking for someone with experience to help think through and implement solutions.

...am looking for a skilled developer proficient in Apache Camel, Spring Boot, Kafka, and Ksqldb. The goal is to integrate Salesforce, ERP, and SAP using a cloud to cloud rest api approach. The successful applicant will have extensive experience leveraging these technologies to implement both staging and production deployment. Key Responsibilities: - Build a middleware solution for cloud to cloud rest api integration. - Ensure this solution adheres to best practice for handling transactions. Specifics: - An existing REST API for the Salesforce integration is present. - The Salesforce integration needs to support order management functionality. Ideal Skills: - In-depth knowledge of Apache Camel, Spring Boot, Kafka, Ksqldb. - Prior experience with Salesforce, ERP, and ...

...(IngressRouteTCP). The primary goal for this project is to leverage these components for efficient routing and connecting on kafka from external port 9093 or 9094 by hostname. Example client command: "kcat -b -L -J" I have already independently deployed a Minikube cluster with Strimzi and Traeffik, everything works fine inside the cluster, but to connect from the outside you need to connect via kafka :cluster-ip, and I can't do it. I know how to create self-signed certificates and a trusted certificate authority, so it shouldn't be difficult to quickly generate and and Key Responsibilities: - Configurate strimzi kafka :cluster-ip and tls:true - Configurate traefik kind: IngressRouteTCP - Configurate TLS self signed certificate

Por favor, Cadastre-se ou Faça Login para ver os detalhes.