Enhancing Data Infrastructure for Trading Platform

₹12500-37500 INR

Fechado

Publicado há aproximadamente 2 meses

₹12500-37500 INR

Pago na entrega

Project Status Report: Data Architecture Evolution Implemented 1. Infrastructure (Docker) We have successfully set up the core infrastructure using docker-compose. The following services are running and healthy: Kafka (Port 9094): For high-throughput market data ingestion. Redis (Port 6379): For real-time "hot" data storage (latest prices). ClickHouse (Port 8123): For "warm" data storage (historical OHLC data for charts). MinIO (Port 9002): S3-compatible object storage for "cold" data archival (raw ticks). Elasticsearch & Kibana: For system observability and logs. 2. Kafka Producer Service Created KafkaProducerService in NestJS. Configured to connect to the local Kafka broker. Ready to publish messages to topics. In Progress / Next Steps 1. Ingestion Layer (Binance Integration) Goal: Stream live market data from Binance instead of using simulated data. Implement Binance WebSocket: Connect to Binance's btcusdt@trade stream. Publish to Kafka: Forward incoming trades to the market_ticks Kafka topic using the KafkaProducerService . 2. Hot Storage (Redis) Goal: Provide sub-millisecond price updates to the frontend. Redis Consumer: Create a service to consume from Kafka market_ticks. Update Cache: Store the latest price in Redis keys (e.g., PRICE:BTCUSDT). Pub/Sub: Publish updates to a Redis channel for the WebSocket gateway to broadcast to the frontend. 3. Warm Storage (ClickHouse) Goal: Store historical data for fast charting. Table Schema: Define ticks and ohlc tables in ClickHouse. Batch Ingestion: Create a consumer to batch insert Kafka messages into ClickHouse. 4. Cold Storage (S3/MinIO) Goal: Archive raw data for compliance and backtesting. Parquet Archiver: Periodically flush data to Parquet files and upload to MinIO. 5. Frontend Integration Goal: Visualize the live data. WebSocket Client: Update the React frontend to listen to the new real-time stream. Chart API: Update FinancialChart to fetch history from the new ClickHouse-backed API. 6. Account Management (New) Goal: Enforce account limits and expiry. Demo Account: Limit to 1 per user, valid for 15 days. Purchased Accounts: Limit to 2 per user. 7. Aurora Credits (Gamification) Goal: Incentivize user engagement. Earning Logic: Implement rules for Login Streak, Trades, Profit, etc. Spending Logic: Cap credit usage at 40% of challenge fee. 8. Account Packages Goal: Configure tiered challenges. Standard: Balanced risk (80% profit split). Premium: Higher risk/reward (90% profit split). Pro: 100% payout flagship. Seeding: Populate database with these package configurations. 9. Educational Video Platform (New) Goal: Host and stream educational content securely. Storage/CDN: Use MinIO (local) / S3+CloudFront (prod). Access Control: Ensure only users with active accounts can view videos (Signed URLs). Uploads: Admin interface for uploading content.

ID do Projeto: 40166734

Sobre o projeto

9 propostas

Projeto remoto

Ativo há 16 dias

Quer ganhar algum dinheiro?

Endereço de e-mail

Benefícios de ofertar no Freelancer

Defina seu orçamento e seu prazo

Seja pago pelo seu trabalho

Descreva sua proposta

É grátis para se inscrever e fazer ofertas em trabalhos

9 freelancers estão ofertando em média ₹16.111 INR for esse trabalho

@abdf2010

Hello there, I reviewed your project Enhancing Data Infrastructure for Trading Platform and understood the requirements at a high level. I focus on delivering clear, stable, and maintainable solutions aligned with the actual scope, I can work with Python, Big Data Sales, Hadoop and follow a clean development process with proper structure and error handling. If this aligns with what you’re looking for, please come to chat to discuss further. Best regards

₹12.500 INR em 7 dias

5,0

(2 avaliações)

4,3

@anirudhat

Hi! I'd be happy to help with this. I work efficiently and communicate clearly throughout. Available to start now - feel free to message me with any questions.

₹12.500 INR em 1 dia

3,7

(10 avaliações)

3,4

@mariamz24

Hello Sarthak A., We went through your project description and it seems like our team is a great fit for this job. We are an expert team which have many years of experience on Python, Big Data Sales, Hadoop, Redis, Elasticsearch, Docker, Data Integration, Data Architecture, API Development, Kibana Lets connect in chat so that We discuss further. Regards

₹12.500 INR em 7 dias

0,0

(0 avaliações)

0,0

@connectsurendram

Hello, I have strong experience designing and implementing end-to-end data pipelines for real-time and historical analytics. I can build a fully automated workflow that ingests live data, processes it reliably, and delivers ready-to-use dashboards and reports. What I will deliver: Real-time data ingestion using Kafka and WebSockets Automated data processing and cleaning Hot storage with Redis for ultra-fast updates Historical analytics using ClickHouse Power BI dashboards with automated refresh Export of dashboards to presentation-ready PowerPoint files Complete logging, documentation, and clear setup guide I focus on clean architecture, scalability, and zero manual intervention once the system is live. I will also provide clear documentation so you can modify data sources or visuals easily in the future. I am ready to start immediately and can share a clear execution plan before development begins. Thank you for your time and consideration. Surendra Meshram

₹20.000 INR em 10 dias

0,0

(0 avaliações)

0,0

@gsinfotechopcp4

GSINFOTECH OPC Pvt. Ltd. – Your Trusted Tech Partner Based in New Delhi, GSINFOTECH OPC Pvt. Ltd. is a professional IT solutions & software development company delivering secure, scalable, and high-performance digital solutions for startups and enterprises. We help businesses convert ideas into powerful, market-ready products. Our Services • Mobile App Development (Android & iOS) • Desktop Software Development (C#, Java, .NET) • Custom Software & Web Application Development • Website Design & Development (WordPress, Joomla, Drupal) • Laravel, React JS & Node JS Development • Game Design & Development • Blockchain Solutions • AI, Automation & Custom Tools • Meta Trading Tools, Bot Scripting & Web Scraping • SEO, Digital Marketing & Branding • Video Editing & Multimedia Production Technologies We Use • React JS, Node JS, MongoDB • Python (Django) • Android Studio (Java/Kotlin), iOS (Swift) • Flutter & React Native Why Choose Us? ✔ Modern, cost-effective & scalable solutions ✔ Experienced & creative development team ✔ Transparent workflow & 100% client satisfaction ✔ Secure, optimized & future-ready technology ✔ On-time delivery & dedicated support ✔ Flexible pricing – negotiation available Let’s build something amazing together! Hire GSINFOTECH OPC Pvt. Ltd. to take your project to the next level.

₹12.500 INR em 7 dias

0,0

(0 avaliações)

0,0

@viglundurm

Hi, I’ve reviewed your data architecture evolution project and am impressed by the scope—from real-time ingestion (Binance/Kafka) to multi-tiered storage (Redis/ClickHouse/MinIO) and React frontend integration. As a Full-Stack Developer with strong experience in .NET Core/Web APIs, React, and database-driven systems, I can help you: Build robust Binance → Kafka → Redis/ClickHouse pipelines Develop real-time WebSocket APIs for frontend price streaming Implement account management, gamification (Aurora Credits), and package logic Update the React frontend to consume new real-time data and chart APIs I’m comfortable in Dockerized environments and keen to work with Kafka, ClickHouse, and event-driven architecture. To ensure I deliver precisely what you need, let me ask: What’s the highest priority: real-time data pipeline completion, frontend integration, or account/features rollout? Do you have existing API specs or data schemas for ClickHouse and the gamification system? What’s your preferred development and deployment workflow (branching, staging, observability)? Ready to accelerate your platform’s evolution. Let’s discuss next steps.

₹12.500 INR em 7 dias