📚🧪 Traffic Sentinel is a learning-focused POC that explores a scalable IoT architecture using Fog nodes and Apache Flink to process 📷 IP camera streams, powered by YOLO for intelligent 🚗 traffic monitoring on highways. 🛣️

Apache Flink examples designed to be run by AWS Kinesis Data Analytics (KDA).

Simple example for access count in Apache logs using Apache Flink

A Makeshift data infrastructure setup for datafirstjobs.com.

A streaming data pipeline uses Kafka as the backbone and Flink for data processing and transformations. Kafka Connect is used for writing the streams to S3 compatible blob stores and Redis (low latency KV store for real-time ML inference). Spark is used for the batch job to backfill the ml feature data.

Setup for realtime data streaming using Kafka, Flink, Pinot, MySQL, Postgres and Superset

Automated deployment of an Apache Flink cluster in your Grid'5000 reserved nodes.

Real Time Ad Performance & User Engagement Tracking Platform

we are thrilled to announce our new PoC project aimed at providing a complete real-time extraction, transformation, and exposure architecture for the new provincial transportation systems.

AutostreamPipe: LLM Assisted Automatic Stream Processing Pipeline Generation

This project demonstrates practical strategies for detecting and mitigating Kafka Poison Pills. A malformed or problematic messages that can break consumers.

This project demonstrates a practical end-to-end solution for building a real-time data lake. It showcases how to integrate Apache Kafka, Flink, Hadoop, and Apache Iceberg to stream data from a FastAPI application into a scalable, reliable data lake architecture.

🍔 Build a production-grade data pipeline for food delivery, enabling real-time insights and features through PostgreSQL, Kafka, and Flink.

Proyek Real-Time Web Sessionization with PyFlink membangun pipeline streaming menggunakan Apache Flink (PyFlink – Table API) untuk melakukan sessionization data web secara real-time. Pipeline ini membaca event JSON dari Kafka, mengelompokkan aktivitas pengguna berdasarkan IP dan host dengan session gap lima menit.

This project demonstrates data cleaning, processing with Apache Spark and Apache Flink, both locally and on AWS EMR.

This project showcases a real-time data streaming pipeline using Apache Flink, Apache Spark, and Grafana. It streams data, stores it in Parquet format, and performs aggregations for insights, with seamless visualization via Grafana dashboards.

Stream Processing of website click data using Kafka and monitored and visualised using Prometheus and Grafana

End-to-end data pipeline: PostgreSQL CDC (Debezium) → Kafka → Flink SQL → OpenSearch (search) + Redis (features). Local Docker stack, sample data, and verification scripts.