Real-Time Network Traffic Anomaly Detection
A production-ready system that ingests live network traffic via Kafka and Apache Spark, applies multiple ML algorithms to detect intrusions and DDoS attacks in real-time, and surfaces findings through an interactive Streamlit dashboard.
View on GitHubSimulated real-time anomaly detection — alerts stream live and model performance metrics update automatically.
2.4M
Packets Analyzed
last hour
347
Anomalies Detected
last hour
1.8%
False Positive Rate
ensemble avg.
42ms
Avg. Latency
detection time
Anomaly Detections Over Time
Detections per model, rolling 10-minute window
- Isolation Forest
- Autoencoder
- K-Means
- DBSCAN
Model Performance Metrics
Precision, Recall & F1 Score on CICIDS2017 benchmark
- Precision %
- Recall %
- F1 Score
Real-Time Streaming Pipeline
Kafka + Apache Spark architecture ingests live network traffic and processes packets in real-time with sub-second latency.
Anomaly Detection Models
Ensemble of Isolation Forest, Autoencoder (deep learning), K-Means, and DBSCAN clustering for robust intrusion detection.
Multiple Data Sources
Supports the CICIDS2017 benchmark dataset and live Wireshark/Scapy packet captures for both retrospective and live analysis.
Live Streamlit Dashboard
Interactive dashboard with real-time alerts, anomaly statistics, and model performance metrics updated continuously.
Docker & CI/CD
Fully containerized with Docker Compose and automated GitHub Actions pipeline for testing and deployment.
ETL & Feature Engineering
Structured ETL pipeline for connection logs, packet parsing, and advanced feature extraction aligned with OSI model layers.
An ensemble of four algorithms provides complementary coverage across different anomaly types.
Isolation Forest
Unsupervised anomaly detection
Isolates anomalies by randomly partitioning the feature space. Highly effective for high-dimensional network traffic data.
Autoencoder (Neural Network)
Reconstruction-based detection
Learns a compressed representation of normal traffic. High reconstruction error signals an anomaly or intrusion.
K-Means Clustering
Cluster-based outlier detection
Groups traffic flows into clusters; data points far from cluster centroids are flagged as suspicious.
DBSCAN
Density-based detection
Identifies anomalies as low-density regions in the feature space, robust to noise and arbitrary cluster shapes.
Ingest
CICIDS2017 dataset or live Wireshark packet capture via Scapy/PyShark
Stream
Kafka producer publishes traffic flows; Spark/Flink consumer processes in real-time
Detect
Ensemble ML models score each flow; anomalies trigger live alerts
Visualize
Streamlit dashboard displays metrics, alerts, and model performance in real-time