Log Processing & Visualization Tool
LogMerge
A concurrent log processing tool in Go with a real-time web dashboard, role-based auth, and SSE live streaming.
Problem
Developers needed a way to aggregate, filter, and visualize log data from multiple sources in real time without relying on heavy external tools.
Solution
Built a concurrent pipeline using a goroutine-per-file reader and a fan-in pattern that merges the resulting channels into one time-ordered stream. Added pluggable filters (level, keyword, regex), a real-time web dashboard with SSE live streaming, and role-based authentication with CSRF protection and per-IP login rate limiting.
System Architecture & Data Pipeline
Concurrency Pipeline: Fan-in concurrency pattern using Goroutines per log source and non-blocking Go channels, broadcasting live telemetry to the web dashboard via Server-Sent Events (SSE).
Highlights
- Built a concurrent log processing pipeline using a goroutine-per-file reader and a fan-in pattern to merge multiple log sources into one time-ordered stream.
- Built a real-time web dashboard with SSE live streaming, role-based authentication (admin/viewer), CSRF protection, and per-IP login rate limiting.
- Covered the pipeline, filters, and auth modules with 30+ unit and integration tests, and packaged the app with a multi-stage Docker build.
Impact
Built a Go application with goroutine-based concurrency, a role-based auth system (bcrypt, CSRF protection, rate limiting), and a real-time SSE dashboard. Covered by 30+ unit and integration tests, with Docker support for deployment.