Projects
Things I've built
Personal projects are where I try the ideas that would be irresponsible to test first on someone's production data. Everything here is public, runnable and documented.
Flagship · MIT licensed · all 20 build phases complete
OpenLakehouse
A working Databricks-style data and AI platform that runs entirely on your own machine. Not a tutorial, not a diagram — real Spark jobs writing real Iceberg tables, real Trino queries against them, real Kafka change-data-capture streams, a real orchestrator and a real model registry, behind one login and one UI. Thirty-nine Docker services, one command.
Why it exists
Most "learn the modern data stack" guides ask you to stitch together a dozen SaaS trials or fight half-broken local installs. This is one command that gives you the whole thing — so you can learn it, prototype on it, or demo it without paying a cloud bill or handing your data to a vendor.
Lakehouse core
Apache Spark writing Iceberg tables through an Apache Polaris REST catalog, stored on MinIO in bronze → silver → gold layers.
Query & BI
Trino as the SQL engine with saved queries and history; Apache Superset dashboards embedded directly in the app.
No-code pipeline builder
A visual graph — source → transform → quality gate → destination — compiling down to real Spark SQL against the live catalog.
Streaming & CDC
Kafka in KRaft mode with Debezium capturing Postgres changes in real time into the lakehouse.
Orchestration
Dagster schedules and sensors driving Spark jobs and dbt-core models built against Trino.
Quality & lineage
Not-null, unique, range, regex, schema, freshness and row-count checks with scoring, plus an auto-derived lineage graph and OpenMetadata catalog.
ML & notebooks
MLflow experiment tracking and model registry, with Jupyter pre-wired to the same Spark cluster.
Security
Keycloak issuing real OIDC/JWT tokens, five RBAC roles enforced server-side on every API call, OpenBao for secrets, Traefik as the single gateway.
Observability
Prometheus, Grafana, Loki and an OpenTelemetry collector pre-provisioned, so a slow Trino query traces back to the request that caused it.
AI assistant
A local LLM through Ollama wired into the platform for in-app help — no data leaves the machine.
Inside the platform
More work
Other projects
Smaller builds, each aimed at a specific problem I wanted to understand properly.
Self-Healing Sentiment Pipeline
A sentiment analysis pipeline that repairs itself. When a task fails, a local LLM inspects the error, decides on a fix and retries — automated recovery instead of an engineer woken at 3 a.m.
End-to-End Distributed Lakehouse
A distributed lakehouse on open table formats: Iceberg for tables, Nessie for git-style branching and rollback of data, Trino for federated queries, MinIO underneath.
AWS · dbt · Snowflake
The full path from raw files in S3 through dbt transformations in Snowflake, ending in a CatBoost model served through Streamlit — the whole analytics loop, not just ingestion.
Quay Crane Monitoring
Real-time monitoring for port crane operations, turning IoT sensor streams into operational insight and predictive maintenance signals. Built from what I learned at a container terminal.
Real-Time Data Streaming
High-throughput streaming pipelines on Kafka, working through the patterns that matter at volume — partitioning, consumer groups, backpressure and exactly-once handling.
Elden Ring Data Pipeline
A complete ETL and visualization pipeline over Elden Ring game data. The fundamentals are identical to any enterprise pipeline; the dataset is considerably more fun to debug.