Overview
Icelake is a sovereign European AI-driven data research workbench. Public datasets — weather, air quality, sovereign maps without tracking, geo and IoT — are already loaded and refreshed for you, alongside any data you ingest yourself via Prometheus, Loki, OpenTelemetry, or MQTT. Everything lands as Parquet in the open Apache Iceberg table format on S3, and is queryable with DuckDB through SQL, LogQL, PromQL, and AI MasterMind in natural language. Built on open standards so your data stays portable and independent from proprietary providers.
Architecture
Section titled “Architecture”Icelake follows a pipeline architecture designed for high-throughput, low-latency observability:
- Multi-Protocol Ingestion — Prometheus remote write, Loki push API, and OTLP protobuf
- S3-Native Storage — All data stored as Parquet files on S3-compatible object storage
- Apache Iceberg — Open table format with snapshot isolation and ACID transactions
- Query Engine — DuckDB-powered SQL via pgwire, LogQL for logs, PromQL for metrics
- Background Compaction — Automatic merging of small files for optimal query performance
Technology Stack
Section titled “Technology Stack”| Technology | Purpose | Benefits |
|---|---|---|
| Rust | Core Runtime | Memory safety, zero-cost abstractions, speed |
| Apache Iceberg | Open Table Format | Snapshot isolation, ACID, portable to any engine |
| DuckDB | Query Engine | Sub-millisecond analytical queries |
| PostgreSQL | Catalog Backend | Reliable, transactional Iceberg catalog |
| S3 | Object Storage | Cost-efficient, durable, scalable storage |
| Parquet | Data Format | Columnar, compressed, query-optimized |
Getting Started
Section titled “Getting Started”- Architecture — Understand the data flow and multi-tenant model
- Prometheus — Configure Prometheus remote write ingestion
- Loki & LogQL — Set up log ingestion and LogQL queries
- OpenTelemetry — Ingest via OTLP protobuf endpoints
What’s New
Section titled “What’s New”- Geo & Public Datasets — DWD weather warnings, OpenAQ air quality, and Overture places joinable against your own data
- MQTT & IoT — Configure MQTT ingestion with TTN LoRaWAN auto-parsing
- AI MasterMind — Natural language chat analytics with DuckDB
- Admin Dashboard — Manage data sources, teams, and analytics
- Home Assistant — HACS integration for smart home metrics
- Query Interfaces — Grafana, pgwire SQL, REST API, and Loki Query API
Community & Support
Section titled “Community & Support”- GitHub — Contribute to the project and report issues
- Discord — Join our community for discussions and support
- Documentation — Comprehensive guides and API references