SQL Stream Processing to Iceberg Tables

Problem Context

Teams using Kafka, Kinesis, or RabbitMQ need to transform streamed events into structured data in Iceberg tables on S3. They lack a tool that lets them write SQL transformations directly on the stream and auto-insert results into Iceberg without heavy DE work.

Pain Points

Current solutions require manual setup (e.g., Kafka Streams + Iceberg SDK) or lack SQL support. Users waste time writing boilerplate code, debugging connectors, and maintaining pipelines. Small DE teams struggle to keep up with demand, leading to delays or technical debt.

Impact

Delays in data processing slow down analytics, reporting, and decision-making. Manual errors in transformations or Iceberg writes risk data corruption. Teams lose productivity to repetitive setup tasks instead of focusing on core work.

Urgency

The problem is urgent because stream processing is a daily task, and manual workarounds are unsustainable. Without a dedicated tool, teams either fall behind or overburden their DE teams, creating a bottleneck for growth.

Target Audience

Operations teams, small data engineering teams, and mid-size tech companies using Kafka, Kinesis, or RabbitMQ to process events into Iceberg tables. Similar pain points exist in fintech, e-commerce, and SaaS companies relying on real-time data pipelines.

Solution Approach

A micro-SaaS that lets users write SQL transformations on streams (Kafka/Kinesis/RabbitMQ) and auto-insert results into Iceberg tables on S3. It abstracts away connectors, stream processing, and Iceberg writes, so users focus only on SQL—just like DBT but for streams.

Key Features

1. Stream Connectors: Plug into Kafka, Kinesis, or RabbitMQ with zero code.
2. Iceberg Auto-Writes: Results are automatically inserted into Iceberg tables in S
4. Monitoring Dashboard: Track stream health, transformation errors, and Iceberg write status in real time.

User Experience

Users connect their stream source, write SQL transformations in a familiar editor, and deploy. The tool handles the rest—processing events, running SQL, and writing to Iceberg. They get a live dashboard to monitor everything, with alerts for failures. No DE team needed.

Differentiation

Unlike DBT (batch-only) or Kafka Streams (code-heavy), this tool is designed *specifically- for stream-to-Iceberg SQL. It’s simpler than manual setups, more powerful than no-code tools, and avoids vendor lock-in by supporting multiple stream sources and Iceberg.

Scalability

Starts with a single stream and table, then scales to multiple streams, transformations, and Iceberg tables. Users can add more streams or tables without reconfiguring the entire pipeline. Pricing scales with event volume (e.g., $50/mo for 10–100 events/sec).

Expected Impact

Teams save 5–10 hours/week on manual setup and debugging. Data pipelines become more reliable, reducing errors in transformations and Iceberg writes. DE teams focus on high-value work instead of maintenance, accelerating product development.