Streaming architecture
A governed event pipeline pattern for near-real-time KPIs, attribution, segmentation, and analytical serving.
50M+
daily streaming events
30%
Spark runtime reduction
25%
compute savings
The failure mode
Campaign and digital-event sources needed reliable near-real-time processing without losing replayability, schema control, or warehouse-ready analytical structure.
Event Sources
Streaming Ingestion
Lakehouse Processing
Warehouse Serving
Event Sources
Digital events land through Kafka, Event Hubs, or Kinesis with source metadata, event timestamps, and replay controls.
Streaming Ingestion
Spark Structured Streaming performs checkpointed ingestion with watermarking, deduplication, schema evolution, and dead-letter handling.
Lakehouse Processing
Delta Lake or Iceberg stores raw and standardized event layers while curated aggregations feed Gold-layer serving models.
Warehouse Serving
Snowflake and dbt publish governed marts for attribution, campaign performance, and near-real-time KPI consumption.
Clickstream and campaign events could arrive late or out of order, so simple load-time processing would distort near-real-time KPIs.
Used Spark Structured Streaming with event-time watermarks, checkpointing, and replay controls so late events could be corrected without corrupting downstream marts.
The platform needed safe retries after dependency failures, schema changes, and partial stream interruptions.
Standardized business keys, incremental MERGE logic, and dead-letter paths so replay and backfills could run without duplicate downstream state.
Teams needed both raw event retention for reprocessing and fast curated KPI access for attribution and optimization.
Kept raw and standardized event layers in the lakehouse while publishing Snowflake/dbt Gold marts for governed analytical consumption.