Apache Flink on Google Kubernetes Engine (GKE): Architecture for Apache Iceberg Streaming

TL;DR

• Flink on GKE with RocksDB state and GCS-backed checkpoints delivers exactly-once Iceberg streaming with millisecond latencies.
• A single Flink job can fan out to multiple sinks, but consider two jobs if you need failure isolation or differing SLAs.
• Spark Structured Streaming does have a state store; Flink’s native state + timers + watermarks offer finer-grained, lower-latency control.
• For Google-native stacks, Flink + Iceberg on GCS and Pub/Sub is a robust, cost-effective alternative to Dataflow where you need table ACID semantics.
• Use the Flink K8s Operator, Workload Identity, local SSDs for RocksDB, and an Iceberg REST/Nessie catalog for an enterprise-ready deployment.

Abstract & Executive Observations

Enterprises need streaming architectures that do more than move bytes from A to B. They need exactly-once semantics into a governed lakehouse table format, ability to replay with schema evolution, and real-time fan-out to downstream services. Apache Flink on Google Kubernetes Engine (GKE) with Apache Iceberg on Google Cloud Storage (GCS) achieves this, while maintaining production-grade operability and TCO discipline.

We see a pattern: teams start with batch-oriented lakes, add streaming via message buses, and then struggle with consistent, queryable history. Iceberg’s table abstraction changes the game by making streaming outputs instantly available to engines like Trino, Spark, BigQuery BigLake external tables and Flink SQL, with atomic commits and schema evolution. Flink adds event-time correctness, low-latency stateful processing, and exactly-once sinks that Spark’s micro-batch engine approximates but does not match at sub-second latencies.

Two misconceptions persist. First, “Spark has no state store.” It does: Spark Structured Streaming uses a StateStore persisted to DFS and now offers a RocksDB-backed store. Second, “Flink must run one job per sink.” In practice, a single Flink topology can branch and sink to Iceberg and Pub/Sub simultaneously; however, operational isolation may argue for two jobs.

Our thesis: for streaming into Iceberg on GCP, Flink on GKE is the reference architecture when you need low-latency event-time correctness, complex state, and ACID table writes. Spark Structured Streaming remains strong when you are already heavily invested in Spark and accept micro-batch latency. Apache Beam/Dataflow suits unified batch-stream pipelines and managed ops, though first-class Iceberg integration usually lags behind Flink’s.

The Current Architectural Challenge

Organisations are converging on lakehouse patterns: open-table formats atop object storage, with multi-engine access. Iceberg offers atomic commits, hidden partitioning, schema and partition evolution, and snapshot isolation. The challenge is building a streaming write path that is consistent, observable, and resilient under production pressures.

Legacy approaches fail for several reasons:

• Batch-centric ingestion pipelines create hours of data latency and brittle backfills that break downstream SLAs.
• Direct object-writes without a table format force query engines to rely on directory conventions, leading to data corruption risks, inconsistent schemas, and costly compactions.
• Naïve streaming sinks lack exactly-once guarantees. Replays or failures cause duplicates or partial writes, violating governance and leading to opaque reprocessing cycles.

In GCP, teams often start with Pub/Sub and Dataflow or Spark on Dataproc. While both can land data into GCS, turning those files into ACID table commits is non-trivial. Beam has evolving support for Iceberg, but production-grade, operationally complete integrations tend to arrive later than in Flink. Spark Structured Streaming to Iceberg works well for micro-batch, but requires careful tuning to avoid small files and manage commit frequency; achieve exactly-once semantics; and handle complex CDC.

Operationally, we see hidden costs: - State management at scale: storing tens to hundreds of GB of state needs an efficient local store (RocksDB) and fast, incremental checkpoints. Using only remote state or in-memory heaps collapses under load. - Scaling under uneven traffic: maintaining event-time correctness with late data requires watermarking, idleness detection, and backpressure-aware scaling. Over-simplified autoscaling produces instability and checkpoint storms. - Fan-out and isolation: teams want to write to Iceberg and simultaneously publish to an outbound bus. Coupling these in one topology simplifies code but makes failure domains intertwined; splitting them increases complexity but improves resilience.

The architecture we recommend balances these concerns with GKE-native ops, strong security via Workload Identity, and a catalogue approach (Iceberg REST or Nessie) that gives atomic commits and simplified metadata management on GCS.

Technical Visualization 1: End-to-end streaming on GKE with Iceberg and Pub/Sub

Figure 1: Producers emit to Pub/Sub. Flink on GKE consumes, processes with state in local RocksDB, checkpoints to GCS, commits to Iceberg via REST/Nessie catalog on GCS, and fans out to another Pub/Sub topic for real-time consumers.

Architecture Blueprint 2: Checkpoint and commit sequencing for exactly-once

Figure 2: Flink’s checkpoint barrier aligns operator state. After state and files are durable, the Iceberg commit is made and sources are acknowledged, delivering exactly-once semantics end-to-end.

Implementation Detail 3: Decide single job vs two jobs for fan-out

Figure 3: A decision flow for fan-out strategy. One topology is simpler; two topologies isolate failure and scale independently.

Operational Flow 4: Core components in a Flink-on-GKE deployment

Figure 4: Kubernetes objects and external services with their relationships and responsibilities.

Data Model 5: Stream-to-table event lifecycle

Figure 5: Each record’s progression through ingestion, validation, Iceberg commit and optional fan-out, including error paths.

Governance Framework 6: Delivery milestones for a production platform

Figure 6: A pragmatic roll-out timeline aligning architecture, security, platform build and operational readiness.

Risk Matrix 7: Operational risks and controls as entities

Figure 7: Mapping risks to controls and evidential artefacts provides auditability for the streaming platform.

System Context 8: GCP ecosystem integration

Figure 8: The platform in the context of surrounding GCP services and operational facilities.

Implementation & Orchestration Strategy

Delivering a resilient Flink-to-Iceberg stack on GKE hinges on five decisions: cluster topology, state management, catalog design, security, and operational automation.

• Cluster Mode — Job vs Session: For production, we prefer Job Clusters per pipeline using the Flink Kubernetes Operator. Each job ships alongside its dependencies, isolates upgrades, and scales independently. Session clusters add density but couple lifecycle and failover.

• State Backend — RocksDB vs HashMap: Flink supports a heap-based backend and RocksDB. For non-trivial windows, joins or CEP we choose RocksDB for bounded memory and fast, incremental checkpoints. Enable the changelog state backend to reduce checkpoint I/O bursts where appropriate. Back state with local SSD to improve compaction throughput; checkpoint to GCS.

• Catalog — REST or Nessie: On GCP, deploy the Iceberg REST Catalog or Nessie as a small service (GKE or Cloud Run). For multi-writer concurrency and atomic commits, Nessie brings versioned metadata and optimistic concurrency control. Store data and metadata in GCS, aligning with BigLake external access where applicable.

• Security — Workload Identity and CMEK: Use GKE Workload Identity to grant least-privilege access to GCS buckets and Pub/Sub topics. Encrypt GCS buckets with Customer-Managed Keys and define retention policies for checkpoint and data prefixes. Mount secrets via the Secrets Store CSI driver where needed.

• Fan-out — One or Two Topologies: A single Flink job can branch streams and write to Iceberg and Pub/Sub. If the downstream Pub/Sub has different SLOs, or you require strict failure isolation, emit to an intermediate topic and let a second job publish or transform further. Exactly-once across both paths typically requires an outbox pattern with idempotent consumers.

Example: a minimal FlinkDeployment using the Kubernetes Operator.

apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: iceberg-streaming-job
spec:
  image: gcr.io/your-proj/flink-iceberg:1.17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "4"
    state.backend: rocksdb
    state.checkpoints.dir: "gs://your-bucket/flink-checkpoints/job-123"
    state.savepoints.dir: "gs://your-bucket/flink-savepoints/job-123"
    state.backend.changelog.enabled: "true"
    execution.checkpointing.interval: "15000"
    execution.checkpointing.unaligned: "true"
  serviceAccountName: flink-wi-sa
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
    replicas: 1
    highAvailability:
      type: kubernetes
  taskManager:
    resource:
      memory: "8192m"
      cpu: 2
    replicas: 6
    volumes:
      - name: rocksdb-ssd
        hostPath:
          path: /mnt/disks/ssd0
    volumeMounts:
      - name: rocksdb-ssd
        mountPath: /var/flink/rocksdb
  job:
    jarURI: local:///opt/flink/usrlib/iceberg-job.jar
    args: ["--catalogUri", "http://iceberg-rest:8181", "--inTopic", "projects/x/topics/in", "--outTopic", "projects/x/topics/out"]
    upgradeMode: stateless

Key Flink configuration call-outs: - Checkpointing interval: 10–30 seconds balances latency and metadata churn. Unaligned checkpoints help under backpressure. - RocksDB directory: Point to a local SSD for performance; persist only via incremental checkpoints to GCS. - Watermarks: Define robust event-time watermarks with idleness detection to avoid lag on sparse partitions.

Example: Flink SQL to write to Iceberg and Pub/Sub in a single job.

CREATE CATALOG lake WITH (
  'type'='rest',
  'uri'='http://iceberg-rest:8181',
  'warehouse'='gs://your-warehouse'
);

CREATE TABLE lake.ops.events (
  user_id BIGINT,
  event_time TIMESTAMP(3),
  action STRING,
  payload STRING,
  WATERMARK FOR event_time AS event_time - INTERVAL '30' SECOND
) PARTITION BY (days(event_time))
WITH (
  'write.upsert.enabled'='true',
  'write.distribution-mode'='hash',
  'format-version'='2'
);

CREATE TABLE out_pubsub (
  data STRING
) WITH (
  'connector'='pubsub',
  'topic'='projects/x/topics/out',
  'format'='json'
);

-- Process and branch
INSERT INTO lake.ops.events
SELECT user_id, event_time, action, payload
FROM source_pubsub;

INSERT INTO out_pubsub
SELECT TO_JSON(ROW(user_id, action, event_time))
FROM source_pubsub
WHERE action IN ('ALERT','NOTIFY');

Iceberg table design guidance: - Partitioning: Use time-based transforms (hour/day) combined with domain keys (e.g., user_id bucketing) only when necessary. Over-partitioning increases small files and commit overhead. - Sorting: Set write.distribution-mode=range and write.target-file-size-bytes to optimise read performance for downstream engines. - Compaction: Run Flink or Iceberg’s built-in compaction actions off-peak, or schedule via an Operator job.

Spark Structured Streaming comparison in the context of a 5-minute window: - Spark supports stateful operations with watermarks and triggers. It uses a StateStore persisted to DFS and, since 3.4+, a RocksDB state store plugin is available. Five-minute windows with event-time and watermarking are supported, but latency aligns to micro-batch intervals and commit cycles. - Flink executes true continuous streaming with event-time timers, per-key state, and low-latency firing. Late-data handling is configured with allowed lateness and custom triggers. - For Iceberg streaming, Spark writes are coordinated per micro-batch; Flink commits align with checkpoints, often giving finer granularity and lower end-to-end latency.

Beam/Dataflow positioning: - Beam’s unified model is powerful across runners. On GCP, Dataflow brings managed autoscaling and operational simplicity. Iceberg integration exists via community IOs or file sinks plus table commits, but is generally less mature than Flink’s first-party integration, and exactly-once semantics for table commits require careful pipeline design.

Operational essentials on GKE: - Autoscaling: Use GKE node autoscaling and Flink’s reactive scaling for TaskManagers. For strict SLOs, bound maximum parallelism per job to avoid reshuffle storms. - Observability: Expose Flink metrics to Prometheus; ship logs to Cloud Logging; alert on checkpoint duration, number of pending checkpoints, backpressure, and out-of-orderness. - Reliability: Run the catalog service with HA and backups. Buckets should use Object Versioning for metadata prefixes to protect against accidental deletions. Enforce network egress via Private Google Access and VPC-SC if regulated.

Security, Performance, and Trade-offs

• Security: Workload Identity reduces long-lived secrets. Use IAM Conditions to restrict bucket paths. Apply NetworkPolicies to limit egress from JobManager/TaskManagers. For compliance, turn on CMEK for GCS and validate key rotation. Catalogue endpoints should enforce mTLS or OIDC and least-privilege access to the warehouse prefix.

• Performance: The largest gains come from local SSDs for RocksDB, incremental checkpoints, and careful state TTLs. Tune RocksDB options via Flink config: block cache size, compaction threads, and disable overly aggressive compactions under high ingest. For Iceberg, set target file size to 128–512 MB and enable fanout writer when high parallelism produces small files. Avoid overly granular partitions. Remember that GCS has request-rate limits; batch and combine manifests, and prefer fewer, larger files.

• Trade-offs:

• A single job with dual sinks is simpler but couples failure domains; you cannot achieve global exactly-once across Iceberg and Pub/Sub unless both sinks coordinate two-phase commit and the source supports transactional acknowledgements. Pub/Sub acknowledgements can be timed post-commit, but the Pub/Sub sink itself is usually at-least-once.
• Two jobs with an intermediate topic add cost and latency, but isolate backpressure and allow independent SLAs.
• Spark vs Flink: Spark micro-batch suits analytics-centric teams and integrates cleanly with existing Spark jobs, but has higher latency and backpressure behaviour driven by batch intervals. Flink offers finer control for streaming-native workloads. Beam/Dataflow reduces ops burden but may lag on cutting-edge Iceberg features.

On state backends: Flink’s RocksDB is the de facto choice for large state. The in-heap HashMap backend works for small state and low-latency scenarios but lacks durability under memory pressure. Alternative or custom state stores are technically possible via Flink’s pluggable state backend interfaces and the changelog backend, yet production support centres on RocksDB and heap; external remote databases are not recommended due to latency and consistency constraints.

Parting Thoughts on Industry Direction

Over the next 12–24 months we expect three shifts. First, Iceberg’s REST and Nessie catalogues will become the default on GCP, enabling multi-writer concurrency and simplified governance. Second, Flink on Kubernetes will continue to displace bespoke streaming stacks as operators mature and reactive scaling stabilises. Third, cross-engine optimisation will improve: engines will read Iceberg partition stats and sort orders more intelligently, cutting costs without changing pipelines.

For organisations standardising on GCP, Flink on GKE with Iceberg on GCS is a powerful and open architecture that balances performance, operability and vendor independence. If you want to benchmark your workloads across Flink, Spark and Beam, or need a design authority to land this with the right SLOs and guardrails, we can help. We invite you to engage MNB to review your current streaming estate and share real performance baselines from production environments.

Recommended Reading

• Designing resilient stateful streaming on Kubernetes with Flink and RocksDB (/insights/cloud/related-topic-1/)
• Iceberg table optimisation strategies on object storage (/insights/cloud/related-topic-2/)
• Exactly-once semantics in practice across Pub/Sub, Kafka and lakehouse sinks (/insights/cloud/related-topic-3/)

About the Research Team

The MNB Technology Research Team consists of senior architects and engineers with backgrounds spanning global banking, healthcare, retail, defence, and public sector. Our work focuses on enterprise AI, secure distributed systems, and resilient operational delivery.

Work with Us

Ready to make · new · better?

Contact us to discuss how this architecture applies to your specific environment.