Yy9088 Stack
📖 Tutorial

The Silent Upgrade: How Kubernetes Image Promotion Got a Modern Makeover

Last updated: 2026-05-01 14:36:04 Intermediate
Complete guide
Follow along with this comprehensive guide

Every container image you pull from registry.k8s.io arrives thanks to kpromo, the Kubernetes image promoter. This tool copies images from staging registries to production, signs them with cosign, replicates signatures across over 20 regional mirrors, and generates SLSA provenance attestations. If kpromo breaks, no Kubernetes release ships. Recently, the team rewrote its core from scratch, deleted 20% of the codebase, made it dramatically faster, and nobody noticed. That was the goal.

A Brief History

The image promoter began in late 2018 as an internal Google project by Linus Arver. The aim was to replace manual, Googler-gated image copying into k8s.gcr.io with a community-owned GitOps workflow. Push to a staging registry, open a PR with a YAML manifest, get reviewed and merged, and automation handles the rest. KEP-1734 formalized this.

The Silent Upgrade: How Kubernetes Image Promotion Got a Modern Makeover

In early 2019, the code moved to kubernetes-sigs/k8s-container-image-promoter and grew quickly. Stephen Augustus consolidated multiple tools (cip, gh2gcs, krel promote-images, promobot-files) into a single CLI called kpromo. The repo was renamed to promo-tools. Adolfo García Veytia (Puerco) added cosign signing and SBOM support. Tyler Ferrara built vulnerability scanning. Carlos Panato maintained the project. 42 contributors made ~3,500 commits across 60+ releases. It worked, but by 2025 the codebase carried seven years of incremental additions. The README noted duplication, multiple techniques, and many TODOs.

Problems to Solve

Production promotion jobs for Kubernetes core images regularly took over 30 minutes and often failed with rate limit errors. The core promotion logic became a monolith hard to extend and test, making new features painful. On the SIG Release roadmap, two work items lingered: "Rewrite artifact promoter" and "Make artifact validation more robust." Discussions at meetings and KubeCons led to open research spikes on project board #171 capturing eight questions.

The Phased Rewrite

In February 2026, the team opened issue #1701 to rewrite the artifact promoter pipeline. All eight spikes were answered in a single tracking issue. The rewrite was phased so each step could be reviewed, merged, and validated independently.

Phase 1: Rate Limiting (#1702)

Rewrote rate limiting to properly throttle all registry operations with adaptive backoff.

Phase 2: Interfaces (#1704)

Put registry and auth operations behind clean interfaces for independent swapping and testing.

Phase 3: Pipeline Restructuring

The pipeline was restructured into composable stages, allowing easier extension for provenance and vulnerability scanning. This modular approach reduced coupling and simplified testing.

Conclusion: Invisible Improvements

The rewrite made kpromo faster, more reliable, and easier to maintain. The 20% code deletion and new architecture ensure future features can be added without breaking existing workflows. The team succeeded in making the improvements invisible to users — exactly as intended.