Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

GitImage

GitImage is a Kubernetes operator that turns a git URL into a running pod. You point a GitImage at a repository; every commit is built in-cluster by BuildKit inside an isolated pod, stored in an OCI registry or in S3, and injected into your workloads by digest.

apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: myapp, namespace: team-a }
spec:
  url: git@github.com:you/myapp.git
  ref: main
  secretRef: { name: myapp-git }
# in any Deployment, StatefulSet, DaemonSet, Job or CronJob pod template
metadata:
  annotations:
    gitimage.dev/image.web: myapp     # container "web" runs GitImage "myapp"
spec:
  containers:
    - name: web
      image: placeholder              # replaced by the built digest at admission

What you get:

  • No CI, no registry required. Builds run in your cluster; with S3 storage, nodes pull from a node-local read-only server and no registry exists at all.
  • Pods never fail on a missing image. A pod created before its build finishes waits as SchedulingGated, then starts with the digest.
  • Every commit rolls out. Deployments, StatefulSets and DaemonSets get the new digest automatically; each pod is pinned to an exact commit.
  • Builds are isolated. Kata VMs, gVisor, user namespaces or (opt-in) privileged; the controller refuses to start rather than fall back to a weaker mode.
  • Builds never hold credentials they do not need. Git credentials stay in the tenant namespace; storage credentials never leave the controller.

Current release: 0.1.71. Source: github.com/DiverOfDark/gitimage.

Getting started

You need Kubernetes 1.30+ with containerd, Helm 3, and either an OCI registry or an S3 bucket.

1. Pick an isolation mode

ClusterUse
Nodes with /dev/kvm and Kata installedmode: runtimeClass, runtimeClassName: kata-qemu, privilegedInSandbox: true
gVisor installed (GKE Sandbox, runsc)mode: runtimeClass, runtimeClassName: gvisor
Sysbox installed (sysbox-deploy-k8s)mode: runtimeClass, runtimeClassName: sysbox-runc, rootfulInSandbox: true
Kubernetes 1.33+ on VMs or bare metalmode: userns
Trusted repositories onlymode: privileged, allowPrivileged: true

See Isolation modes.

2. Install

Registry storage:

kubectl create namespace gitimage-system
kubectl -n gitimage-system create secret docker-registry registry-push \
  --docker-server=ghcr.io --docker-username=<user> --docker-password=<token>

helm install gitimage oci://ghcr.io/diverofdark/charts/gitimage --version 0.1.71 \
  -n gitimage-system \
  --set buildPolicy.isolation.mode=runtimeClass \
  --set buildPolicy.isolation.runtimeClassName=gvisor \
  --set storage.type=registry \
  --set storage.registry.url=ghcr.io/<you>/builds \
  --set storage.registry.pushSecret=registry-push

S3 storage (no registry):

kubectl -n gitimage-system create secret generic gitimage-s3 \
  --from-literal=AWS_ACCESS_KEY_ID=... --from-literal=AWS_SECRET_ACCESS_KEY=...

helm install gitimage oci://ghcr.io/diverofdark/charts/gitimage --version 0.1.71 \
  -n gitimage-system \
  --set buildPolicy.isolation.mode=userns \
  --set storage.type=s3 --set storage.s3.bucket=gitimage \
  --set storage.s3.endpoint=https://s3.example.com --set storage.s3.secret=gitimage-s3

The controller runs an isolation check on start. kubectl -n gitimage-system get pods shows Running when the mode works; CrashLoopBackOff with IsolationUnavailable in the log when it does not.

3. Build something

Build pods need the privileged Pod Security level in their namespace (baseline with Sysbox; see Security model).

kubectl create namespace demo
kubectl label namespace demo pod-security.kubernetes.io/enforce=privileged
kubectl apply -n demo -f - <<'YAML'
apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: hello }
spec:
  url: https://github.com/<you>/<public-repo-with-a-Dockerfile>.git
  ref: main
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: hello }
spec:
  selector: { matchLabels: { app: hello } }
  template:
    metadata:
      labels: { app: hello }
      annotations: { gitimage.dev/image.web: hello }
    spec:
      containers: [{ name: web, image: placeholder }]
YAML

kubectl -n demo get gi -w        # Revision, Ready, Status (current BuildKit step)
kubectl -n demo get pods -w      # SchedulingGated, then Running

How it works

  1. Resolve. The controller lists the remote’s refs (git protocol v2 ls-refs, nothing fetched) every interval, or at once when a push hook arrives. A full sha is used as is, else a tag, else a branch; only a branch can move.
  2. Decide. If the commit did not change the build context’s git tree (monorepos), the last image is reused and nothing is built.
  3. Build. A Job in the tenant namespace: an init container clones exactly that commit at depth 1 (git credentials are mounted only here), then BuildKit builds it under the chosen isolation mode. Progress (step 20/42 backend (+1 parallel): RUN cargo build) is copied into status and into Events on waiting pods. The count is over every step of every stage, since stages build in parallel; the step named is the one running longest.
  4. Push. BuildKit pushes the image and its layer cache to the controller’s push endpoint with a token valid only for <namespace>/<gitimage> and only while that Job runs. The endpoint writes to the registry or S3.
  5. Admit. The webhook sees every pod annotated with gitimage.dev/image.<container>. If the image for the pinned (or current) commit exists, it writes the digest into the container right away; otherwise it adds the scheduling gate gitimage.dev/build.
  6. Release. When the build lands, the controller patches the digest into each gated pod and removes the gate. It also pins the new commit into every Deployment, StatefulSet and DaemonSet that uses the GitImage (gitimage.dev/revision.<container>, and gitimage.dev/digest.<container> so a rebuild of the same commit rolls too), which rolls them.
 git push ──> controller ──> build Job (tenant ns) ──> push endpoint ──> registry | S3
                  │                                                         │
                  └── status.image ──> webhook / gate releaser ──> pods <───┘ (pull)

Workloads in gitimage-system

WorkloadReplicasDoes
gitimage-controller2, one leader (Lease)reconcilers, gate release, rollouts, GC on the leader; push endpoint and hooks on every replica
gitimage-webhook2admission (gates, digests, cross-tenant guard); reads a GitImage cache, never the API on the request path
gitimage-nodeDaemonSet, s3 mode onlyread-only OCI server on 127.0.0.1:25050 for containerd

The GitImage resource

gitimage.dev/v1alpha1, kind GitImage, namespaced, short name gi.

apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: myapp, namespace: team-a }
spec:
  url: git@github.com:you/myapp.git   # ssh (git@host:path, ssh://...) or https
  ref: main                           # branch | tag | full sha | refs/heads/... | refs/tags/...
  secretRef: { name: myapp-git }      # optional, same namespace
  interval: 5m                        # poll interval; hooks trigger earlier
  build:
    context: .                        # repo-relative build directory
    dockerfile: Dockerfile            # relative to context
    args: {}                          # see Credentials, build secrets and args
    secrets: []
    resources: { requests: { cpu: "1", memory: 2Gi }, limits: { memory: 4Gi } }
    onNewCommit: wait                 # wait | cancel

Status

FieldMeaning
latestRevisionnewest commit seen on the ref
revisioncommit whose image is current
image<pull host>/<namespace>/<name>@sha256:...
builds[]newest first, up to 10: revision, outcome (Built, Reused, Superseded, Failed), digest, contextTree, sourceRevision, reason, attempt (the build Job’s), finishedAt
webhookURLpush-hook URL to paste into the git host (needs hooks.baseURL)
handledRebuildthe last gitimage.dev/rebuild value whose rebuild finished; a different annotation value is a rebuild still to do
conditionsReady, Building (message = current BuildKit step), Failed (reason)

Failure reasons: ResolveFailed, SecretMissing, CloneFailed, BuildFailed, PushFailed, DeadlineExceeded, PodCreationFailed, ReconcileError.

ReconcileError means the controller itself failed, for example because the API server rejected the build Job or a quota forbids it; the message is the API server’s. The controller retries every 30 seconds, and the next successful reconcile clears the condition. If the GitImage has no image yet, Ready is False with the same reason, so kubectl get gi shows it.

Behaviour

  • One build at a time per GitImage. With onNewCommit: wait (default) the running build finishes, then only the newest commit is built; with cancel the running build is deleted.
  • A failed build is not retried automatically. Push a new commit or bump the annotation: kubectl annotate gi myapp gitimage.dev/rebuild=$(date +%s) --overwrite.
  • Any change of gitimage.dev/rebuild rebuilds the current commit, whatever its last outcome (Built, Reused, Failed, Superseded); see Rebuilding a commit.
  • Pod-level failures (eviction, node loss) are retried twice.
  • Deleting the GitImage deletes its Jobs and hook Secret; stored images are left to GC.

Rebuilding a commit

Set gitimage.dev/rebuild on the GitImage to a new value to build the current commit again without a new commit:

kubectl annotate gi myapp gitimage.dev/rebuild=$(date +%s) --overwrite
  • The rebuild always runs a build Job: it never ends as Reused, even if the context did not change, and it runs even when the commit’s earlier Jobs are already gone.
  • The value reaches the Dockerfile as the build arg GITIMAGE_REBUILD (empty without the annotation). BuildKit otherwise reuses every cached layer; a Dockerfile that declares ARG GITIMAGE_REBUILD misses the cache from that line on, so the steps after it run again.
  • While it runs, status.image and status.revision stay on the previous image and workloads keep running it. When it succeeds, the commit’s record gets the new digest and workloads roll to it (the roller pins gitimage.dev/digest.<container> next to the revision). If the digest is unchanged, nothing rolls.
  • If it fails, the failure is recorded (Failed condition, BuildFailed event) and the previous image stays current: the Failed record keeps its digest.
  • A request made while a build runs (of this commit, or of an older one with onNewCommit: wait) waits for that build, then rebuilds. status.handledRebuild takes the value once the rebuild’s result is recorded.
  • A rebuild that has started always finishes and is recorded: removing the annotation, or setting it back to status.handledRebuild, does not cancel it, and a newer value rebuilds again after it.
  • Only the value matters, not what it is; a request is never repeated by itself. Do not keep the annotation in a GitOps-managed manifest, or a sync that resets it starts a rebuild.

Daily rebuilds

A site that filters content by date (posts published in the future) has to be rebuilt even when no commit changed. Declare the arg before the build step:

FROM node:24 AS build
WORKDIR /src
COPY package*.json ./
RUN npm ci
COPY . .
# Changes on every rebuild: the build below runs again, npm ci stays cached.
ARG GITIMAGE_REBUILD
RUN npm run build

And bump the annotation from a CronJob in the GitImage’s namespace (kubectl annotate reads the object first, so the Role needs get and patch):

apiVersion: v1
kind: ServiceAccount
metadata: { name: site-rebuild, namespace: team-a }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: site-rebuild, namespace: team-a }
rules:
  - apiGroups: [gitimage.dev]
    resources: [gitimages]
    resourceNames: [site]
    verbs: [get, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: site-rebuild, namespace: team-a }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: site-rebuild }
subjects: [{ kind: ServiceAccount, name: site-rebuild, namespace: team-a }]
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: site-rebuild, namespace: team-a }
spec:
  schedule: "5 0 * * *"            # daily, 00:05
  timeZone: Etc/UTC
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          serviceAccountName: site-rebuild
          restartPolicy: Never
          containers:
            - name: annotate
              image: docker.io/alpine/kubectl:1.37.2
              command: [sh, -c, 'kubectl annotate gi site gitimage.dev/rebuild=$(date +%s) --overwrite']

Running workloads

Annotate the pod template (not the workload’s own metadata):

spec:
  template:
    metadata:
      annotations:
        gitimage.dev/image.web: myapp      # container "web" -> GitImage "myapp"
        gitimage.dev/image.worker: myapp   # several containers may use one GitImage
    spec:
      containers:
        - { name: web, image: placeholder }
        - { name: worker, image: placeholder, command: [worker] }

Init containers can be bound the same way. The GitImage must be in the pod’s namespace.

You wantDo
Automatic rolloutsnothing: Deployments, StatefulSets and DaemonSets get gitimage.dev/revision.<container> pinned on each new commit, and gitimage.dev/digest.<container> on each new image (a rebuild of the same commit rolls too)
No automatic rolloutsgitimage.dev/rollout: manual on the workload’s metadata
Jobs and CronJobssupported; each new pod gets the current revision at creation
A specific commitset ref to the sha (or a tag)

A pod whose template pins a digest (gitimage.dev/digest.<container>) that the GitImage’s status does not show yet, because a rebuild just finished, waits gated for a moment rather than start on the commit’s previous image. A pin that never matches (you changed gitimage.dev/revision.<container> by hand, or a manual workload’s commit was rebuilt since) is dropped after a minute: the pod gets the revision’s current digest. Remove the digest annotation when you pin a revision by hand.

A pod admitted before its image exists stays SchedulingGated (not Pending on a node, never ImagePullBackOff). While it waits, it gets Events like Building myapp@3f2c9e1: step 4/12 RUN npm ci, and GitImageNotFound if the GitImage does not exist yet.

A pod may only reference GitImage images of its own namespace; the webhook denies anything else.

Image scanners

The template keeps placeholder; only pods get the built image. Scanners that read templates instead of pods, such as Trivy Operator for Deployments, StatefulSets and DaemonSets, scan the placeholder. Two chart values fix that:

  • controller.pinTemplateImage: true: the roller writes the built image into each bound container of the template, together with the revision annotation. It checks on every reconcile, so workloads created after a build and templates reset by an apply are pinned again. Turning it on rolls each workload once.
  • webhook.crossTenantNamespaces: [trivy-system]: Trivy Operator in filesystem mode runs the scanned image in its own namespace, which the cross-tenant guard otherwise denies.

With ArgoCD, the managedFieldsManagers ignore rule and RespectIgnoreDifferences from GitOps with ArgoCD keep the pinned image on sync. Helm and kubectl apply reset it to the placeholder, which rolls the workload, and the roller then pins it again, which rolls it once more.

Credentials, build secrets and args

Git credentials

spec.secretRef names a Secret in the GitImage’s namespace (Flux format):

TransportKeys
sshidentity (OpenSSH private key: ed25519, ecdsa or rsa; no passphrase), known_hosts (required: unknown host keys are refused)
httpsusername, password (a token works; username defaults to git)
kubectl -n team-a create secret generic myapp-git \
  --from-file=identity=./deploy_key --from-literal=known_hosts="$(ssh-keyscan github.com)"

The controller reads it to list refs; the clone init container of each build mounts it read-only. It is never copied anywhere else. Over plain http:// no credentials are sent.

Build secrets

For private package registries, without baking credentials into layers:

build:
  secrets: [{ id: npmrc, secretRef: { name: npm-token, key: npmrc } }]   # key optional if the Secret has one key
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci

The secret exists in tmpfs for that RUN only; it is never in a layer, the cache or the history. Repository code in that step can read it, so give secrets only to repositories you trust.

Build args

build:
  args:
    NODE_ENV: { value: production }
    SENTRY_RELEASE: { valueFrom: { secretKeyRef: { name: sentry, key: release } } }

Build args end up in the image history; use build secrets for real secrets.

Commit metadata

Every build also gets three args from GitImage, set after build.args so they cannot be overridden:

ArgValue
GITIMAGE_REVISIONfull commit sha
GITIMAGE_COMMIT_TIMEthe commit’s committer time, unix seconds
GITIMAGE_REBUILDthe GitImage’s gitimage.dev/rebuild annotation, empty without one

The build sees no .git (the clone is one commit deep), so these are how a Dockerfile learns which commit it is building. Declare the ones you use:

ARG GITIMAGE_REVISION=unknown
ARG GITIMAGE_COMMIT_TIME=0

GITIMAGE_COMMIT_TIME only grows along a branch that is not rewritten, so it works as a monotonic version number (an Android versionCode, for example: it stays under Android’s 2100000000 cap until mid-2036).

Undeclared, they change nothing: BuildKit ignores unused args. Declared, they change on every commit and so invalidate the cache of every later RUN in that stage: declare them in the stage that needs them, as late as possible.

GITIMAGE_REBUILD changes only when the annotation does. Declare it just before the steps that must run again on a rebuild without a new commit (a site build that filters posts by date, for example); everything above it stays cached:

ARG GITIMAGE_REBUILD
RUN npm run build

Push hooks

Polling every interval always works. For builds within seconds of a push:

  1. Expose the hooks Service: hooks.baseURL: https://gitimage.example.com and hooks.ingress.enabled: true (or your own Ingress/Gateway to Service gitimage-hooks:8080, path prefix /hooks/).
  2. Copy the URL: kubectl get gi myapp -o jsonpath='{.status.webhookURL}'.
  3. Add it as a push webhook in GitHub, GitLab, Gitea or Forgejo (any content type, no secret).

The URL carries a random token (Secret <name>-gitimage-hook; delete it to rotate). A hook only triggers a re-check of the ref: its payload is never trusted. Hooks for one repository wake every GitImage in that namespace with the same repository URL.

Monorepos

Create one GitImage per image, all with the same url and a different build.context:

spec: { url: git@github.com:you/mono.git, build: { context: services/api } }
---
spec: { url: git@github.com:you/mono.git, build: { context: services/web } }

A commit that does not change the git tree of context is recorded as Reused: the previous image is pinned to the new commit and nothing is built. The tree id comes from the host’s API for GitHub, GitLab and Gitea/Forgejo over https; otherwise the build Job’s clone step compares it and stops before BuildKit starts.

GitOps with ArgoCD

GitImage writes two things ArgoCD would report as drift:

  • the gitimage.dev/revision.<container> and gitimage.dev/digest.<container> pod-template annotations, and with controller.pinTemplateImage the bound containers’ image (field manager gitimage-controller);
  • the caBundle of the gitimage MutatingWebhookConfiguration.
spec:
  syncPolicy:
    syncOptions: [ServerSideApply=true, RespectIgnoreDifferences=true]
  ignoreDifferences:
    - { group: apps, kind: Deployment, managedFieldsManagers: [gitimage-controller] }
    - { group: apps, kind: StatefulSet, managedFieldsManagers: [gitimage-controller] }
    - { group: apps, kind: DaemonSet, managedFieldsManagers: [gitimage-controller] }
    # for the Application that installs the chart:
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: gitimage
      jqPathExpressions: ['.webhooks[].clientConfig.caBundle']

Sync order does not matter: pods created before their GitImage wait gated.

Isolation modes

buildPolicy.isolation.mode is required and has no default. At startup the controller runs a check Job with exactly the build pod’s isolation settings: BuildKit lists its workers, then builds a two-layer image (from its own busybox, nothing is pulled) whose second layer hard-links a file of the first while it is mapped. If either fails, the controller exits with code 3 and logs IsolationUnavailable; a failed hard link names the fix (gVisor: snapshotter: native). Nothing falls back to a weaker mode, and nothing builds.

ModeValuesBuild podVerified in CI
KataruntimeClass, runtimeClassName: kata-qemu, privilegedInSandbox: truerootful BuildKit, privileged inside the VMkind + kata-deploy (KVM runners)
gVisorruntimeClass, runtimeClassName: gvisor, snapshotter: nativerootless BuildKit, seccomp/AppArmor unconfined inside the sandboxkind + runsc
SysboxruntimeClass, runtimeClassName: sysbox-runc, rootfulInSandbox: truerootful BuildKit, unprivileged, hostUsers: false; Pod Security baselinekind + Sysbox 0.7.1 (hack/e2e/sysbox.sh)
User namespacesusernsrootless BuildKit, hostUsers: falsenot on kind (see below)
Privilegedprivileged + allowPrivileged: truerootful BuildKit, privileged on the nodekind

Kata and privilegedInSandbox. Kata’s shared filesystem drops the file capabilities that rootless BuildKit’s newuidmap needs, so BuildKit runs rootful and privileged inside the guest. That is only safe when the runtime keeps privileged in the guest: containerd’s privileged_without_host_devices = true for the Kata handler (kata-deploy sets it). Check your node’s containerd config before enabling it.

gVisor and snapshotter: native. BuildKit’s overlay layers live on the build pod’s state volume, and gVisor treats volumes as shared with the outside world. In that mode it cannot copy up a lower-layer file that a running process has memory-mapped: the copy-up fails with ENODEV. Package managers hit this routinely, e.g. apt-get upgrading a library that apt-get itself has loaded stops with unable to make backup link of '...libcrypto.so.3' before installing new version: No such device. The native snapshotter copies layers instead of stacking them, so nothing is copied up; every operation works, but builds are slower and use more disk (a large multi-stage build took about twice as long as under Kata). Things that do not help (tested with runsc release-20261116.0): file-access-mounts=exclusive on the runsc handler avoids the ENODEV but crashes the whole sandbox when the copying process has the file mapped itself; per-pod gVisor mount hints turn the state emptyDir into a RAM-backed tmpfs; ephemeral PVCs reach gVisor as the same kind of volume (filesystem mode) or not at all (block mode: no ext4); fuse-overlayfs does not run in gVisor. For heavy builds prefer Kata. The isolation check builds exactly this case, so gVisor without native stops the controller at startup rather than failing the first build that upgrades a package.

Sysbox and rootfulInSandbox. Sysbox runs the pod in a user namespace, gives root in the pod its capabilities there and virtualizes procfs/sysfs, so rootful BuildKit works without privileged: true or unconfined seccomp/AppArmor (Sysbox applies its own seccomp filter and ignores AppArmor profiles). rootfulInSandbox: true selects the rootful image and sets hostUsers: false; it needs mode runtimeClass and excludes privilegedInSandbox. Sysbox must not be combined with privileged containers. Requirements (Sysbox docs): Sysbox 0.7+ (sysbox-deploy-k8s), Kubernetes user namespaces (on by default from 1.33) and containerd 2.0.5+, so hostUsers: false reaches the runtime; GitImage does not set the CRI-O-only io.kubernetes.cri-o.userns-mode annotation older setups used. sysbox-deploy-k8s refuses Kubernetes versions it was not tested with (1.32-1.35 for Sysbox 0.7.1), documents Ubuntu nodes, and sets host sysctls.

Sysbox in kind. hack/e2e/sysbox.sh installs the Sysbox 0.7.1 package by hand into the kind worker (Kubernetes 1.37, containerd 2.3; the installer would refuse 1.37 and change the sysctls of the host kernel, which kind nodes share), remounts the node’s /sys read-write and removes kind’s bind mounts over /sys/devices/virtual/dmi/id/product_*: with either in place the kernel refuses the container’s sysfs mount (sysfs mount through procfd to sys caused: operation not permitted). CI always runs it (matrix include, independent of E2E_ISOLATIONS); the nodes need user namespaces enabled.

Kata on Talos (virtiofs without xattrs). Kata shares filesystem pod volumes (emptyDir, and PVCs in Filesystem mode) into the VM over virtiofs. When that share lacks extended-attribute support (Talos’ Kata extension), BuildKit fails with failed to create hash for ...: operation not supported. Use a raw block volume instead: Kata attaches volumeMode: Block PVCs to the VM as a virtio-blk disk, so BuildKit gets a real filesystem.

build:
  stateVolume:
    block:
      storageClassName: ceph-block   # any class that supports volumeMode: Block
      size: 20Gi
      fsType: ext4                   # or ext3

Each build pod gets a fresh ephemeral PVC (deleted with the pod), or with persistent the GitImage’s own PVC, formatted once. A format-state init container (formatImage, default debian:bookworm-slim) runs mkfs, and the BuildKit container mounts the device at its state dir before starting, so this needs privileged or privilegedInSandbox. The isolation check runs with the same volume. medium: Memory (tmpfs) also works but counts against the build’s memory limit.

userns in kind. runc cannot mount sysfs for a user-namespaced pod when the node itself is a container, so kind cannot test this mode; use it on VM or bare-metal nodes with Kubernetes 1.33+.

Hosts without user namespaces. gVisor’s gofer, rootless BuildKit and userns mode all create user namespaces; with user.max_user_namespaces=0 (the Talos default) they fail (fork/exec /proc/self/exe: no space left on device). Kata with privilegedInSandbox and privileged mode do not need them.

Use the RuntimeClass’s scheduling.nodeSelector to keep Kata builds on nodes that have it.

Storage modes

storage.typeImages live inPods pull from
registry<url>/<namespace>/<gitimage>:<sha> in any OCI registrythe registry
s3an S3 bucket: repos/<namespace>/<gitimage>/{blobs,manifests,tags}localhost:25050 on each node

Builds never see storage credentials: BuildKit pushes to the controller’s push endpoint (Service gitimage-push:5000, TLS with the controller’s CA), which writes to the backend.

registry

pushSecret (dockerconfigjson, in the release namespace) is used by the controller only. With a private registry set pullSecret: it is copied into every namespace with a GitImage as gitimage-pull and added to gated pods. GC deletes manifests; blob GC is the registry’s own.

s3

Works with Ceph RGW, SeaweedFS, AWS S3, R2 and other S3 APIs (pathStyle: true for most self-hosted ones). gitimage-node is a hostNetwork DaemonSet bound to 127.0.0.1:25050 only; containerd allows plain HTTP to localhost with no node configuration (verified with containerd 2.3). Change node.port if 25050 is taken on your nodes.

Add a lifecycle rule that expires objects under uploads/ after a day: they are abandoned multipart uploads of crashed pushes. Blobs over 5 GiB are not supported.

Garbage collection

Hourly on the leader: keep the 5 newest builds per GitImage, every digest a pod runs, the build cache and in-flight builds; delete unreferenced manifests, and in s3 mode blobs older than 1 h.

Helm values

ValueDefaultMeaning
buildPolicy.isolation.moderequiredruntimeClass, userns or privileged
buildPolicy.isolation.runtimeClassNameRuntimeClass for runtimeClass mode
buildPolicy.isolation.allowPrivilegedfalsemust be true for privileged
buildPolicy.isolation.privilegedInSandboxfalserootful BuildKit inside the sandbox (Kata)
buildPolicy.isolation.rootfulInSandboxfalserootful BuildKit, unprivileged, hostUsers: false (Sysbox); runtimeClass only, excludes privilegedInSandbox
buildPolicy.egress.allowInternettruebuild pods may reach the internet
buildPolicy.egress.denyClusterCIDRstrueexcept network.clusterCIDRs
buildPolicy.egress.extraRules[]NetworkPolicy egress rules added for build pods, each with to; see In-cluster registry mirrors
network.clusterCIDRs[]pod and service CIDRs, IPv4 and IPv6
storage.typeregistryregistry or s3
storage.registry.url, .pushSecret, .pullSecret, .insecureregistry mode
storage.s3.bucket, .endpoint, .region, .secret, .readOnlySecret, .pathStyles3 mode
build.image / build.rootfulImagemoby/buildkit:v0.34.2-rootless / moby/buildkit:v0.34.2BuildKit images
build.activeDeadline1hper build
build.platforms[linux/amd64]more than one builds a multi-arch index
build.cachemaxmax, min or off
build.labels, build.extraOpts, build.defaultResourcesapplied to every build
build.stateVolume.medium, .sizeLimit""emptyDir for BuildKit’s state; Memory = tmpfs
build.stateVolume.blockunset{storageClassName, size, fsType: ext4|ext3, formatImage}: raw block ephemeral PVC per build, formatted and mounted in the pod (Kata on Talos)
build.stateVolume.labels, .annotations{}extra metadata on the state PVCs (block or persistent), e.g. velero.io/exclude-from-backup: "true"
build.stateVolume.persistentunset{size, storageClassName}: one PVC per GitImage kept between builds; see Persistent build cache
build.buildkitdConfig""appended to the build pods’ buildkitd.toml (registry mirrors, private registries)
hooks.baseURL, hooks.ingress.*push hooks
ui.enabled, ui.baseURL, ui.auth.*, ui.ingress.*falsestatus UI on port 8082; ui.auth.mode is none or oidc
controller.replicas, webhook.replicas2
controller.extraEnv[]extra env for the controller
controller.pinTemplateImagefalsealso write the built image into workload templates, for scanners (Workloads)
webhook.crossTenantNamespaces[]namespaces whose pods may use any namespace’s GitImage images (image scanners)
node.port, node.healthPort25050, 25051gitimage-node host ports
podMonitor.enabledfalsePrometheus Operator PodMonitor
logFormat, logLeveljson, info,kube=warn
extraObjects[]extra manifests installed with the release

Persistent build cache

By default every build starts BuildKit on an empty state volume and gets its cache from the registry (build.cache): a step that misses the cache first downloads every layer below it, and RUN --mount=type=cache (cargo, npm, ccache) always starts empty. With build.stateVolume.persistent each GitImage gets a PVC, <name>-buildkit, that its builds reuse:

build:
  stateVolume:
    persistent:
      size: 20Gi
      storageClassName: local-path   # default class if unset
  • Cache. Cache hits are local, nothing is downloaded, and cache mounts keep their contents. BuildKit’s garbage collection keeps the state within 80% of size. Keep build.cache on: it fills a new PVC (first build, after a reset or a settings change).
  • One build at a time. The PVC is ReadWriteOnce and BuildKit locks its state dir, so with onNewCommit: cancel the next build starts once the cancelled build’s pod is gone (status: waiting for the cancelled build of <sha> to stop).
  • Placement. Builds of a GitImage run where its PVC can be attached: its zone, or its node with local-path. Network block storage adds a few seconds of attach time per build.
  • Reset. kubectl delete pvc <name>-buildkit; the controller creates a new one for the next build. It also recreates the PVC itself when the isolation layout (rootful or rootless), size, storageClassName or block.fsType change, with a StateVolumeRecreated Event.
  • Block mode. With block too, the PVC is volumeMode: Block, sized by persistent, and formatted only when it holds no filesystem; e2fsck -p repairs it after a killed build. Give block only fsType/formatImage then: block: { fsType: ext4 }.
  • Backups. The PVCs hold only a cache. Keep them out of backups with build.stateVolume.labels (Velero: velero.io/exclude-from-backup: "true") or .annotations (k8up: k8up.io/backup: "false"); PVCs that already exist get them on their next build. A PVC restored anyway points at the old GitImage’s UID: the controller replaces it rather than reuse it.
  • Security. A commit can leave files in the state that later builds of the same GitImage use, as it already can through the registry cache; other GitImages never see it. Reset the PVC after building an untrusted commit.

Not combinable with medium or sizeLimit. The PVC is deleted with its GitImage.

In-cluster registry mirrors

Build pods may reach the internet but not network.clusterCIDRs. A mirror in build.buildkitdConfig served by an ingress controller in the same cluster resolves to its LoadBalancer IP, which the CNI translates to the controller’s pod IPs, so the build policy drops the connection. BuildKit then waits on the mirror for about two minutes per base image ([internal] load metadata for docker.io/...) and falls back to the upstream registry, rate limits included. Allow the ingress controller’s pods:

buildPolicy:
  egress:
    extraRules:
      - to:
          - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: traefik } }
            podSelector: { matchLabels: { app.kubernetes.io/name: traefik } }
        ports: [{ port: 8443, protocol: TCP }]   # the pods' port, not the Service's

The rules are added to the build pods’ NetworkPolicy as written; each needs to, and unknown fields fail the config. Build pods can then reach every host that ingress controller serves.

extraObjects

Anything else the install needs, as maps or templated strings (tpl is applied):

extraObjects:
  - apiVersion: objectbucket.io/v1alpha1
    kind: ObjectBucketClaim
    metadata: { name: gitimage, namespace: "{{ .Release.Namespace }}" }
    spec: { bucketName: gitimage, storageClassName: ceph-bucket }
  - |
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: notes
      labels: {{- include "gitimage.labels" . | nindent 8 }}
    data: { pushHost: {{ include "gitimage.pushHost" . }} }

The chart writes gitimage-config from these values; gitimage check-config --config <file> validates a rendered config with the controller’s strict parser.

Security model

AssetWho can read it
Git credentialsthe controller (get on the named Secret, in memory) and the clone init container of that GitImage’s builds
Build secretsthe BuildKit container of that GitImage’s builds, as tmpfs for one RUN
Storage credentialsthe controller (and read-only ones in gitimage-node); never build pods
Push tokensone per build Job, HS256, scoped to <namespace>/<gitimage>, valid only while that Job runs
  • Pod Security admission. Build pods are privileged in Kata (inside the VM) and privileged modes, and use unconfined seccomp/AppArmor in rootless modes, so every namespace with a GitImage needs pod-security.kubernetes.io/enforce: privileged. With Sysbox (rootfulInSandbox: true) build pods are unprivileged, add no capabilities and set no unconfined profiles, so baseline is enough; restricted rejects them (no allowPrivilegeEscalation: false, capabilities.drop: [ALL] or RuntimeDefault seccomp). The release namespace needs privileged in every mode (gitimage-node uses hostNetwork). With a stricter level the build fails within seconds with PodCreationFailed and the admission message.
  • RBAC. The controller gets get/create/update/delete on Secrets, never list or watch.
  • Admission. The mutating webhook is called only for pods carrying a gitimage.dev/image.* annotation (CEL matchConditions), so a webhook outage blocks only GitImage pods. A second entry denies any pod that references another namespace’s GitImage image.
  • Egress. Build pods get a NetworkPolicy: DNS, the push endpoint, the internet minus the cluster CIDRs, and any buildPolicy.egress.extraRules.
  • TLS. The controller runs its own CA (Secret gitimage-ca), issues the webhook and push certificates, renews them 30 days before expiry and patches the webhook caBundle.
  • Supply chain. Release images are signed with cosign (keyless) and carry an SBOM and provenance.

Status UI

A read-only web page served by the controller: every GitImage with its state, the live build log grouped by BuildKit step, the workloads that use an image and the pods waiting for one.

ui:
  enabled: true
kubectl -n gitimage-system port-forward svc/gitimage-ui 8082
# open http://localhost:8082

The UI changes nothing in the cluster. To retry a build, it shows the kubectl annotate command to run.

Sign-in

ui.auth.modeWho sees whatUse it for
none (default)Every visitor sees what the controller’s ServiceAccount sees: all namespaces, all build logs.kubectl port-forward, or behind a proxy that authenticates. Do not expose it otherwise.
oidcEach user sees what cluster RBAC allows them.A shared URL.

OIDC

The UI signs the user in (authorization code flow with PKCE) and sends their ID token to the Kubernetes API server as the bearer token of every request it makes for them. The API server authenticates the token and applies RBAC, the same model as Headlamp. The controller’s own permissions are never used for a signed-in user.

Requirements:

  1. The API server trusts the issuer (--oidc-* flags or a structured AuthenticationConfiguration), and the UI’s client id is one of its accepted audiences.

  2. The client has the redirect URI <ui.baseURL>/auth/callback.

  3. Users have RBAC. To see a GitImage in a namespace:

    To seeNeeds
    It in the listlist on gitimages.gitimage.dev in the namespace (or cluster-wide)
    Its pageget on that GitImage
    Workloadslist on deployments, statefulsets, daemonsets
    Waiting podslist on pods
    Build logslist and get on pods, get on pods/log

    What a user may not list is shown as a notice on the page, not as an error.

ui:
  enabled: true
  baseURL: https://gitimage.example.com
  auth:
    mode: oidc
    oidc:
      issuer: https://auth.example.com     # exactly as the API server has it
      clientID: "312345678901234567"       # quote numeric ids; with helm use --set-string
      existingSecret: gitimage-ui-oidc     # keys: clientSecret, optionally clientID
  ingress:
    enabled: true
    host: gitimage.example.com

A session lasts as long as the ID token. When it expires the browser is sent through the issuer again, which is invisible while the issuer’s own session is alive. The session is an encrypted cookie, so both controller replicas accept it and nothing is stored server-side.

If the API server rejects the token, the UI says so. The usual cause is an audience mismatch: the API server accepts only the client ids listed in its configuration.

Limits

  • Logs live one hour. They are read from the build pod, and build Jobs are removed one hour after they finish. After that the page shows the failure message kept in status.builds.
  • Build output is visible to everyone who may read pod logs in the namespace. A Dockerfile that prints a build argument prints it there too. Use build.secrets for credentials; secret mounts never reach the log.
  • The page of a GitImage shows its push-hook URL to anyone who may get the GitImage, the same as kubectl get gi -o yaml.

Developing the UI

The web app lives in ui/ (Svelte 5, Vite, Tailwind) and is embedded into the binary at build time. Without a built ui/dist, the binary serves a placeholder page.

cd ui && npm ci
npm run dev                                        # against mock data, no cluster needed
GITIMAGE_UI_PROXY=http://localhost:8082 npm run dev   # against a port-forwarded controller
npm run check && npm run build                     # then cargo build embeds ui/dist

Troubleshooting

SymptomLook at
Controller in CrashLoopBackOffkubectl -n gitimage-system logs deploy/gitimage-controller --previous: IsolationUnavailable names the failing mode; kubectl -n gitimage-system logs job/gitimage-isolation-check-<hash>
Pod stays SchedulingGatedkubectl get gi <name>: no image yet; kubectl describe pod: GitImageNotFound or build progress Events
Failed / ResolveFailedwrong URL, ref, credentials or host key (host key ... not in known_hosts)
Failed / PodCreationFailedthe namespace’s Pod Security level: label it pod-security.kubernetes.io/enforce=privileged (baseline with rootfulInSandbox)
Failed / ReconcileErrorthe message is the API server’s answer to the controller (an invalid Job, a quota, missing RBAC); the controller log has the same error
Failed / BuildFailedkubectl logs job/<name>-<sha7> -c buildkit
Failed / PushFailedcontroller logs (storage errors), then the build log
Annotated pod is created without a gate or digestkubectl get mutatingwebhookconfiguration gitimage -o yaml: caBundle set? webhook pods ready?
Every build spends ~2 min in load metadata for docker.io/...a registry mirror build pods cannot reach, often one behind an in-cluster ingress controller: In-cluster registry mirrors
Pull fails in s3 modekubectl -n gitimage-system get ds gitimage-node: NotReady means S3 is unreachable or port 25050 in use on 127.0.0.1

Logs are JSON by default; --set logFormat=text for humans, logLevel=info,gitimage=debug for more.

Example: Talos homelab

Kata on nodes with /dev/kvm, Ceph RGW (Rook) for storage, no registry, ArgoCD.

buildPolicy:
  isolation: { mode: runtimeClass, runtimeClassName: kata, privilegedInSandbox: true }
storage:
  type: s3
  s3:
    bucket: gitimage
    endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc
    secret: gitimage        # created by the ObjectBucketClaim below
    pathStyle: true
network:
  clusterCIDRs: [10.244.0.0/16, 10.96.0.0/12]
extraObjects:
  - apiVersion: objectbucket.io/v1alpha1
    kind: ObjectBucketClaim
    metadata: { name: gitimage, namespace: "{{ .Release.Namespace }}" }
    spec: { bucketName: gitimage, storageClassName: ceph-bucket }

Before enabling privilegedInSandbox, confirm the Kata handler’s containerd config has privileged_without_host_devices = true.

Architecture

One Rust binary (gitimage) with subcommands controller, webhook, node, clone, crdgen, check-config, built on kube-rs, axum, gix and russh. No git or ssh binary, no OpenSSL.

CrateRole
gitimage-apiCRD types, annotation keys, the strict Config
gitimage-gitref resolution, gix fetch and checkout, russh ssh transport with known_hosts checks, host tree APIs
gitimage-ociOCI distribution routes (push endpoint and node server), S3 / registry / memory storage, push tokens
gitimage-controllerreconciler state machine, Job builder, gate releaser, roller, hooks, GC, isolation check, TLS bootstrap, status UI API (ui/, with the web app from the top-level ui/ embedded by build.rs)
gitimage-webhookpure mutate(pod) -> patch, guard, axum handler
gitimage-nodelocalhost-only read-only OCI server
gitimage-tlsCA and certificates, hot-reloading TLS listeners

The design documents (PRD and implementation design) record the decisions and trade-offs.

Testing

cargo test --workspace                 # unit, storage, router, snapshot tests
./hack/oci-conformance.sh              # OCI distribution-spec conformance
./hack/check-chart-config.sh           # chart renders configs the controller accepts
./hack/e2e/run.sh s3 gvisor            # kind e2e: registry|s3 x privileged|gvisor|kata-qemu|sysbox
KEEP=1 ./hack/e2e/run.sh registry privileged   # keep the cluster for debugging
E2E_PERSISTENT=1 ./hack/e2e/run.sh registry gvisor   # with build.stateVolume.persistent

The e2e environment is a kind cluster with its own kubeconfig (hack/.kubeconfig) plus a registry, SeaweedFS (S3) and Gitea as containers on the kind network. The suite drives Gitea through its API and covers: commit to running digest, rollouts, tags, monorepo reuse, hooks, build secrets, private repos over ssh with host-key checks, the cross-tenant guard, failed builds and rebuilds (of failed and of built commits), GC, and IsolationUnavailable.

Releasing

Every push to main runs one pipeline: checks (fmt, clippy, tests, chart, cargo-deny), OCI conformance, one build per architecture, the e2e matrix, and only if all of them pass, a release:

  • version <major>.<minor>.<run number> (major.minor from Cargo.toml);
  • multi-arch image ghcr.io/diverofdark/gitimage:<version> and :latest, cosign-signed;
  • Helm chart oci://ghcr.io/diverofdark/charts/gitimage --version <version>;
  • a GitHub release with binaries and notes;
  • this documentation, published to GitHub Pages.

Runners: the repository variable RUNS_ON names a self-hosted runner label for every job (E2E_RUNS_ON overrides it for e2e only); unset means GitHub-hosted. Pull requests from forks always use GitHub-hosted runners, and arm64 builds always use GitHub’s arm runner. E2E_ISOLATIONS (JSON list) adds kata-qemu where runners have /dev/kvm (sysbox always runs, via the matrix include); DOCKERHUB_PREFIX (e.g. harbor.example.com/docker-hub/) makes self-hosted jobs pull every Docker Hub image through a path-style proxy cache: job containers, kind’s node image, the e2e helpers, containerd in kind (hosts.toml with override_path) and BuildKit (mirrors). E2E_GITHUB_ISOLATIONS (JSON list) keeps some isolations on GitHub-hosted runners — gVisor, rootless BuildKit and Sysbox need user namespaces, which Talos disables by default (user.max_user_namespaces=0).