GitImage
GitImage is a Kubernetes operator that turns a git URL into a running pod. You point a
GitImage at a repository; every commit is built in-cluster by BuildKit inside an isolated pod,
stored in an OCI registry or in S3, and injected into your workloads by digest.
apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: myapp, namespace: team-a }
spec:
url: git@github.com:you/myapp.git
ref: main
secretRef: { name: myapp-git }
# in any Deployment, StatefulSet, DaemonSet, Job or CronJob pod template
metadata:
annotations:
gitimage.dev/image.web: myapp # container "web" runs GitImage "myapp"
spec:
containers:
- name: web
image: placeholder # replaced by the built digest at admission
What you get:
- No CI, no registry required. Builds run in your cluster; with S3 storage, nodes pull from a node-local read-only server and no registry exists at all.
- Pods never fail on a missing image. A pod created before its build finishes waits as
SchedulingGated, then starts with the digest. - Every commit rolls out. Deployments, StatefulSets and DaemonSets get the new digest automatically; each pod is pinned to an exact commit.
- Builds are isolated. Kata VMs, gVisor, user namespaces or (opt-in) privileged; the controller refuses to start rather than fall back to a weaker mode.
- Builds never hold credentials they do not need. Git credentials stay in the tenant namespace; storage credentials never leave the controller.
Current release: 0.1.71. Source: github.com/DiverOfDark/gitimage.
Getting started
You need Kubernetes 1.30+ with containerd, Helm 3, and either an OCI registry or an S3 bucket.
1. Pick an isolation mode
| Cluster | Use |
|---|---|
Nodes with /dev/kvm and Kata installed | mode: runtimeClass, runtimeClassName: kata-qemu, privilegedInSandbox: true |
gVisor installed (GKE Sandbox, runsc) | mode: runtimeClass, runtimeClassName: gvisor |
Sysbox installed (sysbox-deploy-k8s) | mode: runtimeClass, runtimeClassName: sysbox-runc, rootfulInSandbox: true |
| Kubernetes 1.33+ on VMs or bare metal | mode: userns |
| Trusted repositories only | mode: privileged, allowPrivileged: true |
See Isolation modes.
2. Install
Registry storage:
kubectl create namespace gitimage-system
kubectl -n gitimage-system create secret docker-registry registry-push \
--docker-server=ghcr.io --docker-username=<user> --docker-password=<token>
helm install gitimage oci://ghcr.io/diverofdark/charts/gitimage --version 0.1.71 \
-n gitimage-system \
--set buildPolicy.isolation.mode=runtimeClass \
--set buildPolicy.isolation.runtimeClassName=gvisor \
--set storage.type=registry \
--set storage.registry.url=ghcr.io/<you>/builds \
--set storage.registry.pushSecret=registry-push
S3 storage (no registry):
kubectl -n gitimage-system create secret generic gitimage-s3 \
--from-literal=AWS_ACCESS_KEY_ID=... --from-literal=AWS_SECRET_ACCESS_KEY=...
helm install gitimage oci://ghcr.io/diverofdark/charts/gitimage --version 0.1.71 \
-n gitimage-system \
--set buildPolicy.isolation.mode=userns \
--set storage.type=s3 --set storage.s3.bucket=gitimage \
--set storage.s3.endpoint=https://s3.example.com --set storage.s3.secret=gitimage-s3
The controller runs an isolation check on start. kubectl -n gitimage-system get pods shows
Running when the mode works; CrashLoopBackOff with IsolationUnavailable in the log when it
does not.
3. Build something
Build pods need the privileged Pod Security level in their namespace (baseline with
Sysbox; see
Security model).
kubectl create namespace demo
kubectl label namespace demo pod-security.kubernetes.io/enforce=privileged
kubectl apply -n demo -f - <<'YAML'
apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: hello }
spec:
url: https://github.com/<you>/<public-repo-with-a-Dockerfile>.git
ref: main
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: hello }
spec:
selector: { matchLabels: { app: hello } }
template:
metadata:
labels: { app: hello }
annotations: { gitimage.dev/image.web: hello }
spec:
containers: [{ name: web, image: placeholder }]
YAML
kubectl -n demo get gi -w # Revision, Ready, Status (current BuildKit step)
kubectl -n demo get pods -w # SchedulingGated, then Running
How it works
- Resolve. The controller lists the remote’s refs (git protocol v2
ls-refs, nothing fetched) everyinterval, or at once when a push hook arrives. A full sha is used as is, else a tag, else a branch; only a branch can move. - Decide. If the commit did not change the build context’s git tree (monorepos), the last image is reused and nothing is built.
- Build. A Job in the tenant namespace: an init container clones exactly that commit at
depth 1 (git credentials are mounted only here), then BuildKit builds it under the chosen
isolation mode. Progress (
step 20/42 backend (+1 parallel): RUN cargo build) is copied intostatusand into Events on waiting pods. The count is over every step of every stage, since stages build in parallel; the step named is the one running longest. - Push. BuildKit pushes the image and its layer cache to the controller’s push endpoint
with a token valid only for
<namespace>/<gitimage>and only while that Job runs. The endpoint writes to the registry or S3. - Admit. The webhook sees every pod annotated with
gitimage.dev/image.<container>. If the image for the pinned (or current) commit exists, it writes the digest into the container right away; otherwise it adds the scheduling gategitimage.dev/build. - Release. When the build lands, the controller patches the digest into each gated pod
and removes the gate. It also pins the new commit into every Deployment, StatefulSet and
DaemonSet that uses the GitImage (
gitimage.dev/revision.<container>, andgitimage.dev/digest.<container>so a rebuild of the same commit rolls too), which rolls them.
git push ──> controller ──> build Job (tenant ns) ──> push endpoint ──> registry | S3
│ │
└── status.image ──> webhook / gate releaser ──> pods <───┘ (pull)
Workloads in gitimage-system
| Workload | Replicas | Does |
|---|---|---|
gitimage-controller | 2, one leader (Lease) | reconcilers, gate release, rollouts, GC on the leader; push endpoint and hooks on every replica |
gitimage-webhook | 2 | admission (gates, digests, cross-tenant guard); reads a GitImage cache, never the API on the request path |
gitimage-node | DaemonSet, s3 mode only | read-only OCI server on 127.0.0.1:25050 for containerd |
The GitImage resource
gitimage.dev/v1alpha1, kind GitImage, namespaced, short name gi.
apiVersion: gitimage.dev/v1alpha1
kind: GitImage
metadata: { name: myapp, namespace: team-a }
spec:
url: git@github.com:you/myapp.git # ssh (git@host:path, ssh://...) or https
ref: main # branch | tag | full sha | refs/heads/... | refs/tags/...
secretRef: { name: myapp-git } # optional, same namespace
interval: 5m # poll interval; hooks trigger earlier
build:
context: . # repo-relative build directory
dockerfile: Dockerfile # relative to context
args: {} # see Credentials, build secrets and args
secrets: []
resources: { requests: { cpu: "1", memory: 2Gi }, limits: { memory: 4Gi } }
onNewCommit: wait # wait | cancel
Status
| Field | Meaning |
|---|---|
latestRevision | newest commit seen on the ref |
revision | commit whose image is current |
image | <pull host>/<namespace>/<name>@sha256:... |
builds[] | newest first, up to 10: revision, outcome (Built, Reused, Superseded, Failed), digest, contextTree, sourceRevision, reason, attempt (the build Job’s), finishedAt |
webhookURL | push-hook URL to paste into the git host (needs hooks.baseURL) |
handledRebuild | the last gitimage.dev/rebuild value whose rebuild finished; a different annotation value is a rebuild still to do |
conditions | Ready, Building (message = current BuildKit step), Failed (reason) |
Failure reasons: ResolveFailed, SecretMissing, CloneFailed, BuildFailed, PushFailed,
DeadlineExceeded, PodCreationFailed, ReconcileError.
ReconcileError means the controller itself failed, for example because the API server
rejected the build Job or a quota forbids it; the message is the API server’s. The controller
retries every 30 seconds, and the next successful reconcile clears the condition. If the
GitImage has no image yet, Ready is False with the same reason, so kubectl get gi shows it.
Behaviour
- One build at a time per GitImage. With
onNewCommit: wait(default) the running build finishes, then only the newest commit is built; withcancelthe running build is deleted. - A failed build is not retried automatically. Push a new commit or bump the annotation:
kubectl annotate gi myapp gitimage.dev/rebuild=$(date +%s) --overwrite. - Any change of
gitimage.dev/rebuildrebuilds the current commit, whatever its last outcome (Built,Reused,Failed,Superseded); see Rebuilding a commit. - Pod-level failures (eviction, node loss) are retried twice.
- Deleting the GitImage deletes its Jobs and hook Secret; stored images are left to GC.
Rebuilding a commit
Set gitimage.dev/rebuild on the GitImage to a new value to build the current commit again
without a new commit:
kubectl annotate gi myapp gitimage.dev/rebuild=$(date +%s) --overwrite
- The rebuild always runs a build Job: it never ends as
Reused, even if the context did not change, and it runs even when the commit’s earlier Jobs are already gone. - The value reaches the Dockerfile as the build arg
GITIMAGE_REBUILD(empty without the annotation). BuildKit otherwise reuses every cached layer; a Dockerfile that declaresARG GITIMAGE_REBUILDmisses the cache from that line on, so the steps after it run again. - While it runs,
status.imageandstatus.revisionstay on the previous image and workloads keep running it. When it succeeds, the commit’s record gets the new digest and workloads roll to it (the roller pinsgitimage.dev/digest.<container>next to the revision). If the digest is unchanged, nothing rolls. - If it fails, the failure is recorded (
Failedcondition,BuildFailedevent) and the previous image stays current: theFailedrecord keeps its digest. - A request made while a build runs (of this commit, or of an older one with
onNewCommit: wait) waits for that build, then rebuilds.status.handledRebuildtakes the value once the rebuild’s result is recorded. - A rebuild that has started always finishes and is recorded: removing the annotation, or
setting it back to
status.handledRebuild, does not cancel it, and a newer value rebuilds again after it. - Only the value matters, not what it is; a request is never repeated by itself. Do not keep the annotation in a GitOps-managed manifest, or a sync that resets it starts a rebuild.
Daily rebuilds
A site that filters content by date (posts published in the future) has to be rebuilt even when no commit changed. Declare the arg before the build step:
FROM node:24 AS build
WORKDIR /src
COPY package*.json ./
RUN npm ci
COPY . .
# Changes on every rebuild: the build below runs again, npm ci stays cached.
ARG GITIMAGE_REBUILD
RUN npm run build
And bump the annotation from a CronJob in the GitImage’s namespace (kubectl annotate reads
the object first, so the Role needs get and patch):
apiVersion: v1
kind: ServiceAccount
metadata: { name: site-rebuild, namespace: team-a }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: site-rebuild, namespace: team-a }
rules:
- apiGroups: [gitimage.dev]
resources: [gitimages]
resourceNames: [site]
verbs: [get, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: site-rebuild, namespace: team-a }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: site-rebuild }
subjects: [{ kind: ServiceAccount, name: site-rebuild, namespace: team-a }]
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: site-rebuild, namespace: team-a }
spec:
schedule: "5 0 * * *" # daily, 00:05
timeZone: Etc/UTC
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
serviceAccountName: site-rebuild
restartPolicy: Never
containers:
- name: annotate
image: docker.io/alpine/kubectl:1.37.2
command: [sh, -c, 'kubectl annotate gi site gitimage.dev/rebuild=$(date +%s) --overwrite']
Running workloads
Annotate the pod template (not the workload’s own metadata):
spec:
template:
metadata:
annotations:
gitimage.dev/image.web: myapp # container "web" -> GitImage "myapp"
gitimage.dev/image.worker: myapp # several containers may use one GitImage
spec:
containers:
- { name: web, image: placeholder }
- { name: worker, image: placeholder, command: [worker] }
Init containers can be bound the same way. The GitImage must be in the pod’s namespace.
| You want | Do |
|---|---|
| Automatic rollouts | nothing: Deployments, StatefulSets and DaemonSets get gitimage.dev/revision.<container> pinned on each new commit, and gitimage.dev/digest.<container> on each new image (a rebuild of the same commit rolls too) |
| No automatic rollouts | gitimage.dev/rollout: manual on the workload’s metadata |
| Jobs and CronJobs | supported; each new pod gets the current revision at creation |
| A specific commit | set ref to the sha (or a tag) |
A pod whose template pins a digest (gitimage.dev/digest.<container>) that the GitImage’s
status does not show yet, because a rebuild just finished, waits gated for a moment rather than
start on the commit’s previous image. A pin that never matches (you changed
gitimage.dev/revision.<container> by hand, or a manual workload’s commit was rebuilt since)
is dropped after a minute: the pod gets the revision’s current digest. Remove the digest
annotation when you pin a revision by hand.
A pod admitted before its image exists stays SchedulingGated (not Pending on a node, never
ImagePullBackOff). While it waits, it gets Events like
Building myapp@3f2c9e1: step 4/12 RUN npm ci, and GitImageNotFound if the GitImage does not
exist yet.
A pod may only reference GitImage images of its own namespace; the webhook denies anything else.
Image scanners
The template keeps placeholder; only pods get the built image. Scanners that read templates
instead of pods, such as Trivy Operator for Deployments, StatefulSets and DaemonSets, scan the
placeholder. Two chart values fix that:
controller.pinTemplateImage: true: the roller writes the built image into each bound container of the template, together with the revision annotation. It checks on every reconcile, so workloads created after a build and templates reset by an apply are pinned again. Turning it on rolls each workload once.webhook.crossTenantNamespaces: [trivy-system]: Trivy Operator in filesystem mode runs the scanned image in its own namespace, which the cross-tenant guard otherwise denies.
With ArgoCD, the managedFieldsManagers ignore rule and RespectIgnoreDifferences from
GitOps with ArgoCD keep the pinned image on sync. Helm and kubectl apply reset it
to the placeholder, which rolls the workload, and the roller then pins it again, which rolls it
once more.
Credentials, build secrets and args
Git credentials
spec.secretRef names a Secret in the GitImage’s namespace (Flux format):
| Transport | Keys |
|---|---|
| ssh | identity (OpenSSH private key: ed25519, ecdsa or rsa; no passphrase), known_hosts (required: unknown host keys are refused) |
| https | username, password (a token works; username defaults to git) |
kubectl -n team-a create secret generic myapp-git \
--from-file=identity=./deploy_key --from-literal=known_hosts="$(ssh-keyscan github.com)"
The controller reads it to list refs; the clone init container of each build mounts it
read-only. It is never copied anywhere else. Over plain http:// no credentials are sent.
Build secrets
For private package registries, without baking credentials into layers:
build:
secrets: [{ id: npmrc, secretRef: { name: npm-token, key: npmrc } }] # key optional if the Secret has one key
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
The secret exists in tmpfs for that RUN only; it is never in a layer, the cache or the history.
Repository code in that step can read it, so give secrets only to repositories you trust.
Build args
build:
args:
NODE_ENV: { value: production }
SENTRY_RELEASE: { valueFrom: { secretKeyRef: { name: sentry, key: release } } }
Build args end up in the image history; use build secrets for real secrets.
Commit metadata
Every build also gets three args from GitImage, set after build.args so they cannot be
overridden:
| Arg | Value |
|---|---|
GITIMAGE_REVISION | full commit sha |
GITIMAGE_COMMIT_TIME | the commit’s committer time, unix seconds |
GITIMAGE_REBUILD | the GitImage’s gitimage.dev/rebuild annotation, empty without one |
The build sees no .git (the clone is one commit deep), so these are how a Dockerfile learns
which commit it is building. Declare the ones you use:
ARG GITIMAGE_REVISION=unknown
ARG GITIMAGE_COMMIT_TIME=0
GITIMAGE_COMMIT_TIME only grows along a branch that is not rewritten, so it works as a
monotonic version number (an Android versionCode, for example: it stays under Android’s
2100000000 cap until mid-2036).
Undeclared, they change nothing: BuildKit ignores unused args. Declared, they change on every
commit and so invalidate the cache of every later RUN in that stage: declare them in the
stage that needs them, as late as possible.
GITIMAGE_REBUILD changes only when the annotation does. Declare it just before the steps that
must run again on a rebuild without a new commit (a site
build that filters posts by date, for example); everything above it stays cached:
ARG GITIMAGE_REBUILD
RUN npm run build
Push hooks
Polling every interval always works. For builds within seconds of a push:
- Expose the hooks Service:
hooks.baseURL: https://gitimage.example.comandhooks.ingress.enabled: true(or your own Ingress/Gateway to Servicegitimage-hooks:8080, path prefix/hooks/). - Copy the URL:
kubectl get gi myapp -o jsonpath='{.status.webhookURL}'. - Add it as a push webhook in GitHub, GitLab, Gitea or Forgejo (any content type, no secret).
The URL carries a random token (Secret <name>-gitimage-hook; delete it to rotate). A hook only
triggers a re-check of the ref: its payload is never trusted. Hooks for one repository wake every
GitImage in that namespace with the same repository URL.
Monorepos
Create one GitImage per image, all with the same url and a different build.context:
spec: { url: git@github.com:you/mono.git, build: { context: services/api } }
---
spec: { url: git@github.com:you/mono.git, build: { context: services/web } }
A commit that does not change the git tree of context is recorded as Reused: the previous
image is pinned to the new commit and nothing is built. The tree id comes from the host’s API for
GitHub, GitLab and Gitea/Forgejo over https; otherwise the build Job’s clone step compares it and
stops before BuildKit starts.
GitOps with ArgoCD
GitImage writes two things ArgoCD would report as drift:
- the
gitimage.dev/revision.<container>andgitimage.dev/digest.<container>pod-template annotations, and withcontroller.pinTemplateImagethe bound containers’image(field managergitimage-controller); - the
caBundleof thegitimageMutatingWebhookConfiguration.
spec:
syncPolicy:
syncOptions: [ServerSideApply=true, RespectIgnoreDifferences=true]
ignoreDifferences:
- { group: apps, kind: Deployment, managedFieldsManagers: [gitimage-controller] }
- { group: apps, kind: StatefulSet, managedFieldsManagers: [gitimage-controller] }
- { group: apps, kind: DaemonSet, managedFieldsManagers: [gitimage-controller] }
# for the Application that installs the chart:
- group: admissionregistration.k8s.io
kind: MutatingWebhookConfiguration
name: gitimage
jqPathExpressions: ['.webhooks[].clientConfig.caBundle']
Sync order does not matter: pods created before their GitImage wait gated.
Isolation modes
buildPolicy.isolation.mode is required and has no default. At startup the controller runs a
check Job with exactly the build pod’s isolation settings: BuildKit lists its workers, then
builds a two-layer image (from its own busybox, nothing is pulled) whose second layer
hard-links a file of the first while it is mapped. If either fails, the controller exits with
code 3 and logs IsolationUnavailable; a failed hard link names the fix (gVisor:
snapshotter: native).
Nothing falls back to a weaker mode, and nothing builds.
| Mode | Values | Build pod | Verified in CI |
|---|---|---|---|
| Kata | runtimeClass, runtimeClassName: kata-qemu, privilegedInSandbox: true | rootful BuildKit, privileged inside the VM | kind + kata-deploy (KVM runners) |
| gVisor | runtimeClass, runtimeClassName: gvisor, snapshotter: native | rootless BuildKit, seccomp/AppArmor unconfined inside the sandbox | kind + runsc |
| Sysbox | runtimeClass, runtimeClassName: sysbox-runc, rootfulInSandbox: true | rootful BuildKit, unprivileged, hostUsers: false; Pod Security baseline | kind + Sysbox 0.7.1 (hack/e2e/sysbox.sh) |
| User namespaces | userns | rootless BuildKit, hostUsers: false | not on kind (see below) |
| Privileged | privileged + allowPrivileged: true | rootful BuildKit, privileged on the node | kind |
Kata and privilegedInSandbox. Kata’s shared filesystem drops the file capabilities that
rootless BuildKit’s newuidmap needs, so BuildKit runs rootful and privileged inside the
guest. That is only safe when the runtime keeps privileged in the guest: containerd’s
privileged_without_host_devices = true for the Kata handler (kata-deploy sets it). Check your
node’s containerd config before enabling it.
gVisor and snapshotter: native. BuildKit’s overlay layers live on the build pod’s state
volume, and gVisor treats volumes as shared with the outside world. In that mode it cannot copy
up a lower-layer file that a running process has memory-mapped: the copy-up fails with ENODEV.
Package managers hit this routinely, e.g. apt-get upgrading a library that apt-get itself
has loaded stops with unable to make backup link of '...libcrypto.so.3' before installing new version: No such device. The native snapshotter copies layers instead of stacking them, so
nothing is copied up; every operation works, but builds are slower and use more disk (a large
multi-stage build took about twice as long as under Kata). Things that do not help (tested with
runsc release-20261116.0): file-access-mounts=exclusive on the runsc handler avoids the
ENODEV but crashes the whole sandbox when the copying process has the file mapped itself;
per-pod gVisor mount hints turn the state emptyDir into a RAM-backed tmpfs; ephemeral PVCs
reach gVisor as the same kind of volume (filesystem mode) or not at all (block mode: no ext4);
fuse-overlayfs does not run in gVisor. For heavy builds prefer Kata. The isolation check
builds exactly this case, so gVisor without native stops the controller at startup rather
than failing the first build that upgrades a package.
Sysbox and rootfulInSandbox. Sysbox runs the pod in
a user namespace, gives root in the pod its capabilities there and virtualizes procfs/sysfs, so
rootful BuildKit works without privileged: true or unconfined seccomp/AppArmor (Sysbox applies
its own seccomp filter and ignores AppArmor profiles). rootfulInSandbox: true selects the
rootful image and sets hostUsers: false; it needs mode runtimeClass and excludes
privilegedInSandbox. Sysbox must not be combined with privileged containers. Requirements
(Sysbox docs): Sysbox 0.7+ (sysbox-deploy-k8s), Kubernetes user namespaces (on by default
from 1.33) and containerd 2.0.5+, so hostUsers: false reaches the runtime; GitImage does not
set the CRI-O-only io.kubernetes.cri-o.userns-mode annotation older setups used.
sysbox-deploy-k8s refuses Kubernetes versions it was not tested with (1.32-1.35 for Sysbox 0.7.1), documents Ubuntu nodes, and sets host sysctls.
Sysbox in kind. hack/e2e/sysbox.sh installs the Sysbox 0.7.1 package by hand into the
kind worker (Kubernetes 1.37, containerd 2.3; the installer would refuse 1.37 and change the
sysctls of the host kernel, which kind nodes share), remounts the node’s /sys
read-write and removes kind’s bind mounts over /sys/devices/virtual/dmi/id/product_*: with
either in place the kernel refuses the container’s sysfs mount (sysfs mount through procfd to sys caused: operation not permitted). CI always runs it (matrix include, independent of
E2E_ISOLATIONS); the nodes need user namespaces enabled.
Kata on Talos (virtiofs without xattrs). Kata shares filesystem pod volumes (emptyDir,
and PVCs in Filesystem mode) into the VM over virtiofs. When that share lacks
extended-attribute support (Talos’ Kata extension), BuildKit fails with
failed to create hash for ...: operation not supported. Use a raw block volume instead: Kata
attaches volumeMode: Block PVCs to the VM as a virtio-blk disk, so BuildKit gets a real
filesystem.
build:
stateVolume:
block:
storageClassName: ceph-block # any class that supports volumeMode: Block
size: 20Gi
fsType: ext4 # or ext3
Each build pod gets a fresh ephemeral PVC (deleted with the pod), or with
persistent the GitImage’s own PVC, formatted once. A format-state init container
(formatImage, default debian:bookworm-slim) runs mkfs, and the BuildKit container mounts
the device at its state dir before starting, so this needs privileged or
privilegedInSandbox. The isolation check runs with the same volume. medium: Memory (tmpfs)
also works but counts against the build’s memory limit.
userns in kind. runc cannot mount sysfs for a user-namespaced pod when the node itself is a container, so kind cannot test this mode; use it on VM or bare-metal nodes with Kubernetes 1.33+.
Hosts without user namespaces. gVisor’s gofer, rootless BuildKit and userns mode all
create user namespaces; with user.max_user_namespaces=0 (the Talos default) they fail
(fork/exec /proc/self/exe: no space left on device). Kata with privilegedInSandbox and
privileged mode do not need them.
Use the RuntimeClass’s scheduling.nodeSelector to keep Kata builds on nodes that have it.
Storage modes
storage.type | Images live in | Pods pull from |
|---|---|---|
registry | <url>/<namespace>/<gitimage>:<sha> in any OCI registry | the registry |
s3 | an S3 bucket: repos/<namespace>/<gitimage>/{blobs,manifests,tags} | localhost:25050 on each node |
Builds never see storage credentials: BuildKit pushes to the controller’s push endpoint
(Service gitimage-push:5000, TLS with the controller’s CA), which writes to the backend.
registry
pushSecret (dockerconfigjson, in the release namespace) is used by the controller only. With a
private registry set pullSecret: it is copied into every namespace with a GitImage as
gitimage-pull and added to gated pods. GC deletes manifests; blob GC is the registry’s own.
s3
Works with Ceph RGW, SeaweedFS, AWS S3, R2 and other S3 APIs (pathStyle: true for most
self-hosted ones). gitimage-node is a hostNetwork DaemonSet bound to 127.0.0.1:25050 only;
containerd allows plain HTTP to localhost with no node configuration (verified with containerd
2.3). Change node.port if 25050 is taken on your nodes.
Add a lifecycle rule that expires objects under uploads/ after a day: they are abandoned
multipart uploads of crashed pushes. Blobs over 5 GiB are not supported.
Garbage collection
Hourly on the leader: keep the 5 newest builds per GitImage, every digest a pod runs, the build cache and in-flight builds; delete unreferenced manifests, and in s3 mode blobs older than 1 h.
Helm values
| Value | Default | Meaning |
|---|---|---|
buildPolicy.isolation.mode | required | runtimeClass, userns or privileged |
buildPolicy.isolation.runtimeClassName | RuntimeClass for runtimeClass mode | |
buildPolicy.isolation.allowPrivileged | false | must be true for privileged |
buildPolicy.isolation.privilegedInSandbox | false | rootful BuildKit inside the sandbox (Kata) |
buildPolicy.isolation.rootfulInSandbox | false | rootful BuildKit, unprivileged, hostUsers: false (Sysbox); runtimeClass only, excludes privilegedInSandbox |
buildPolicy.egress.allowInternet | true | build pods may reach the internet |
buildPolicy.egress.denyClusterCIDRs | true | except network.clusterCIDRs |
buildPolicy.egress.extraRules | [] | NetworkPolicy egress rules added for build pods, each with to; see In-cluster registry mirrors |
network.clusterCIDRs | [] | pod and service CIDRs, IPv4 and IPv6 |
storage.type | registry | registry or s3 |
storage.registry.url, .pushSecret, .pullSecret, .insecure | registry mode | |
storage.s3.bucket, .endpoint, .region, .secret, .readOnlySecret, .pathStyle | s3 mode | |
build.image / build.rootfulImage | moby/buildkit:v0.34.2-rootless / moby/buildkit:v0.34.2 | BuildKit images |
build.activeDeadline | 1h | per build |
build.platforms | [linux/amd64] | more than one builds a multi-arch index |
build.cache | max | max, min or off |
build.labels, build.extraOpts, build.defaultResources | applied to every build | |
build.stateVolume.medium, .sizeLimit | "" | emptyDir for BuildKit’s state; Memory = tmpfs |
build.stateVolume.block | unset | {storageClassName, size, fsType: ext4|ext3, formatImage}: raw block ephemeral PVC per build, formatted and mounted in the pod (Kata on Talos) |
build.stateVolume.labels, .annotations | {} | extra metadata on the state PVCs (block or persistent), e.g. velero.io/exclude-from-backup: "true" |
build.stateVolume.persistent | unset | {size, storageClassName}: one PVC per GitImage kept between builds; see Persistent build cache |
build.buildkitdConfig | "" | appended to the build pods’ buildkitd.toml (registry mirrors, private registries) |
hooks.baseURL, hooks.ingress.* | push hooks | |
ui.enabled, ui.baseURL, ui.auth.*, ui.ingress.* | false | status UI on port 8082; ui.auth.mode is none or oidc |
controller.replicas, webhook.replicas | 2 | |
controller.extraEnv | [] | extra env for the controller |
controller.pinTemplateImage | false | also write the built image into workload templates, for scanners (Workloads) |
webhook.crossTenantNamespaces | [] | namespaces whose pods may use any namespace’s GitImage images (image scanners) |
node.port, node.healthPort | 25050, 25051 | gitimage-node host ports |
podMonitor.enabled | false | Prometheus Operator PodMonitor |
logFormat, logLevel | json, info,kube=warn | |
extraObjects | [] | extra manifests installed with the release |
Persistent build cache
By default every build starts BuildKit on an empty state volume and gets its cache from the
registry (build.cache): a step that misses the cache first downloads every layer below it,
and RUN --mount=type=cache (cargo, npm, ccache) always starts empty. With
build.stateVolume.persistent each GitImage gets a PVC, <name>-buildkit, that its builds
reuse:
build:
stateVolume:
persistent:
size: 20Gi
storageClassName: local-path # default class if unset
- Cache. Cache hits are local, nothing is downloaded, and cache mounts keep their contents.
BuildKit’s garbage collection keeps the state within 80% of
size. Keepbuild.cacheon: it fills a new PVC (first build, after a reset or a settings change). - One build at a time. The PVC is
ReadWriteOnceand BuildKit locks its state dir, so withonNewCommit: cancelthe next build starts once the cancelled build’s pod is gone (status:waiting for the cancelled build of <sha> to stop). - Placement. Builds of a GitImage run where its PVC can be attached: its zone, or its node
with
local-path. Network block storage adds a few seconds of attach time per build. - Reset.
kubectl delete pvc <name>-buildkit; the controller creates a new one for the next build. It also recreates the PVC itself when the isolation layout (rootful or rootless),size,storageClassNameorblock.fsTypechange, with aStateVolumeRecreatedEvent. - Block mode. With
blocktoo, the PVC isvolumeMode: Block, sized bypersistent, and formatted only when it holds no filesystem;e2fsck -prepairs it after a killed build. GiveblockonlyfsType/formatImagethen:block: { fsType: ext4 }. - Backups. The PVCs hold only a cache. Keep them out of backups with
build.stateVolume.labels(Velero:velero.io/exclude-from-backup: "true") or.annotations(k8up:k8up.io/backup: "false"); PVCs that already exist get them on their next build. A PVC restored anyway points at the old GitImage’s UID: the controller replaces it rather than reuse it. - Security. A commit can leave files in the state that later builds of the same GitImage use, as it already can through the registry cache; other GitImages never see it. Reset the PVC after building an untrusted commit.
Not combinable with medium or sizeLimit. The PVC is deleted with its GitImage.
In-cluster registry mirrors
Build pods may reach the internet but not network.clusterCIDRs. A mirror in
build.buildkitdConfig served by an ingress controller in the same cluster resolves to its
LoadBalancer IP, which the CNI translates to the controller’s pod IPs, so the build policy
drops the connection. BuildKit then waits on the mirror for about two minutes per base image
([internal] load metadata for docker.io/...) and falls back to the upstream registry, rate
limits included. Allow the ingress controller’s pods:
buildPolicy:
egress:
extraRules:
- to:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: traefik } }
podSelector: { matchLabels: { app.kubernetes.io/name: traefik } }
ports: [{ port: 8443, protocol: TCP }] # the pods' port, not the Service's
The rules are added to the build pods’ NetworkPolicy as written; each needs to, and unknown
fields fail the config. Build pods can then reach every host that ingress controller serves.
extraObjects
Anything else the install needs, as maps or templated strings (tpl is applied):
extraObjects:
- apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata: { name: gitimage, namespace: "{{ .Release.Namespace }}" }
spec: { bucketName: gitimage, storageClassName: ceph-bucket }
- |
apiVersion: v1
kind: ConfigMap
metadata:
name: notes
labels: {{- include "gitimage.labels" . | nindent 8 }}
data: { pushHost: {{ include "gitimage.pushHost" . }} }
The chart writes gitimage-config from these values; gitimage check-config --config <file>
validates a rendered config with the controller’s strict parser.
Security model
| Asset | Who can read it |
|---|---|
| Git credentials | the controller (get on the named Secret, in memory) and the clone init container of that GitImage’s builds |
| Build secrets | the BuildKit container of that GitImage’s builds, as tmpfs for one RUN |
| Storage credentials | the controller (and read-only ones in gitimage-node); never build pods |
| Push tokens | one per build Job, HS256, scoped to <namespace>/<gitimage>, valid only while that Job runs |
- Pod Security admission. Build pods are privileged in Kata (inside the VM) and privileged
modes, and use unconfined seccomp/AppArmor in rootless modes, so every namespace with a
GitImage needs
pod-security.kubernetes.io/enforce: privileged. With Sysbox (rootfulInSandbox: true) build pods are unprivileged, add no capabilities and set no unconfined profiles, sobaselineis enough;restrictedrejects them (noallowPrivilegeEscalation: false,capabilities.drop: [ALL]orRuntimeDefaultseccomp). The release namespace needsprivilegedin every mode (gitimage-node uses hostNetwork). With a stricter level the build fails within seconds withPodCreationFailedand the admission message. - RBAC. The controller gets
get/create/update/deleteon Secrets, neverlistorwatch. - Admission. The mutating webhook is called only for pods carrying a
gitimage.dev/image.*annotation (CELmatchConditions), so a webhook outage blocks only GitImage pods. A second entry denies any pod that references another namespace’s GitImage image. - Egress. Build pods get a NetworkPolicy: DNS, the push endpoint, the internet minus the
cluster CIDRs, and any
buildPolicy.egress.extraRules. - TLS. The controller runs its own CA (Secret
gitimage-ca), issues the webhook and push certificates, renews them 30 days before expiry and patches the webhookcaBundle. - Supply chain. Release images are signed with cosign (keyless) and carry an SBOM and provenance.
Status UI
A read-only web page served by the controller: every GitImage with its state, the live build log grouped by BuildKit step, the workloads that use an image and the pods waiting for one.
ui:
enabled: true
kubectl -n gitimage-system port-forward svc/gitimage-ui 8082
# open http://localhost:8082
The UI changes nothing in the cluster. To retry a build, it shows the kubectl annotate
command to run.
Sign-in
ui.auth.mode | Who sees what | Use it for |
|---|---|---|
none (default) | Every visitor sees what the controller’s ServiceAccount sees: all namespaces, all build logs. | kubectl port-forward, or behind a proxy that authenticates. Do not expose it otherwise. |
oidc | Each user sees what cluster RBAC allows them. | A shared URL. |
OIDC
The UI signs the user in (authorization code flow with PKCE) and sends their ID token to the Kubernetes API server as the bearer token of every request it makes for them. The API server authenticates the token and applies RBAC, the same model as Headlamp. The controller’s own permissions are never used for a signed-in user.
Requirements:
-
The API server trusts the issuer (
--oidc-*flags or a structuredAuthenticationConfiguration), and the UI’s client id is one of its accepted audiences. -
The client has the redirect URI
<ui.baseURL>/auth/callback. -
Users have RBAC. To see a GitImage in a namespace:
To see Needs It in the list listongitimages.gitimage.devin the namespace (or cluster-wide)Its page geton that GitImageWorkloads listondeployments,statefulsets,daemonsetsWaiting pods listonpodsBuild logs listandgetonpods,getonpods/logWhat a user may not list is shown as a notice on the page, not as an error.
ui:
enabled: true
baseURL: https://gitimage.example.com
auth:
mode: oidc
oidc:
issuer: https://auth.example.com # exactly as the API server has it
clientID: "312345678901234567" # quote numeric ids; with helm use --set-string
existingSecret: gitimage-ui-oidc # keys: clientSecret, optionally clientID
ingress:
enabled: true
host: gitimage.example.com
A session lasts as long as the ID token. When it expires the browser is sent through the issuer again, which is invisible while the issuer’s own session is alive. The session is an encrypted cookie, so both controller replicas accept it and nothing is stored server-side.
If the API server rejects the token, the UI says so. The usual cause is an audience mismatch: the API server accepts only the client ids listed in its configuration.
Limits
- Logs live one hour. They are read from the build pod, and build Jobs are removed one
hour after they finish. After that the page shows the failure message kept in
status.builds. - Build output is visible to everyone who may read pod logs in the namespace. A
Dockerfile that prints a build argument prints it there too. Use
build.secretsfor credentials; secret mounts never reach the log. - The page of a GitImage shows its push-hook URL to anyone who may
getthe GitImage, the same askubectl get gi -o yaml.
Developing the UI
The web app lives in ui/ (Svelte 5, Vite, Tailwind) and is embedded into the binary at build
time. Without a built ui/dist, the binary serves a placeholder page.
cd ui && npm ci
npm run dev # against mock data, no cluster needed
GITIMAGE_UI_PROXY=http://localhost:8082 npm run dev # against a port-forwarded controller
npm run check && npm run build # then cargo build embeds ui/dist
Troubleshooting
| Symptom | Look at |
|---|---|
| Controller in CrashLoopBackOff | kubectl -n gitimage-system logs deploy/gitimage-controller --previous: IsolationUnavailable names the failing mode; kubectl -n gitimage-system logs job/gitimage-isolation-check-<hash> |
Pod stays SchedulingGated | kubectl get gi <name>: no image yet; kubectl describe pod: GitImageNotFound or build progress Events |
Failed / ResolveFailed | wrong URL, ref, credentials or host key (host key ... not in known_hosts) |
Failed / PodCreationFailed | the namespace’s Pod Security level: label it pod-security.kubernetes.io/enforce=privileged (baseline with rootfulInSandbox) |
Failed / ReconcileError | the message is the API server’s answer to the controller (an invalid Job, a quota, missing RBAC); the controller log has the same error |
Failed / BuildFailed | kubectl logs job/<name>-<sha7> -c buildkit |
Failed / PushFailed | controller logs (storage errors), then the build log |
| Annotated pod is created without a gate or digest | kubectl get mutatingwebhookconfiguration gitimage -o yaml: caBundle set? webhook pods ready? |
Every build spends ~2 min in load metadata for docker.io/... | a registry mirror build pods cannot reach, often one behind an in-cluster ingress controller: In-cluster registry mirrors |
| Pull fails in s3 mode | kubectl -n gitimage-system get ds gitimage-node: NotReady means S3 is unreachable or port 25050 in use on 127.0.0.1 |
Logs are JSON by default; --set logFormat=text for humans, logLevel=info,gitimage=debug
for more.
Example: Talos homelab
Kata on nodes with /dev/kvm, Ceph RGW (Rook) for storage, no registry, ArgoCD.
buildPolicy:
isolation: { mode: runtimeClass, runtimeClassName: kata, privilegedInSandbox: true }
storage:
type: s3
s3:
bucket: gitimage
endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc
secret: gitimage # created by the ObjectBucketClaim below
pathStyle: true
network:
clusterCIDRs: [10.244.0.0/16, 10.96.0.0/12]
extraObjects:
- apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata: { name: gitimage, namespace: "{{ .Release.Namespace }}" }
spec: { bucketName: gitimage, storageClassName: ceph-bucket }
Before enabling privilegedInSandbox, confirm the Kata handler’s containerd config has
privileged_without_host_devices = true.
Architecture
One Rust binary (gitimage) with subcommands controller, webhook, node, clone,
crdgen, check-config, built on kube-rs, axum, gix and russh. No git or ssh binary, no
OpenSSL.
| Crate | Role |
|---|---|
gitimage-api | CRD types, annotation keys, the strict Config |
gitimage-git | ref resolution, gix fetch and checkout, russh ssh transport with known_hosts checks, host tree APIs |
gitimage-oci | OCI distribution routes (push endpoint and node server), S3 / registry / memory storage, push tokens |
gitimage-controller | reconciler state machine, Job builder, gate releaser, roller, hooks, GC, isolation check, TLS bootstrap, status UI API (ui/, with the web app from the top-level ui/ embedded by build.rs) |
gitimage-webhook | pure mutate(pod) -> patch, guard, axum handler |
gitimage-node | localhost-only read-only OCI server |
gitimage-tls | CA and certificates, hot-reloading TLS listeners |
The design documents (PRD and implementation design) record the decisions and trade-offs.
Testing
cargo test --workspace # unit, storage, router, snapshot tests
./hack/oci-conformance.sh # OCI distribution-spec conformance
./hack/check-chart-config.sh # chart renders configs the controller accepts
./hack/e2e/run.sh s3 gvisor # kind e2e: registry|s3 x privileged|gvisor|kata-qemu|sysbox
KEEP=1 ./hack/e2e/run.sh registry privileged # keep the cluster for debugging
E2E_PERSISTENT=1 ./hack/e2e/run.sh registry gvisor # with build.stateVolume.persistent
The e2e environment is a kind cluster with its own kubeconfig (hack/.kubeconfig) plus a
registry, SeaweedFS (S3) and Gitea as containers on the kind network. The suite drives Gitea
through its API and covers: commit to running digest, rollouts, tags, monorepo reuse, hooks,
build secrets, private repos over ssh with host-key checks, the cross-tenant guard, failed builds
and rebuilds (of failed and of built commits), GC, and IsolationUnavailable.
Releasing
Every push to main runs one pipeline: checks (fmt, clippy, tests, chart, cargo-deny), OCI
conformance, one build per architecture, the e2e matrix, and only if all of them pass, a
release:
- version
<major>.<minor>.<run number>(major.minor fromCargo.toml); - multi-arch image
ghcr.io/diverofdark/gitimage:<version>and:latest, cosign-signed; - Helm chart
oci://ghcr.io/diverofdark/charts/gitimage --version <version>; - a GitHub release with binaries and notes;
- this documentation, published to GitHub Pages.
Runners: the repository variable RUNS_ON names a self-hosted runner label for every job
(E2E_RUNS_ON overrides it for e2e only); unset means GitHub-hosted. Pull requests from forks
always use GitHub-hosted runners, and arm64 builds always use GitHub’s arm runner.
E2E_ISOLATIONS (JSON list) adds kata-qemu where runners have /dev/kvm (sysbox always
runs, via the matrix include);
DOCKERHUB_PREFIX (e.g. harbor.example.com/docker-hub/) makes self-hosted jobs pull every
Docker Hub image through a path-style proxy cache: job containers, kind’s node image, the e2e
helpers, containerd in kind (hosts.toml with override_path) and BuildKit (mirrors).
E2E_GITHUB_ISOLATIONS (JSON list) keeps some isolations on GitHub-hosted runners — gVisor,
rootless BuildKit and Sysbox need user namespaces, which Talos disables by default
(user.max_user_namespaces=0).