A symptom → cause → fix runbook for the traps that actually bite when deploying
and operating zaentrum. Each section is a single trap: the symptom (the log
line or error you see), the cause, and the fix (the commands).
The platform is operator-managed — one operator renders the whole
stack from a single Zaentrum custom resource. Most fixes below are day-2
operations against a running install; the reference demo details are in
self-hosting.md.
Symptom — the deploy:zaentrum-demo CI job fails at its pre-flight guard:
ns zaentrum-demo missing — run 'oc apply -f zaentrum-demo/bootstrap.yaml' as cluster-admin first
(or the sibling guard Zaentrum CRD missing — install the operator first).
Cause — the guard is a plain kubectl get ns zaentrum-demo / kubectl get crd zaentrums.zaentrum.io run with the group-level OC_TOKEN. If the namespace and
CRD do exist, the guard only fails for one of two reasons:
OC_TOKEN has expired. It is a deployer ServiceAccount token; if it was
minted short-lived it will silently lapse and every kubectl call under it
returns unauthorized, which the guard surfaces as "missing."
The deployer SA lacks the cluster-scoped reads. Both guards are
cluster-scoped gets that the namespace-admin RoleBinding does not grant.
They are covered by the resourceNames-scoped ClusterRolezaentrum-demo-ns-get in zaentrum-demo/bootstrap.yaml, which grants
exactly get namespaces/zaentrum-demo and
get customresourcedefinitions/zaentrums.zaentrum.io. If bootstrap was never
applied (or was applied without that block), the guard fails even with a valid
token.
Fix — mint a long-lived deployer token and update the group CI variable:
Terminal window
# Long-lived deployer token. The deployer SA lives in your deploy namespace.
If the grant is missing rather than the token, (re-)apply the bootstrap as
cluster-admin — this is a one-time cluster-scoped bootstrap, not something CI
can do:
Terminal window
ocapply-fzaentrum-demo/bootstrap.yaml
Caveat — false positive when testing an SA token locally.oc --token=""
(empty or unset) does not fail; oc silently falls back to your admin
kubeconfig, so the guard passes and the token looks fine. Always test the
actual token value explicitly, e.g.:
Symptom — after setting storage.kafkaPvc (see trap 6),
the operator's server-side apply on the Kafka Deployment fails:
Deployment.apps "kafka" is invalid: spec.template.spec.volumes[…]:
Forbidden: may not specify more than 1 volume type
Cause — the existing Kafka Deployment already carries an emptyDir volume
for the log dir. Server-side apply cannot cleanly transform that same volume into
a persistentVolumeClaim in place — the merge leaves both volume types on one
volume entry, which the API server rejects.
Fix — delete the Kafka Deployment once so the operator recreates it clean
from the chart with the PVC-backed volume:
Terminal window
ocdeletedeploykafka-nzaentrum-demo
The operator re-renders it on the next reconcile. This is safe for the config
switch itself; be aware it restarts the broker (see trap 6
for what survives).
3. Every transcode fails: Driver does not support the required nvenc API version
Symptom — every transcode job aborts. In the transcoder logs:
Driver does not support the required nvenc API version. Required: 13.1 Found: 13.0
or, on a mismatched older ffmpeg:
Unrecognized option 'spatial_aq'
Cause — the transcoder's bundled ffmpeg NVENC build does not match the GPU
node's Nvidia driver. The image ships a prebuilt NVENC-enabled ffmpeg static
build, pinned in transcoder/Dockerfile via FFMPEG_BUILD_URL. Tracking
master-latest drifts to bleeding-edge NVENC SDKs: a mid-2026 master build began
requiring nvenc API 13.1 (driver ≥ 610) and aborted every encode on nodes running
driver 580 / nvenc 13.0 (RTX 3090). A build that is too new for the driver
fails with the "required nvenc API version" error; a mismatched build can also
reject encoder flags (spatial_aq).
Fix — pin FFMPEG_BUILD_URL to a release branch, never master-latest.
The default in transcoder/Dockerfile is the ffmpeg 7.1 release branch:
The n7.1 branch links an NVENC SDK compatible with driver ≥ ~550 and takes only
bugfix backports, so it never bumps the driver floor out from under the cluster.
Override for a mirror or a different pin with a build arg:
Symptom — one or more pods sit in ContainerCreating indefinitely. oc describe pod shows only a Scheduled event and nothing after it — no
Pulling, no FailedMount, no progress. The node's kubelet is effectively hung.
Cause — the same NFS export is mounted twice inside one pod. Mounting one
NFS export at two paths in a single pod hangs the kubelet's mount operation, which
stalls the pod at ContainerCreating with no further events.
Fix — mount the export once at a single parent path and reference
sub-paths beneath it, rather than mounting the same export at multiple mount
points. In this platform the media library is a single parent-mount at
/var/lib/katalog; consumers use sub-paths of that one mount instead of adding a
second volume for the same export. Fix the offending workload's volume/mount spec
(or the CR/chart values that generate it) so the export appears exactly once, then
delete the stuck pod so it reschedules:
Symptom — login fails, or services log OIDC issuer-validation errors when
talking to the identity provider from inside the cluster (issuer URL unreachable,
or the discovered issuer does not match).
Cause — split-horizon DNS. The public HTTPS issuer host (e.g. the
hostname on the CR) resolves to a public/edge address that in-cluster pods
cannot route to, or that does not point back at the cluster's router. Token
issuer validation requires that the same public host resolve to the OKD router
from inside the cluster.
Fix — set network.issuerHostAliasIP on the Zaentrum CR. The operator
injects a matching hostAliases entry into the workloads so the public issuer
host resolves to the router IP from inside the cluster, satisfying issuer
validation without changing the public URL. Example CR fragment:
spec:
network:
issuerHostAliasIP:<router-ip>
Apply the CR change (via CI for the demo) and let the operator reconcile; the
affected pods pick up the hostAliases on their next roll.
6. A fresh scan's first item never advances / topics gone after a Kafka restart
Symptom — after a Kafka pod restart (including the deliberate delete in
trap 2),
a fresh scan's first item never advances through the pipeline, or expected topics
are missing.
Cause — the media pipeline is pure event-driven: stage handoffs are Kafka
events keyed by item_id
(scan → stube.catalog.item.discovered → enrich → …enriched → analyze → …analyzed → transcode → …transcoded → package). The bundled single-node KRaft
broker defaults to an emptyDir log dir, so a pod restart/reschedule wipes
its topics and consumer offsets — in-flight handoffs for the affected keys are
lost.
Fix — back the broker's log dir with a PVC so topics survive a restart. Set
storage.kafkaPvc (and pin the pod to a node with storage.kafkaNode,
since the PVC is node-local) on the Zaentrum CR:
spec:
storage:
kafkaPvc:<pvc-name># e.g. kafka-data — empty => ephemeral emptyDir
kafkaNode:<node-hostname># pin the broker to the node holding the PV
For the demo, the node-local PV and its pre-created host dir are established by
zaentrum-demo/bootstrap.yaml (a pv-...-kafka PersistentVolume, a host dir on
the node, and the <node> the broker is pinned to).
Self-heals otherwise. Even on emptyDir, the platform recovers on its own:
topics auto-create on first produce, and katalog-managerretries
transient produce errors. So a lost topic reappears and the pipeline resumes —
kafkaPvc just makes it durable instead of relying on recovery.
Symptom — signing in as admin (realm zaentrum) on a fresh install
immediately asks for a new password (UPDATE_PASSWORD required action) instead
of signing you in.
Cause — this is a fresh realm import, not an error. On first import the
realm seeds admin with a temporary, one-time password and the UPDATE_PASSWORD
required action. Existing realms are not re-imported, so this only happens on
a genuinely fresh identity store.
Fix — sign in with the one-time password, choose your own, then sign in
normally. Unless secrets.external is set, the operator generated the one-time
password for this install and keeps it in Secret zaentrum-keycloak-admin, key
realm-admin-password (first run):
The CR's SecretsGenerated condition names the same Secret. On an install made
from an earlier chart it is False, reason PublishedDefaults: its Secrets still
hold the values that chart shipped to every install, and each has to change where
it is used first — see
the platform's Secrets
in the operator's README. On the demo the Secret is CI-created
(DEMO_KC_ADMIN_PW / DEMO_REALM_ADMIN_PW CI variables). Nothing needs to be
"repaired" — it is expected first-login behavior.
8. Sign-in fails over plain http: "Cookie not found"
Symptom — the platform's pages load, but signing in does not: Keycloak answers the
login form with
Cookie not found. Please make sure cookies are enabled in your browser.
or the form keeps coming back. It happens from a phone, a TV or another computer — over
http://<lan-ip>, http://<name>.local or any other plain-http name — and never on the
machine that reaches the platform as http://zaentrum.localhost. The Android phone and TV
apps fail earlier: they refuse to connect over plain http at all.
Cause — Keycloak sets its login session's cookies (AUTH_SESSION_ID, KC_RESTART)
Secure; SameSite=None, over plain http too. A browser stores and sends a Secure cookie
only over https, or to a localhost name, which counts as a secure context. Anywhere else
the cookie never comes back with the form, and Keycloak has lost the login session.
Fix — serve the platform over https, with a certificate the devices trust (the Android
apps accept public certificate authorities only): terminate TLS in front of the ingress, set
identity.issuerScheme: https, and resolve the issuer in-cluster with
network.issuerHostAliasIP (trap 5). See
self-hosting.md. The appliance serves plain
http only, so sign in to it on the machine it runs on.
Symptom — https://<hostname>/auth/admin/, or the realm's console at
/auth/admin/zaentrum/console/, answers 404 instead of Keycloak's admin console, while
signing in to the platform works.
Cause — the admin console and the admin API are not published on the public host by
default. The Ingress and the Routes send Keycloak only what people sign in through,
/auth/realms and /auth/resources; every other path falls to the portal, which has no
such page.
Fix — reach the console through a port-forward, on local port 8080, as the master
realm's bootstrap admin (Secret zaentrum-keycloak-admin, keys username and password),
then switch to the realm zaentrum:
On the appliance the port-forward runs inside the container, which needs
-p 127.0.0.1:8080:8080 when it starts — see
the admin console. Or publish the console with
identity.exposeAdminConsole: true on the CR, knowing that it then answers anyone who can
reach the public host.