Upgrades & Maintenance

Use kip upgrade for routine platform updates. Review its scope and disruption warning below before running it. For initial setup, see Installation.

kip upgrade

Upgrades the cluster to the versions pinned in this kip build. Before making changes, it checks the cluster's data for compatibility and asks before applying repairs; see Checking before an upgrade. Then three things happen, in order:

1. Kipper CRDs are updated (so newer console features that need new CRD fields work).
2. Console and console-api are restarted to pull the latest images. console-api also gets the pod security settings this kip build ships (non-root, no privilege escalation, dropped capabilities, a read-only root filesystem), so a cluster installed a while ago ends up with the same posture as a fresh install. Anything you added to the Deployment by hand, such as an extra volume, is kept. Each component is then waited for: kip only moves on once the new pods are actually running, and stops with the reason if one of them cannot start. That matters because a rollout keeps the previous pod serving while the new one fails, so without the wait a broken upgrade looks exactly like a working one.
3. Cluster system components (Traefik, Longhorn, KEDA, Loki, Prometheus, Grafana, Velero, Zot, security middleware) are reconciled. Each chart is re-applied at the version pinned in kip, and helm-controller upgrades it in place. Longhorn is the exception: a plain upgrade re-applies the Longhorn release the cluster already runs, with the settings this kip pins, and --longhorn moves it to a newer release. See Longhorn upgrades. The Zot step restarts k3s only when the server's registry CA certificate or registries.yaml changes, or when an earlier attempt leaves a pending marker. The marker ensures that a retry loads any changes left by an interrupted run. It is cleared after any required restart and node readiness check succeed.

On a cluster that uses a *.kipper.run name, the upgrade also records that name and the address the cluster registers with on the ClusterIdentity, which is where the console reads them from when it renews the registration. It keeps the address already recorded rather than replacing it, and tells you when the cluster answers somewhere else: a server that has genuinely moved needs a fresh registration, because the gateway ties a name to one address. A cluster that has never registered is left alone and told so, since only you know whether it should hold a kipper.run name.

On a single-node cluster whose pod network still runs VXLAN, a full upgrade first moves it to WireGuard, which kip node add needs. This is a one-time change. kip saves the k3s configuration, restarts k3s on the new backend, and waits up to five minutes each for the network and for the workloads that were ready before. If either check fails, it attempts to restore the previous configuration and restart k3s again. The restart leaves running containers alone; the Kubernetes API, kip commands, console actions and any app that calls the API are unavailable meanwhile (usually 10 to 30 seconds, longer on small or busy servers), and the whole change takes about a minute. If a check fails, kip waits up to five minutes per step and then attempts to restore the previous configuration. Review the migration prompt before using --yes to approve it in unattended upgrades. --skip-system skips migration but still recovers interrupted network changes. A server whose kernel cannot load the wireguard module keeps VXLAN with a warning, and the rest of the upgrade runs.

Your apps and services are not directly touched, but step 3 can briefly disrupt running workloads if a chart upgrade rolls pods. For that reason kip prompts before running step 3 and refuses to proceed in non-interactive contexts without --yes.

Clusters installed before operator login existed

A cluster installed before Kipper configured the API server has no authenticator, so it rejects every login token while still accepting the admin certificate. kip cluster ca status reports it as an authentication config that names no issuer. The arguments that fix it live on the server rather than in the cluster, so only an upgrade, which reaches the host over SSH, can add them.

Every upgrade checks, and repairs a cluster that needs it:

  ...  The API server is missing the arguments this kip installs. Adding them and restarting k3s once; workloads keep running through it.
  ✔  API server arguments, and k3s restarted on them
  ...  Operator login against dex.shop.kipper.run
  ✔  Operator login configured. Run 'kip auth login', then 'kip auth kubeconfig'

k3s restarts once, which interrupts the control plane for a few seconds. containerd and your pods run through it, so apps keep serving. kip then waits up to two minutes for the API server to answer.

One component can notice: kube-state-metrics rebuilds its view of every object when the API server returns, and on a cluster still carrying the old 64 Mi limit that burst can OOM-kill it for a few minutes before it settles. The current limit is applied by the system-component step of a full kip upgrade, which runs after this one, so a cluster being repaired for the first time may see that blip once. --skip-system skips the resize entirely, and the next full upgrade applies it.

If the API server does not come back from a restart that changed the configuration, kip restores the file it replaced and restarts k3s on that. A copy of the previous file stays at /etc/rancher/k3s/config.yaml.kipper-bak either way.

A restart that changed no configuration is different, and kip restores nothing there. Later upgrades restart k3s to load a new audit policy, and the backup on disk may be months old and predate edits you have made since, so putting it back to recover from a failed restart would undo your work rather than kip's. The error names the file and leaves the decision with you.

Read the recovery result before retrying. It distinguishes a successful rollback, a rollback whose restart also failed, and an unreachable host whose state needs inspection. The previous configuration remains at /etc/rancher/k3s/config.yaml.kipper-bak.

For custom or partially configured kube-apiserver-arg blocks, Kipper prints the required additions and stops for manual review. If Dex is unreachable, it reports that operator authentication could not be configured and preserves certificate access.

Audit logging arrives with the same block, writing to /var/lib/rancher/k3s/server/logs/audit.log under the policy fresh installs use: metadata only, never request or response bodies, capped at 100 MB per file with 10 kept for 30 days.

A cluster already carrying these arguments keeps them, and the arguments themselves are not rewritten. It can still restart: an upgrade that changes the audit policy loads it by restarting k3s, and says so before it does.

kip upgrade --check            # report what the upgrade would fix, change nothing
kip upgrade                    # default, prompts before system components
kip upgrade --skip-system      # only steps 1 and 2 (Kipper console layer)
kip upgrade --k3s              # also upgrade k3s to the release pinned by this kip version
kip upgrade --longhorn         # also move Longhorn to the release pinned by this kip version
kip upgrade --yes              # all three steps, no component prompt (for automation)
kip upgrade --yes --repair-autoscaling  # also apply autoscaling fixes without asking

Flags

Flag	Default	Description
--skip-system	false	Skip the cluster components (Traefik, Longhorn, KEDA, Velero, Zot, monitoring). CRDs, console, and the cluster's own trust material are still updated, and Traefik is restarted when its copy of the hop certificate changes. Use this in production to avoid touching component versions.
--yes	false	Confirm cluster-component upgrades and removal of app routes that collide with installed platform routes. Required for these steps in non-interactive contexts (CI, scripts). Autoscaling fixes are confirmed separately, so a run without a terminal also needs --repair-autoscaling when there are fixes to apply
--k3s	false	Also upgrade k3s to the release pinned by this kip version, before the CRDs and console: the server first, then each ready worker. Stops before upgrading k3s when the server is more than one Kubernetes minor behind the pin, a node runs a newer release, or a worker would cross the limit of three minor versions behind. With --check, reports the planned upgrade. See k3s upgrades
--longhorn	false	Also upgrade Longhorn to the release pinned by this kip version, one minor at a time. Needs external backup storage. Before each step it checks every volume, takes a Velero backup and saves Longhorn's objects, then waits for every volume to settle. Running it again resumes a step that did not finish. Cannot be combined with --skip-system. See Longhorn upgrades
--check	false	Run the upgrade's data checks and report what it would fix, without changing anything. Needs no SSH key. Exits non-zero when something needs a decision. See Checking before an upgrade
--repair-autoscaling	false	Apply the autoscaling fixes the upgrade lists without asking. Required in non-interactive contexts when there are fixes to apply
--skip-autoscaling-check	false	Upgrade although some apps' autoscaling settings need a decision, leaving those apps as they are
--image-tag	none	Test a commit's console, instance proxy and data mover images using its full 40-character SHA. Requires --skip-system to prevent the system component upgrade from restoring the released authz image. Apps using the instance proxy roll out when its image changes. Omit this flag on the next kip upgrade to restore released images and roll out those apps again. See Testing a branch on a test cluster
--seed-credential-grants	false	Grant each shared git credential the projects whose apps already reference it, without asking. Only acts on a cluster that predates credential allow-lists
--ssh-key	inherited from ~/.kip/config.yaml	SSH private key for connecting to the cluster host. If unset, falls back to KIP_SSH_KEY env, then the saved cluster.ssh_key, then your ssh-agent. Needed for upgrades, including --skip-system, to reconcile the cluster's trust material. --check needs no SSH key

Checking before an upgrade

Upgrade kip first, then run the check before every upgrade:

kip upgrade --check

The check lists findings and proposed fixes without changing the cluster. It reads apps, Deployments and Ingresses through the Kubernetes API, so it needs no SSH key. It exits non-zero if a check fails or an app needs a decision; otherwise it exits zero.

The check covers app autoscaling settings. Older apps can have invalid policies or stored counts outside their bounds:

  Checking this cluster's data against the rules of kip v0.24.0...

  !   The new rules refuse these apps' autoscaling settings. Once the new console-api
      is running, kip upgrade changes them as shown, and nothing running changes:
      - shop-prod/api: autoscaling is on and it stores replicas 1, outside its bounds of 2 to 5, so the stored count becomes the running count when the fix is written.
          replicas: 1 → 3 (the running count when the fix is written)
  ✗   These apps need a decision, and kip upgrade does not change them:
      - shop-prod/worker: autoscaling is off, but its Deployment is set to 4 while it stores 3. Any reconcile restores 3.
          Set the count you want with 'kip app scale worker --project shop --environment prod --replicas N', with N from 2 to 5.

Route names

kip upgrade --check lists route name conflicts and explains which routes need attention:

• A route conflicts with an installed Kipper component on the same port. For example, app system-console-api in namespace kipper on port 8080 conflicts with the console API. The upgrade removes conflicting app routes managed by Kipper. Use a different app name to keep the app reachable. The check exits with a non-zero status, and kip upgrade requires confirmation. Use --yes to confirm without a prompt. Manually remove conflicting routes that Kipper does not manage.
• A route uses a name reserved for a Kipper component that is absent or uses a different port. The existing route stays available, but its request figures are unavailable. kip ai refuses to install a component while its route name is in use by another namespace, regardless of port. Rename or remove the conflicting route before installing it.
• Apps or services have conflicting route names. The upgrade keeps their existing routes. If their ports match, requests can reach the wrong app or service. Give each a distinct name. In the same namespace, a removed route cannot be recreated while the conflicting app or service remains.

The last two findings are advisory. After renaming conflicting apps or services, run kip upgrade to recheck the names so new request figures can appear. Figures recorded while names conflicted cannot be separated.

What the upgrade does with the findings

kip upgrade checks autoscaling settings before its first change and prints the proposed fixes. If there are autoscaling repairs to apply, it asks:

  Apply these fixes? [y/N]

Answering no stops the upgrade with nothing changed. --repair-autoscaling answers yes without asking, for automation. A run without a terminal that has fixes to apply stops unless that flag is given. An app that needs a decision stops the upgrade too, before anything changes, unless you pass --skip-autoscaling-check, which upgrades and leaves those apps as they are.

The fixes are written late in the upgrade, once the new console-api is running and no old console-api pod remains. Versions predating stops would restart an app that the repair marks as stopped. If kip cannot confirm that the new console-api is serving alone, it writes no fix, says why, and finishes the rest of the upgrade; running kip upgrade again applies them. Each app is read again before its fix is written. If the proposed fix no longer matches what you approved, the app is left alone and reported for another kip upgrade --check. For a stored-count repair, consent covers the Deployment's latest desired count as long as it remains within bounds. A failed write names the app, and the next run picks up from there.

What the repair changes

Repairs apply to apps with valid bounds. Each fix is a single App update, shown with its values before and after. Here, the Deployment count means its desired replicas, not the number of ready pods:

The app	The fix
Stopped, with a stored count outside its bounds	Autoscaling on: the count moves into the bounds (a start uses the minimum anyway). Autoscaling off: the bounds widen to include the count it starts at
Deployment count and stored count both 0, with autoscaling off	A stop is recorded with the reason "scaled to 0 before Kipper version" and the count is set to the minimum. The app stays at 0, its route serves the stopped page, and listings show it as Stopped. kip app start starts it at the minimum
Autoscaling on, Deployment count within bounds, stored count outside them	The stored count becomes the Deployment count at write time
Autoscaling off, Deployment count matching the stored count, both outside bounds	The bounds widen to include the count, because moving the count into them would scale the app

These repairs preserve the Deployment's desired count and leave autoscalers in place. The new controller can separately start autoscaled apps at zero, as explained below.

Apps that need a decision

The check names the command to run for each of these, and the upgrade changes none of them:

• The app's Deployment cannot be read, or is missing while the app is not stopped.
• The autoscaling block is invalid: a minimum above the maximum, a minimum or maximum below 1, a minimum above 1 without a maximum, autoscaling on without a maximum or without a target, or a negative target while autoscaling is on.
• Autoscaling is on, but the Deployment's nonzero desired count lies outside the bounds. Check whether a stale autoscaler or another writer is changing it.
• Autoscaling is off, but the Deployment's desired count differs from the stored one.
• No fix could satisfy the rules, for example a stopped app with autoscaling off and a stored count of 0.

Autoscaled apps at 0 pods start during the upgrade

An autoscaled app whose Deployment was set to 0 without a stop starts during the upgrade. The new console-api starts it at its minimum as soon as it runs, because the autoscaler cannot scale up from zero. The check and the prompt list these apps separately. To keep one at 0, stop it before upgrading:

kip app stop web --project shop --environment test

Stops exist since Kipper v0.23.0. On an older cluster the upgrade starts the app, so run kip app stop on it once the upgrade finishes.

Between a release and your upgrade

console-api runs a :latest image that is pulled whenever its pod starts. A console-api pod that restarts for any other reason after a release, such as a node reboot, runs the new release before kip upgrade has repaired anything or updated the CRDs. In that window the new controller reports invalid settings and starts autoscaled apps at 0. Until the CRDs are updated, the old schema drops the OOM marker, so memory tracking can filter out a stored memory increase intended for OOM recovery. Upgrade kip as soon as a release is out and run kip upgrade --check, so the findings are in front of you before that can happen.

Upgrade scope

Check this table before upgrading to see which components are included and which need a separate procedure.

Component	kip upgrade	How it moves otherwise
Kipper CRD schemas	Yes, or the upgrade stops	—
Cluster CA and API server trust anchor	Yes, over SSH, even with --skip-system	—
Recorded gateway identity (*.kipper.run)	Yes	—
console-api image and pod hardening	Yes	—
console image	Yes	—
kipper-authz image	Yes	—
console-api RBAC	Yes	—
Project operator roles (viewer, deployer, owner)	Yes	—
Shared git credential allow-lists, where never set	Once, on consent (prompt, or --seed-credential-grants), from the apps that reference them	kip credentials allow
App autoscaling settings the new rules refuse	On consent (prompt, or --repair-autoscaling), after the console-api rollout	kip app autoscale, kip app scale
Traefik, KEDA, Velero, Zot	Yes	—
Longhorn settings, at the installed release	Yes	—
Longhorn release	With --longhorn	See Longhorn upgrades
Loki, Prometheus, Grafana	Yes, when enabled	—
Security-header middleware, build isolation	Yes	—
cert-manager DNS resolvers	Yes	—
cert-manager version	No	Re-run kip install
Dex (version and configuration)	No	See the warning below
Console deployment manifest	No	Image moves on upgrade; the manifest does not
Initial admin ClusterRoleBinding	No, deliberately	Managed as you add and remove admins
k3s	No	kip upgrade --k3s
API server arguments and operator login	Yes, over SSH, even with --skip-system	—
Host firewall, service hardening	No	kip cluster harden
Kernel sysctls	No	Fresh install only
Running apps	No rollout for these platform changes	New readiness and shutdown settings apply on the next pod rollout

The following components need additional explanation.

Apps adopt new readiness and shutdown settings on their next pod rollout. These settings alone do not restart apps during an upgrade. Run kip app restart <app> to apply them immediately. If a check is selected, new pods must pass it before receiving traffic. Pods created before the upgrade lack the shutdown delay; replacement pods use it on supported clusters.

Downgrading to a controller that removes these settings can trigger app rollouts. If the cluster rejects the server-side dry run used to identify rollout changes, Kipper preserves the existing platform settings and records an AdoptionUnavailable warning event on the App.

The CRD schemas move unless this kip is older than the cluster. Two things stop that, and both refuse before anything is written, so nothing is ever partially applied:

• The cluster has objects stored under an API version this kip does not declare. Applying would strand them.
• The schemas were last written by a newer kip. Every upgrade records which version wrote them, so an older CLI can tell it is about to replace a newer schema with an older one and prune whatever the newer one added. A cluster last upgraded before this recording existed carries no such marker, and is allowed through on the first check alone.

Either way the remedy is the same: upgrade kip, then run it again. A kip built from source reports a version that cannot be ordered against a release, so the second check announces that it was skipped rather than passing quietly.

The cluster CA and the trust anchor the API server verifies logins against are reconciled over SSH on every upgrade, including with --skip-system, because they are this cluster's own identity rather than a component version. With --skip-system it is the only part that reaches the host, which is why --ssh-key matters even for an upgrade you expected to be API-only. It is announced before it runs.

Dex. Its configuration lives in a ConfigMap that kip install renders from install-time values, and the console writes users into that same ConfigMap. So re-running kip install on a cluster removes every user account created through the console since. There is no supported way to move Dex on an existing cluster today, and re-running the installer is not a workaround for it.

The initial admin binding is left alone on purpose. Its subject list changes as admins are added and removed, so re-applying the install-time copy would reset the cluster to a single admin and revoke everyone else.

Re-running kip install is not a general upgrade path

It is idempotent for some things and destructive for others. It replaces the Velero credentials Secret in place, which is why it is the documented way to rotate backup keys, and it moves k3s and cert-manager onto the versions this kip pins. It also re-renders the Dex ConfigMap, losing console-created users, and re-runs the rest of the install. Use it for the specific tasks named in these docs, not as a way to catch up a cluster.

k3s deserves its own note: kip upgrade --k3s moves a cluster onto the k3s release this kip version pins without re-running the install. See k3s upgrades. Re-running kip install also moves the server onto that release, and it restarts k3s on every run so that it loads the configuration kip wrote. For a server more than one Kubernetes minor behind, the install keeps the installed version and warns that the intermediate minor upgrades are needed first.

To check component versions after an upgrade, run kip platform versions. It shows the kip version recorded on the App CRD, the console-api build recorded at startup, each node's Kubernetes version, and the pod template images of every Deployment, StatefulSet and DaemonSet in Kipper's system namespaces.

k3s upgrades

kip upgrade --k3s upgrades the cluster to the k3s release pinned by this kip version, then continues with the rest of the upgrade. The k3s step preserves the other settings created by kip install, including console users, the admin password and the k3s configuration.

kip upgrade --check --k3s   # report the planned move
kip upgrade --k3s           # move the server, then each ready worker

After recovering any interrupted network change and before upgrading k3s, kip checks the server's version and the reported versions of recorded workers. It stops when the server is more than one Kubernetes minor behind the pin, because Kubernetes upgrades the control plane one minor at a time, and when the server or a worker runs a newer release, because kip prevents k3s downgrades. It also stops while a worker is still joining. Workers follow the skew rules below.

The server moves first. kip runs the k3s install script for the pinned release and restarts k3s, then waits up to five minutes for the server's node to report Ready on that release. Restarting k3s pauses the Kubernetes API, usually for under a minute, while running pods keep serving. A server whose node already reports the pinned release keeps running, and one whose binary was replaced by an interrupted attempt is restarted. The workers then follow as described below, and a failed step names the command to run again.

Before applying the CRDs and console, kip waits up to two minutes for the API to accept kip's credentials again. An API server that authenticates operators through Dex can reject their tokens for a few seconds after a restart while it reconnects to Dex and initializes authentication.

On a cluster with workers, kip upgrade --k3s and rerunning kip install check every recorded worker's reported k3s version before upgrading k3s. Each command stops if a worker runs a newer version than the server will run. Use a kip release that installs at least that worker's version.

It also stops if the server upgrade would leave a worker more than three Kubernetes minor versions behind when it was previously within that limit. Three minor versions is the limit in the Kubernetes version skew policy for kubelets at version 1.25 or later. Bring such a worker to the server's current version first with kip node add <host>. Workers already beyond that limit are handled by the worker upgrade that follows.

Once the server is on its new version, kip upgrades workers recorded as ready to the server's k3s version, one at a time in name order. Before upgrading each worker, it checks that the host is the recorded machine and that its installed k3s matches either the version its Node reports or the target version from an earlier attempt. It then reruns the k3s install script with the worker's saved join settings. k3s-agent restarts while existing pods keep running, without a drain. kip waits up to five minutes for each worker to report Ready on the target version, including workers already on that version, and stops at the first upgrade failure or readiness timeout. Running the same command again resumes the upgrades and skips workers already reporting Ready on the server's version. Workers recorded as joining, being removed or being wiped are left unchanged. kip holds the same lock as kip node add and kip node remove while upgrading workers, so it stops if another command holds that lock.

By default on Windows, and with --via-server, the install runs as a job on the server. The machine running the CLI upgrades the workers after the job finishes. kip install --local upgrades the server only.

Longhorn upgrades

Longhorn holds the data of every service and app volume on a Longhorn cluster, so kip moves it only when you ask, with --longhorn. A plain kip upgrade re-applies the release the cluster already runs.

kip upgrade --check            # lists the planned Longhorn steps
kip upgrade --longhorn         # runs them

Longhorn upgrades one minor version at a time and cannot be downgraded. kip moves it to the latest patch of each minor in turn: from 1.7.3, the release earlier kip versions installed, that is 1.8.2, 1.9.2, 1.10.2 and then 1.11.3. Longhorn 1.11 needs Kubernetes 1.34 or later on every node, so run kip upgrade --k3s first on an older cluster, or pass both flags.

--longhorn needs backups on external storage. Backups in in-cluster MinIO live on the Longhorn storage the upgrade changes, so kip refuses to start on such a cluster, and on one with backups turned off. kip also resolves the backup endpoint and refuses one that leads back into the cluster: a node's address, a Service's address or endpoint, an address in the pod or service range, or a loopback or link-local address. kip resolves the endpoint's name from your machine, so a cluster DNS override pointing the same name back into the cluster is outside this check's scope. Configure external storage first by running kip install again with --backup-storage-bucket and its related flags. Read Re-running install before you do, because a re-run also resets users created in the console.

Keep these off the cluster before the first step, because recovery from a failed step may require restoring onto a fresh cluster:

• the backup repository key, from kip backup repository-key export --to <file>.
• the SSE-C key, if backups are encrypted with one, from kip backup sse-key export --to <file>.
• the credentials of the backup bucket.
• a restore you have verified, with kip backup drill <namespace> for each namespace with data. kip backup drill does not run on the nano profile, so on nano restore a backup onto a separate recovery cluster instead.

The confirmation lists all four requirements. --yes skips the prompt, so scripts must ensure all four are met.

Each step then runs the same way:

1. kip checks that Longhorn and every volume have settled: the manager, instance managers and CSI components have finished rolling out, the manager reports the current release, no volume is faulted or stuck attaching, and no replica is rebuilding. A node marked out of service, as a fenced node is, runs no instance manager, so the step waits until you bring the node back with kip node add or remove it with kip node remove --force. It also checks that every node meets the next release's Kubernetes floor, that every engine in use meets its minimum engine version, and, before the step to 1.10, that no Longhorn CRD still stores v1beta1 objects. Longhorn 1.10 removes v1beta1, so that one requires the migration described in the Longhorn v1.10.0 release notes. During a step, Longhorn starts another instance manager on every node. It requests the node's instanceManagerCPURequest when set to a positive value, or the guaranteed-instance-manager-cpu share of the node's allocatable CPU (12% by default). The old manager stays until no volume uses it. For an attached volume, this means waiting until it detaches, so a node can run two managers after the first step. kip refuses the step if a node has too little unrequested CPU for the new manager. Otherwise, the node would refuse the manager and the step would time out with volumes still on the old engine. The error suggests which services' CPU requests to lower when possible, or recommends adding CPU. It also explains when an older instance manager still reserves CPU on the node. A resumed step skips this check. If it times out waiting for an instance manager, kip checks CPU again and names nodes with a shortage, with the same suggestion. This diagnostic requires readable cluster data and skips nodes where the new manager's pod has already been admitted.
2. kip rechecks the backup location and verifies that a regular container in a running pod mounts every app volume in the backup scope, because file backups require these mounts. A volume used only by a stopped app blocks the step, and so does the volume of a service kip node fence stopped. Start the app with kip app start, start the service with kip service start, or remove the volume. kip then takes a Velero backup in the scope of kip backup create, kept for 30 days, and waits for it to complete. A backup that fails stops the step. kip then checks the cluster again.
3. kip saves Longhorn's volumes, replicas, engines and settings as YAML under ~/.kip/longhorn-upgrades/<cluster>/, readable only by you.
4. kip records the step on the cluster and applies the next release's manifest.
5. kip waits for every component to run the new release, lets Longhorn move each volume onto the new engine, and waits for every volume to settle again. From Longhorn 1.10 on, kip then removes the engine images of older releases that nothing uses any more. Longhorn deploys each on every node until it is deleted, and 1.10 is the first release that refuses to delete one in use.

Only one run changes Longhorn at a time. While applying Longhorn, every kip upgrade holds the same lock as the kip node commands. If another command holds the lock, the upgrade stops and names that command. kip upgrade --longhorn holds the lock throughout the Longhorn upgrade, which can take hours. The kip node commands refuse to run until the lock is released. To remove or fence a worker that fails during a step, stop the upgrade with Ctrl-C first. The lock is released about a minute later, or up to six minutes later if a command started by the upgrade is still running on the server. Stopping the upgrade is safe because the next kip upgrade --longhorn resumes the step.

Each wait lasts up to 45 minutes, so a full run from 1.7.3 can take a few hours on a busy cluster. A step that does not finish in time stops with what is still pending, with longhorn-manager's last log lines when a reported problem mentions the manager and its logs are available. Running kip upgrade --longhorn again resumes a started version change with its existing backup, after checking that no volume is faulted or uses the v2 data engine and that the new release's engine, Kubernetes, CRD and host requirements still hold. If neither the manager DaemonSet nor the running release shows a recorded step's target, the step starts again with every check and a new backup. An unfinished engine catch-up also restarts with a new backup. A plain kip upgrade in between re-applies the release an unfinished step was moving to, or the release the cluster runs when that step never applied, and says so.

A shared volume in Longhorn's stock longhorn class wants three copies, so on a cluster with fewer than three nodes it stays degraded while it is attached. Longhorn cannot move such a volume's engine while it is in use, so kip lets it keep its older engine as long as each node holds a working copy and the new release still supports that engine, and names it at the end. Longhorn moves it once the cluster has a node for every copy: add nodes with kip node add, then run kip upgrade --longhorn again. If a later release no longer supports the engine, that step stops before starting and names the volume.

kip pins these Longhorn settings on every Longhorn cluster, on install and on every upgrade: update checks and usage reporting off, fast-replica-rebuild-enabled off (Longhorn issue 14096), a node drain that waits while Longhorn moves a volume's last copy off the node, data locality and replica auto-balance off, RWX fast failover off, pods on a node that goes down left in place, and volume creation with degraded availability allowed. The last five are Longhorn's current defaults, explicitly set so that upgrades through four consecutive releases preserve them. Longhorn's own production guidance turns degraded creation off. Kipper keeps it on so that a single-node cluster can create volumes that want more copies than it has nodes. kip never sets a Longhorn backup target, because backups go through Velero. The same step marks Longhorn's stock longhorn class as non-default, so k3s's local-path stays the cluster's only default class. Kipper specifies a class on claims it creates directly. The default applies to claims that omit the class: those created by the charts that kip ai installs, and Loki's on clusters installed by v0.38.0 or earlier.

Rocky Linux and AlmaLinux 9.7. On Rocky Linux 9.7, Longhorn's volumes stop attaching (Longhorn issue 13606), and no Longhorn release carries a fix yet. kip holds every host at that level or later: Rocky Linux or AlmaLinux 9.7 or a later 9.x, and any EL9 kernel from 5.14.0-611, which also covers CentOS Stream 9. A new Longhorn cluster and kip node add refuse such a host, kip upgrade warns about each such node, --longhorn refuses to start, and kip doctor warns. Keep Longhorn nodes on 9.6 or earlier, or use version 10, until a Kipper release adopts a fixed Longhorn.

Downgrading console-api

Traefik serves the hop certificate of a *.kipper.run cluster from its own copy, the Secret kipper-hop-cert in the traefik namespace, and console-api keeps that copy up to date. An older console-api, installed by an older kip with --skip-system or by using --image-tag with an older commit, does not maintain the copy. If it rotates the hop key, Traefik keeps serving the retired key, and the cluster stops routing through the kipper.run gateway once the old key's proof runs out. kip doctor reports this as Traefik's copy holding a different key.

Once the downgrade has finished and only older console-api pods are running, remove the copy and restart Traefik. Traefik then serves the hop certificate through its default TLSStore, which the older console-api maintains. If you remove the copy earlier, a console-api that maintains it recreates it. Any kip upgrade run by a kip that maintains the copy also recreates it, including one using --image-tag with an older commit.

A finished rollout can still leave a newer console-api pod terminating. The function below therefore waits until every console-api pod belongs to the current revision, printing the pods it is waiting for, and then removes the copy. It prints the current revision's image for reference and leaves the copy unchanged if any read before deletion fails. At the end, it displays any recreated copy so you can check that deletion persisted. Each rollout wait has a five-minute timeout. The wait for pods from earlier revisions to exit has no timeout. Run it while no other console-api rollout is in progress, and rerun it if it stops partway:

remove_traefik_copy() (
  local rev hash pods
  kubectl -n kipper-system rollout status deployment console-api --timeout=300s || return 1
  if ! rev=$(kubectl -n kipper-system get deployment console-api \
      -o jsonpath='{.metadata.annotations.deployment\.kubernetes\.io/revision}') || [ -z "$rev" ]; then
    echo "STOP: could not read console-api's revision. The copy is unchanged."; return 1
  fi
  if ! hash=$(kubectl -n kipper-system get replicaset -l app=console-api \
      -o jsonpath="{range .items[?(@.metadata.annotations.deployment\.kubernetes\.io/revision==\"$rev\")]}{.metadata.labels.pod-template-hash}{' '}{end}"); then
    echo "STOP: could not read console-api's ReplicaSets. The copy is unchanged."; return 1
  fi
  hash=${hash% }
  if [ -z "$hash" ] || [ "$hash" != "${hash% *}" ]; then
    echo "STOP: could not find exactly one current console-api ReplicaSet. The copy is unchanged."; return 1
  fi
  if ! kubectl -n kipper-system get deployment console-api -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'; then
    echo "STOP: could not read console-api's image. The copy is unchanged."; return 1
  fi
  while :; do
    if ! pods=$(kubectl -n kipper-system get pods -l "app=console-api,pod-template-hash!=$hash" -o name); then
      echo "STOP: could not list console-api's pods. The copy is unchanged."; return 1
    fi
    [ -z "$pods" ] && break
    echo "waiting for: $pods"
    sleep 2
  done
  kubectl -n traefik delete secret kipper-hop-cert --ignore-not-found &&
  kubectl -n traefik rollout restart deployment traefik &&
  kubectl -n traefik rollout status deployment traefik --timeout=300s &&
  kubectl -n traefik get secret kipper-hop-cert --ignore-not-found
)
remove_traefik_copy

The last command prints nothing when the copy is gone. If it prints the Secret, a console-api or kip that maintains the copy recreated it. Check the image above and any concurrent kip upgrade, then rerun the function once the downgrade has finished. Until a console-api or kip that maintains the copy creates it again, a rebooted Traefik serves the hop certificate only once it has loaded the TLSStore, as it did before the copy existed.

Repeat these steps after every kip upgrade by a kip that maintains the copy while an older console-api stays in place. The same steps recover a cluster that is already serving a retired key. An older kip upgrade without --skip-system needs no extra step, because it reapplies a Traefik chart that does not read the copy. Upgrading console-api again to a version that maintains the copy creates or repairs it.

Recovering from a failed chart upgrade

If a system component upgrade fails (helm release ends in failed state), subsequent kip upgrade runs may hang on helm uninstall --wait while helm-controller tries to reset the release. The fix is to drop the stale helm release secrets and let helm-controller do a fresh install:

ssh root@<cluster-host>
kubectl -n kube-system delete job helm-install-<chart>
kubectl -n kube-system delete pods -l helmcharts.helm.cattle.io/chart=<chart> --force --grace-period=0
kubectl -n <chart-target-ns> delete secret -l owner=helm,name=<chart>
kubectl -n kube-system annotate helmchart <chart> kip.kipper/reapply="$(date +%s)" --overwrite

When a node dies

What happens depends on which node is lost and whether its disk survives.

A worker. Services that ran on it stay down until the worker returns or you fence it with kip node fence <worker>. When a node dies in the worker guide walks through the fence, the copies check and bringing the worker back. Use the fence, never kubectl delete pod --force, because the fence makes sure the old host can no longer write to its volumes.

The server. The server runs the Kubernetes control plane and holds every single-copy volume, so the cluster stops while it is down. Workers keep their data. When the host comes back with its disk, the cluster resumes on its own.

The server's disk. Losing the server's disk loses the cluster's state, every single-copy volume and in-cluster backups. Install a new cluster and restore from a backup in external storage. A service with two copies also has a copy on a worker, and that copy can hold newer data than the last backup. Longhorn can export it from the worker's disk, as described below.

A Postgres dump volume. The dump volume of a service with consistent backups always has one copy. If Longhorn reports it faulted, consistent dumps fail and the database pod cannot start again until the dump volume is replaced. kip service info shows a warning with the command that replaces it. kip service recreate-dump <name> deletes only the dump volume and restarts the pod. Kipper never deletes it on its own.

Export a volume from a worker's copy

Longhorn keeps the full data of a volume in each copy, so one copy on a surviving worker is enough to read it, even with the server gone. This follows Longhorn's export from a single replica, with extra steps that keep the worker's copy unchanged. It may be the only copy left, and starting the Longhorn engine on a directory writes to it, so run the engine only on a separate working copy.

1. Assess the copy before changing it. On the worker, find the copy under /var/lib/longhorn/replicas/. Each directory is named after the volume, which is the PV name, plus an eight-character suffix. kip service info <name> showed the volume's nodes before the loss, but omitted its PV name. With the server gone, the cluster can no longer provide that name. Use the Size field of volume.meta, which gives the service's storage size in bytes, and the head image's modification time to help identify the directory. Confirm which service the volume belongs to by inspecting its files after mounting it in step 5. A Postgres volume holds a pgdata directory. Read the evidence now, because starting the engine rewrites volume.meta and the head image's modification time:

ls /var/lib/longhorn/replicas/
cat <replica directory>/volume.meta
ls -l --time-style=full-iso <replica directory>/

None of these signs proves the copy is current, so treat a copy you are unsure about as possibly stale and compare it with your newest backup:

• "Rebuilding": true in volume.meta means the copy was still being rebuilt and is incomplete. Do not use it.
• The modification time of the volume-head-*.img file is the last write this copy received. A time well before the server was lost means the copy had stopped receiving writes.
• What Kipper reported before the loss: kip service info and kip status showing all copies healthy mean the copy was current at that point.

Note the volume's size in bytes from the Size field of volume.meta.

2. Stop everything that holds the copy open. lsof +D <replica directory> usually lists a longhorn process for the replica still running in the worker's instance manager. The cluster is gone with the server's disk, so stop k3s and every container on the worker with /usr/local/bin/k3s-killall.sh, then run lsof +D <replica directory> again. Continue only when it prints nothing. k3s on the worker cannot start again without its server, so the steps below run the engine with Docker instead.

3. Make an untouched copy. Copy the whole directory to another disk. If the worker has only one disk, use another directory on it. If it has no room, copy to another host. Keep that copy unchanged. The image files are sparse, so preserve that format when copying them. Each copy then takes about as much space as the data on the volume:

mkdir -p /mnt/spare
cp -a --sparse=always <replica directory> /mnt/spare/
# Use this alternative if the worker has no room.
rsync -aS <replica directory> recovery-host:/srv/replicas/

Make a second copy of that one to work on. Every step below uses the working copy only. If something goes wrong, make a new working copy from the untouched one:

mkdir -p /mnt/work
cp -a --sparse=always /mnt/spare/<replica directory name> /mnt/work/

4. Start the engine on the working copy. It needs a Linux host with open-iscsi, like any Longhorn node, and the engine image of the Longhorn release that wrote the copy, which is v1.11.3 for this kip. The surviving worker already has open-iscsi. Its k3s cannot run containers without the server, so install Docker there and start the engine with it. Docker pulls the engine image from Docker Hub, about 170 MB. <volume name> is the replica directory's name without its eight-character suffix, and <size in bytes> is the Size from step 1:

apt-get install -y docker.io
docker run -d --name longhorn-recovery --privileged \
  -v /dev:/host/dev -v /proc:/host/proc -v <working copy directory>:/volume \
  longhornio/longhorn-engine:v1.11.3 launch-simple-longhorn <volume name> <size in bytes>
timeout 120 sh -c 'until [ -b /dev/longhorn/<volume name> ]; do sleep 1; done'

The block device takes about ten seconds to appear. If the wait times out, docker logs longhorn-recovery shows why the engine did not start.

On a working k3s node, such as a node in the new cluster, a static pod can run the engine instead of Docker. Write it to /var/lib/rancher/k3s/agent/pod-manifests/longhorn-recovery.yaml. launch-simple-longhorn expects the host's /dev and /proc at /host/dev and /host/proc:

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-recovery
  namespace: longhorn-system
spec:
  hostPID: true
  restartPolicy: Never
  containers:
    - name: engine
      image: longhornio/longhorn-engine:v1.11.3
      securityContext:
        privileged: true
      command: ["launch-simple-longhorn"]
      args: ["<volume name>", "<size in bytes>"]
      volumeMounts:
        - { name: dev, mountPath: /host/dev }
        - { name: proc, mountPath: /host/proc }
        - { name: data, mountPath: /volume }
  volumes:
    - { name: dev, hostPath: { path: /dev } }
    - { name: proc, hostPath: { path: /proc } }
    - { name: data, hostPath: { path: <working copy directory> } }

5. Mount it read-only and copy the data off. The volume appears as /dev/longhorn/<volume name>. Kipper formats volumes as ext4, and a plain mount -o ro still replays an unclean journal, which writes to the device. For a first look, noload skips the replay:

mkdir -p /mnt/recovery
mount -o ro,noload /dev/longhorn/<volume name> /mnt/recovery

Skipping the replay can leave the last writes before the loss missing or inconsistent. Reading a file written shortly before the loss can fail with Structure needs cleaning. Before copying data off, always replay the journal, including for a database. Run umount /mnt/recovery, then stop the engine with docker rm -f longhorn-recovery or by removing the static pod manifest. Make a fresh working copy as in step 3 and start the engine on it as in step 4. Mount it with mount -o ro /dev/longhorn/<volume name> /mnt/recovery, omitting noload. The journal then replays on that working copy only. Copy the files you need.

For a Postgres service, the data directory is pgdata at the top of the volume, owned by the image's postgres user. That user's ID is 70 in Kipper's default Alpine image and 999 in the Debian image used below. When started as root, the official image adjusts ownership of the working data copy to its own postgres user. Kipper passes the TLS and access settings on the command line, leaving the data directory's configuration files usable by the official image during recovery. Start the same major version on a copy, wait for it to finish crash recovery, and take a dump. The user is kipper and Kipper's database is app. psql -l lists any other databases the service holds. Each one needs its own dump:

mkdir /mnt/pg-copy
cp -a /mnt/recovery/pgdata/. /mnt/pg-copy/
docker run -d --name pg-recovery -e PGDATA=/data -v /mnt/pg-copy:/data postgres:16
timeout 600 sh -c "until docker logs pg-recovery 2>&1 | grep -q 'ready to accept connections'; do sleep 1; done"
docker exec pg-recovery psql -U kipper -l
docker exec pg-recovery pg_dump -U kipper -Fc app > /mnt/app.dump

If the wait times out, check docker logs pg-recovery for startup errors, such as a major version mismatch.

During crash recovery, Postgres uses its write-ahead log to restore committed transactions present in this copy. Load the dump into a service on the new cluster with kip service import <name> --file app.dump.

6. Unmount, then stop the engine. If you started Postgres, run docker rm -f pg-recovery. Then run umount /mnt/recovery. Finally, stop the engine with docker rm -f longhorn-recovery or by removing the static pod manifest. Stopping the engine while the filesystem is mounted leaves the mount pointing at a device that no longer exists.

Host maintenance and recovery

• DNS repair restores the configured host resolvers.
• Host hardening applies firewall and service defaults.
• Node repair updates storage-related host settings.
• Node reboot backs up and reboots the server to apply kernel or service updates, then waits for healthy nodes and components and for previously ready apps to become ready again.
• Worker rejoin: a worker that shows NotReady while k3s-agent logs not authorized can have missing or expired agent certificates. Run kip node add <host> again with the original join options; it rejoins the worker with a fresh token and keeps its name and data. If that fails, follow the cause it reports.
• Domain changes move the cluster’s serving identity.
• Cluster removal removes the installation and cluster data.