CLI Reference

Use this reference to look up commands and flags. For workflows, start with the guides in the sidebar. Run kip <command> --help for the options available in your installed version.

• Installation reference covers kip install, server requirements, and initial setup.
• Upgrades & Maintenance covers kip upgrade, its flags, kip upgrade --check, kip upgrade --longhorn and upgrade recovery.

kip cluster

Manage cluster configurations on your local machine.

kip cluster export > file.kip        # export connection details for sharing
kip cluster add file.kip              # import a cluster
kip cluster add file.kip --set-current # import and switch to it
kip cluster list                      # list all clusters
kip cluster use <name>                # switch active cluster
kip cluster remove <name>             # remove local config
kip cluster uninstall <name>          # wipe the cluster off the remote host
kip cluster domain <domain>           # move to a custom domain (no-lockout cutover)
kip cluster domain <domain> --ack-sso-callbacks # confirm SSO callback URLs are updated
kip cluster domain --sync             # finish an interrupted domain change
kip cluster domain --rollback         # return to the previous domain
kip cluster domain --repair           # rewrite local config from the cluster
kip cluster hosts                     # show the hostnames kip uses for a cluster
kip cluster hosts --dex dex.example.com # correct one without contacting the cluster
kip cluster dns repair                # restore the curated DNS resolvers on every node and restart CoreDNS
kip cluster encrypt-secrets           # encrypt the cluster's Secrets at rest (restarts k3s)
kip cluster storage                   # storage mode, the copies offer and the nodes that can hold copies
kip cluster storage --max-copies 2    # let deployers create services with two copies
kip cluster storage --over-provisioning 200 # let Longhorn schedule up to twice the disk space after its reserve
kip cluster access show               # the operator access list and a check of every node
kip cluster access allow --here       # restrict SSH and the API to listed operator addresses
kip platform restart <component>      # restart a platform or cluster component

kip cluster uninstall

Wipes Kipper from a remote Linux server. Runs k3s's own uninstall script over SSH, sweeps the data directories Kipper writes outside k3s (Longhorn volumes, Zot blobs, AI bundle data), and removes the cluster from your local kip config. Recorded workers are wiped over SSH first. Each wipe checks that the host's machine ID matches its recorded ID, protecting machines at reused addresses. Workers are selected from the cluster's node records, or the local copy if those records cannot be read. Workers found only in the cluster's records require separate confirmation and are skipped with --yes. Workers whose cleanup is unconfirmed are listed with their recorded machine IDs. Check that /etc/machine-id on each host matches before running /usr/local/bin/k3s-agent-uninstall.sh there.

kip cluster uninstall storefront                  # interactive (prompts for cluster name)
kip cluster uninstall storefront --yes            # skip the confirmation prompt
kip cluster uninstall storefront --keep-local-config  # wipe host, keep local entry
kip cluster uninstall storefront --ssh-key ~/.ssh/kipper_ed25519

The command prompts you to type the cluster name to confirm, so you cannot tear down a cluster by reflex. Pass --yes only for automation.

A cluster on a free *.kipper.run subdomain also hands that name back to the gateway. The name is released after the host is wiped, using a credential read from the cluster beforehand: the gateway will only release a name to whoever holds its token, and that token lives on the cluster the wipe destroys. The name is free again straight away, so a rebuild of the same server can claim it back immediately. That also means links you published under it can be claimed by someone else, so move them before you uninstall if they still matter.

If the cluster cannot be reached, kip falls back to a copy of that credential it keeps in ~/.kip/config.yaml, recorded on an earlier command. When neither is available the command says so and asks whether to wipe anyway. Answering no is the safe choice: wiping leaves the name registered with nothing able to release it, and installing on that host again cannot serve on it. The gateway stops routing such a name about a week after the cluster's last heartbeat, the registration lapses after 30 days without contact, and the name comes free 90 days after that, so waiting it out means waiting four months. --yes skips this question as well as the typed-name one, so scripted teardown consents to stranding a name it could not release.

If the server itself cannot be reached, kip offers to hand the name back without it, provided a credential for that name is recorded locally. Say yes only when the server is really gone: a cluster that is merely unreachable is still serving, and releasing its name takes it off the air. --yes never takes this offer, because a script cannot tell those two apart.

When the host is wiped but the gateway will not take the name back, the cluster stays in ~/.kip/config.yaml and the command tells you to run it again. That entry is deliberate, because it holds the only credential that can still release the name, and it is why kip cluster list can show a server you have already wiped. The re-run goes straight to the gateway: it does not touch the host, so it works just as well on a server you have since destroyed.

kip cluster uninstall storefront
#   storefront was already wiped. Releasing its gateway name.
#
#   ✔  Gateway name released
#   ✔  Local config entry for storefront removed

This is destructive. All cluster state and persistent volume data on the host is removed.

On each host whose firewall kip manages, the uninstall removes the rules kip added: ports 80, 443 and 6443, pod access to the kubelet (10250) and node-exporter (9100), and the rules for the cluster's other nodes. The operator tunnel's UDP port is scheduled to close about ten seconds after its host is wiped, giving an uninstall session through the tunnel time to finish. Rules for ports used by SSH are kept. On UFW it also saves the routed-traffic policy as deny, which UFW applies the next time it starts, for example after a reboot. On firewalld it deletes the entire kipper-pods, kipper-peers and kipper-server zones and kip's two forwarding policies, including anything added to them by hand.

The firewall stays running or stopped, so a host where it was on keeps it on. SSH stays allowed. Rules you added yourself stay unless they match a kip rule or belong to one of the deleted firewalld zones or policies. kip leaves the firewall alone and prints a warning when another firewall change is pending on the host. It also skips firewalld cleanup if the running configuration differs from the saved configuration, because reloading to apply the removal would discard unsaved rules.

The ownership file, /etc/kipper/firewall-managed or /etc/kipper/firewalld-managed, stays, so a later kip install on the host manages the firewall again and adds its rules back. On firewalld that install replaces the public zone, as it always does, so add your own public-zone rules again afterwards. Delete the ownership file to manage the firewall yourself from then on.

If the rules cannot be removed, the uninstall prints a warning and continues. It also removes the operator access list from each wiped host, so a reused host never boots with a list from a cluster that no longer exists. OS hardening, such as disabling rpcbind, remains in place because it is a general OS security improvement unrelated to k3s.

Use --keep-local-config when you plan to reinstall immediately, so the existing cluster name and kubeconfig path stay in ~/.kip/config.yaml for the new install to refresh.

kip cluster access

Restricts new connections to every node's SSH ports and to the Kubernetes API (6443) to a list of operator prefixes. Ports 80 and 443 stay public. The list is opt-in. A cluster without one is unchanged. See Operator access list for how it works, what happens at boot and how to recover from the provider console.

kip cluster access show                       # the list, recent changes and a check of every node
kip cluster access show --json                # the same as JSON
kip cluster access allow --here               # add the address this workstation connects from
kip cluster access allow 203.0.113.0/24 2001:db8::/48 --note office
kip cluster access revoke 198.51.100.7/32     # remove prefixes
kip cluster access apply                      # load the recorded list on every node again
kip cluster access apply --accept-older-record # restore an older record's list after reviewing it
kip cluster access reset --yes                # remove the list and open SSH and the API
kip cluster access device add laptop          # reach SSH and the API through a WireGuard tunnel
kip cluster access device add laptop --device-only # then accept only registered devices
kip cluster access device list
kip cluster access device remove laptop
kip cluster access device repin <node>        # trust a reinstalled node's new WireGuard key
kip cluster access via auto|tunnel|direct     # how this workstation reaches the nodes
kip cluster access ssh-config                 # OpenSSH Host blocks through the tunnel
kip cluster access forward [--port 16443]     # a local port to the API through the tunnel
kip cluster access kubeconfig [--port 16443]  # a kubeconfig context for that port

show, allow, revoke, apply, reset and the device subcommands take --ssh-key. Every command that reaches the nodes accepts --via, though commands specifically for the tunnel always use it. A bare address is a /32 or /128. 0.0.0.0/0 and ::/0 are refused, because reset is the way to open the ports. --here adds an IPv4 address as a /32 and an IPv6 address as the /64 around it. Each change is loaded on the workers first and the server last. A node keeps the change only after a new SSH connection works from a listed address. The server also checks that the API still answers. kip refuses a list that would not admit your workstation, revoking the last entry while no operator device is active, any change while a node is unreachable or a worker's removal is unfinished, and any change while the API does not answer from your workstation.

show exits 1 unless every node enforces the record. It reports nodes it cannot reach as not verified, a live table that differs from the saved files, a node that booted closed, a failed kipper-access service, an enabled nftables.service, and a saved table the installed nft would reject at the next boot. When an access list is present, a failed SSH or API check for an address family with no prefix in the list appears as a note and leaves the exit status unchanged, because your network may have no route for that family. reset removes the list from every reachable node, including a worker whose removal has not finished, and deletes the record only when every node has been reset. Run it again once an unreachable node is back. Run it from your workstation, because on the cluster's own host kip cannot see the address you connect from.

kip platform restart

Triggers a rolling restart of a platform or cluster component, for when it has stale configuration or needs to pick up changes.

If the Kubernetes API request fails or the API rejects your login, kip restarts the component over SSH by running kubectl on the server with the cluster's SSH key. Login can fail when Traefik serves its fallback certificate after a restart. Other API errors, such as a permission denial, are reported without retrying over SSH.

kip platform restart dex           # restart identity provider
kip platform restart console       # restart web console
kip platform restart console-api   # restart console API
kip platform restart traefik       # restart ingress controller

kip platform versions

Shows the cluster's component versions. The header shows the kip version recorded on Kipper's App CRD, the console-api build, each node's Kubernetes version, and your local kip version. The Kubernetes version is shown once when all nodes agree. Source builds preserve an existing CRD release stamp, so it can refer to an earlier CRD write. At startup, console-api records the commit it was built from because its image uses the moving latest tag. A rollback to a console-api build that predates this recording leaves the newer commit in place unless the record is cleared.

Below the header, kip lists every Deployment, StatefulSet and DaemonSet in Kipper's system namespaces with its ready count and pod template images. Init container images are marked (init). Velero uses init containers to load its storage plugin.

A workload marked updating may still have pods from an earlier pod template, so some of them may run the previous image. READY counts pods that pass their readiness checks.

kip platform versions
kip platform versions --cluster staging

kip cluster env

Sets environment variables on a cluster component and restarts it to pick up the changes.

kip cluster env console-api LOG_LEVEL=debug
kip cluster env console-api LOG_LEVEL=debug FEATURE_X=enabled

See Team Access for full documentation.

kip tunnel

Opens a secure tunnel from your machine to a service running in the cluster. Use this to connect desktop database clients (DBeaver, TablePlus, pgAdmin) to databases that are not exposed to the internet.

kip tunnel mydb                        # PostgreSQL on localhost:5432
kip tunnel cache                       # Redis on localhost:6379
kip tunnel mydb --local-port 15432     # custom local port

Flag	Required	Default	Description
--local-port	No	Same as service port	Local port to listen on
--port	No	Auto-detected from the pod	Remote container port
--project	No	Saved project, else every project	Project name
--environment	No	Saved environment, but only when using the saved project	Target environment
--kind	No	Any kind	Restrict to app, function, or service

The tunnel forwards to a pod that is Ready. When a workload is running but no replica is ready, kip tunnel says so instead of forwarding, because a tunnel to a pod that cannot serve produces connection errors that look like a broken application.

See Naming one workload below, and Team Access for full documentation.

kip exec

Opens an interactive shell or runs a command inside a running container.

kip exec myapp                         # interactive shell
kip exec myapp -- cat /app/config.yaml # run a single command
kip exec mydb -- psql -U kipper app   # SQL session in a database pod

Flag	Required	Default	Description
--project	No	Saved project, else every project	Project name
--environment	No	Saved environment, but only when using the saved project	Target environment
--kind	No	Any kind	Restrict to app, function, or service

Unlike kip tunnel, kip exec will enter a pod that is running but not ready, since a container failing its readiness probe is usually the one you want a shell in.

See Naming one workload below, and Team Access for full documentation.

Naming one workload

kip exec and kip tunnel both take a workload name, and that name has to identify exactly one workload. Where it matches more than one, they list the matches and stop rather than pick one for you.

A project is authoritative when you give one, either with --project and --environment or through kip project use. The search is confined to it, and a project that holds no workload of that name is an error rather than a reason to look in someone else's project. Without a project, the search covers the whole cluster.

Two things commonly match more than once. The same app name in several environments is the ordinary case, since each environment is its own namespace:

$ kip exec backend
Error: "backend" matches more than one workload:
  app/blog-prod
  app/blog-test
Name the one you mean with --project, plus --environment if the project has environments.

An app and a service can also share a name inside one namespace, which naming the project does not resolve. Use --kind for that:

$ kip exec backend --project blog --environment prod
Error: "backend" matches more than one workload in blog-prod:
  app/blog-prod
  service/blog-prod
Name the one you mean with --kind app or --kind service.

$ kip exec backend --project blog --environment prod --kind service
  Connecting to blog-prod/backend-0...

kip status

Shows cluster health, node status, and component availability.

kip status

A component with no ready replicas (or no available replicas for a Deployment) is marked … and listed as starting when its pods appear to be starting up. Assigned nodes must be Ready. Pending pods can qualify before scheduling or container status is reported. Finished and deleting pods are ignored. If startup persists, check pod events, init containers, and readiness probes. ✗ means the health check found neither a healthy component nor evidence of startup; possible causes include missing pods, crash loops, image pull failures, scheduling failures, nodes that are not Ready, and lookup errors.

A node fenced with kip node fence shows as OutOfService and never counts as Ready. When a node is not Ready or is fenced, a Stranded section lists the Kipper services with a single-node volume on it, the nodes that hold a current copy of their data, and the command to run: kip node fence <node> for a node that is not fenced, and the rejoin or forced removal for one that is. The section also lists services kip node fence left stopped, with the kip service start command for each.

The Backups section reports the health of the newest finished backup of each active schedule, using the same checks as kip doctor.

It also audits the DNS resolvers your cluster forwards external queries to. Those live in a file on the server, so kip status makes a best-effort SSH connection as the cluster's SSH user (reusing the key from install, then your SSH agent or ~/.ssh/id_ed25519) to read it. Three things are checked: the file is still a safe set (IPv4 entries only, at most three), the entries still match the resolvers the cluster was configured with (dns_resolvers in ~/.kip/config.yaml, or the default public set), and each resolver actually accepts connections from the server. Each problem gets its own warning, and drift or an unsafe hand-edit is fixed with kip cluster dns repair. If the server can't be reached over SSH, the section reports that the check could not run, and the rest of the status still prints.

On a cluster with workers, the host checks (DNS resolvers, pending restarts, read-only volume mounts, attempts to switch UFW off after a failed firewall rollback, and operator access) cover the server and every recorded worker. Results identify each node by its recorded name. kip finds workers in node records read through the Kubernetes API. If those records are missing, unreadable or invalid, it uses the local copy and warns when the read or validation fails. It runs all of a worker's checks as one SSH command to reduce connections against the worker's firewall rate limit. Each worker's resolvers are probed from that worker. kip cluster dns repair restores the resolver file on the server and every worker. Before reporting a worker's results, kip confirms that the address still answers with the worker's recorded machine ID. A worker that cannot be reached, or that answers as another machine, is listed as not checked and the rest of the status still prints. Operator access checks require an SSH connection to the server to read the cluster's access record. On a single-node cluster, results retain their layout without node-name prefixes.

On a cluster with Longhorn storage, a Storage section appears when the server's roomiest eligible disk has less than 5Gi available for scheduling new volumes. This is the default volume size for kip volume create.

Two storage copies sections appear when they have something to list. The first names the services, volumes and project shared storage whose copies are unset, with the copy counts reported by Longhorn when available, and how to record them. The second names those whose copies need attention: fewer healthy copies than set, a volume held back at creation (for example, two copies asked for while fewer than two nodes can hold them), a set count the volume does not have yet, or copies Kipper could not check because it could not read the node holding them. A copy counts as healthy when its data is current and its node exists, is ready and is not fenced.

kip doctor

kip doctor

Diagnoses a cluster that installed but does not work and provides a way to report failed checks. It first reads the cluster over SSH as the cluster's SSH user, reusing the key from install. If SSH is unavailable, it reads through the Kubernetes API. Using SSH lets it diagnose problems even when the API rejects your login. This can happen after a restart when Traefik serves its built-in fallback certificate: the API server can no longer load Dex's discovery document, so commands using your API login fail, including kip status.

The checks are grouped by where a problem could occur:

• Host: the server answers over SSH, k3s is running, and how long ago the server booted.
• Kubernetes: the API server answers, it accepts your login, the nodes are Ready, and Traefik, CoreDNS, cert-manager, Dex and the Console API are running. On a Longhorn cluster it warns about each node at the Rocky Linux or AlmaLinux 9.7 level or later, where Longhorn's volumes may stop attaching (see Longhorn upgrades).
• Backups: the newest finished backup of each active schedule. A failed one fails the check. A partially failed one warns and points to kip backup describe. A schedule that has not run for more than two days for daily or faster schedules, or two weeks for weekly schedules, also warns. Staleness is not judged for other cadences, such as a monthly schedule. kip status prints the same result.
• Gateway hop (kipper.run clusters only): the hop certificate is in place, Traefik mounts a copy with the same certificate, the default TLSStore points Traefik at it, port 443 actually serves it, the gateway heartbeat renews the name, and the cluster registers the address kip has on record.
• kipper.run gateway (kipper.run clusters only): the gateway is up and routes the cluster's name to it.

Checks that cannot run are marked as not checked. For example, reading the hop certificate requires SSH access. When a check fails, kip doctor prints a link to a prefilled GitHub issue form and a report to paste into it, as a failed kip install does. The link contains only check names and statuses, the kip version, your machine's OS and architecture, the server's distribution and version, the sizing profile and the names of the flags you set. The report replaces recognised addresses, hostnames and credentials with placeholders. The command exits with status 1 when a check fails.

Flag	Meaning
--ssh-key	Private key for the server, if it is not the one recorded at install.

kip cluster dns repair

Restores the curated DNS resolver file on every recorded node and restarts CoreDNS to pick it up. This fixes the resolver drift kip status warns about. It also synchronizes node records with the local cluster configuration and leaves other installation settings unchanged. kip writes the file on the server first, then on each worker one at a time, and restarts CoreDNS once every node has it. It stops at the first node it cannot reach or write, before the restart, and names that node. Run the command again to finish, starting from the server.

kip cluster dns repair

  ✔  Restored resolvers on server-1 (203.0.113.10): 1.1.1.1, 8.8.8.8, 9.9.9.9
  ✔  Restored resolvers on worker-1 (203.0.113.11): 1.1.1.1, 8.8.8.8, 9.9.9.9
  ✔  CoreDNS restarted to pick them up

The resolvers come from dns_resolvers in ~/.kip/config.yaml (set via --dns-resolver at install time), or the default public set when none are configured.

kip cluster encrypt-secrets

Turns on k3s secrets encryption on the server of an older cluster and rewrites every stored Secret with the new key. A cluster installed by this version of kip already encrypts its Secrets. Running the command there rotates the key once, and later runs change nothing. The API server is unavailable for a short time at each k3s restart. There are usually two restarts. Recovering interrupted re-encryption may require an additional restart. See Secrets at rest for the steps and for what encryption covers.

kip cluster encrypt-secrets

kip cluster storage

Shows the storage mode, the most copies of a volume the cluster offers and how many nodes can hold copies. With --max-copies, sets the offer. A cluster offers one copy until an admin runs kip cluster storage --max-copies 2, after which deployers may create services with --copies 2. Existing volumes keep their copies when the offer changes. Two copies need Longhorn storage, so the offer of two is refused on local storage. On a cluster with fewer than two ready nodes that can hold copies, the offer is recorded with a warning. New volumes with two copies need two such nodes. An admin can add a worker with kip node add or let an existing worker hold copies with kip node storage <node> --copies.

kip cluster storage
kip cluster storage --max-copies 2

The offer is recorded in the PlatformConfig and enforced by the kipper-storage-copies-offer admission policy, which kip renders on install, upgrade and every --max-copies change. Admins are not bound by it. If the command stops between the two writes, run it again. See Two copies of a service's data.

On a Longhorn cluster, the output also estimates the capacity of the server's roomiest disk for new single-copy volumes and shows Longhorn's over-provisioning percentage. A volume with two copies also needs room on another node that can hold copies. Longhorn counts each volume at its full size, regardless of how much data it holds. It schedules volumes up to the over-provisioning percentage of each disk's space after its reserve. When no scheduling capacity remains, even with plenty of free disk space, kip service add and kip volume create refuse to create volumes. A drill stops waiting after two minutes of scheduling failures with zero placed copies. Run the command again once there is room. --over-provisioning sets the percentage, and it must be at least 100:

kip cluster storage --over-provisioning 200

At 200, the volumes on a disk may add up to twice its space after the reserve. Writes fail once the volumes together fill the physical disk, so raise the percentage only while the volumes hold much less data than their configured sizes. Every flag is checked before anything changes. kip upgrade keeps the value.

On a Longhorn cluster, the output also lists unfinished moves to Longhorn, service volumes still on local storage, and the local copies kept after a move, the same as kip status.

kip cluster storage longhorn

Switches a local-storage cluster to Longhorn in place. Nothing stops while it runs.

kip cluster storage longhorn [--ssh-key <path>]

The command needs at least 6000 MiB of RAM on the server, 12% of each node's allocatable CPU available after accounting for requests from its pods and pending pods that could run there, a platform upgraded with the same kip version, more than 25% of the filesystem holding /var/lib/longhorn free, and an OS that can hold Longhorn volumes. If CPU is short, the command suggests service CPU reductions on that node that would make room when possible. You apply those changes yourself. The command prepares the host, installs Longhorn, and records Longhorn as the storage mode. It then requests a console-api restart to enable the Longhorn volume watch and lists the service volumes still on local storage with the command that moves each one. The platform's own volumes stay on local storage. Run it again to continue after an interruption. See Grow in place.

kip cluster storage prune

Deletes the local copies that kip service storage longhorn kept after a move.

kip cluster storage prune [<service> [--project <project>] [--environment <env>]] [--now] [--delete-unproven] [--yes] [--ssh-key <path>]

It lists each kept copy with its service, size, age and the reason it is kept, asks once, and deletes a copy when its service still has its data on the Longhorn volume the move verified, its pod has been Ready on that volume for at least an hour, and either a backup made after the move holds the service's data or a day has passed since the move. The backup counts only when it completed, has a future expiry and is not being deleted, and backed up the service's data volume from the verified claim in that same backup, matched by the backup's UID. A Postgres service with consistent backups always waits the day, because Velero counts hooks per backup rather than per pod and the dump's manifest is only inside the backup, so no backup shows that the service's own dump succeeded. Copies whose service and data claim are both gone are deleted without these checks and listed in their own section of the prompt. It checks again right before deleting each copy. Copies of unfinished moves, and of services that lost or replaced their data volume, are kept.

Flag	Effect
<service>	Limits the command to that service's copy. Without --project or --environment, copies of that service name must exist in only one project.
--now	Skips the wait for a backup or a day, so the local copy stops being the fallback early. The health check still applies.
--delete-unproven	Deletes the named copy before its service passes the health check, for example a service you stopped or one that has been Ready on Longhorn for less than an hour. Requires <service> and waives only the health check. The wait for a backup or a day still applies unless --now is also passed.
--yes	Skips the question.

kip node add

A cluster starts with one server. kip node add joins workers that add CPU and memory for apps, functions, jobs and services. The server remains the only control plane and ingress node, so the cluster still depends on it for availability.

kip node add <host> [--ssh-key <path>] [--ssh-user <user>] [--private-network <cidr> | --public] [--firewall | --firewall=false] [--no-ssh-rate-limit] [--no-copies] [--accept-stale-data] [--yes]

The command prepares the host (kernel modules, kubelet hardening, Longhorn's storage packages, host hardening and the firewall) and joins it as a k3s worker on the server's k3s version. It refuses a host at the Rocky Linux or AlmaLinux 9.7 level or later, where Longhorn's volumes stop attaching (Longhorn issue 13606, see Longhorn upgrades).

• SSH user: kip logs in to the new node as the cluster's SSH user. Pass --ssh-user when the node's image uses a different login, such as ec2-user on an AlmaLinux worker joining a Rocky Linux server. kip saves that user in the node's record and uses it for later connections and the node's block in kip cluster access ssh-config. The node keeps that user when it joins again without the flag. A user other than root needs passwordless sudo.
• Networking: kip uses a shared private network when available, or public addresses otherwise. The first join sets the cluster network mode. Applying new network settings restarts k3s on the server. --yes confirms network changes and recovery without prompting. Use --private-network to select a private network that carries the default route, as can happen for cloud VMs in the same VPC, or --public to use public addresses. A worker already recorded in a private-mode cluster can rejoin or be added again without --private-network. If automatic selection finds no private path, kip checks the path between the server's and worker's recorded private addresses and reuses it if the routing, connectivity and network overlap checks pass. On the first join, if no other private route qualifies and you pass neither --private-network nor --public, kip checks this network for private connectivity and network overlap. If it passes, kip stops before the host joins and shows the --private-network value to use. When the hosts share a private route but the server cannot reach the new host's SSH port on it, or the new host cannot reach the API on port 6443, kip stops and names the failed connection. Open the port, or pass --public to join over public addresses instead. The server must already use WireGuard for its pod network. Run kip upgrade first if needed.
• Firewall: By default, kip sets up a host firewall on a new node only when it manages the server's firewall. Use --firewall or --firewall=false to override this default. A kip-managed firewall allows 51820/udp and 10250/tcp from the other nodes and 9100/tcp from the server. On workers, kip opens only SSH to all sources. When configuring UFW for a worker, kip removes saved public rules for TCP ports 80, 443 and 6443 that exactly match its server rules, preserving every SSH port reported by sshd -T. Identical rules added by an operator are also removed. On a node without a kip-managed firewall, your rules must allow that traffic, restrict those ports to the listed sources, keep 8888/tcp closed to outside traffic, and limit access to any NodePort services. Existing kip-managed firewalls get updated peer rules on every join, even with --firewall=false, and keep their SSH rate limiting setting. kip checks operator-managed firewalld rules on existing nodes and, when firewall setup is enabled, on the joining node. A UFW firewall that kip does not manage on the server or another existing node stops the join, even with the ports open. Switch it off first. On the server, run kip cluster harden if kip should manage it. On a new node, such a firewall stops the join unless you pass --firewall=false. A new Ubuntu or Debian worker inherits the server's kip-managed UFW setting, with SSH rate limiting as the default. --no-ssh-rate-limit allows SSH without rate limiting when kip configures the joining node's UFW firewall. It replaces any existing SSH rate limit, for example one left by an earlier install. A UFW rate limit blocks connections from an address that attempts six or more SSH connections within 30 seconds. Because this can cause an SSH timeout, kip node add suggests checking for that rule when a connection times out. firewalld hosts allow SSH without rate limiting. Applying kip-managed firewalld rules restarts the active k3s service (k3s-agent on a worker). On the server, this temporarily interrupts API access.
• Provider firewalls: A cloud provider's firewall or security group must allow 51820/udp and 10250/tcp in both directions between the nodes, 9100/tcp from the server to each worker, and 6443/tcp from each worker to the server. Allow either node to initiate traffic on 51820/udp and 10250/tcp, even if the join succeeds without these rules. A stateful security group allows replies to WireGuard traffic initiated by the server, so a join can pass while the server's inbound 51820/udp rule is missing. Once the connection tracking entry expires, for example after a long idle period or a reboot, the provider can block tunnel traffic initiated by the worker.
• Join checks: before allowing new workloads, kip checks pod DNS, connectivity to CoreDNS on the server, Longhorn volume reads and writes, host authentication to the cluster registry, containerd's registry mirror configuration and direct registry access, and kubelet access from the server. The worker stays cordoned until the node and its Longhorn storage pods are ready. kip then replaces the cordon with a kipper.run/checking taint so Longhorn can attach the test volume. During validation, the taint blocks scheduling of pods that do not tolerate it. To resume an interrupted join, rerun the command with the same options.
• Fenced workers: for a worker fenced with kip node fence, kip node add <host> rejoins it in a fixed order. It makes sure the fence's drop-in is in place and no node processes or volume mounts remain, and it reboots the host unless the host has booted since the fence. The reboot point is recorded first, so a rejoin interrupted during the reboot resumes after it. In private mode, if a worker responds at an address different from the one its node publishes, kip records the new address. This can happen, for example, when a cloud provider assigns a new public IP. After the rejoin, kip runs the add process again so the node publishes the new address. In public mode, the other nodes reach the worker at its old address, so kip refuses to rejoin it at the new address. kip then cordons the node, removes the drop-in, starts the k3s agent, waits for the node to report Ready from the new boot and removes the out-of-service taint. The usual join checks follow, and the node is uncordoned. Last, kip starts the services the fence stopped: each starts when every one of its volumes has a current copy on some node, or when you accept the older data, interactively or with --accept-stale-data. The others stay stopped for kip service start. Unless the worker moved, the rejoin leaves the network, firewall and access settings as they are. A worker whose fence is unfinished is refused until kip node fence completes, and a fenced worker whose removal has started is refused until kip node remove --force finishes. A host that joins without a record but still carries a fence drop-in, for example after a forced removal, has the drop-in removed before its agent is installed. A host that answers at a fenced worker's address with another machine ID is refused until you remove that worker with kip node remove <worker> --force.
• Records: kip records nodes in the kipper-nodes ConfigMap in kipper-system. kip upgrade, kip cluster harden and kip install require records for every cluster node. Use kip node add to adopt an unrecorded worker. Adopt all unrecorded nodes before joining a new node.
• Operator access list: on a cluster with an operator access list, the server and every existing worker must already enforce it, and your workstation's address must be on it. kip loads the list on the server and each existing worker before their peer firewall changes, and on the new worker right after it registers. While the node joins, the server admits the new host's addresses on 6443. An abandoned join leaves that entry until the next kip cluster access apply or reboot. kip node remove drops the removed node from the other nodes' lists and notes any node without one.
• Join credentials: each join attempt uses a fresh k3s bootstrap token valid for one hour. The command attempts to delete it after registration or a join failure. If cleanup fails, the token expires on its own. Once joined, a worker authenticates with its own agent certificates. k3s issues them for one year and renews them at each agent start while they are still valid, so a worker that restarts its agent at least once a year (kip node reboot <worker> does this) keeps them current. A worker whose certificates have expired or gone missing cannot reconnect: it shows NotReady and k3s-agent logs failed to retrieve configuration from server: not authorized. Run kip node add <host> again for that worker with the original join options. It mints a fresh join token, repeats the join and keeps the node's name and the data on its disks; workloads are kept off the node while the join is checked. If the rerun fails, follow the cause it reports.
• Maintenance: kip upgrade performs host operations on the server only. kip cluster uninstall wipes the recorded workers before the server. kip cluster harden, kip cluster dns repair and kip node repair-host act on every recorded node: the server first, then each worker one at a time. Each stops at the first node it cannot reach or change, names that node and leaves the rest for a rerun. On a worker, kip cluster harden applies the worker firewall: SSH open to all sources, the cluster ports open to the other nodes, and no public 80, 443 or 6443. kip node reboot <worker> drains and reboots a worker, and kip node remove <worker> removes one. Rerunning kip install upgrades k3s on the server, then brings each worker recorded as ready to the server's version (see k3s upgrades). kip cluster ca, kip cluster domain, kip auth reset-password and kip install --local act on the server only. When the first Rocky Linux, AlmaLinux or CentOS Stream node joins, kip node add turns on Velero's privileged file system backups, which SELinux hosts need, and waits for the backup agents to restart before it checks the worker.
• Copies: once the worker has joined, kip runs the disk check of kip node storage on it and, when the disk passes, lets it hold copies of services created with two copies. A refused or failed check allows the join to complete without enabling copies on the worker. The output names the command to run. --no-copies skips the check.
• Availability: ingress stays on the control-plane node. Kipper configures each database, registry and in-cluster backup volume to keep one copy on the server. Workers add CPU and memory; high availability requires additional data copies and control-plane redundancy.
• Data placement: kip upgrade and kip node add set server-only scheduling for these volumes and report copies still on workers. kip status also lists these copies. For a pinned volume, kip node reboot <worker> moves the copy to the server when the server has room for it. Otherwise the reboot names the volume and stops before cordoning the worker. If kip cannot read Longhorn's state, or capacity changes after that check, the drain blocks until --drain-timeout, then fails, and kip cancels the reboot. For volumes reported as not pinned, run kip upgrade to retry pinning; a reboot may otherwise move the copy to another worker. Permanent loss of a worker holding the only copy can lose that data. kip service info <name> shows where the service runs and where its data is stored.

kip node storage

Checks a worker's disk and lets it hold copies of services created with two copies, or stops it taking new ones.

kip node storage worker-1 --copies
kip node storage worker-1 --copies --ignore-slow-disk
kip node storage worker-1 --copies=false

With --copies, kip installs fio on the worker when it is missing and runs etcd's fio recipe twice in /var/lib/longhorn: 22 MiB written in 2300-byte blocks with a fdatasync after each, capped at 60 seconds per run. It also times 200 TCP connections from the worker to the server's kubelet port, and 20 pings to the server over the pod network, the path Longhorn's copies travel. The ping results are reported and recorded for information. The worst measurements from the two fio runs determine the disk-check result:

Result	When	Effect
Refused, no override	the slowest sync takes at least half of Longhorn's engine-replica-timeout (4 s with the default of 8 s)	the worker gets no copies
Refused	1 in 100 syncs takes more than 20 ms, any sync takes more than 1 s, or fio cannot write 22 MiB within 60 s	--ignore-slow-disk accepts the disk and records the override
Warning	1 in 100 syncs takes more than 10 ms, or 1 in 100 connections to the server takes more than 2 ms	the worker gets copies

Local SSD and NVMe disks pass with a wide margin. Network block storage, such as a cloud provider's attachable volume mounted at /var/lib/longhorn, is usually refused. kip records the figures, the time of the check and any override in the node's record, and tags the node kipper-replica in Longhorn when it passes. A check that cannot run leaves the node as it was. Running the command again repeats the check. When a repeated check refuses a worker that already takes copies, the worker keeps taking them, the refusal says so, and --copies=false stops new ones.

--copies=false removes the tag, so Longhorn places no new copies on the worker. It is refused while the worker still holds copies, because moving them off a node is not available yet. Reduce the services and volumes on it to one copy with kip service update <name> --copies 1 first. The server is always eligible to hold copies, so this command accepts workers only.

Flag	Required	Description
--copies	Yes	--copies to check the disk and allow copies, --copies=false to stop new copies
--ignore-slow-disk	No	Accept a disk refused as slow. The override is recorded with the node
--ssh-key	No	SSH private key. Overrides KIP_SSH_KEY and cluster.ssh_key

kip node remove

Removes a worker from the cluster. The server cannot be removed. kip cluster uninstall removes the whole cluster.

kip node remove worker-1
kip node remove 203.0.113.11 --yes

kip records the worker as being removed, stops Longhorn from placing volume copies on it and asks Longhorn to move the copies already there. It waits until no volume has its only healthy copy on the worker, then cordons and drains it. It checks that the host still has the machine ID recorded for the worker, uninstalls k3s, wipes Kipper's data and removes the firewall rules kip added there, as kip cluster uninstall does. It checks that k3s is gone before deleting the node. It then removes the worker's address from the firewall on every remaining node whose firewall kip manages, names each node whose firewall it does not manage so you can remove the address there yourself, and deletes the worker's record.

If a step fails, the worker stays recorded as being removed. Fix the cause and run the command again to resume, even if the node itself has already been deleted. Without --force, the worker's host has to be reachable over SSH for the first run and for every rerun, because kip only deletes the node after it has checked the host's machine ID and wiped it.

To keep a worker whose removal stopped, run kip node add for it again, which also lifts Longhorn's eviction request. You can keep the worker until kip records it as wiping, just before it prints "Uninstalling k3s" or, with --force, starts deleting the node. After that, finish the removal and add the host again. To keep a worker while its removal is still running, stop the removal first.

Without --force, kip starts a removal only while the worker's host answers with its recorded machine ID. A host that is gone for good needs --force. While a removal is unfinished, kip blocks adding other nodes, rebooting the worker, hardening or reinstalling the cluster and starting another removal. These operations could change peer rules or schedule work on the worker. Only one kip node add or kip node remove runs on a cluster at a time. A second command is refused with a message naming the running command. A stopped command waits up to five minutes for its outstanding steps to finish. Each drain round has a timeout of at most two minutes. While that cleanup is running, a new command is refused with a message explaining the wait.

When Longhorn cannot place a copy elsewhere, for example because the other nodes lack disk space, the command stops before draining and names the volumes. Evicted pods lose their emptyDir data.

Before asking for confirmation, kip also names volumes that retain a healthy copy elsewhere but have no eligible destination for the copy on the worker. For example, this happens with a volume that has two copies when only the server and this worker can hold copies. Those volumes stay degraded until a node that can hold the copy is ready.

For a worker whose host is powered off or gone for good, --force deregisters it without changing anything on the host. It first checks whether the host still answers, without a password prompt and with short connection and command timeouts. If the connection or machine ID check fails, or the address now belongs to another machine, kip deletes the worker's node, requests deletion of its Longhorn node, removes its address from the kip-managed firewalls of the remaining nodes and drops its record. It refuses when the host still answers with the worker's machine ID, because forced removal could leave k3s running on that host. Remove it without --force instead. When a volume has a replica on the worker and no healthy copy elsewhere, its only data may be on the worker, so --force also needs --accept-data-loss. A copy elsewhere counts only when its data is current and its node is ready and not fenced. The same rule applies when checking whether a resumed removal may wipe the host. The host keeps its k3s credentials for the cluster, so wipe it with /usr/local/bin/k3s-agent-uninstall.sh or reinstall it before it comes back. Otherwise, its agent or a kip command already in transit to the host when an earlier run was interrupted can register the node again. To resume an interrupted forced removal, use --force again and include --accept-data-loss if it was needed. A rerun without --force takes the normal path and needs the host.

A fenced worker requires --force for removal. If its fence is unfinished, complete the fence before removing it. When the host of a fenced worker still answers over SSH, kip removes it only while the fence still holds it: the drop-in is in the loaded k3s agent unit, and no agent, container, Longhorn process or Longhorn mount is left. Otherwise kip refuses and tells you to fence the worker again or finish its rejoin. This check applies to a resumed removal as well. A service the fence stopped whose other copies may be older blocks the removal until you decide about it with kip service start <name> --accept-stale-data or rejoin the worker, because removing the worker loses the newer data on it. Services whose only copy is on the worker are listed with how to restore them afterwards from a backup taken before the fence, and --accept-data-loss covers them as above.

Flag	Required	Description
--yes	No	Remove without asking for confirmation
--force	No	Remove a worker whose host is powered off or gone for good, without changing anything on the host
--accept-data-loss	No	With --force, remove the worker even when a volume's only data is on it
--timeout	No	Maximum wait for Longhorn to move copies and, separately, for the drain (default 30m)
--ssh-key	No	SSH private key. Overrides KIP_SSH_KEY and cluster.ssh_key

kip node fence

Fences a worker that stopped reporting, so that its old pods can never write to their volumes again, then lets Kubernetes start its services on other nodes. When a node dies explains when to use it.

kip node fence worker-1
kip node fence worker-1 --powered-off
kip node fence worker-1 --accept-stale-data

The worker must have been NotReady for at least a minute. kip then:

1. Records the worker as fencing, with your user name, the time, the method and the host's boot ID, after the SSH and confirmation checks below but before changing the host.
2. Connects to the worker over SSH without a password prompt. When the worker answers with its recorded machine ID, kip asks for confirmation unless you pass --yes, writes /etc/systemd/system/k3s-agent.service.d/90-kipper-fence.conf, reloads systemd and reads the loaded unit's assert list to check that the drop-in took effect. The drop-in asserts that its own path does not exist, so every start of the agent fails for as long as it is there, after a reboot as well. kip then runs k3s-killall.sh and checks that the agent is inactive or failed and that no container shim, Longhorn process or Longhorn device mount is left, and waits up to three minutes for the node to report NotReady. When the worker does not answer, kip asks whether the host is powered off and stays off until you run kip node add, unless you pass --powered-off. kip refuses to proceed if the server can still reach the worker's kubelet port or cannot check it. The check requires the worker's recorded address to resolve to an IPv4 address on the server and the probe to run successfully.
3. Checks each Kipper service with a single-node Longhorn volume on the worker. A copy is current when it is not failed and became healthy after it last failed. A service whose every volume has a current copy on another node is left to Kubernetes. Any other service is stopped by scaling its StatefulSet to zero, which kip records on the StatefulSet with the replica count to restore. A service paused by a migration whose data transfer runs on the worker is checked too, and kip records the replica count the migration saved. A migration keeps a service stopped while it carries the fence stop annotation. When its other copies may be older, kip shows each one with its timestamps and asks whether to start on them, or accepts --accept-stale-data as confirmation. Your answer is recorded and reused only while those copies are unchanged. A service with no copy on another node stays stopped. To bring it back without the worker, delete it with kip service delete <name> --delete-data and then restore it with kip backup restore <backup> --namespace <namespace>, because a restore skips objects that still exist. Use a backup taken before the fence, while the service ran. A later backup holds none of its files and restores the service stopped. After stopping the services and before applying the taint, kip prevents Kipper pods outside StatefulSets from restarting with single-node volumes. It marks migration data transfers failed and suspends Jobs. Pods without an owner need no further action. A pod with any other existing owner blocks the fence until you delete that owner. These checks apply only to pods with the app.kubernetes.io/managed-by=kipper label. kip warns about other pods with single-node Longhorn volumes on the worker. The taint lets Kubernetes start them using whichever copy Longhorn has available.
4. Taints the node node.kubernetes.io/out-of-service=nodeshutdown:NoExecute and records it as fenced. Kubernetes then deletes the node's pods and detaches their volumes.
5. After a minute, deletes what Kubernetes left behind: for each remaining Kipper pod with a volume, first its volume attachments, waiting up to two minutes for them to go, then the pod with no grace period. Before every deletion it checks again that the node is NotReady and tainted, and it stops when that changes or an attachment stays. It never removes finalizers.
6. Starts the stopped services you accepted, then waits up to --timeout in total for the services to run on other nodes and reports where each one runs, or why it stays stopped.

Run the command again to resume an interrupted fence. On a fenced worker, it repeats the service check, the cleanup and the report, which is how --accept-stale-data is given later. If a fenced worker that was powered off answers over SSH again, the rerun installs the drop-in and stops k3s on it. A rerun is refused once a rejoin has removed the taint.

While a worker is fenced, kip status and the console show it as OutOfService, and it never counts as Ready. kip node add <host> rejoins it. kip node remove <worker> --force removes it for good. kip node reboot refuses it, and k3s upgrades leave it alone.

kip disables automatic node-down pod deletion and fast RWX failover with two Longhorn settings: node-down-pod-deletion-policy is do-nothing, and rwx-volume-fast-failover is false. Normal RWX share-manager failover remains available. The fence warns when the first setting has another value. Do not delete a dead node's pods with kubectl delete pod --force: Kubernetes can force-detach volumes after a six-minute deletion timeout on an unhealthy node, without any of the fence's checks.

Flag	Required	Description
--powered-off	No	Confirm that the unreachable worker is powered off and stays off until kip node add
--yes	No	Stop k3s on a worker that answers over SSH without asking for confirmation
--accept-stale-data	No	Start services whose only other copies may be older than the data on the worker
--timeout	No	How long to wait for the worker's services to run on other nodes (default 10m)
--ssh-key	No	SSH private key. Overrides KIP_SSH_KEY and cluster.ssh_key

kip node list

Lists all nodes in the cluster with role, status, version, and IP. When a worker is recorded, the table also shows each node's network mode (public or private), whether it may hold copies of volumes, and the address kip reaches it at over SSH. The COPIES column shows yes for the server and for a worker that passed the disk check, no for one that was refused or stopped, and - for one never checked. A node without a record shows - in every recorded column.

kip node list

kip node reboot

Reboots a server or worker and waits for the cluster and previously ready apps or workloads to be ready again. Use it to apply a new kernel or service updates listed under Pending restarts in kip status.

kip node reboot
kip node reboot --yes --skip-backup
kip node reboot worker-1

Flag	Required	Description
--yes	No	Reboot without asking for confirmation
--skip-backup	No	Reboot without taking a backup first
--ssh-key	No	SSH private key for the node; overrides KIP_SSH_KEY and cluster.ssh_key
--timeout	No	How long to wait for node, component, and app readiness after reboot (default 20m)
--drain-timeout	No	Maximum wait for a worker to drain, including volume copies to other nodes (default 30m, minimum 1s)
--ignore-degraded-volumes	No	Reboot even when an attached volume is not healthy or its health cannot be read

Apps on a single-server cluster are unavailable during the reboot. For a server, the command:

1. Reads the server's boot ID over SSH. Stops if the server cannot be reached.
2. Asks for confirmation, unless you pass --yes.
3. Unless --skip-backup is set, takes a backup using the defaults from kip backup create and waits up to 30 minutes for completion. A failed or timed-out backup cancels the reboot.
4. Records apps with at least one ready replica, then reboots the server. If listing apps fails, it warns and skips the app readiness checks. If console-api uses :latest, it may start with a newer image. Before rebooting, run kip upgrade --check to check for required data updates.
5. Waits until the server reports a new boot ID. If it still reports the old ID after three minutes, the command stops with instructions for investigating the cause.
6. Waits for Kubernetes to report the new boot ID and for every node and component from kip status to pass two consecutive readiness checks.
7. Waits for each app recorded in step 4 to have a non-terminating pod that became ready after the reboot was scheduled. Ready, non-terminating pods on workers also count, even if they became ready before the server reboot. If time runs out, it lists apps still waiting for readiness or reports an error if readiness could not be checked.
8. Runs the host checks from kip status over SSH, including Pending restarts. These checks also run if cluster or app readiness times out, even when the API server is unavailable.

On a cluster with workers, the server reboot checks volume health before confirmation and again before rebooting, as described below. A single-server cluster skips this check.

When kip cannot read the node list, use kip node reboot --skip-backup to reboot the server without a backup. Add --yes for unattended use. kip retries the node list before rebooting. If workers are found, it applies the volume health checks described below, including --ignore-degraded-volumes and the exception for excess requested replicas. If the list is still unavailable, kip warns and proceeds. Volume replicas on the server are unavailable during the reboot.

A fenced worker cannot be rebooted: its k3s must stay stopped until kip node add rejoins it.

To reboot a worker, name it by its node name or address. Without an argument, or with the server's name or address, the command reboots the server as described above. For a worker, the command:

1. Checks the worker's machine ID against its recorded ID and requires Longhorn's drain policy to copy last healthy replicas to another node (kip upgrade configures this policy).
2. Lists volumes with healthy replicas on the worker and shows whether healthy copies exist elsewhere. Requires attached Longhorn volumes to report healthy before proceeding, because rebooting while a volume is degraded can reduce its availability further. Longhorn rebuilds degraded volumes on its own when healthy replicas and node capacity are available. Allow time for rebuilding, then rerun the command. A volume requesting more replicas than the cluster has nodes only triggers a warning when every node holds a healthy copy of it. A volume that has lost one of those copies still blocks. Use --ignore-degraded-volumes to proceed with unhealthy volumes or when volume health cannot be checked. For each volume whose healthy copies are all on the worker, checks for another node that can take a copy under Longhorn's scheduling rules. The node must be ready, and the node and disk must accept replicas, meet the volume's tag requirements, report themselves schedulable and have the space Longhorn requires. If every candidate is ruled out, the command names the volume, explains why each node was rejected and stops before cordoning. Without a destination for the copy, the drain would time out. This check repeats before cordoning. Each volume is checked against current capacity, so passing the check leaves the final placement decision to Longhorn. If the check fails to read Longhorn's state, the command warns and allows the drain to proceed, which still waits for Longhorn to copy the replicas.
3. Asks for confirmation and takes a backup, as for the server.
4. Records fully ready Deployments, StatefulSets and DaemonSets with at least one desired pod and checks volume health again, since it can change during the confirmation and backup. Then it cordons and drains the worker. Controllers can recreate evicted pods on other nodes; DaemonSet and mirror pods stay. Evicted pods lose their emptyDir data. Longhorn copies volumes with their last healthy replica on the worker to another node before the drain finishes.
5. Checks that every volume with a healthy replica still on the worker also has a healthy copy elsewhere. If draining or this check fails, the command reports the error, attempts to lift any cordon it set, and cancels the reboot. Unmanaged pods block the drain; --drain-timeout limits the wait.
6. Reboots the worker and waits for its new boot ID.
7. Lifts any cordon it set, then waits for every node and component to be ready and for the recorded workloads to be fully ready again. A worker that was already cordoned stays cordoned.
8. Lists attached volumes still reporting unhealthy, without waiting for recovery. A volume requesting more replicas than the cluster has nodes allows the next reboot with a warning if every node holds a healthy copy.
9. Runs the Pending restarts check on the worker.

The --timeout period starts after the reboot is scheduled; backup and drain time are separate. Boot times vary by host. If the wait times out, run kip status to check progress. If SSH remains unreachable, check the host through your provider’s console.

kip auth verify

Proves your OIDC identity authenticates and authorizes against the cluster, the same check the installer runs inline. Run it after a headless install, or any time you want to confirm the login path works end to end.

kip auth verify

  ✔  Authenticated and authorized as oidc:admin@shop.kipper.run

It exits non-zero when the API server rejects your token or an admin identity is denied access, so it doubles as the CI signal that the authenticator is live. It never changes your kubeconfig.

kip auth kubeconfig

Converts a legacy or admin kubeconfig to authenticate through your kip auth login session. Fresh installations normally use this per-user configuration already.

kip auth kubeconfig

  ...  Checking that your login reaches this cluster (up to a minute)

  ✔  /Users/anna/.kip/clusters/shop.kipper.run.yaml now authenticates as your OIDC identity
     kubectl runs `kip auth kubectl-token` for short-lived tokens.

The check comes first because the file being replaced is often the only credential that reaches the cluster from this machine. It is the same proof kip auth verify makes: your session token is sent to the API server, which has to accept it as you and grant you access. A cluster that answers anything else keeps the credential it has.

  ✗  This cluster did not accept your login: the API server rejected the token
     /Users/anna/.kip/clusters/shop.kipper.run.yaml is unchanged, so it still reaches the cluster.
     Run 'kip cluster ca status' to see what the API server has loaded,
     and 'kip auth verify' to re-check the login once it is fixed.

this cluster does not accept your login, kubeconfig unchanged

A rejected token usually means the API server has no authenticator for your issuer. kip cluster ca status says so directly, reporting that the authentication config names no issuer.

The command also preserves the existing kubeconfig when it cannot reach the API server:

  ⚠  Could not reach the API server to check your login: Get "https://203.0.113.10:6443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 203.0.113.10:6443: i/o timeout
     /Users/anna/.kip/clusters/shop.kipper.run.yaml is unchanged and still works.

login could not be checked, kubeconfig unchanged

Sign in with kip auth login before running it:

  ✗  Your login could not be checked, so /Users/anna/.kip/clusters/shop.kipper.run.yaml is unchanged.

not authenticated. Run: kip auth login

After this, the kubeconfig carries no credential at all: kubectl obtains a token valid for a few minutes each time it needs one, every action in the Kubernetes audit log names your email, and removing a person's access means removing their account rather than rotating certificates. The admin certificate stays on the server as the break-glass credential (see Architecture).

The file also names the cluster it signs in against, so run this once per cluster on each machine that holds one. kip cluster domain keeps that name current when a cluster's domain changes.

kip auth kubectl-token

The credential plugin the rewritten kubeconfig calls. kubectl runs it automatically whenever it needs a token; there is normally no reason to run it by hand.

kip auth kubectl-token --cluster-domain shop.kipper.run

{"apiVersion":"client.authentication.k8s.io/v1","kind":"ExecCredential","status":{"token":"eyJhbGciOi…","expirationTimestamp":"2026-07-21T09:15:00Z"}}

--cluster-domain names the cluster whose session to serve, and kip writes it into every kubeconfig it renders, so kubectl passes it for you. It has to come from the file because kubectl tells a credential plugin nothing about which kubeconfig invoked it: each kubeconfig therefore signs in against its own cluster whatever kip cluster use is set to, which is what lets you work across two clusters in two terminals.

kip cluster use still decides which cluster the rest of kip talks to. It has no say over kubectl.

When the session has expired entirely it prints session expired. Run: kip auth login on stderr, which kubectl surfaces verbatim. A kubeconfig written before this flag existed prints this kubeconfig does not say which cluster it authenticates to. Regenerate it with: kip auth kubeconfig, and no token is issued until you do.

kip auth reset-password

Generates a new admin password, writes it to Dex, displays the new credentials, and restarts Dex.

kip auth reset-password

Uses your local session or the authentication configured in the cluster's kubeconfig. If the kubeconfig requires a session and none is available, kip retrieves the server's admin credentials over SSH and keeps them in memory.

Pass --ssh-key <key> to use SSH explicitly. Resetting the password requires permission to read and update Dex's ConfigMap and patch its deployment, or SSH access to the server as root or as the cluster's SSH user with passwordless sudo.

Flag	Required	Default	Description
--ssh-key	No	KIP_SSH_KEY, then the key saved for the cluster	Private key for resetting the password over SSH

kip discover

Find Kipper-labeled workloads on the cluster that have no owning Kipper CR. Read-only.

kip discover

Kipper considers a Service or App or Volume or Function to "exist" when its CR exists. A Deployment, StatefulSet, or PVC carrying app.kubernetes.io/managed-by=kipper without a matching CR is drift. It will not show up in kip service list or in the console, even though it occupies cluster resources.

kip discover lists each orphan and prints a suggested kip command that recreates it as a proper CR with the workload's current settings. Run the suggested command to bring the orphan under management. The controller adopts the existing workload to match the new CR, no deletion needed. Edit the suggested command first if you want to change anything.

For Deployments, the suggestion uses kip app deploy with --image, --port, --memory, --cpu, --env, --secret, --replicas. For StatefulSets, kip service add plus the service type (postgres, redis, and so on), with --storage, --memory, --cpu. For PVCs, kip volume create with --size. Functions print a comment instead, because the source code is not on the workload and has to come from kip function create.

Workloads in the kipper-system namespace are skipped. The console, console-api, and zot legitimately have no owning CR.

kip cert list

Lists every certificate on the current cluster with its host, current state, the age of the last state change, and cert-manager's reason when it is not ready. The list includes platform hosts and app routes.

kip cert list

kip cert email

Shows or updates the Let's Encrypt email used for TLS certificates. This is the email cert-manager uses when registering with Let's Encrypt for automatic certificate issuance.

kip cert email                    # show current email
kip cert email admin@example.com  # update email and renew stuck certs

When updating, the command re-registers with Let's Encrypt using the new email and triggers renewal for any certificates that are stuck or failed. Certificates usually come through within 1-2 minutes.

See Domains & SSL: Troubleshooting certificates for common certificate issues.

kip ai

kip ai covers two related things: choosing which AI provider Kipper itself uses (for log analysis, Dockerfile generation, diagnostics), and installing a private LLM stack inside the cluster that your apps can call.

kip ai configure / kip ai status

Configure which AI provider Kipper uses. Supports Claude (Anthropic), OpenAI, and Ollama (self-hosted).

kip ai configure                                     # interactive setup
kip ai configure --provider claude --key sk-ant-...  # non-interactive
kip ai configure --provider ollama                   # self-hosted (no key needed)
kip ai status                                        # show current config and bundle health

Flag	Required	Default	Description
--provider	No	—	AI provider: claude, openai, ollama
--key	No	—	API key (not needed for Ollama)
--model	No	—	Model override
--ollama-url	No	http://localhost:11434	Ollama server URL

kip ai admin create

Seed the first LibreChat admin account after kip ai install. The bundle ships with open registration off, so an admin must be created once before anyone can log in. Runs npm run create-user inside the running librechat pod via the Kubernetes API. No kubectl required.

kip ai admin create --email you@example.com --name 'Your Name' --password 'a-strong-password'
kip ai admin create --email you@example.com --name 'Your Name' --username alice --password '...'

Flag	Required	Default	Description
--email	Yes	—	Admin email address
--password	Yes	—	At least 8 characters
--name	Yes	—	Display name shown in the chat UI
--username	No	local part of --email	LibreChat username

kip ai install / kip ai uninstall

Install or remove the in-cluster AI bundle (Ollama and LibreChat). The full walkthrough is on the AI Bundle page.

kip ai install                            # detect tier, pick a model, install
kip ai install --host chat.example.com    # override the chat hostname
kip ai install --model qwen2.5:7b-instruct-q4_K_M
kip ai uninstall                          # remove the bundle and wipe its data

kip ai uninstall is destructive: it deletes the workload, the PVCs (model cache, chat history, MongoDB), credentials, and the kipper-ai namespace. Take a blocking snapshot first with kip ai backup --name pre-uninstall --wait if you want to preserve any of it. Uninstall refuses by default while a Kipper AI backup is still in flight; pass --force to override (you'll get an unrestorable snapshot if you do).

Flag	Command	Default	Description
--host	install	chat.<cluster-domain>	External chat UI hostname
--model	install	tier-appropriate Qwen 2.5	Ollama model tag to preload
--pvc-size	install	10/30/60 GiB by tier	Model cache PVC size
-y, --yes	install	false	Skip the auto-configure prompt

AI bundle health on the Platform page

The Platform page surfaces a drift check for each installed AI bundle. The check reads the kipper-ai-bundle-state and kipper-rag-bundle-state ConfigMaps in kipper-ai and confirms every expected workload (Ollama, LibreChat, AnythingLLM, Qdrant, plus their Ingresses) still exists.

If a bundle was installed but a workload has disappeared, the panel renders the missing resources and points at the kip ai install command that reconciles. This is the same diagnosis path that used to require ssh + kubectl get.

kip ai backup / kip ai restore

Velero-backed snapshot of the AI bundle. Velero is a Kipper system component, so no extra setup is needed.

kip ai backup                            # auto-name, exits after 60s warmup
kip ai backup --name pre-upgrade         # named, exits after 60s warmup
kip ai backup --name pre-upgrade --wait  # block until completion (CI scripts)
kip ai backup show --name pre-upgrade    # detailed status (phase, items, errors)
kip ai backup list
kip ai backup delete --name pre-upgrade           # exits after 60s warmup
kip ai backup delete --name pre-upgrade --wait    # block until Backup CRs are gone
kip ai restore --name pre-upgrade        # requires kipper-ai uninstalled first

Flag	Command	Default	Description
--name	backup	timestamped	Snapshot name
--wait	backup	false	Block until completion instead of exiting after the 60s warmup
--wait	backup delete	false	Block until both Backup CRs are gone instead of exiting after the 60s warmup
--name	backup show	—	Snapshot to inspect (required)
--name	backup delete	—	Snapshot to delete (required)
--name	restore	—	Snapshot to restore (required)

kip app info

Shows an app's status, scaling, and CPU and memory settings.

kip app info backend --project shop --environment prod

  App: backend

  Status:  running, 2/2 ready
  Image:   ghcr.io/acme/api:v1
  Nodes:   server-1 (1 pod), worker-1 (1 pod)

  Autoscaling: on (CPU 70%)
  Desired: 2 (set by autoscaling)   Min: 2   Max: 5
  Ready: 2
  cpu: target 70%, current 12%
  Last scaled: 2026-10-10 20:56 UTC (2h ago)

  Resources (profile jvm, from the pod template)
  CPU:     request 300m, limit 1
           the autoscaler tracks CPU, so Kipper leaves the CPU request alone
  Memory:  request 2Gi, limit 2Gi
           sized by Kipper; latest recommendation: request 1536Mi, limit 2Gi

  Conditions
  ✔  ServiceBindingsReady: BindingsInjected
           every declared service binding was injected
  ✘  BindingDatabasesReady: DatabaseCreateFailed
           binding "db": creating "shop" in service "db": no running pod found for service "db"

On a cluster with workers, the Nodes line names the nodes the app's pods run on, with the number of pods on each and any pods not yet scheduled. Finished pods are excluded. A single-node cluster leaves it out.

The Conditions section shows the app's controller-reported conditions: ✔ for True, ✘ for False, and ⚠ for an unknown status. Read each reason and message for context: RolloutComplete: False can mean pods are still being replaced, while RouteReady: True can include a warning about a shared route name.

Flag	Required	Description
--project	No	Project name
--environment	No	Target environment

The resource lines show the app's pod-template values. During a rollout, some pods may still use earlier values. Each resource includes a sizing explanation:

• automatic: Kipper manages sizing and shows its latest recommendation when it differs from the template. Recommendations take effect when sizing rules allow. Paused tuning keeps the current size; autoscaler targets restrict changes to the tracked resource.
• bounded: you set a request below a limit, and Kipper moves the request between them.
• fixed: Kipper uses the size you set.
• held: ownership is unknown, so Kipper keeps the values until you set them.

For automatic and bounded resources, the sizing explanation accounts for autoscaling: Kipper keeps tracked CPU requests unchanged and increases tracked memory only after an out-of-memory kill. If the policy is invalid and you lack permission to read the autoscaler, sizing is shown as unknown.

See Your own CPU and memory values.

kip app update

Updates a deployed app's image, resource settings, or health check. Changes to the pod configuration trigger a rolling update.

kip app update backend --image ghcr.io/acme/api:v2.1.0
kip app update backend --profile jvm
kip app update backend --memory-request 512Mi --memory-limit 2Gi
kip app update backend --tuning auto
kip app update backend --health-path /actuator/health/readiness --health-startup-timeout 600

Flag	Required	Description
--image	No*	New container image
--profile	No*	Resource profile: lightweight, standard, compute-heavy, memory-heavy, or jvm
--memory / --cpu	No*	Set request and limit to the same value
--memory-request / --cpu-request	No*	Minimum request for automatic tuning
--memory-limit / --cpu-limit	No*	Container limit and maximum request for automatic tuning
--tuning auto	No*	Clear the CPU and memory values so Kipper sizes the app on its own
--health	No*	Health check type: http, tcp, none (disable the app readiness check), or auto to remove a declared check
--health-path	No*	Path for an http check, e.g. /ready. Selects http when --health is not given
--health-port	No*	Port to check, when it is not the app port
--health-startup-timeout	No*	Seconds from pod creation to readiness before the pod is reported as stuck, including image pulls (default 300)
--health-timeout	No*	Timeout for each check in seconds (default 2)
--closed	No*	Publish only the route's public paths; use --closed=false to reopen. See closing a route
--project	No	Project name
--environment	No	Target environment

* Supply at least one update flag: --image, --profile, a CPU or memory flag, --tuning auto, --redirect-from, --internal-path, --public-path, --closed, --health or a --health-* flag.

A request below its limit defines a range for automatic tuning. A request or limit supplied alone sets a fixed size; a request above its limit is rejected. Use either --memory or the separate memory request/limit flags. The same rule applies to CPU. --tuning auto cannot be combined with CPU or memory values. Changing the profile preserves explicit resource values. See Your own CPU and memory values.

The health flags also work with kip app deploy. Updates merge into the existing check; switching types clears unsupported settings. Changing the readiness probe triggers a rollout, while changing only the startup timeout leaves pods running. See Health checks and rollouts.

kip app route

Shows, sets or removes an app's public route.

kip app route show backend --project shop
kip app route set backend --host shop.example.com --path /api
kip app route remove backend --yes

show prints the host, path, URL and configured route settings: authentication, internal and public paths, redirects, the CSP allowlist and rate limit. It also displays the controller's RouteReady condition when available. Routes without an explicit host use a hostname derived from the app name, environment and cluster domain.

set updates the host and path you provide while preserving other route settings. Updating an existing route's host or path keeps pods running. A new route uses the supplied host or the default hostname, with / as the default path. Adding a route may restart pods to add Kipper's instance proxy.

For kipper.run, use the cluster's domain (such as lab.kipper.run) or a name such as shop--lab.kipper.run. The gateway routes these names to your cluster, and its wildcard certificate covers one label under kipper.run. For a custom host, point its DNS record at the server so its certificate can be issued. set waits briefly for the controller and reports whether publication is confirmed.

remove deletes the route and its settings, then waits briefly for the route's Ingresses to be removed. The app remains accessible inside the cluster and through links. Pods may restart to remove the instance proxy. Adding a route later uses defaults rather than restoring the removed settings. The command lists the settings being removed and asks for confirmation. Pass --yes to skip the prompt; this flag is required without an interactive terminal.

Flag	Command	Description
--host	set	Hostname to serve the app on
--path	set	Path the app is served under (default / for a new route)
--yes, -y	remove	Remove without asking
--project	all	Project name
--environment	all	Target environment

kip app scale

Sets the replica count for a deployed application. Give the count as the second argument or with --replicas. A count is required to prevent accidental scaling to zero.

kip app scale backend 3

Flag	Required	Description
--replicas	If the second argument is omitted	Number of replicas. Equivalent to the second argument.
--project, --environment	No	The app's project and environment

For a stopped app, the new replica count takes effect on start. Use kip app stop to take an app out of service while preserving its count and recording a reason.

For an app with bounds, counts outside them (including 0) are refused with a message naming the bounds. While autoscaling is on, every count is refused, because the autoscaler sets it. kip app deploy --replicas follows the same bounds on an existing app; on an autoscaled app it stores the count for when autoscaling is off and prints a note saying so.

kip app autoscale

Configures automatic scaling for an app. The minimum and maximum are bounds on the app's replica count whether the autoscaler or you set it. See Autoscaling.

kip app autoscale backend --min 2 --max 10 --cpu 70
kip app autoscale backend --max 15
kip app autoscale backend --status
kip app autoscale backend --off
kip app autoscale backend --remove

Flag	Required	Description
--min	No	Minimum number of replicas. 1 on a first enable
--max	No	Maximum number of replicas. 5 on a first enable
--cpu	No	Target CPU use as a percentage of the request. 0 removes the target
--memory	No	Target memory use as a percentage of the request. 0 removes the target
--status	No	Show desired replicas, bounds, ready pods, current use and autoscaling problems
--off	No	Switch autoscaling off, retaining the desired count within the bounds
--remove	No	Remove the bounds from an app whose autoscaling is off
--project, --environment	No	The app's project and environment

Any of --min, --max, --cpu or --memory switches autoscaling on. Flags you leave out keep their stored values. Missing bounds default to 1 and 5. When neither target is set, CPU defaults to 70%. A change to the bounds moves the stored count into them and prints the move.

--off keeps the count the autoscaler last set, moved into the bounds if it lies outside them. A stopped app keeps its stored count. A failed Deployment read also falls back to the stored count and prints a warning. --remove refuses while autoscaling is on; pass --off --remove to do both. --status takes precedence over the other flags. When the App's AutoscalingReady condition is not True, it adds a line starting ⚠ Autoscaling is not ready ( with the reason and message the cluster reports.

--off and --remove ignore --min, --max, --cpu and --memory and print a notice naming the flags they ignored. To change a setting and switch off, run the change first and --off after it.

The cluster refuses a change that leaves the autoscaling block invalid, and switching off counts as a change. On an app whose stored minimum is above its maximum, --off (and so --off --remove) fails with minReplicas must not exceed maxReplicas. Set valid bounds with --min and --max first, then switch off. An app stored with autoscaling on, a minimum above 1 and no maximum fails --off with set maxReplicas with minReplicas, or leave both out to remove the bounds, so first set a maximum no lower than the stored minimum, for example kip app autoscale backend --max 5 for a minimum of up to 5. In both cases kip names the command that fixes the block. See When settings do not add up.

kip app stop / kip app start

kip app stop shuts down an app’s pods. kip app start resumes the app using its saved configuration.

kip app stop backend --reason "not needed until the next campaign"
kip app start backend

Flag	Command	Description
--reason	stop	Optional stop reason, up to 500 bytes. On an already stopped app, only an explicit --reason changes the reason
--for-migration	stop	Marks a new stop as a migration write freeze, which stays on the source. Existing stop records keep their original type
--project, --environment	both	The app's project and environment

Stopping preserves the app’s configuration, replica count, autoscaling settings and data. Its route returns a "this app is stopped" page. For an app scaled to zero without a stop record and without autoscaling, use kip app scale to raise the count. See Stopping and starting an app.

kip app env / kip app secret

Manage environment variables and secrets for an application.

kip app env set backend LOG_LEVEL=debug       # set env var
kip app env list backend                      # list with values visible
kip app env delete backend LOG_LEVEL          # remove

kip app secret set backend DATABASE_URL       # interactive hidden prompt
kip app secret list backend                   # keys only, values masked
kip app secret reveal backend DATABASE_URL    # show a single value
kip app secret rollback backend DATABASE_URL  # restore previous value
kip app secret delete backend DATABASE_URL    # remove

Every command that changes configuration saves it without restarting the workload; add --restart to apply it immediately. See Secrets and configuration.

Secrets can also be set at deploy time with --secret on kip app deploy (repeatable; a bare KEY prompts with hidden input), so the app never starts without them.

See Secrets & Environment for full documentation.

kip app link / kip app unlink

Connect apps so one can reach another via a URL.

kip app link domain-service api-gateway             # internal URL (backend-to-backend)
kip app link domain-service webapp --public          # public URL (for frontend apps)
kip app unlink domain-service api-gateway            # removes DOMAIN_SERVICE_URL

Use --public when linking to a frontend app that runs in the browser. It injects the target's public HTTPS URL instead of the internal Kubernetes DNS.

See Deploying Apps: Linking apps for details.

kip service

Manage stateful services (databases, caches) with persistent storage.

kip service add postgres --name mydb         # deploy PostgreSQL
kip service add redis --name cache           # deploy Redis
kip service add postgres --name mydb --copies 2 # keep two copies of the data on two nodes
kip service update mydb --copies 2 --wait    # convert a running service to two copies
kip service list                             # list all services
kip service info mydb                        # show connection details, password masked
kip service info mydb --show-secrets         # include the password
kip service share mailhog --expires 72h      # shareable link to a service web UI
kip service import mydb --file dump.sql      # load a database dump
kip service export mydb --file nightly.dump  # dump a database to a local file
kip service restore-dump mydb --to mydb-new  # restore mydb's consistent dump from a backup into an empty service
kip service recreate-dump mydb               # replace a faulted dump volume, data volume left unchanged
kip service credentials                      # check every service owns its credentials
kip service rotate-password mydb             # new password in the engine, the Secret and every bound app
kip service delete mydb --delete-data        # delete the service and its volume
kip service start mydb                       # start a service kip node fence left stopped

kip service add --copies 2 keeps two copies of the service's data, each on a different node that can hold copies. It is refused on local storage, on a cluster that offers one copy (see kip cluster storage), and when fewer than two nodes can hold copies. The default is one copy. On a cluster with in-cluster backups, kip warns once that a second copy does not replace external backups. kip service update <name> --copies N [--wait] changes the copies of an existing service, which needs the project owner role. Requesting two copies runs the same checks as creation, unless the volume already has at least two copies. If the cluster switched to Longhorn but the service's data is still on local storage, move that data to Longhorn before requesting two copies. kip saves the requested copy count. Lowering the copies stops the service: Kipper lowers the count only while the volume is attached on the server with nothing able to write to it, Longhorn removes the extra copy away from the server, and Kipper starts the service again once the copy is gone. A volume created without Longhorn's revision counter also stops the service once when its copies are raised, to switch the counter on, and the change says so while the service is stopped. Setting the copies back to 2 before Longhorn has removed the copy starts the service again at two copies. If Longhorn has placed the server copy for the change already, Kipper keeps it, and Longhorn removes one of the other copies first, so the volume ends at two. On a cluster that offers one copy, setting them back to 2 needs an admin. Without --wait, it waits up to 30 seconds to print the controller's first response. --wait follows the conversion until every copy is healthy, for up to three hours. While waiting, the command fails if another client changes the copy count or the object is deleted and created again. A blocked conversion exits non-zero with the reason. A wait for Kubernetes or Longhorn, such as for the volume to detach after the stop or for Longhorn to remove the extra copy, is reported as pending, and --wait keeps waiting through it until its limit. The copies cannot change again while Longhorn builds a copy, or while the service moves to Longhorn. See Changing the copies of a running service.

kip service share accepts --project and --environment to target a service outside the active project, --expires (up to 720h) for the link lifetime, and --label for a note in the listing. Use --list to see a service's links, --revoke <id> to kill one, and --revoke-all plus --rotate-key to contain a leak.

kip service rotate-password takes --yes to skip the confirmation, plus --project and --environment. Without --project or --environment, it finds the service by name across the cluster. See Rotating a service password.

kip service restore-dump takes --to (required, an existing Postgres service without tables), --backup (default: the latest completed backup that covers the namespace), --project and --environment. See Restore a Postgres dump into a new service.

kip service info prints a warning line while a Postgres service's dump volume is faulted, with the command that replaces it. kip service recreate-dump refuses unless Longhorn reports the dump volume as faulted and the dump volume, StatefulSet and any existing pod belong to the service. It also refuses while the service is moving to Longhorn, while a backup of the namespace or a restore into it is still running, or if it cannot read the backup and restore status. It repeats all these checks after confirmation, even when --yes skips the prompt. It then deletes the dump volume, restarts the pod and waits up to three minutes for the controller's new dump volume. A pod on a node that is down stays terminating, and kip names kip node fence as the next step. It takes --yes to skip the confirmation, plus --project and --environment. Without --project or --environment, it finds the service by name across the cluster. See Consistent database dumps.

See Stateful Services for full documentation.

kip service start

Starts a service that kip node fence stopped. The service starts at once when every one of its volumes has a current copy on a node that is not fenced. A rejoined worker's copies count too. When its copies may be older than the data on the fenced worker, kip shows each copy with its timestamps and asks whether to start on them, or accepts --accept-stale-data as confirmation. Writes made since those copies were current are lost. A service whose only copy is on the fenced worker starts after the worker rejoins. Once its volumes no longer exist, for example after the worker was removed, kip says how to restore it from a backup taken before the fence and, on a terminal, offers to clear the fence's stop mark. The service stays stopped either way. The command takes --project, --environment and --ssh-key, and without --project or --environment it finds the service by name across the cluster. It holds the node lifecycle lease, so it never runs alongside a fence or a rejoin. It refuses a service whose move to Longhorn is unfinished and names the command that finishes it.

kip service storage longhorn

Moves a service's data volume from local storage onto a single-copy Longhorn volume, on a cluster that kip cluster storage longhorn switched to Longhorn.

kip service storage longhorn <name> [--project <project>] [--environment <env>] [--no-backup] [--rollback] [--yes] [--ssh-key <path>]

The service is stopped for the whole copy. The prompt estimates the downtime. kip keeps every other pod off the volume, copies the data into a new Longhorn volume at least as large as the declared size and the data, flushes it, and lets Longhorn detach the volume. A separate verify pod then compares every file of the copy, read from a fresh mount, with the original, and kip starts the service on the new volume. The copy pod may use up to 1 GiB of memory, the verify pod up to 128 MiB. A file or directory name containing a line break stops the copy before anything is copied, because the comparison cannot represent it. A database dump volume is recreated empty on Longhorn. The old local directory is kept until kip cluster storage prune.

• Backups: on a cluster with backups, a completed backup covering the service's project is required, and no backup, restore or migration transfer may be running. On a cluster without backups, --no-backup moves the service anyway.
• Room: the Longhorn volume must fit on the server. kip checks this before stopping the service, and again once the service has stopped if it wrote more data in between. If the copy still runs out of room, kip grows the volume once, to the larger of 125% of the planned size and the data plus 30%, after checking Longhorn's room and the quota again, including the dump volume a service with consistent backups gets on Longhorn at the new size. A copy that runs out of room on the grown volume leaves the service stopped for --rollback.
• Node and quota: before stopping the service, kip checks that the server keeps room for the larger of the copy pod, the verify pod and the service once the service's pods are gone, and that the namespace's ResourceQuota admits the larger of the copy and verify pods, which run one at a time. Scoped quotas count only the pods their scopes cover, and kip refuses the move if it cannot evaluate a scope, naming the affected quota. The node check is a point-in-time check of requests, not a reservation.
• Interruptions: run the command again and it continues from the step it reached, after checking again that the service is stopped and other pods are kept off its volume. A finished copy is kept unless another pod mounted the new volume or its protection was lifted in the meantime, in which case kip copies the data again. A move that resumes after its copy was verified and before it committed verifies the copy again, and copies again if it no longer matches. Until the move commits, just before the service starts on Longhorn, --rollback undoes it and starts the service on its local volume. After the commit the move can only be finished. If the move record is missing, --rollback rebuilds it from the remaining volumes.
• Deleting: kip service delete and project deletion are blocked while the move is unfinished, and an admission policy refuses to delete the namespace. A new move stops before stopping the service if its namespace or project is already being deleted.
• Confirmation: the command asks before it stops the service. --yes skips the question, and without a terminal it is required.

Without --project or --environment, it finds the service by name across the cluster. It holds the node lifecycle lease.

kip project

Manage projects and environments.

kip project create blog --environments test,acc,prod
kip project create blog --display-name "example.com Domain Platform" --environments test,acc,prod
kip project list
kip project add-env blog prod
kip project remove-env blog staging
kip project delete blog
kip project update blog --copies 1 --wait                      # set the copies of the project's shared storage

kip project remove-env deletes the matching namespace and everything in it. You'll be asked to type the environment name to confirm. See Projects & Environments for details.

kip project update

kip project update <project> --copies 1 [--wait] sets the copies of a project's shared storage, the shared-uploads volume of each environment, and follows the conversion like kip volume update. The project must have shared storage. Shared storage keeps one copy in this release, so 2 is refused, except to stop a reduction to one that is blocked or pending. The volumes then keep the copies they have. Changing a project's spec needs cluster admin rights.

kip project members

Manage who can access a project and what they can do.

kip project members list blog
kip project members add blog jordan@acme.com deployer
kip project members remove blog jordan@acme.com

Roles are owner, deployer, or viewer. See Project Members for what each role can do.

kip project use

Set a persistent project context for the current cluster, so the rest of the kip commands do not need --project and --environment flags every time.

kip project use blog           # active project: blog, default environment
kip project use blog/test      # active project: blog, environment: test
kip project use blog test      # same, with a space instead of a slash
kip project use --clear              # forget the active project on this cluster

The active project is stored per cluster in ~/.kip/config.yaml. After setting it, kip service list, kip volume list, kip function list, every kip job command and every kip app command, including kip app deploy, kip app logs, kip app rebuild and kip app webhook, resolve to the active project's namespace automatically. kip app promote is the exception: it moves an app between environments, so it always takes --project. kip cluster list shows the active project on each cluster line.

Explicit --project flags still win. Passing --project other-name switches that single command to the other project; the persisted environment is not carried over so a different project never inherits a stale environment.

See Projects & Environments for full documentation.

kip app promote

Promote an app from one environment to the next (copies the image tag only).

kip app promote backend --from test --to acc --project blog
kip app promote --all --from acc --to prod --project blog

See Projects & Environments for full documentation.

kip function

Manage serverless functions. Functions scale to zero when idle and spin up on demand. Alias: kip fn.

kip function create process-image --image myregistry/processor:v1 --port 8080 --project blog
kip function create db-sync --trigger postgres --source mydb --query "SELECT * FROM events WHERE processed = false" --project blog
kip function create cache-worker --trigger redis --source cache --list jobs --project blog
kip function list --project blog
kip function info process-image --project blog
kip function logs process-image --project blog
kip function logs process-image --project blog -f
kip function delete process-image --project blog

kip function info <name> shows a function's status, trigger, image and URL. On a cluster with workers it also names the nodes its pods run on, as kip app info does.

kip function create flags

Flag	Required	Default	Description
--image	Exactly one of --image or --code-file	—	Container image for the function
--code-file	Exactly one of --image or --code-file	—	Source file for an inline function
--runtime	With --code-file	—	Runtime for an inline function: node or python
--trigger	No	http	Trigger type: http, cron, postgres, mysql, redis, minio
--schedule	With --trigger cron	—	Cron expression, for example "0 2 * * *"
--port	No	8080	Port the function listens on
--source	No	—	Service name for event triggers
--query	No	—	SQL query for postgres/mysql triggers
--mark-done	No	—	SQL to mark rows as processed
--list	No	—	Redis list name for redis triggers
--bucket	No	—	MinIO bucket name for minio triggers
--project	No	Active project, otherwise the namespace of a function with this name, otherwise default	Project name
--environment	No	—	Target environment

See Serverless Functions for full documentation.

kip job

Run one-off tasks and scheduled jobs. Job commands use the active project unless you specify --project. If no project is active or specified, they use the default project.

kip job run --name migrate --image myapp:latest --command "npm run migrate" --project blog --environment test
kip job schedule --name cleanup --image myapp:latest --command "python cleanup.py" --cron "0 3 * * *"
kip job list --project blog
kip job history cleanup
kip job delete cleanup

See Jobs & Scheduled Tasks for full documentation.

kip volume

Create shared persistent volumes that can be mounted by multiple apps. Backed by Longhorn with ReadWriteMany access.

kip volume create uploads --size 5Gi --project blog --environment test
kip volume mount uploads webapp --path /data --project blog --environment test
kip volume unmount uploads webapp --project blog --environment test
kip volume list --project blog
kip volume update uploads --copies 1 --wait --project blog --environment test
kip volume delete uploads --delete-data --project blog --environment test

kip volume list shows each volume's copies: 1, not set for a volume created before Kipper managed copies, or - on an older cluster. kip volume update <name> --copies 1 [--wait] converts a volume to one copy on the server, for example a volume from an earlier release that has three. The conversion waits until no app mounts the volume, stops the volume's backup holder and attaches the volume on the server, where Longhorn removes the extra copies once Kipper lowers the count. While an app still mounts it, the conversion reports Blocked, names the pods and gives kip volume update <name> --copies 2, which stops the change. Shared volumes keep one copy in this release. A request for 2 is accepted only to stop a blocked or pending reduction to one. The volume then keeps the copies it has, and if Kipper has already lowered Longhorn's requested count while holding the volume, it raises that count back to the number of copies present. The status and --wait count the copies present against the requested two copies. A Longhorn count of three from an earlier release stays at three, because Kipper lowers a count only while it holds the volume for a reduction to one. Canceling the reduction while Kipper builds the server's copy keeps that copy once Longhorn has placed it on the server, so the volume keeps every copy it had plus the server's, for example two copies when it had one. A server copy Longhorn has not placed yet is dropped from the count. On a cluster that offers one copy, only an admin can stop it.

kip volume create flags

Flag	Required	Default	Description
--size	No	5Gi	Volume size
--copies	No	1	Copies of the data. Shared volumes keep one copy in this release, so 2 is refused at creation. kip volume update --copies 2 is accepted only to stop a blocked or pending reduction
--project	When no project is active	Active project	Project name
--environment	No	—	Target environment

Before creating the volume, kip volume create checks whether the server has capacity for the volume's copy, using Longhorn's reserved space, over-provisioning and minimal-available settings. If capacity is insufficient, it refuses creation and reports the node with the most room and its remaining capacity. If Longhorn's state cannot be read, it prints a warning and proceeds with creation.

kip volume mount flags

Flag	Required	Default	Description
--path	No	/data	Mount path inside the container
--project	No	Active project, otherwise the namespace of the named app	Project name
--environment	No	—	Target environment

Mounts are recorded on the volume and applied to the app automatically, so they survive image updates and redeployments. kip volume unmount removes the mount again; the volume and its data stay available for other apps. Deleting a volume needs --delete-data because it permanently destroys the data.

While backups are configured, each shared volume has a holder pod, volume-holder-<name>, that keeps it mounted so backups include its files even when no app mounts it. A project's shared storage also has a holder named shared-uploads-holder. kip volume list prints a warning line for a volume whose holder cannot run, and kip project list prints one for each environment whose shared-storage holder cannot run. See Shared volume backups.

See Storage for full documentation.

kip user

Manage cluster users and roles. Kipper has three cluster roles:

• admin: manage the cluster, settings, users and platform components.
• deployer: sign in and use the console's AI assistant.
• viewer: sign in to the console.

In the console, admins can work in every project. Other users get access to project apps, services and data through project membership.

kip user list
kip user add dev@example.com --role deployer
kip user add pm@example.com --role viewer --password secret123
kip user invite --email dev@example.com --role deployer      # invite a developer
kip user invite --email ops@example.com --role admin --expires 24h
kip user role dev@example.com admin                # change role
kip user monitoring grant dev@example.com          # let them open Grafana
kip user monitoring revoke dev@example.com
kip user remove dev@example.com
kip user import --from-backup daily-20261101030000 # bring accounts back after recovery onto a new cluster
kip user import dex-snapshot.yaml                  # bulk-import Dex users from a snapshot
kip user import dex-snapshot.yaml --restart-dex    # also roll Dex so the new config takes effect

kip user monitoring

Grant or revoke Grafana access for an existing Kipper user:

kip user monitoring grant <email>
kip user monitoring revoke <email>

This grants access to every project's metrics and logs, plus Explore and dashboard editing. Cluster admins have access automatically; revoking an explicit grant does not remove their access.

Use kip user list to check the MONITORING column. See Accessing Grafana for sign-in and permission details.

kip user import

Adds console accounts and sign-in connectors from a backup or a captured snapshot to this cluster. Backup imports also add roles.

kip user import --from-backup <backup> [--restart-dex]
kip user import <snapshot.yaml> [--restart-dex]

With --from-backup, kip restores the backup's Dex configuration, connector client secrets and roles into a temporary namespace, merges them, and removes the namespace again. Use it after recovering onto a new cluster, where the restore keeps the new installation's accounts. For imported connectors whose callback URL matches the source Dex issuer's callback, kip updates the URL to this cluster's Dex callback. When the URL changes, kip prints it so you can update those providers' app settings. Custom callback URLs stay unchanged. The backup must cover the dex and kipper-system namespaces. See Console accounts after recovery.

With a snapshot file, kip merges the staticPasswords and connectors blocks of a captured dex-config. The file can be either a full ConfigMap manifest or the raw Dex config YAML directly. Connectors whose client secret is stored outside the config are skipped, and kip lists them so you can enable them again in the console.

Accounts, connectors and roles the cluster already has always win, so the install admin cannot get overwritten with stale data.

Dex reads the new configuration when it restarts. Pass --restart-dex to roll the Dex Deployment once the import finishes.

kip user add flags

Flag	Required	Default	Description
--role	No	deployer	Role: admin, deployer, or viewer
--password	No	prompted	Password (prompted interactively if not provided)

kip user invite flags

Flag	Required	Default	Description
--email	Yes	—	Address of the person being invited. The account is created under it, and only that address can accept the invite
--role	No	deployer	Role: admin, deployer, or viewer
--expires	No	48h	Expiry: 24h, 48h, 7d

Invites are tied to the supplied email address. The CLI prints a link you can share even when email delivery is unconfigured.

Create the account with an appropriate cluster role, then assign access to individual projects. For a project deployer:

kip user invite --email jordan@example.com --role viewer
# After Jordan accepts the invitation:
kip project members add acme-shop jordan@example.com deployer

Project access depends on membership; cluster admins have access to every project.

See Team Access for full documentation.

kip 2fa

Manage two-factor authentication for console users. Destructive operations, meaning starting a cluster migration and applying its cutover, require a TOTP code on top of the admin login, and enrolling a factor requires a one-time code that only these host-level commands can issue.

kip 2fa bootstrap admin@example.com    # issue a one-time enrollment code
kip 2fa remove admin@example.com       # remove a factor (lost phone, no recovery codes)

kip 2fa bootstrap

Issues a single-use enrollment code, valid for 15 minutes:

kip 2fa bootstrap admin@example.com

  ✔  Enrollment code for admin@example.com

     K7QT-M3XP-9WLC-R2VD

  Valid for 15 minutes, single-use.
  Enter it in Console → Settings → Two-factor authentication to enroll.

The user enters the code in Console → Settings → Two-factor authentication, scans the QR code with an authenticator app, and confirms. Issuing a new code for the same email replaces any unused one. Enrollment is gated on this code because it can only be issued with kubeconfig access; a stolen console login alone can never enroll a device.

kip 2fa remove

Deletes a user's enrolled factor, leaving the account unenrolled. This is the recovery path when the phone is gone and no recovery codes are left. Re-enrollment needs a fresh bootstrap code, and the new factor waits the full eligibility period (7 days by default) before it can authorize a migration.

See Cluster Migration for the full 2FA and migration security model.

kip registry

Manage container registry credentials. A credential is stored once in kipper-system and carries an allow-list of projects. When a workload in an allowed project runs an image from that registry, Kipper stages a pull secret in the workload's namespace, scoped to that single registry. Build containers run user code, so they carry no registry credentials and pull base images anonymously.

kip registry add --server ghcr.io --username myuser --password ghp_token123 --allow-project acme
kip registry add --server registry.git.example.com --username deploy --password secret --allow-project acme
kip registry allow ghcr-io --project shop      # adds shop, keeps acme
kip registry revoke ghcr-io --project shop     # removes shop, keeps the rest
kip registry list
kip registry remove ghcr-io

Flag	Required	Default	Description
--server	Yes	—	Registry server (e.g. ghcr.io, registry.git.example.com)
--username	For a new registry	—	Registry username or token name
--password	For a new registry	—	Registry password or access token
--name	No	auto-generated	Credential name (derived from the server's host if omitted)
--allow-project	No	none	Project allowed to pull with this credential (repeatable; replaces the allow-list)

A credential is used only by projects on its allow-list, so grant at least one. Re-running kip registry add for an existing name updates just the flags you pass, so granting a project keeps the stored password.

--allow-project replaces the allow-list. Naming one project takes every other away, which is what it has always done; it now prints what it removed so you can see it happen. To add a project without disturbing the others, use kip registry allow, and to take one away use kip registry revoke:

kip registry allow ghcr-io --project shop --project blog
kip registry revoke ghcr-io --project blog

allow checks that the project exists and refuses a name the cluster does not have, since a project name is matched exactly at pull time and a typo would be stored as a grant that can never work. revoke takes any name, because it is also how you remove one that should never have been there.

Pointing an existing credential at a different registry needs --password. The credential is addressed by its name, so changing the server would otherwise hand the new registry the password stored for the old one.

The console's registry settings edit the password and the server. Who may pull with a credential is changed with the commands above, and the settings API refuses a request that would change the allow-list on a credential that already exists.

When a workload in a granted project runs an image from that registry, the credential is staged into the workload's namespace as a pull secret, where the project's members can read it. Grant a credential only to projects that may share that registry login. When projects must stay isolated, create a separate registry account per project and add each as its own credential; a project's workloads then use the credential granted to them.

kip credentials

Read back the git and container-registry credentials stored in the cluster. Useful if you no longer have a copy of a token and need to reuse it somewhere else, such as another cluster or a CI pipeline.

kip credentials list                          # masked overview, both types, with allowed projects
kip credentials list --type git               # masked overview, git only
kip credentials get git-acme-tools            # plaintext token to stdout
kip credentials get ghcr-io --type registry   # disambiguate if names collide
kip credentials get --app blog --project acme --environment prod   # an app's own git token
kip credentials allow git-acme-tools --project acme    # let a project build with it
kip credentials revoke git-acme-tools --project acme   # stop it building with it

Granting a project

A shared git credential is usable only by the projects on its allow-list, so a new one builds nothing until you allow a project. Granting never asks for the token: it changes who may use the credential, not what it is.

kip credentials allow git-acme-tools --project acme
kip credentials allow git-acme-tools --project acme --project blog
kip credentials revoke git-acme-tools --project blog

A build refused for want of a grant says so and names the command:

git credential "git-acme-tools" is not allowed for project "acme". Allow it with 'kip credentials allow git-acme-tools --project acme'

Revoking leaves running apps alone. They keep the image they have, and the next build for that project is refused.

Container registry credentials have their own allow-list, granted with kip registry add --allow-project as described above. That flag replaces the list rather than adding to it, so name every project that should keep access.

The console's credential settings edit the token and the server. Who may build with an existing credential is changed with the commands above; allow checks that the project exists, and revoke takes any name, since it is also how you remove one that should never have been there. The settings API accepts a credential's allow-list when creating it, and refuses a request that would change the list on one that is already there.

kip credentials list shows each credential's allowed projects, so you can check a grant landed where you meant it.

On clusters that predate credential allow-lists, kip upgrade previews the projects whose apps reference each undecided shared credential. Accept the prompt to grant those pairs, or decline and grant access later with kip credentials allow. The migration runs once per cluster.

For automation, --seed-credential-grants accepts the previewed grants. A run with no terminal grants nothing unless that flag is set.

Rotate or replace shared credentials before or after an upgrade. If a credential changes during the upgrade, Kipper reports it and leaves its grants for you to review. After a rollback to a version older than 0.14, check kip credentials list and restore missing grants explicitly.

A token configured per app (under the app's Git settings) lives in a secret in the app's namespace, separate from the named credentials above. It does not show up in kip credentials list or the global credentials screen. To read it back, pass --app with the project and environment instead of a credential name. Kipper finds the secret the app references for its git source and prints that token.

The console masks credential values by default. To reveal one in the browser, click the eye icon next to a credential in Settings and re-enter your password. On the CLI, kip credentials get prints plaintext to stdout without prompting, so make sure nothing is watching your terminal.

CLI credential reads require Kubernetes permission to read the relevant Secret. Named credentials live in kipper-system; an app’s own token lives in its project namespace.

Flag	Required	Default	Description
--type	No	—	Restrict or disambiguate: git or registry

kip backup

Create, list, and restore cluster backups using Velero.

kip backup create                                                   # backup user namespaces (system namespaces excluded by default)
kip backup create weekly --project blog                       # backup one project
kip backup create --project blog --environment prod --ttl 720h  # 30-day retention
kip backup create everything --include-system                       # also back up system namespaces
kip backup list
kip backup describe velero-daily-apps-20260413030000                # how a backup ended, its volume backups, errors and warnings
kip backup restore weekly-20260413
kip backup restore weekly-20260413 --namespace blog-prod      # restore one namespace
kip backup schedules                                                # list scheduled backups
kip backup storage resize --size 30Gi                               # grow the in-cluster MinIO volume
kip backup storage set --bucket example-new-backups --region hel1 ...  # send future backups to another empty external bucket
kip backup storage retire-previous                                  # forget the previous backup location
kip backup repository-key export --to ~/example-backups.key         # save the key that encrypts the backups
kip backup repository-key reset                                     # delete every backup and start again with a new key
kip backup drill blog-prod                                          # restore blog-prod's services and shared volumes into an isolated namespace and count their rows
kip backup sse-key export --to ~/example-backups-sse.key            # save the SSE-C key, on clusters whose backups use SSE-C
kip backup mirror add uploads --source s3://... --target s3://... ...  # copy an S3 bucket daily into a locked bucket
kip backup mirror status uploads                                    # last run, preflight result and copy totals
kip backup mirror remove uploads                                    # stop the mirror. Both buckets stay as they are

kip backup drill estimates the Longhorn or local storage it needs, counting each restored volume once per estimated copy on this cluster. It warns about restored volumes with fewer copies than configured.

The nano profile blocks kip backup drill because running a second copy of a namespace's services exceeds a small server's memory capacity. Use kip service restore-dump to check one Postgres service's backup. See Test a restore with a drill.

kip backup repository-key export saves the volume backup encryption key to a new file with mode 0600. The command asks for confirmation, which you can skip with --yes. On Windows, the file inherits its folder's permissions. It never overwrites an existing file. Keep the key outside the cluster because a recovery install needs it to read the volume backups. See The backup repository key.

kip backup sse-key export does the same for the SSE-C key of a cluster whose backups use SSE-C, writing it as base64 text that --backup-storage-sse-c-key-file reads back. With --previous kipper-backup-sse-key-previous-<timestamp>, it exports the old key that kip saved when a key file replaced an unused SSE-C key. See SSE-C.

kip backup repository-key reset replaces the key. It deletes every backup in the bucket, because the old ones cannot be read with a new key, then takes a full backup from each schedule. It asks for the bucket's name before deleting anything. Use --yes to skip the prompt. Like kip cluster encrypt-secrets, it requires SSH access to the server. See The backup repository key.

kip backup storage resize grows the in-cluster MinIO backup volume and waits up to five minutes for it to report the requested capacity. Run it again with the same size to resume waiting. The size must be at least the volume's current storage request. Manage external backup storage through its provider.

kip backup storage set sends future backups to another external bucket that holds no backups, using a new repository key. It saves the old and new keys to files first. Run it again with the same flags to continue an interrupted move, or pass --abort to cancel one before Velero switches. kip backup storage retire-previous forgets the old bucket. It requires a completed backup created in the new bucket after the move. See Move backups to another bucket.

Flag	Required	Default	Description
--bucket	Yes	—	The new bucket. It must hold no backups
--region	Yes	—	The new bucket's region
--endpoint	No	—	S3-compatible endpoint URL. Omit for AWS S3
--credentials	No	~/.aws/credentials	AWS-style credentials file for the new bucket
--profile	No	default	Profile in the credentials file
--sse-c	If the cluster uses SSE-C	false	Encrypt the new bucket's objects with SSE-C. A cluster that already uses SSE-C keeps its key, and a cluster without SSE-C gets a new key
--export-new-key-to	Yes	—	File to save the new repository key to
--export-previous-key-to	Yes	—	File to save the old bucket's repository key to
--export-sse-c-key-to	With --sse-c	—	File to save the SSE-C key to
--ssh-key	No	—	SSH private key for the server
--yes	No	false	Move without asking for confirmation
--abort	No	false	Cancel a move before the switch begins

A default kip backup create excludes system namespaces (kube-system, kube-public, kube-node-lease, traefik, longhorn-system, keda, monitoring, velero). The daily schedule uses the same exclusion. Without it, Velero would try to back up its own MinIO PVC and the backup would hang. Pass --include-system if you really want a backup of those namespaces too.

Every backup also skips cert-manager's transient issuance objects (CertificateRequests, Orders, Challenges) and Dex's signing keys and sessions, including when --include-system is set. cert-manager recreates its issuance objects on demand, and restoring stale issuance state can interfere with certificate renewal. Excluding Dex's keys and sessions prevents their reuse from backups to create or renew console sessions.

kip backup mirror

kip backup mirror add <name> copies an S3 bucket once per schedule into a bucket at another endpoint that is versioned and carries an Object Lock default retention in compliance mode. It first runs the mirror's checks against both buckets from your machine and creates cluster resources only after all checks pass: the endpoints differ, source versioning is enabled, the target's default retention is COMPLIANCE and at least --retention-days, a canary written to the target is locked and cannot be deleted by version, and the source key is denied writing and deleting the canary key in the source. It then creates the Secret backup-mirror-<name> holding both keys and a CronJob of the same name in the velero namespace, then attempts to add the Prometheus alert KipperBackupMirrorStale. Each run repeats the checks, then runs rclone sync and rclone check.

kip backup mirror add uploads \
  --source s3://eu-1.objects.example.com/app-uploads --source-region eu-1 \
  --target s3://eu-2.objects.example.com/app-uploads-backup --target-region eu-2 \
  --source-credentials ~/.aws/mirror --target-credentials ~/.aws/mirror \
  --retention-days 30
kip backup mirror status [name]
kip backup mirror remove uploads

Flag	Required	Default	Description
--source, --target	Yes	—	Buckets as s3://endpoint/bucket
--source-region, --target-region	Yes	—	Region as the provider names it
--source-credentials, --target-credentials	Yes	—	AWS-format credentials files. The same file may serve both sides
--source-profile, --target-profile	No	default	Profile within each credentials file
--retention-days	Yes	—	Minimum default retention of the target bucket, in days
--schedule	No	0 3 * * *	Cron schedule, in the cluster's time zone
--max-delete	No	-1 (off)	Maximum objects one run may delete from the target. Exceeding it fails the run without a retry
--image-tag	No	—	Run the mirror image built from this commit (full 40-character SHA)

status shows the schedule, the last run and its outcome, the time since the last success, the preflight result and the last sync's totals, for one mirror or all of them. remove deletes the CronJob, the Secret and the alert and leaves both buckets untouched. To change a mirror, remove it and add it again. See Mirror an S3 bucket into a locked bucket for the bucket setup and the restore procedure.

kip backup create flags

Flag	Required	Default	Description
--project	No	—	Backup a specific project only
--environment	No	—	Backup a specific environment only
--ttl	No	168h (7 days)	Backup retention period
--include-system	No	false	Include system namespaces (kube-*, traefik, longhorn-system, monitoring, keda, velero). Off by default because Velero recurses into its own MinIO PVC otherwise.

kip backup restore flags

Flag	Required	Default	Description
--namespace	No	—	Restore only a specific namespace
--namespace-mapping	No	—	Restore only the source namespace into the target namespace (format: source:target). If set, --namespace must match the source.
--resources	No	—	Restore only specific resource types (comma-separated)
--allow-partial	No	false	Restore a PartiallyFailed backup despite potentially missing data or resources
--repair-credentials	No	false	Re-attach restored service credentials Secrets without prompting
--cluster-resources	No	false	Also restore projects and their resource adjustments. Existing ones remain unchanged.
--include-ingresses	No	false	With --namespace-mapping, also restore the copy's Ingresses, which claim the same hostnames as the source
--cancel	No	—	Cancel the named restore while it is InProgress instead of starting one. Takes no backup name. See Cancel a stuck restore.
--force	No	false	With --cancel, allow a Velero restart that also fails InProgress backups and other restores

Restore a Completed backup, or use --allow-partial for a PartiallyFailed backup. For partial backups, kip lists volumes with failed or canceled backups when details are available. Backups in other phases cannot be restored; wait for pending or running backups to finish.

kip backup restore waits up to four hours for the restore to finish. After a Completed or PartiallyFailed restore, it checks the restored service credentials Secrets (see Restore progress and credentials). Press Ctrl-C to stop waiting; the restore continues in the cluster. The command returns an error for Failed, FailedValidation, or PartiallyFailed restores, even if credential repair succeeds.

kip platform

Manage system component sizing, including the observability stack (Prometheus, Grafana, Loki). See Platform Resources for the full reference.

kip platform status                          # active profile + per-component state
kip platform versions                        # Kipper, Kubernetes and component image versions
kip platform resize prometheus --memory 2Gi  # set a manual memory override
kip platform disable loki                    # turn a component off
kip platform enable loki                     # turn it back on
kip platform restart prometheus              # rolling restart
kip platform profile show                    # current profile
kip platform profile set large               # change profile
kip platform tuning show                     # active resource tuning mode
kip platform tuning expert                   # stop automatic resource changes
kip platform tuning auto                     # resume automatic tuning

kip blueprint

Browse and install application blueprints. Blueprints are pre-built templates for common application stacks.

kip blueprint list
kip blueprint info wordpress
kip blueprint install wordpress --project myblog --set domain=blog.example.com

kip blueprint install flags

Flag	Required	Default	Description
--project	No	—	Project name (overrides template)
--environment	No	—	Target environment
--set	No	—	Parameter values (key=value, repeatable)

kip apply

Apply a kipper.yaml manifest to the cluster. This is the declarative way to manage Kipper resources: the manifest is the desired spec, and on update a field left out of it is cleared.

kip apply                                         # apply ./kipper.yaml
kip apply -f myapp.yaml                           # apply a specific file
kip apply -f kipper/                              # apply all manifests in a directory
kip apply --dry-run                               # preview changes without applying
kip apply --project blog --environment prod  # override project/environment

Flag	Required	Default	Description
-f, --file	No	kipper.yaml	Path to manifest file or directory
--dry-run	No	false	Print what would be applied without making changes
--project	No	from manifest	Override the project name
--environment	No	from manifest	Override the environment

Before writing anything, kip apply checks Longhorn's capacity for the volumes the manifest adds, using the same check as kip service add. This covers new services' volumes, new shared volumes and dump volumes for existing Postgres services whose consistentBackups the manifest enables.

kip diff

Show differences between a kipper.yaml manifest and the live cluster state. Useful for reviewing changes before applying.

kip diff                               # diff ./kipper.yaml against live state
kip diff -f myapp.yaml                 # diff a specific file
kip diff --project blog          # override project

Flag	Required	Default	Description
-f, --file	No	kipper.yaml	Path to manifest file or directory
--project	No	from manifest	Override the project name
--environment	No	from manifest	Override the environment

kip export

Export the current cluster state as a kipper.yaml manifest. Useful for backing up configuration or migrating between clusters.

kip export --project blog                                # export to stdout
kip export --project blog -o kipper.yaml                 # export to file
kip export --project blog --split -o blog-exports  # export every env, one file per env, into a directory

Flag	Required	Default	Description
--project	Yes	—	Project name
--environment	No	—	Export a specific environment only. Mutually exclusive with --split.
-o, --output	No	stdout	Output file. With --split, this is the output directory.
--split	No	false	Read the project's spec.environments and write one manifest per env into --output. The directory is created if it does not exist.

kip init

Generate a kipper.yaml manifest from a blueprint template.

kip init --blueprint wordpress --set domain=blog.example.com
kip init --blueprint nodejs -o kipper.yaml

Flag	Required	Default	Description
--blueprint	Yes	—	Blueprint name to use as template
--set	No	—	Parameter values (key=value, repeatable)
-o, --output	No	kipper.yaml	Output file