Alerts

Kipper tracks important cluster events (resource changes, OOM kills, stuck pods) and surfaces them through an in-console alerting system with optional Slack integration.

The bell icon

The bell icon in the sidebar shows a badge with the number of unread alerts. Click it to open the alerts panel, which lists the most recent 50 alerts sorted newest first.

Each alert shows:

• Severity: critical (red), warning (yellow), or info (green)
• App and namespace: which workload triggered the alert
• Action: what happened (memory doubled, CPU increased, stuck pod deleted)
• Reason: why the action was taken (OOM kill detected, usage at 92%)
• Timestamp: when the event occurred

What triggers alerts

Alerts come from the resource controller. Failure alerts (crash loops, read-only volumes, unready nodes, failed jobs, stalled rollouts) and the autoscaling warnings run whether the controller is in auto or expert mode, because an operator needs to know about those either way. The alerts about resource changes below only appear in auto mode, since expert mode makes no such changes.

Resource adjustments

The controller checks CPU and memory usage every 60 seconds. High usage must exceed 80% throughout the current observation window, which holds up to three checks. A fresh window needs one reading for an app or function, or two for a service. Decreases require usage below 20% for three consecutive checks and at least two replicas. Resource-change alerts show the current and recommended values.

For a resource with user bounds, the controller adjusts only the request within that range. A proposal that resolves to the current allocation produces no resize alert, so an idle app at its floor stays quiet. See Your own CPU and memory values.

OOM kills

For automatically sized memory, an out-of-memory (OOM) kill triggers a recommendation to double memory, subject to the OOM cap, and a critical alert. Recovery starts on the first detection, without waiting for usage readings.

For user-set or held memory, Kipper preserves the limit and raises a critical OOMKilled at your limit alert. You decide whether to increase it.

Stuck pods

If a pod remains in ContainerCreating state for more than 5 minutes, the controller deletes it (allowing Kubernetes to recreate it) and creates a warning alert.

Stalled rollouts

When a rollout exceeds its Kubernetes progress deadline, the controller raises a warning alert with the available diagnostic details. These may identify a scheduling problem, a quota limit, or pods that have not become ready. See When a rollout waits for troubleshooting steps.

Node resource pressure

When total memory usage across all pods reaches 80% of the combined allocatable memory of all nodes, the controller generates a warning alert naming up to three workloads using the most memory. At 90% or higher, the alert is marked critical.

Nodes that are down

When a node is not Ready, or is fenced, the controller raises a critical alert on its next check and repeats it at most once every 30 minutes while the node stays down. When the node hosts Kipper services with a single-node Longhorn volume, the alert names each service, the nodes that hold a current copy of its data, and the command kip node fence <node> to run once you know the node is down for good. The copy locations are left out when Longhorn's replicas cannot be read. The alert for a fenced node names kip node add to rejoin it and kip node remove <node> --force to remove it. See When a node dies.

Faulted dump volumes

When Longhorn reports the dump volume of a Postgres service with consistent backups as faulted, the controller raises a dump volume faulted warning with the command that replaces the dump volume. Consistent dumps fail while the dump volume is faulted, and the database pod cannot start again until the dump volume is replaced. The warning fires once per fault and can fire for a new fault after the dump volume has recovered or been replaced. After a console-api restart, the controller uses the 50 stored alerts to avoid repeating warnings for faults already reported.

Autoscaling

For an app with autoscaling, two situations raise a warning:

• autoscaling at maximum: the app runs its maximum number of pods and the autoscaler reports that the maximum stops it from adding more (ScalingLimited with reason TooManyReplicas). The warning fires once per episode and again only after the app has dropped below its maximum. If the load is expected to last, raise the maximum, or the request of each metric the autoscaler tracks, which the warning names. The app's Traffic and scaling charts help you compare traffic, resource use and scaling around the warning.
• autoscaling not ready: the App's AutoscalingReady condition is False. The alert names the reason and the message, for example an invalid policy or an autoscaler that could not be written. It fires once per reason, and again after the condition has returned to True or been removed.

Container restarts, such as those after a node reboot, can cause CPU spikes that trigger scaling. For 10 minutes after the latest detected container restart, scale changes appear only in the resource log and new at-maximum warnings are deferred. After the hold expires, the warning becomes eligible if the app is still at its maximum and the autoscaler still requests more pods. New pods created by scaling out do not trigger this hold.

An ordinary scale out or scale in is recorded as an info alert and in the capacity panel's Scaling activity. Scale changes are recorded in auto mode. When the controller first sees an autoscaler, for example after a console-api restart, a scale within the last three minutes is recorded as a scale from the minimum, unless the resource log already holds it. After a console-api restart, a warning whose alert is still among the stored 50 is not repeated. The exception is an at-maximum limit that lifted and returned while the app stayed at its maximum, which can be reported once more after a restart.

Default profile application

When a new app has no resource requests configured, the controller applies profile defaults and creates an informational alert recording which profile was used.

Crash loops

A crash-looping container generates hourly warnings for six hours, then a critical alert and daily reminders. This keeps recurring failures visible while preserving space in the 50-alert history.

After ten minutes of healthy running, the episode closes. An escalated episode also produces an all-clear alert.

Kipper suppresses crash-loop and stuck-rollout alerts for stopped apps and resets their crash-loop history. After the app starts again, any new crash loop begins a fresh episode.

Read-only volumes

When a container with a writable persistent volume fails, Kipper checks its previous logs for read-only filesystem errors. A matching message produces a critical ReadOnlyFilesystem alert containing the log evidence and mounted volumes.

The message identifies a write failure, but may not identify which filesystem failed. Check the affected mount before recreating the pod. See Recovering a read-only volume.

Where alerts go

Alerts are delivered to Slack when a webhook is configured, and by email to the cluster admins when one is not. With neither configured they are stored in the bell and go nowhere else, which means only somebody who already has the console open will see them.

Both the console's Settings page and kip status say which of the three is happening:

$ kip status

  Alerts:
    ⚠  not leaving this cluster
       Alerts are stored in the console bell and delivered nowhere. Add a
       Slack webhook or SMTP server under Settings in the console.

Configure a channel under Settings in the console. See Configuration for the Slack setup.

Dismissing alerts

Click Dismiss in the alerts panel to mark all current alerts as read. The unread count on the bell icon resets to zero. Dismiss is per-user, so each team member has their own read/unread state.

Dismissed alerts remain in the panel’s history. New alerts increase the unread count again.

Storage

Alerts are stored in a Kubernetes ConfigMap (kipper-alerts in the kipper-system namespace). The system retains the most recent 50 alerts, automatically discarding older ones. No external database is required.

Recovering a read-only volume

A Longhorn volume holds an iSCSI session between the node and the pod serving the volume's data. When that session drops, the block device underneath the mounted filesystem fails, and ext4 answers by remounting the filesystem read-only.

The pod does not notice until it writes. A database then dies on its first write, gets restarted by Kubernetes into the same read-only mount, and dies again. Every signal reads as an ordinary crash loop: the volume is attached, the pod is scheduled, and the container's own log is the only place that says why.

Restarting the container does not fix it. The mount belongs to the pod, so kubelet has to detach and re-attach the volume, and that only happens when the pod is recreated:

kip service restart <name>

The volume's data is untouched. The pod is recreated and its volume reattached.

To confirm what is happening first:

kip service list           # Shows crash-looping services and their restart counts.
kip status

kip status reads mounts over SSH on the cluster host and every recorded worker and names any node it could not check. The alert above runs in the cluster and sees pods on every node.

What causes the session to drop

The common trigger is a security update. Debian's needrestart restarts daemons holding a patched library, and restarting iscsid takes every Longhorn session on the node with it.

Kipper’s protection covers restarts initiated by needrestart. An upgrade of the open-iscsi package can restart iscsid through its own package script, so plan storage-related package upgrades carefully.

Kipper writes /etc/needrestart/conf.d/50-kipper-storage.conf during kip install and kip node add to stop that, leaving iscsid, systemd-networkd and systemd-resolved for you to restart at a time you choose. A node added by an older kip does not have the file. Repair every recorded node, or one node by address:

kip node repair-host
kip node repair-host --host <address>

On Rocky Linux, AlmaLinux and CentOS Stream, Kipper enables iscsid and reads pending restarts from dnf needs-restarting. The needrestart drop-in applies only to Ubuntu and Debian. Schedule iscsi-initiator-utils updates during planned downtime: package scripts may restart iscsid and disrupt mounted volumes.

kip status reports services that still need a restart to load updated libraries:

  Pending restarts:
    ⚠  storage services awaiting restart: iscsid.service
       These services need a restart to load updated libraries.
       Restarting them can disrupt Longhorn volumes. Schedule downtime
       and reboot with 'kip node reboot' instead of restarting these
       services directly.

Apply the updates during planned downtime with kip node reboot. Restarting iscsid on its own can disconnect mounted Longhorn volumes.