Resource Management
Kipper automatically manages CPU and memory for your apps so you do not have to think about Kubernetes resource requests and limits. It monitors actual usage and adjusts allocations to match. It scales up when apps need more and scales down when they are over-provisioned.
You can set CPU and memory bounds for each workload. The request sets the minimum allocation and the limit sets the maximum. Kipper adjusts the request within that range and keeps the limit fixed. See Your own CPU and memory values.
Auto mode (default)
A background controller monitors resource usage via metrics-server every 60 seconds. When it detects sustained high or low usage, it adjusts CPU and memory requests and limits automatically.
How it works
| Condition | Threshold | Action |
|---|---|---|
| High usage | Above 80% throughout the current observation window (up to 3 checks) | Increase by 50% |
| Low usage | Below 20% for 3 consecutive checks, with at least 2 replicas | Halve (with minimums) |
| OOM kill | Immediate | Double memory (capped, see OOM memory cap) |
| Stuck pod | In ContainerCreating for 5+ minutes | Delete pod to trigger recreation |
Routine adjustments use recent readings to smooth out temporary load changes. With a fresh history, an increase can follow one high reading for an app or function, or two for a service. Decreases require three low readings. OOM recovery and the saturation override below handle urgent increases.
The table describes automatic sizing. For a resource with user bounds, Kipper adjusts only the request and keeps it within the range you set. See Your own CPU and memory values.
After Kipper recommends a memory increase to recover from an OOM kill, it pauses automatic CPU and memory decreases for that workload for 24 hours. This prevents low usage after a restart from immediately undoing the increase. Any resource change also resets the usage history, so future decisions use readings taken at the new size.
Profile-based minimums
For automatically sized resources, the controller uses profile minimums when proposing adjustments:
| Profile | Min CPU | Min memory |
|---|---|---|
lightweight | 50m | 64 Mi |
standard | 100m | 128 Mi |
compute-heavy | 500m | 256 Mi |
memory-heavy | 100m | 512 Mi |
database | 500m | 1 Gi |
jvm | 100m (limit at least 500m) | 2 Gi |
These minimums apply to automatic sizing recommendations. The jvm profile starts with a 100m CPU request and a 1000m limit, reserving less CPU while allowing bursts during startup. Automatic sizing adjusts the request with usage, down to 100m, and keeps the limit at 500m or higher to preserve startup headroom.
Database services (PostgreSQL, MySQL, MongoDB, OpenSearch) automatically get the database profile.
OOM memory cap
OOM doubling is capped at 50% of the largest node's allocatable memory, with a minimum cap of 8 Gi, because a pod runs on one node. For example, 16 Gi of allocatable memory gives a cap of 8 Gi. This cap limits automatic increases; available memory and scheduling constraints determine whether a pod fits. The 8 Gi minimum can exceed a small node's capacity. If an OOM-killed container is already at or above the cap, the controller raises a critical alert and keeps its allocation.
Automatic adjustments round CPU up to the nearest 50m and memory up to the nearest 64 Mi. User bounds still take precedence. If the resulting allocation is unchanged, the controller skips the adjustment.
Startup grace period
Routine CPU and memory averages exclude pods for 5 minutes after the latest start of a currently running workload container, including native sidecars. This gives restarted containers time to warm up, including after a node reboot.
Containers whose previous run lasted less than 2 minutes are excluded from this calculation to prevent repeated short runs from renewing the grace period. If no running container qualifies, the grace period counts from pod creation. OOM detection remains active throughout.
Saturation override
After the warmup window below, a pod using 95% or more of its CPU limit can trigger an immediate CPU increase. This saturation override bypasses the routine startup grace period and usage history checks.
This helps workloads whose CPU limit slows startup, such as JVM apps performing JIT compilation. Raising the limit before the grace period expires gives startup work more CPU.
The override uses the same pod age as the startup grace period and becomes eligible after 2 minutes. This warmup window reduces the risk of repeatedly replacing pods in response to startup CPU spikes.
Stateful services get extra caution. Restarting a database mid-operation can kill a running restore or bulk import, so a service only receives a saturation bump after staying pinned for 90 seconds of continuous observations, and never from a single hot reading.
The saturation override raises the CPU limit only when CPU is sized automatically. User-set CPU limits stay fixed. Decreases still require three low readings.
Single-replica apps
For single-replica apps, automatic sizing only increases resources. A resource change starts a replacement pod and waits for it to become ready before stopping the old one. With a health check, the replacement must pass it first. The cluster needs capacity for the extra pod. If none is available, lower the request or add capacity to unblock the rollout.
Even after you lower the request, the rollout can remain blocked on a full node because the old pod keeps its CPU and memory until the replacement is ready. In that case, run kip app scale <app> 0 followed by kip app scale <app> 1 to free the old pod's resources, at the cost of a short outage.
The Scale tab in the web console shows a message explaining this when an app has one replica and auto mode is active.
During a rollout
Automatic sizing normally waits for an app's rollout to finish, avoiding repeated pod replacements during startup. Your resource bounds still apply. There are two exceptions:
- A pod cannot be scheduled or is blocked by quota: recommendations may apply only if neither the CPU nor memory request increases. Lower requests may help the pod fit.
- The progress deadline expires or another admission error blocks new pods: recommendations can apply again, subject to the usual bounds and quota checks. For example, more memory may help a pod recover from an out-of-memory failure.
A pod that is slow to become ready postpones recommendations until the rollout finishes or exceeds its progress deadline. Crashes after a completed rollout do not postpone recommendations.
See When a rollout waits for what each state means.
Autoscaling (HPA)
Autoscaling adjusts the number of pods based on CPU and memory utilization. It works independently from the resource controller, which adjusts CPU and memory per pod. Together, they give you both vertical scaling (right-sized pods) and horizontal scaling (right number of pods).
How the two controllers interact
| Concern | Who owns it | What it does |
|---|---|---|
| CPU and memory per pod | Resource controller (auto mode) | Monitors usage, adjusts requests and limits |
| Number of pods (replicas) | HPA (Kubernetes built-in) | Monitors utilization %, scales between min and max |
| Deployment shape (image, env, volumes) | App reconciler | Syncs the Deployment to match the App CR |
When autoscaling is enabled, the App reconciler stops writing spec.replicas to the Deployment and lets the HPA own that field. When autoscaling is disabled, the App reconciler owns replicas again.
The HPA treats CPU metrics from starting and unready pods separately. A health check that reflects startup readiness helps prevent temporary CPU spikes, such as JVM warm-up, from triggering unnecessary scaling.
Choosing a setup
| Workload | Starting point |
|---|---|
| Small internal tool or staging app | Auto mode, one replica |
| App with steady traffic | Auto mode; use multiple replicas for availability during updates |
| App with variable traffic | Auto mode plus HPA |
| App with measured per-pod requirements | Expert mode plus HPA |
| Managed database or cache | Auto mode, single service replica |
With both controllers enabled, the HPA adjusts replica count and the resource controller adjusts per-pod allocations. For resources the HPA tracks, automatic sizing keeps requests and limits stable, except for memory increases after an OOM kill. See how automatic sizing steps back. For untracked resources, sizing allows increases but holds off on decreases while the HPA reports more replicas than its minimum. Resource changes trigger a rollout; allow capacity for replacement pods.
Enabling autoscaling
In the web console, open the app's Scale tab and set Scaling policy to Target tracking. Set the minimum, the maximum and a CPU target percentage, then click Save capacity. See In the console for the rest of the panel.
From the CLI or GitOps:
apiVersion: kipper.run/v1alpha1
kind: App
metadata:
name: api
namespace: blog-prod
spec:
image: registry.git.example.com/api:latest
port: 8080
autoscale:
enabled: true
minReplicas: 2
maxReplicas: 5
cpuTarget: 70The HPA checks metrics every 15 seconds. When average CPU across all pods exceeds the target, it adds pods. When utilization drops, it removes pods (down to minReplicas).
Recommended settings
Set minReplicas to at least 2 when using auto mode. This gives two benefits:
- The resource controller can safely scale resources down. With 2+ replicas, Kubernetes performs a rolling update so at least one pod stays available during the restart
- Your app has basic high availability. If one pod crashes, the other continues serving traffic
A good starting point for most apps:
| Setting | Value |
|---|---|
| Minimum | 2 |
| Maximum | 5 |
| CPU target | 70% |
| Memory target | 0 (disabled) |
Memory-based autoscaling is usually less useful because most applications do not release memory when load drops. CPU-based scaling responds faster to actual load changes.
What happens under the hood
- You enable autoscaling on the App CR
- The App reconciler creates an HPA targeting the app's Deployment
- The HPA reads CPU metrics from metrics-server and adjusts
deployment.spec.replicas - The resource controller independently adjusts CPU and memory requests based on per-pod usage
- When the App reconciler runs (e.g. after an image update), it updates the Deployment template but preserves the replica count set by the HPA
Disabling autoscaling
Set Scaling policy to None (fixed count) in the Scale tab and click Save capacity, or run kip app autoscale <app> --off. The Deployment's desired count is saved within the bounds, and the App reconciler takes over replica management and deletes the HPA. Stopped apps use their stored restart count; a failed Deployment read also falls back to the stored count and produces a warning. See Switching autoscaling off for these fallback rules. The minimum and maximum remain in place. To remove them, clear both in the Scale tab while no scaling policy is set, or run kip app autoscale <app> --remove.
OOM recovery
When a pod exceeds its memory limit and is killed (OOMKilled), the controller recommends a memory increase on the next check, without waiting for usage history. For automatic memory, it doubles the current allocation up to the OOM cap. The workload reconciler then applies the recommendation.
The controller detects OOM kills even when the pod is in a crash loop and has no metrics. It checks the pod's termination state directly from the Kubernetes API, not just from metrics data.
Automatic memory doubling applies only to memory managed by Kipper. For user-set or held memory, Kipper keeps the limit and raises a critical OOMKilled at your limit alert. You can then decide whether to increase it.
Resource profiles
When an app has no resource requests configured, the controller applies defaults based on the app's resource profile label (kipper.run/resource-profile):
| Profile | CPU request | CPU limit | Memory | Use case |
|---|---|---|---|---|
lightweight | 50m | 50m | 64 Mi | Static sites, proxies, lightweight APIs |
standard | 100m | 100m | 128 Mi | Typical web applications (default) |
compute-heavy | 500m | 500m | 256 Mi | Image processing, data transformation |
memory-heavy | 100m | 100m | 512 Mi | Caching layers, in-memory databases, ML inference |
database | 250m | 250m | 256 Mi | PostgreSQL, MySQL, MongoDB, OpenSearch |
jvm | 100m | 1000m | 2 Gi | Java/JVM applications, Spring Boot, heavy runtimes |
The jvm profile is the only burstable profile by default. Most workloads run with request equal to limit (Guaranteed QoS) so they get exactly what they ask for. JVMs are different: they spike during startup and idle the rest of the time, so the request stays low (so the pod schedules) but the limit is high (so JIT can finish). On the same node, you can run six JVM apps reserving 600m total but each one able to burst to a full core during cold start.
If no profile label is set, standard is used. Database services automatically get the database profile.
Your own CPU and memory values
Profiles provide starting values. If an app needs specific resources, set a fixed size or a range. Kipper preserves the values you configure and uses them as bounds:
| You set | What the app gets |
|---|---|
| Nothing | Kipper sizes it on its own, as described under Auto mode |
One size (--memory 1Gi, or request equal to limit) | The same value for request and limit, with no automatic adjustments |
A request below a limit (512Mi and 2Gi) | Kipper adjusts the request within this range and keeps the container limit at your limit |
CPU and memory are independent: you can set a CPU range and leave memory automatic. Your bounds take precedence over profile minimums and the OOM cap.
For example, a memory request of 512Mi and limit of 2Gi lets Kipper reserve between 512 Mi and 2 Gi while the container can use up to 2 Gi. Automatic sizing lowers the request only when the usual scale-down conditions are met. A single-replica app keeps its allocation. When you change the request itself, the workload moves to the new request once any rollout in progress finishes and the project quota allows it. Kipper resumes tuning from the new request. A proposal that clamps to the current allocation produces no resize alert or resource-log entry. Alerts describe recommended changes. The reconciler applies them afterwards.
From the CLI:
kip app update backend --memory-request 512Mi --memory-limit 2Gi # a range
kip app update backend --cpu 500m # a fixed size
kip app update backend --tuning auto # hand both back to Kipper--memory and --cpu set the request and limit to the same value. Use the separate request and limit flags for a range. Supplying just one of those flags also sets a fixed size. The CLI rejects a request above its limit.
kip app deploy accepts --memory <size> and --cpu <size> for fixed sizes. Switching profiles with --profile preserves your explicit values. For services, use kip service update <service> --memory <size> --cpu <size> to set fixed sizes.
From the web console:
When deploying an app, select Custom... from the resource profile dropdown and enter memory and CPU in Kubernetes notation: 256Mi, 1Gi, 4Gi for memory; 250m, 500m, 1, 2 for CPU. That sets a fixed size.
For an existing app, open the Resources tab. Each resource shows whether it is automatic, bounded, fixed or held. The panel also shows the current allocation where available. During a rollout, the reported allocation can reach the new size before all pods are ready.
The sliders set a fixed size by default. Show request controls (advanced) opens all four fields so you can enter a range and choose Save request & limit. With these controls open, a slider changes the limit and keeps the request. Saving sends only the resource pairs you edited, so changing memory preserves automatic CPU sizing. Size automatically returns one resource to automatic sizing.
The service Resources tab shows the sizing mode and provides sliders for fixed sizes. To configure a service range, set its request and limit in kipper.yaml or the Service custom resource. Functions provide all four fields in the Resources section of the function form.
From kipper.yaml or a custom resource:
resources:
memoryRequest: 512Mi
memoryLimit: 2Gi
cpuRequest: 250m
cpuLimit: 250mkip apply treats every resource value in the manifest as an explicit user setting, even when it matches the current value.
Values Kipper cannot attribute
Kipper identifies explicit App values from Kubernetes field ownership (metadata.managedFields). Values set through kip, the current console, kip apply, direct CR edits or GitOps tools count as user settings. Values owned only by the old console-api field manager remain automatic.
An App value whose owner cannot be identified as a user or the old auto-sizer is held. This can happen if a restore fails to recover field ownership. Kipper preserves the value and reports OOM kills without increasing memory. In the console, Keep this value confirms it as your setting; Size automatically returns it to automatic sizing.
Service and function values are always yours, because earlier versions of the auto-sizer never wrote them.
Environment copies and project migrations preserve user settings and held values. They omit automatic values so the target can size itself. kip export also omits automatic values, but includes held values with a message on stderr. Applying the exported file confirms those held values as your settings.
Resource log
Every change the controller makes is logged and visible under Settings in the web console. The log shows:
- Time: when the change happened
- App and namespace: which workload was adjusted
- Action: what changed (increased memory, decreased CPU, applied defaults)
- From / To: old and new values
- Reason: why the change was made (usage at 92%, OOM kill detected)
The system retains the most recent 50 log entries.
Expert mode
Switch to expert mode when you want full control over resource allocation. The auto controller stops making changes, and all CPU and memory values are set manually through the Resources tab in the app detail panel.
Toggle between modes in Settings in the web console (only admins can change the mode), or from the CLI:
kip platform tuning show # print the active mode
kip platform tuning expert # hands off — no automatic resource changes
kip platform tuning auto # default — resources adjust automaticallyThe console uses the same setting through the API:
PUT /api/v1/settings/mode
{"mode": "auto"} // or "expert"Bulk data loads don't need a mode switch: kip service import and kip service export pause tuning for the affected service automatically while they run.
In expert mode, you can still view the resource log to see what the controller changed before you switched.
Alerts
Resource actions generate alerts visible in the console bell icon. Scaling alerts pause during the restart hold:
- Critical (red): OOM kills, emergency memory doubling
- Warning (yellow): resource increases, stuck pod recovery
- Info (green): scale-downs, default profile application
See Alerts for details on the alerting system and Slack integration.
Slack notifications
Resource changes can be forwarded to Slack. See Configuration for setup.
Upgrading from 0.19 or earlier
Earlier versions stored automatic allocations in the same App fields as user settings. This release stores tuning recommendations separately and preserves user bounds. When upgrading:
- Save existing console-set App values again. The old console and auto-sizer shared a field manager, so Kipper treats those values as automatic until you save them after upgrading. Values owned by kip,
kip applyor GitOps tools are recognized immediately. Service and function spec values count as user settings. - Wait for the upgrade to finish before setting bounds. The old auto-sizer can still overwrite newly saved values while console-api rolls out.
- Check your values after a rollback and subsequent upgrade. The old auto-sizer can overwrite them while it runs.
- Upgrade kip as well. Older versions clear explicit App values when changing profiles and update service StatefulSets directly, where the reconciler can overwrite them. An older kip also cannot claim an unchanged automatic App value as a user setting.
What Kipper manages
The auto-sizer tunes Deployments and StatefulSets owned by these Kipper custom resources (kipper.run/v1alpha1):
- Apps: web apps, APIs, frontends
- Services: databases, caches, message queues
- Functions: running function Deployments
Jobs, cron-triggered function runs and one-off function runs use their configured resources when each pod starts. They are not auto-tuned. The auto-sizer also leaves system components such as Traefik, cert-manager, Longhorn and KEDA unchanged.
Project quotas
Project quotas are optional. A new project starts without a tier: its apps simply use whatever the cluster has free, and only the cluster's own capacity limits apply. That is the right mode for most single-team clusters, where the platform should just work without capacity planning.
Assign a tier when you want a ceiling on how much CPU and memory a project's workloads can claim in total, for example on a cluster shared by several teams, where one runaway project must never starve the rest. The ceiling applies per environment namespace, and each environment can override it individually.
Tiers
A tier is a capacity label on the project. It sets the default quota for each of the project's environment namespaces, and it caps how many environments the project can have:
| Tier | CPU requests | CPU limits | Memory requests | Memory limits | Environments |
|---|---|---|---|---|---|
| (no tier) | — | — | — | — | up to 6 |
small | 2 | 6 | 6 Gi | 12 Gi | up to 4 |
medium | 4 | 12 | 12 Gi | 24 Gi | up to 6 |
large | 8 | 24 | 24 Gi | 48 Gi | up to 10 |
Limits are roomier than requests. Requests are what the scheduler reserves; limits only cap bursts. A tier must fit a workload's whole lifecycle, so the ceilings leave room for a rolling update (old and new pods run at once) plus a Git build pod on top of the steady state.
The quota applies per environment namespace. A small project with test and prod environments gets two independent 2-CPU quotas, one per namespace, rather than one shared pool. The project's total granted capacity is therefore the per-environment quota times the number of environments, which is why the tier also caps the environment count. A small project can grant at most 4 × 2 = 8 CPU requests across its four environments.
apiVersion: kipper.run/v1alpha1
kind: Project
metadata:
name: shop
spec:
displayName: "Webshop"
tier: medium
environments:
- name: test
- name: prodThe quota tab in the console
Open a project's settings with the gear icon on its card in the Projects screen. The Quota tab shows the project's tier and, per environment, the four caps with live usage bars. An environment running with an explicit override carries an Override badge; one that has hit a cap shows Over quota.
The Over quota badge appears after a live usage comparison. While usage is unavailable, the console leaves the badge unset and the API reports over_quota: null. Saving a quota can succeed while usage is unavailable; the next successful read supplies it.
Cluster admins can change the tier from the dropdown and edit or reset per-environment overrides with the pencil and reset buttons. Raising a quota hands out cluster capacity, so this stays with admins; project owners and members see the panel read-only. When a new cap would land below what an environment currently uses, the console shows exactly which values are affected and asks for confirmation before applying.
Per-environment overrides
When one environment genuinely needs more than its tier, set an explicit quota on that environment. The override replaces the tier default for that namespace only:
spec:
tier: small
environments:
- name: test
- name: prod
quota:
cpuRequest: "6"
cpuLimit: "12"
memoryRequest: "12Gi"
memoryLimit: "24Gi"Values use Kubernetes quantity notation, the same as app resources: 500m, 2, 4Gi.
The environment limit
Every project has a cap on how many environments it can have. Without a tier the cap is 6, which covers a dev/test/acc/prod pipeline plus a hotfix and a preview environment. A tierless environment has no quota, so the cap is what keeps a scripted loop (say, an environment per feature branch) from filling the cluster with unbounded namespaces before anyone notices.
For tiered projects the tier sets the cap (4 / 6 / 10). Each environment gets its own full tier-sized quota there, so adding environments adds granted capacity, and the cap stops a project owner multiplying the project's footprint past its tier without a cluster admin. Project owners create environments up to the limit themselves. At the limit the console stops offering Add environment and shows the count, and the API rejects an over-limit create.
When a project genuinely needs more environments, a cluster admin sets maxEnvironments on the project, which overrides the default for that project only. Make sure the cluster has the hardware to back the extra environments first, especially for tierless projects where nothing else caps their usage. Assigning a tier is the answer to a different question: it adds managed capacity envelopes, and picking small also lowers the environment cap to 4. Deleting an environment frees a slot straight away.
spec:
tier: small
maxEnvironments: 6 # admin override; small would otherwise allow 4
environments:
- name: test
- name: acc
- name: prodLowering a tier or maxEnvironments below the environments a project already has does not delete anything. The existing environments keep running and keep their quota, new ones are blocked, and the project carries an EnvironmentLimitExceeded status condition until it fits again. The admin quota change asks for confirmation before applying that state, the same way a below-usage quota change does.
What happens when the quota is reached
Running pods keep running. Kubernetes rejects the next pod that would push the namespace over the ceiling, so a deploy, build, or migration fails at admission with a quota error. Scale down something else, raise the tier, or set an environment override to make room.
The auto resource controller respects the ceiling too. When an app needs more memory or CPU than the quota has room for (for example an OOM-killed container that wants its memory doubled), the controller leaves the resources unchanged and raises a critical alert naming the quota instead. Committing the increase would only produce replacement pods that admission rejects.
A manual resource change from the console or CLI runs the same projection before it saves. If raising an app or service beyond what the namespace quota can admit, the request is rejected up front with the exhausted dimension named, rather than saving a value that would stall the rollout at admission.
Existing namespaces
When a quota first arrives on a namespace that already runs more than its tier default (for example after upgrading a cluster that predates quotas), Kipper measures current usage and records a raised quota as an explicit override on the environment, with 25% headroom for rolling updates. Nothing gets evicted and the next deploy still goes through. The override is visible in the Project CR, so you can lower it deliberately once you know what the environment really needs.
Container defaults
Alongside the quota, each tiered or overridden environment gets a LimitRange with small per-container defaults: requests of 25m CPU and 32 Mi memory, limits of 100m CPU and 128 Mi memory. Tierless environments get neither the quota nor the LimitRange. Pods that declare no resources (for example something applied manually with kubectl) are filled in at admission instead of being rejected. Kipper's own workloads always declare explicit resources, so these defaults exist as a safety net. A container that inherits the 128 Mi default limit and needs more will be OOM-killed, which is the signal to give it real resource values.
Project quotas cap CPU and memory only. Storage is not part of the quota; PVC sizes are set per volume and per service.
GitOps
Kipper resources are defined as Custom Resource Definitions (CRDs) under kipper.run/v1alpha1. This means you can manage your entire cluster declaratively with tools like ArgoCD or Flux:
apiVersion: kipper.run/v1alpha1
kind: App
metadata:
name: api
namespace: blog-test
spec:
image: registry.git.example.com/api:v2.1.0
port: 8080
replicas: 2
resources:
profile: jvm
memoryRequest: "4Gi"
memoryLimit: "4Gi"
cpuRequest: "100m"
cpuLimit: "1500m"
env:
LOG_LEVEL: "info"
route:
host: api.example.comApply with kubectl apply -f app.yaml or commit to a Git repo and let your GitOps tool sync it. Kipper's reconcilers ensure the underlying Kubernetes resources (Deployment, Service, Ingress, Secrets) match the CR spec.
Available CRDs: App, Service, Function, Project, Job, Volume.
For a more user-friendly approach, use the kipper.yaml manifest format with kip apply. See the full GitOps guide for details, including ArgoCD and Flux integration examples.