Back to Checks
kubernetes

Controller Manager pod does not use the default --terminated-pod-gc-threshold value

controllermanager_garbage_collection
Service: controllermanagerSeverity: high
OfficialBuilt-inVerified
Categories:cluster-security
Resource Type:Pod

Kubernetes controller manager terminated Pod garbage collection threshold is evaluated. The finding highlights use of the default --terminated-pod-gc-threshold=12500 instead of a value tuned to cluster size and workload churn. The threshold controls when terminated Pods are automatically removed.

Prowler logoProwlerprowler-provider-kubernetes

Risk

Retaining too many terminated Pods strains API server, etcd, and controller memory, reducing control-plane availability. Effects include slow list/watch operations, lagging schedulers, timeouts, and, in worst cases, controller crashes or admin-plane DoS.

Run this check with Prowler

Run in Prowler Cloud
Run
prowler kubernetes --checks controllermanager_garbage_collection

Recommendation

Set a lower, context-appropriate --terminated-pod-gc-threshold to match cluster scale and pod churn, preserving control-plane capacity. Monitor garbage collection and control-plane metrics and adjust proactively. Use ttlSecondsAfterFinished for Jobs to minimize terminated Pods.

Remediation

Other

1SSH to each control-plane node
2Edit the static Pod manifest: /etc/kubernetes/manifests/kube-controller-manager.yaml
3Under the kube-controller-manager container args/command, set: --terminated-pod-gc-threshold=10 (any value not equal to 12500)
4Save the file; the kubelet will automatically restart the controller-manager
5Repeat on all control-plane nodes if using HA

Check Code

from prowler.lib.check.models import Check, Check_Report_Kubernetes
from prowler.providers.kubernetes.services.controllermanager.controllermanager_client import (
    controllermanager_client,
)


class controllermanager_garbage_collection(Check):
    def execute(self) -> Check_Report_Kubernetes:
        findings = []
        for pod in controllermanager_client.controllermanager_pods:
            report = Check_Report_Kubernetes(metadata=self.metadata(), resource=pod)
            report.status = "PASS"
            report.status_extended = f"Controller Manager has an appropriate garbage collection threshold in pod {pod.name}."
            for container in pod.containers.values():
                if "--terminated-pod-gc-threshold=12500" in str(container.command):
                    report.status = "FAIL"
                    report.status_extended = f"Controller Manager has the default garbage collection threshold in pod {pod.name}."
            findings.append(report)
        return findings

References