CIS Kubernetes Benchmark v1.11.0

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the API server pod specification file has permissions of 600 or more restrictive.

By default, the kube-apiserver.yaml file has permissions of 640.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/manifests/kube-apiserver.yaml Verify that the permissions are 600 or more restrictive.

None

Automated

The API server pod specification file controls various parameters that set the behavior of the API server. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the API server pod specification file ownership is set to root:root.

By default, the kube-apiserver.yaml file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/manifests/kube-apiserver.yaml Verify that the ownership is set to root:root.

None

Automated

The API server pod specification file controls various parameters that set the behavior of the API server. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the controller manager pod specification file has permissions of 600 or more restrictive.

By default, the kube-controller-manager.yaml file has permissions of 640.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/manifests/kube-controller-manager.yaml Verify that the permissions are 600 or more restrictive.

None

Automated

The controller manager pod specification file controls various parameters that set the behavior of the Controller Manager on the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the controller manager pod specification file ownership is set to root:root.

By default, kube-controller-manager.yaml file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/manifests/kube-controller-manager.yaml Verify that the ownership is set to root:root.

None

Automated

The controller manager pod specification file controls various parameters that set the behavior of various components of the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the scheduler pod specification file has permissions of 600 or more restrictive.

By default, kube-scheduler.yaml file has permissions of 640.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/manifests/kube-scheduler.yaml Verify that the permissions are 600 or more restrictive.

None

Automated

The scheduler pod specification file controls various parameters that set the behavior of the Scheduler service in the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the scheduler pod specification file ownership is set to root:root.

By default, kube-scheduler.yaml file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/manifests/kube-scheduler.yaml Verify that the ownership is set to root:root.

None

Automated

The scheduler pod specification file controls various parameters that set the behavior of the kube-scheduler service in the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the /etc/kubernetes/manifests/etcd.yaml file has permissions of 600 or more restrictive.

By default, /etc/kubernetes/manifests/etcd.yaml file has permissions of 640.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/manifests/etcd.yaml Verify that the permissions are 600 or more restrictive.

None

Automated

The etcd pod specification file /etc/kubernetes/manifests/etcd.yaml controls various parameters that set the behavior of the etcd service in the master node. etcd is a highly-available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/manifests/etcd.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the /etc/kubernetes/manifests/etcd.yaml file ownership is set to root:root.

By default, /etc/kubernetes/manifests/etcd.yaml file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/manifests/etcd.yaml Verify that the ownership is set to root:root.

None

Automated

The etcd pod specification file /etc/kubernetes/manifests/etcd.yaml controls various parameters that set the behavior of the etcd service in the master node. etcd is a highly-available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/manifests/etcd.yaml

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the Container Network Interface files have permissions of 600 or more restrictive.

NA

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %a <path/to/cni/files> Verify that the permissions are 600 or more restrictive.

None

Manual

Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 <path/to/cni/files>

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the Container Network Interface files have ownership set to root:root.

NA

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G <path/to/cni/files> Verify that the ownership is set to root:root.

None

Manual

Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root <path/to/cni/files>

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the etcd data directory has permissions of 700 or more restrictive.

By default, etcd data directory has permissions of 755.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, stat -c %a /var/lib/etcd Verify that the permissions are 700 or more restrictive.

None

Automated

etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should not be readable or writable by any group members or the world.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, chmod 700 /var/lib/etcd

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the etcd data directory ownership is set to etcd:etcd.

By default, etcd data directory ownership is set to etcd:etcd.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, stat -c %U:%G /var/lib/etcd Verify that the ownership is set to etcd:etcd.

None

Automated

etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should be owned by etcd:etcd.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command: ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, chown etcd:etcd /var/lib/etcd

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the admin.conf file (and super-admin.conf file, where it exists) have permissions of 600.

By default, admin.conf and super-admin.conf have permissions of 600.

Run the following command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/admin.conf On Kubernetes version 1.29 and higher run the following command as well :- stat -c %a /etc/kubernetes/super-admin.conf Verify that the permissions are 600 or more restrictive.

None.

Automated

As part of initial cluster setup, default kubeconfig files are created to be used by the administrator of the cluster. These files contain private keys and certificates which allow for privileged access to the cluster. You should restrict their file permissions to maintain the integrity and confidentiality of the file(s). The file(s) should be readable and writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/admin.conf On Kubernetes 1.29+ the super-admin.conf file should also be modified, if present. For example, chmod 600 /etc/kubernetes/super-admin.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the admin.conf (and super-admin.conf file, where it exists) file ownership is set to root:root.

By default, admin.conf and super-admin.conf file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/admin.conf On Kubernetes version 1.29 and higher run the following command as well :- stat -c %U:%G /etc/kubernetes/super-admin.conf Verify that the ownership is set to root:root.

None.

Automated

As part of initial cluster setup, default kubeconfig files are created to be used by the administrator of the cluster. These files contain private keys and certificates which allow for privileged access to the cluster. You should set their file ownership to maintain the integrity and confidentiality of the file. The file(s) should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/admin.conf On Kubernetes 1.29+ the super-admin.conf file should also be modified, if present. For example, chown root:root /etc/kubernetes/super-admin.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the scheduler.conf file has permissions of 600 or more restrictive.

By default, scheduler.conf has permissions of 640.

Run the following command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/scheduler.conf Verify that the permissions are 600 or more restrictive.

None

Automated

The scheduler.conf file is the kubeconfig file for the Scheduler. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/scheduler.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the scheduler.conf file ownership is set to root:root.

By default, scheduler.conf file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/scheduler.conf Verify that the ownership is set to root:root.

None

Automated

The scheduler.conf file is the kubeconfig file for the Scheduler. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/scheduler.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the controller-manager.conf file has permissions of 600 or more restrictive.

By default, controller-manager.conf has permissions of 640.

Run the following command (based on the file location on your system) on the Control Plane node. For example, stat -c %a /etc/kubernetes/controller-manager.conf Verify that the permissions are 600 or more restrictive.

None

Automated

The controller-manager.conf file is the kubeconfig file for the Controller Manager. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod 600 /etc/kubernetes/controller-manager.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the controller-manager.conf file ownership is set to root:root.

By default, controller-manager.conf file ownership is set to root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c %U:%G /etc/kubernetes/controller-manager.conf Verify that the ownership is set to root:root.

None

Automated

The controller-manager.conf file is the kubeconfig file for the Controller Manager. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown root:root /etc/kubernetes/controller-manager.conf

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that the Kubernetes PKI directory and file ownership is set to root:root.

By default, the /etc/kubernetes/pki/ directory and all of the files and directories contained within it, are set to be owned by the root user.

Run the below command (based on the file location on your system) on the Control Plane node. For example, ls -laR /etc/kubernetes/pki/ Verify that the ownership of all files and directories in this hierarchy is set to root:root.

None

Automated

Kubernetes makes use of a number of certificates as part of its operation. You should set the ownership of the directory containing the PKI information and all files in that directory to maintain their integrity. The directory and files should be owned by root:root.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chown -R root:root /etc/kubernetes/pki/

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that Kubernetes PKI certificate files have permissions of 644 or more restrictive.

By default, the certificates used by Kubernetes are set to have permissions of 644

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c '%a' /etc/kubernetes/pki/*.crt Verify that the permissions are 644 or more restrictive. or ls -l /etc/kubernetes/pki/*.crt Verify -rw------

None

Manual

Kubernetes makes use of a number of certificate files as part of the operation of its components. The permissions on these files should be set to 644 or more restrictive to protect their integrity and confidentiality.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod -R 644 /etc/kubernetes/pki/*.crt

Level 1

1 Control Plane Components

1.1 Control Plane Node Configuration Files

Ensure that Kubernetes PKI key files have permissions of 600.

By default, the keys used by Kubernetes are set to have permissions of 600

Run the below command (based on the file location on your system) on the Control Plane node. For example, stat -c '%a' /etc/kubernetes/pki/*.key Verify that the permissions are 600 or more restrictive. or ls -l /etc/kubernetes/pki/*.key Verify that the permissions are -rw------

None

Manual

Kubernetes makes use of a number of key files as part of the operation of its components. The permissions on these files should be set to 600 to protect their integrity and confidentiality.

Run the below command (based on the file location on your system) on the Control Plane node. For example, chmod -R 600 /etc/kubernetes/pki/*.key

Level 1

1 Control Plane Components

1.2 API Server

Disable anonymous requests to the API server.

By default, anonymous access is enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --anonymous-auth argument is set to false. Alternative Audit Method kubectl get pod -nkube-system -lcomponent=kube-apiserver -o=jsonpath='{range .items[*]}{.spec.containers[*].command} {\}{end}' | grep '--anonymous-auth' | grep -i false If the exit code is '1', then the control isn't present / failed

Anonymous requests will be rejected.

Manual

When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the API server. You should rely on authentication to authorize access and disallow anonymous requests. If you are using RBAC authorization, it is generally considered reasonable to allow anonymous access to the API Server for health checks and discovery purposes, and hence this recommendation is not scored. However, you should consider whether anonymous discovery is an acceptable risk for your purposes.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the below parameter. --anonymous-auth=false

Level 1

1 Control Plane Components

1.2 API Server

Do not use token based authentication.

By default, --token-auth-file argument is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --token-auth-file argument does not exist. Alternative Audit Method kubectl get pod -nkube-system -lcomponent=kube-apiserver -o=jsonpath='{range .items[*]}{.spec.containers[*].command} {\}{end}' | grep '--token-auth-file' | grep -i false If the exit code is '1', then the control isn't present / failed

You will have to configure and use alternate authentication mechanisms such as certificates. Static token based authentication could not be used.

Automated

The token-based authentication utilizes static tokens to authenticate requests to the apiserver. The tokens are stored in clear-text in a file on the apiserver, and cannot be revoked or rotated without restarting the apiserver. Hence, do not use static token-based authentication.

Follow the documentation and configure alternate mechanisms for authentication. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and remove the --token-auth-file=<filename> parameter.

Level 1

1 Control Plane Components

1.2 API Server

This admission controller rejects all net-new usage of the Service field externalIPs.

By default, --enable-admission-plugins=DenyServiceExternalIP argument is not set, and the use of externalIPs is authorized.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the `DenyServiceExternalIPs' argument exist as a string value in --enable-admission-plugins.

When enabled, users of the cluster may not create new Services which use externalIPs and may not add new values to externalIPs on existing Service objects.

Manual

Most users do not need the ability to set the externalIPs field for a Service at all, and cluster admins should consider disabling this functionality by enabling the DenyServiceExternalIPs admission controller. Clusters that do need to allow this functionality should consider using some custom policy to manage its usage.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and append the Kubernetes API server flag --enable-admission-plugins with the DenyServiceExternalIPs plugin. Note, the Kubernetes API server flag --enable-admission-plugins takes a comma-delimited list of admission control plugins to be enabled, even if they are in the list of plugins enabled by default. kube-apiserver --enable-admission-plugins=DenyServiceExternalIPs

Level 1

1 Control Plane Components

1.2 API Server

Enable certificate based kubelet authentication.

By default, certificate-based kubelet authentication is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --kubelet-client-certificate and --kubelet-client-key arguments exist and they are set as appropriate. Alternative Audit kubectl get pod -nkube-system -lcomponent=kube-apiserver -o=jsonpath='{range .items[*]}{.spec.containers[*].command} {\}{end}' | grep '--kubelet-client-certificate' | grep -i false If the exit code is '1', then the control isn't present / failed

You require TLS to be configured on apiserver as well as kubelets.

Automated

The apiserver, by default, does not authenticate itself to the kubelet's HTTPS endpoints. The requests from the apiserver are treated anonymously. You should set up certificate-based kubelet authentication to ensure that the apiserver authenticates itself to kubelets when submitting requests.

Follow the Kubernetes documentation and set up the TLS connection between the apiserver and kubelets. Then, edit API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the kubelet client certificate and key parameters as below. --kubelet-client-certificate=<path/to/client-certificate-file> --kubelet-client-key=<path/to/client-key-file>

Level 1

1 Control Plane Components

1.2 API Server

Verify kubelet's certificate before establishing connection.

By default, --kubelet-certificate-authority argument is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --kubelet-certificate-authority argument exists and is set as appropriate. Alternative Audit kubectl get pod -nkube-system -lcomponent=kube-apiserver -o=jsonpath='{range .items[]}{.spec.containers[].command} {\}{end}' | grep '--kubelet-certificate-authority' | grep -i false If the exit code is '1', then the control isn't present / failed

You require TLS to be configured on apiserver as well as kubelets.

Automated

The connections from the apiserver to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the apiserver does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks.

Follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelets. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. --kubelet-certificate-authority=<ca-string>

Level 1

1 Control Plane Components

1.2 API Server

Do not always authorize all requests.

By default, AlwaysAllow is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --authorization-mode argument exists and is not set to AlwaysAllow.

Only authorized requests will be served.

Automated

The API Server, can be configured to allow all requests. This mode should not be used on any production cluster.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --authorization-mode parameter to values other than AlwaysAllow. One such example could be as below. --authorization-mode=RBAC

Level 1

1 Control Plane Components

1.2 API Server

Restrict kubelet nodes to reading only objects associated with them.

By default, Node authorization is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --authorization-mode argument exists and is set to a value to include Node.

None

Automated

The Node authorization mode only allows kubelets to read Secret, ConfigMap, PersistentVolume, and PersistentVolumeClaim objects associated with their nodes.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --authorization-mode parameter to a value that includes Node. --authorization-mode=Node,RBAC

Level 1

1 Control Plane Components

1.2 API Server

Turn on Role Based Access Control.

By default, RBAC authorization is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --authorization-mode argument exists and is set to a value to include RBAC.

When RBAC is enabled you will need to ensure that appropriate RBAC settings (including Roles, RoleBindings and ClusterRoleBindings) are configured to allow appropriate access.

Automated

Role Based Access Control (RBAC) allows fine-grained control over the operations that different entities can perform on different objects in the cluster. It is recommended to use the RBAC authorization mode.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --authorization-mode parameter to a value that includes RBAC, for example: --authorization-mode=Node,RBAC

Level 1

1 Control Plane Components

1.2 API Server

Limit the rate at which the API server accepts requests.

By default, EventRateLimit is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --enable-admission-plugins argument is set to a value that includes EventRateLimit.

You need to carefully tune in limits as per your environment.

Manual

Using EventRateLimit admission control enforces a limit on the number of events that the API Server will accept in a given time slice. A misbehaving workload could overwhelm and DoS the API Server, making it unavailable. This particularly applies to a multi-tenant cluster, where there might be a small percentage of misbehaving tenants which could have a significant impact on the performance of the cluster overall. Hence, it is recommended to limit the rate of events that the API server will accept.

Follow the Kubernetes documentation and set the desired limits in a configuration file. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml and set the below parameters. --enable-admission-plugins=...,EventRateLimit,... --admission-control-config-file=<path/to/configuration/file>

Level 1

1 Control Plane Components

1.2 API Server

Do not allow all requests.

AlwaysAdmit is not in the list of default admission plugins.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that if the --enable-admission-plugins argument is set, its value does not include AlwaysAdmit.

Only requests explicitly allowed by the admissions control plugins would be served.

Automated

Setting admission control plugin AlwaysAdmit allows all requests and do not filter any requests. The AlwaysAdmit admission controller was deprecated in Kubernetes v1.13. Its behavior was equivalent to turning off all admission controllers.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and either remove the --enable-admission-plugins parameter, or set it to a value that does not include AlwaysAdmit.

Level 1

1 Control Plane Components

1.2 API Server

Always pull images.

By default, AlwaysPullImages is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --enable-admission-plugins argument is set to a value that includes AlwaysPullImages.

Credentials would be required to pull the private images every time. Also, in trusted environments, this might increases load on network, registry, and decreases speed. This setting could impact offline or isolated clusters, which have images preloaded and do not have access to a registry to pull in-use images. This setting is not appropriate for clusters which use this configuration.

Manual

Setting admission control policy to AlwaysPullImages forces every new pod to pull the required images every time. In a multi-tenant cluster users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission control policy, once an image has been pulled to a node, any pod from any user can use it simply by knowing the image’s name, without any authorization check against the image ownership. When this plug-in is enabled, images are always pulled prior to starting containers, which means valid credentials are required.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --enable-admission-plugins parameter to include AlwaysPullImages. --enable-admission-plugins=...,AlwaysPullImages,...

Level 2

1 Control Plane Components

1.2 API Server

Automate service accounts management.

By default, ServiceAccount is set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --disable-admission-plugins argument is set to a value that does not includes ServiceAccount.

None.

Automated

When you create a pod, if you do not specify a service account, it is automatically assigned the default service account in the same namespace. You should create your own service account and let the API server manage its security tokens.

Follow the documentation and create ServiceAccount objects as per your environment. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and ensure that the --disable-admission-plugins parameter is set to a value that does not include ServiceAccount.

Level 2

1 Control Plane Components

1.2 API Server

Reject creating objects in a namespace that is undergoing termination.

By default, NamespaceLifecycle is set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --disable-admission-plugins argument is set to a value that does not include NamespaceLifecycle.

None

Automated

Setting admission control policy to NamespaceLifecycle ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --disable-admission-plugins parameter to ensure it does not include NamespaceLifecycle.

Level 2

1 Control Plane Components

1.2 API Server

Limit the Node and Pod objects that a kubelet could modify.

By default, NodeRestriction is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --enable-admission-plugins argument is set to a value that includes NodeRestriction.

None

Automated

Using the NodeRestriction plug-in ensures that the kubelet is restricted to the Node and Pod objects that it could modify as defined. Such kubelets will only be allowed to modify their own Node API object, and only modify Pod API objects that are bound to their node.

Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to a value that includes NodeRestriction. --enable-admission-plugins=...,NodeRestriction,...

Level 1

1 Control Plane Components

1.2 API Server

Disable profiling, if not needed.

By default, profiling is enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --profiling argument is set to false.

Profiling information would not be available.

Automated

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the below parameter. --profiling=false

Level 1

1 Control Plane Components

1.2 API Server

Enable auditing on the Kubernetes API Server and set the desired audit log path.

By default, auditing is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --audit-log-path argument is set as appropriate.

None

Automated

Auditing the Kubernetes API Server provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. Even though currently, Kubernetes provides only basic audit capabilities, it should be enabled. You can enable it by setting an appropriate audit log path.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --audit-log-path parameter to a suitable path and file where you would like audit logs to be written, for example: --audit-log-path=/var/log/apiserver/audit.log

Level 1

1 Control Plane Components

1.2 API Server

Retain the logs for at least 30 days or as appropriate.

By default, auditing is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --audit-log-maxage argument is set to 30 or as appropriate.

None

Automated

Retaining logs for at least 30 days ensures that you can go back in time and investigate or correlate any events. Set your audit log retention period to 30 days or as per your business requirements.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days: --audit-log-maxage=30

Level 1

1 Control Plane Components

1.2 API Server

Retain 10 or an appropriate number of old log files.

By default, auditing is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --audit-log-maxbackup argument is set to 10 or as appropriate.

None

Automated

Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. For example, if you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate value. --audit-log-maxbackup=10

Level 1

1 Control Plane Components

1.2 API Server

Rotate log files on reaching 100 MB or as appropriate.

By default, auditing is not enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --audit-log-maxsize argument is set to 100 or as appropriate.

None

Automated

Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. If you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. For example, to set it as 100 MB: --audit-log-maxsize=100

Level 1

1 Control Plane Components

1.2 API Server

Set global request timeout for API server requests as appropriate.

By default, --request-timeout is set to 60 seconds.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --request-timeout argument is either not set or set to an appropriate value.

None

Manual

Setting global request timeout allows extending the API server request timeout limit to a duration appropriate to the user's connection speed. By default, it is set to 60 seconds which might be problematic on slower connections making cluster resources inaccessible once the data volume for requests exceeds what can be transmitted in 60 seconds. But, setting this timeout limit to be too large can exhaust the API server resources making it prone to Denial-of-Service attack. Hence, it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml and set the below parameter as appropriate and if needed. For example, --request-timeout=300s

Level 1

1 Control Plane Components

1.2 API Server

Validate service account before validating token.

By default, --service-account-lookup argument is set to true.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that if the --service-account-lookup argument exists it is set to true.

None

Automated

If --service-account-lookup is not enabled, the apiserver only verifies that the authentication token is valid, and does not validate that the service account token mentioned in the request is actually present in etcd. This allows using a service account token even after the corresponding service account is deleted. This is an example of time of check to time of use security issue.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the below parameter. --service-account-lookup=true Alternatively, you can delete the --service-account-lookup parameter from this file so that the default takes effect.

Level 1

1 Control Plane Components

1.2 API Server

Explicitly set a service account public key file for service accounts on the apiserver.

By default, --service-account-key-file argument is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --service-account-key-file argument exists and is set as appropriate.

The corresponding private key must be provided to the controller manager. You would need to securely maintain the key file and rotate the keys based on your organization's key rotation policy.

Automated

By default, if no --service-account-key-file is specified to the apiserver, it uses the private key from the TLS serving certificate to verify service account tokens. To ensure that the keys for service account tokens could be rotated as needed, a separate public/private key pair should be used for signing service account tokens. Hence, the public key should be specified to the apiserver with --service-account-key-file.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --service-account-key-file parameter to the public key file for service accounts: --service-account-key-file=<filename>

Level 1

1 Control Plane Components

1.2 API Server

etcd should be configured to make use of TLS encryption for client connections.

By default, --etcd-certfile and --etcd-keyfile arguments are not set

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --etcd-certfile and --etcd-keyfile arguments exist and they are set as appropriate.

TLS and client certificate authentication must be configured for etcd.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a client certificate and key.

Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the etcd certificate and key file parameters. --etcd-certfile=<path/to/client-certificate-file> --etcd-keyfile=<path/to/client-key-file>

Level 1

1 Control Plane Components

1.2 API Server

Setup TLS connection on the API server.

By default, --tls-cert-file and --tls-private-key-file are presented and created for use.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate.

TLS and client certificate authentication must be configured for your Kubernetes cluster deployment.

Automated

API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic.

Follow the Kubernetes documentation and set up the TLS connection on the apiserver. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the TLS certificate and private key file parameters. --tls-cert-file=<path/to/tls-certificate-file> --tls-private-key-file=<path/to/tls-key-file>

Level 1

1 Control Plane Components

1.2 API Server

Setup TLS connection on the API server.

By default, --client-ca-file argument is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --client-ca-file argument exists and it is set as appropriate.

TLS and client certificate authentication must be configured for your Kubernetes cluster deployment.

Automated

API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic. If --client-ca-file argument is set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.

Follow the Kubernetes documentation and set up the TLS connection on the apiserver. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the client certificate authority file. --client-ca-file=<path/to/client-ca-file>

Level 1

1 Control Plane Components

1.2 API Server

etcd should be configured to make use of TLS encryption for client connections.

By default, --etcd-cafile is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --etcd-cafile argument exists and it is set as appropriate.

TLS and client certificate authentication must be configured for etcd.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a SSL Certificate Authority file.

Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the etcd certificate authority file parameter. --etcd-cafile=<path/to/ca-file>

Level 1

1 Control Plane Components

1.2 API Server

Encrypt etcd key-value store.

By default, --encryption-provider-config is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --encryption-provider-config argument is set to a EncryptionConfig file. Additionally, ensure that the EncryptionConfig file has all the desired resources covered especially any secrets.

None

Manual

etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures.

Follow the Kubernetes documentation and configure a EncryptionConfig file. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --encryption-provider-config parameter to the path of that file: --encryption-provider-config=</path/to/EncryptionConfig/File>

Level 1

1 Control Plane Components

1.2 API Server

Where etcd encryption is used, appropriate providers should be configured.

By default, no encryption provider is set.

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Get the EncryptionConfig file set for --encryption-provider-config argument. Verify that aescbc, kms, or secretbox is set as the encryption provider for all the desired resources.

None

Manual

Where etcd encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the aescbc, kms, and secretbox are likely to be appropriate options.

Follow the Kubernetes documentation and configure a EncryptionConfig file. In this file, choose aescbc, kms, or secretbox as the encryption provider.

Level 1

1 Control Plane Components

1.2 API Server

Ensure that the API server is configured to only use strong cryptographic ciphers.

By default the Kubernetes API server supports a wide range of TLS ciphers

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --tls-cipher-suites argument is set as outlined in the remediation procedure below.

API server clients that cannot support modern cryptographic ciphers will not be able to make connections to the API server.

Manual

TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS cipher suites including some that have security concerns, weakening the protection provided.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the below parameter. --tls-cipher-suites=TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256.

Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_RC4_128_SHA.

Level 1

1 Control Plane Components

1.2 API Server

By default Kubernetes extends service account token lifetimes to one year to aid in transition from the legacy token settings.

By default, this parameter is set to true

Run the following command on the Control Plane node: ps -ef | grep kube-apiserver Verify that the --service-account-extend-token-expiration argument is set to false.

Disabling this setting means that the service account token expiry set in the cluster will be enforced, and service account tokens will expire at the end of that time frame.

Automated

This default setting is not ideal for security as it ignores other settings related to maximum token lifetime and means that a lost or stolen credential could be valid for an extended period of time.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the Control Plane node and set the --service-account-extend-token-expiration parameter to false. --service-account-extend-token-expiration=false

Level 1

1 Control Plane Components

1.3 Controller Manager

Activate garbage collector on pod termination, as appropriate.

By default, --terminated-pod-gc-threshold is set to 12500.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --terminated-pod-gc-threshold argument is set as appropriate.

None

Manual

Garbage collection is important to ensure sufficient resource availability and avoiding degraded performance and availability. In the worst case, the system might crash or just be unusable for a long period of time. The current setting for garbage collection is 12,500 terminated pods which might be too high for your system to sustain. Based on your system resources and tests, choose an appropriate threshold value to activate garbage collection.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, for example: --terminated-pod-gc-threshold=10

Level 1

1 Control Plane Components

1.3 Controller Manager

Disable profiling, if not needed.

By default, profiling is enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --profiling argument is set to false.

Profiling information would not be available.

Automated

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and set the below parameter. --profiling=false

Level 1

1 Control Plane Components

1.3 Controller Manager

Use individual service account credentials for each controller.

By default, --use-service-account-credentials is set to false.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --use-service-account-credentials argument is set to true.

Whatever authorizer is configured for the cluster, it must grant sufficient permissions to the service accounts to perform their intended tasks. When using the RBAC authorizer, those roles are created and bound to the appropriate service accounts in the kube-system namespace automatically with default roles and rolebindings that are auto-reconciled on startup. If using other authorization methods (ABAC, Webhook, etc), the cluster deployer is responsible for granting appropriate permissions to the service accounts (the required permissions can be seen by inspecting the controller-roles.yaml and controller-role-bindings.yaml files for the RBAC roles.

Automated

The controller manager creates a service account per controller in the kube-system namespace, generates a credential for it, and builds a dedicated API client with that service account credential for each controller loop to use. Setting the --use-service-account-credentials to true runs each control loop within the controller manager using a separate service account credential. When used in combination with RBAC, this ensures that the control loops run with the minimum permissions required to perform their intended tasks.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node to set the below parameter. --use-service-account-credentials=true

Level 1

1 Control Plane Components

1.3 Controller Manager

Explicitly set a service account private key file for service accounts on the controller manager.

By default, --service-account-private-key-file it not set.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --service-account-private-key-file argument is set as appropriate.

You would need to securely maintain the key file and rotate the keys based on your organization's key rotation policy.

Automated

To ensure that keys for service account tokens can be rotated as needed, a separate public/private key pair should be used for signing service account tokens. The private key should be specified to the controller manager with --service-account-private-key-file as appropriate.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and set the --service-account-private-key-file parameter to the private key file for service accounts. --service-account-private-key-file=<filename>

Level 1

1 Control Plane Components

1.3 Controller Manager

Allow pods to verify the API server's serving certificate before establishing connections.

By default, --root-ca-file is not set.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --root-ca-file argument exists and is set to a certificate bundle file containing the root certificate for the API server's serving certificate.

You need to setup and maintain root certificate authority file.

Automated

Processes running within pods that need to contact the API server must verify the API server's serving certificate. Failing to do so could be a subject to man-in-the-middle attacks. Providing the root certificate for the API server's serving certificate to the controller manager with the --root-ca-file argument allows the controller manager to inject the trusted bundle into pods so that they can verify TLS connections to the API server.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and set the --root-ca-file parameter to the certificate bundle file`. --root-ca-file=<path/to/file>

Level 1

1 Control Plane Components

1.3 Controller Manager

Enable kubelet server certificate rotation on controller-manager.

By default, RotateKubeletServerCertificate is set to true this recommendation verifies that it has not been disabled.

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that RotateKubeletServerCertificate argument exists and is set to true.

None

Automated

RotateKubeletServerCertificate causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that the there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad. Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. --feature-gates=RotateKubeletServerCertificate=true

Level 1

1 Control Plane Components

1.3 Controller Manager

Do not bind the Controller Manager service to non-loopback insecure addresses.

By default, the --bind-address parameter is set to 0.0.0.0

Run the following command on the Control Plane node: ps -ef | grep kube-controller-manager Verify that the --bind-address argument is set to 127.0.0.1

None

Automated

The Controller Manager API service which runs on port 10252/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the Control Plane node and ensure the correct value for the --bind-address parameter

Although the current Kubernetes documentation site says that --address is deprecated in favour of --bind-address Kubeadm 1.11 still makes use of --address

Level 1

1 Control Plane Components

1.4 Scheduler

Disable profiling, if not needed.

By default, profiling is enabled.

Run the following command on the Control Plane node: ps -ef | grep kube-scheduler Verify that the --profiling argument is set to false.

Profiling information would not be available.

Automated

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file on the Control Plane node and set the below parameter. --profiling=false

Level 1

1 Control Plane Components

1.4 Scheduler

Do not bind the scheduler service to non-loopback insecure addresses.

By default, the --bind-address parameter is set to 0.0.0.0

Run the following command on the Control Plane node: ps -ef | grep kube-scheduler Verify that the --bind-address argument is set to 127.0.0.1

None

Automated

The Scheduler API service which runs on port 10251/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface

Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml on the Control Plane node and ensure the correct value for the --bind-address parameter

Level 1

2 etcd

Configure TLS encryption for the etcd service.

By default, TLS encryption is not set.

Run the following command on the etcd server node ps -ef | grep etcd Verify that the --cert-file and the --key-file arguments are set as appropriate.

Client connections only over TLS would be served.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit.

Follow the etcd service documentation and configure TLS encryption. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameters. --cert-file=</path/to/ca-file> --key-file=</path/to/key-file>

Level 1

2 etcd

Configure TLS encryption for the etcd service.

By default, TLS encryption is not set.

Run the following command on the etcd server node ps -ef | grep etcd Verify that the --cert-file and the --key-file arguments are set as appropriate.

Client connections only over TLS would be served.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit.

Follow the etcd service documentation and configure TLS encryption. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameters. --cert-file=</path/to/ca-file> --key-file=</path/to/key-file>

Level 1

2 etcd

Enable client authentication on etcd service.

By default, the etcd service can be queried by unauthenticated clients.

Run the following command on the etcd server node: ps -ef | grep etcd Verify that the --client-cert-auth argument is set to true.

All clients attempting to access the etcd server will require a valid client certificate.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter. --client-cert-auth=true

Level 1

2 etcd

Do not use self-signed certificates for TLS.

By default, --auto-tls is set to false.

Run the following command on the etcd server node: ps -ef | grep etcd Verify that if the --auto-tls argument exists, it is not set to true.

Clients will not be able to use self-signed certificates for TLS.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and either remove the --auto-tls parameter or set it to false. --auto-tls=false

Level 1

2 etcd

etcd should be configured to make use of TLS encryption for peer connections.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable. By default, peer communication over TLS is not configured.

Run the following command on the etcd server node: ps -ef | grep etcd Verify that the --peer-cert-file and --peer-key-file arguments are set as appropriate. Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

etcd cluster peers would need to set up TLS for their communication.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit and also amongst peers in the etcd clusters.

Follow the etcd service documentation and configure peer TLS encryption as appropriate for your etcd cluster. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameters. --peer-client-file=</path/to/peer-cert-file> --peer-key-file=</path/to/peer-key-file>

Level 1

2 etcd

etcd should be configured for peer authentication.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable. By default, --peer-client-cert-auth argument is set to false.

Run the following command on the etcd server node: ps -ef | grep etcd Verify that the --peer-client-cert-auth argument is set to true. Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

All peers attempting to communicate with the etcd server will require a valid client certificate for authentication.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter. --peer-client-cert-auth=true

Level 1

2 etcd

Do not use automatically generated self-signed certificates for TLS connections between peers.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable. By default, --peer-auto-tls argument is set to false.

Run the following command on the etcd server node: ps -ef | grep etcd Verify that if the --peer-auto-tls argument exists, it is not set to true. Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

All peers attempting to communicate with the etcd server will require a valid client certificate for authentication.

Automated

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster. Hence, do not use self-signed certificates for authentication.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and either remove the --peer-auto-tls parameter or set it to false. --peer-auto-tls=false

Level 2

2 etcd

Use a different certificate authority for etcd from the one used for Kubernetes.

By default, no etcd certificate is created and used.

Review the CA used by the etcd environment and ensure that it does not match the CA certificate file used for the management of the overall Kubernetes cluster. Run the following command on the master node: ps -ef | grep etcd Note the file referenced by the --trusted-ca-file argument. Run the following command on the master node: ps -ef | grep apiserver Verify that the file referenced by the --client-ca-file for apiserver is different from the --trusted-ca-file used by etcd.

Additional management of the certificates and keys for the dedicated certificate authority will be required.

Manual

etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only. Authentication to etcd is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the etcd database.

Follow the etcd documentation and create a dedicated certificate authority setup for the etcd service. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter. --trusted-ca-file=</path/to/ca-file>

Level 1

3 Control Plane Configuration

3.1 Authentication and Authorization

Kubernetes provides the option to use client certificates for user authentication. However as there is no way to revoke these certificates when a user leaves an organization or loses their credential, they are not suitable for this purpose. It is not possible to fully disable client certificate use within a cluster as it is used for component to component authentication.

Client certificate authentication is enabled by default.

Review user access to the cluster and ensure that users are not making use of Kubernetes client certificate authentication.

External mechanisms for authentication generally require additional software to be deployed.

Manual

With any authentication mechanism the ability to revoke credentials if they are compromised or no longer required, is a key control. Kubernetes client certificate authentication does not allow for this due to a lack of support for certificate revocation.

Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates.

The lack of certificate revocation was flagged up as a high risk issue in the recent Kubernetes security audit. Without this feature, client certificate authentication is not suitable for end users.

Level 1

3 Control Plane Configuration

3.1 Authentication and Authorization

Kubernetes provides service account tokens which are intended for use by workloads running in the Kubernetes cluster, for authentication to the API server. These tokens are not designed for use by end-users and do not provide for features such as revocation or expiry, making them insecure. A newer version of the feature (Bound service account token volumes) does introduce expiry but still does not allow for specific revocation.

Service account token authentication is enabled by default.

Review user access to the cluster and ensure that users are not making use of service account token authentication.

External mechanisms for authentication generally require additional software to be deployed.

Manual

With any authentication mechanism the ability to revoke credentials if they are compromised or no longer required, is a key control. Service account token authentication does not allow for this due to the use of JWT tokens as an underlying technology.

Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of service account tokens.

Level 1

3 Control Plane Configuration

3.1 Authentication and Authorization

Kubernetes provides bootstrap tokens which are intended for use by new nodes joining the cluster These tokens are not designed for use by end-users they are specifically designed for the purpose of bootstrapping new nodes and not for general authentication

Bootstrap token authentication is not enabled by default and requires an API server parameter to be set.

Review user access to the cluster and ensure that users are not making use of bootstrap token authentication.

External mechanisms for authentication generally require additional software to be deployed.

Manual

Bootstrap tokens are not intended for use as a general authentication mechanism and impose constraints on user and group naming that do not facilitate good RBAC design. They also cannot be used with MFA resulting in a weak authentication mechanism being available.

Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of bootstrap tokens.

Level 1

3 Control Plane Configuration

3.2 Logging

Kubernetes can audit the details of requests made to the API server. The --audit-policy-file flag must be set for this logging to be enabled.

Unless the --audit-policy-file flag is specified, no auditing will be carried out.

Run the following command on one of the cluster master nodes: ps -ef | grep kube-apiserver Verify that the --audit-policy-file is set. Review the contents of the file specified and ensure that it contains a valid audit policy.

Audit logs will be created on the master nodes, which will consume disk space. Care should be taken to avoid generating too large volumes of log information as this could impact the available of the cluster nodes.

Manual

Logging is an important detective control for all systems, to detect potential unauthorised access.

Create an audit policy file for your cluster.

Level 2

3 Control Plane Configuration

3.2 Logging

Ensure that the audit policy created for the cluster covers key security concerns.

By default Kubernetes clusters do not log audit information.

Review the audit policy provided for the cluster and ensure that it covers at least the following areas :- - Access to Secrets managed by the cluster. Care should be taken to only log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in order to avoid the risk of logging sensitive data. - Modification of pod and deployment objects. - Use of pods/exec, pods/portforward, pods/proxy and services/proxy. For most requests, minimally logging at the Metadata level is recommended (the most basic level of logging).

Increasing audit logging will consume resources on the nodes or other log destination.

Manual

Security audit logs should cover access and modification of key resources in the cluster, to enable them to form an effective part of a security environment.

Consider modification of the audit policy in use on the cluster to include these items, at a minimum.

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the kubelet service file has permissions of 600 or more restrictive.

By default, the kubelet service file has permissions of 640.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet service config file. Please set $kubelet_service_config=<PATH><filename> based on the file location on your system for example: export kubelet_service_config=/etc/systemd/system/kubelet.service.d/kubeadm.conf To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Verify that the permissions are 600 or more restrictive.

None

Automated

The kubelet service file controls various parameters that set the behavior of the kubelet service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the each worker node. For example, chmod 600 /etc/systemd/system/kubelet.service.d/kubeadm.conf

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the kubelet service file ownership is set to root:root.

By default, kubelet service file ownership is set to root:root.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet service config file. Please set $kubelet_service_config=<PATH><filename> based on the file location on your system for example: export kubelet_service_config=/etc/systemd/system/kubelet.service.d/kubeadm.conf To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %U:%G /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Verify that the ownership is set to root:root.

None

Automated

The kubelet service file controls various parameters that set the behavior of the kubelet service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the each worker node. For example, chown root:root /etc/systemd/system/kubelet.service.d/kubeadm.conf

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

If kube-proxy is running, and if it is using a file-based kubeconfig file, ensure that the proxy kubeconfig file has permissions of 600 or more restrictive.

By default, proxy file has permissions of 640.

Find the kubeconfig file being used by kube-proxy by running the following command: ps -ef | grep kube-proxy If kube-proxy is running, get the kubeconfig file location from the --kubeconfig parameter. To perform the audit: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %a <path><filename> Verify that a file is specified and it exists with permissions are 600 or more restrictive.

None

Manual

The kube-proxy kubeconfig file controls various parameters of the kube-proxy service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system. It is possible to run kube-proxy with the kubeconfig parameters configured as a Kubernetes ConfigMap instead of a file. In this case, there is no proxy kubeconfig file.

Run the below command (based on the file location on your system) on the each worker node. For example, chmod 600 <proxy kubeconfig file>

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

If kube-proxy is running, ensure that the file ownership of its kubeconfig file is set to root:root.

By default, proxy file ownership is set to root:root.

Find the kubeconfig file being used by kube-proxy by running the following command: ps -ef | grep kube-proxy If kube-proxy is running, get the kubeconfig file location from the --kubeconfig parameter. To perform the audit: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %U:%G <path><filename> Verify that the ownership is set to root:root.

None

Manual

The kubeconfig file for kube-proxy controls various parameters for the kube-proxy service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the each worker node. For example, chown root:root <proxy kubeconfig file>

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the kubelet.conf file has permissions of 600 or more restrictive.

By default, kubelet.conf file has permissions of 600.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet config file. Please set $kubelet_config=<PATH><filename> based on the file location on your system for example: export kubelet_config=/etc/kubernetes/kubelet.conf To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %a /etc/kubernetes/kubelet.conf Verify that the ownership is set to root:root.Verify that the permissions are 600 or more restrictive.

None

Automated

The kubelet.conf file is the kubeconfig file for the node, and controls various parameters that set the behavior and identity of the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the below command (based on the file location on your system) on the each worker node. For example, chmod 600 /etc/kubernetes/kubelet.conf

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the kubelet.conf file ownership is set to root:root.

By default, kubelet.conf file ownership is set to root:root.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet config file. Please set $kubelet_config=<PATH><filename> based on the file location on your system for example: export kubelet_config=/etc/kubernetes/kubelet.conf To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %U:%G /etc/kubernetes/kubelet.conf Verify that the ownership is set to root:root.

None

Automated

The kubelet.conf file is the kubeconfig file for the node, and controls various parameters that set the behavior and identity of the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the below command (based on the file location on your system) on the each worker node. For example, chown root:root /etc/kubernetes/kubelet.conf

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the certificate authorities file has permissions of 644 or more restrictive.

By default no --client-ca-file is specified.

Run the following command: ps -ef | grep kubelet Find the file specified by the --client-ca-file argument. Run the following command: stat -c %a <filename> Verify that the permissions are 644 or more restrictive.

None

Manual

The certificate authorities file controls the authorities used to validate API requests. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command to modify the file permissions of the --client-ca-file chmod 644 <filename>

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that the certificate authorities file ownership is set to root:root.

By default no --client-ca-file is specified.

Run the following command: ps -ef | grep kubelet Find the file specified by the --client-ca-file argument. Run the following command: stat -c %U:%G <filename> Verify that the ownership is set to root:root.

None

Manual

The certificate authorities file controls the authorities used to validate API requests. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command to modify the ownership of the --client-ca-file. chown root:root <filename>

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that if the kubelet refers to a configuration file with the --config argument, that file has permissions of 600 or more restrictive.

By default, the /var/lib/kubelet/config.yaml file as set up by kubeadm has permissions of 600.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet config yaml file. Please set $kubelet_config_yaml=<PATH><filename> based on the file location on your system for example: export kubelet_config_yaml=/var/lib/kubelet/config.yaml To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %a /var/lib/kubelet/config.yaml Verify that the permissions are 600 or more restrictive.

None

Automated

The kubelet reads various parameters, including security settings, from a config file specified by the --config argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (using the config file location identified in the Audit step) chmod 600 /var/lib/kubelet/config.yaml

Level 1

4 Worker Nodes

4.1 Worker Node Configuration Files

Ensure that if the kubelet refers to a configuration file with the --config argument, that file is owned by root:root.

By default, /var/lib/kubelet/config.yaml file as set up by kubeadm is owned by root:root.

Automated AAC auditing has been modified to allow CIS-CAT to input a variable for the <PATH>/<FILENAME> of the kubelet config yaml file. Please set $kubelet_config_yaml=<PATH><filename> based on the file location on your system for example: export kubelet_config_yaml=/var/lib/kubelet/config.yaml To perform the audit manually: Run the below command (based on the file location on your system) on the each worker node. For example, stat -c %U:%G /var/lib/kubelet/config.yamlVerify that the ownership is set to root:root.

None

Automated

The kubelet reads various parameters, including security settings, from a config file specified by the --config argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (using the config file location identied in the Audit step) chown root:root /etc/kubernetes/kubelet.conf

Level 1

4 Worker Nodes

4.2 Kubelet

Disable anonymous requests to the Kubelet server.

By default, anonymous access is enabled.

If using a Kubelet configuration file, check that there is an entry for authentication: anonymous: enabled set to false. Run the following command on each node: ps -ef | grep kubelet Verify that the --anonymous-auth argument is set to false. This executable argument may be omitted, provided there is a corresponding entry set to false in the Kubelet config file.

Anonymous requests will be rejected.

Automated

When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the Kubelet server. You should rely on authentication to authorize access and disallow anonymous requests.

If using a Kubelet config file, edit the file to set authentication: anonymous: enabled to false. If using executable arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --anonymous-auth=false Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Do not allow all requests. Enable explicit authorization.

By default, --authorization-mode argument is set to AlwaysAllow.

Run the following command on each node: ps -ef | grep kubelet If the --authorization-mode argument is present check that it is not set to AlwaysAllow. If it is not present check that there is a Kubelet config file specified by --config, and that file sets authorization: mode to something other than AlwaysAllow. It is also possible to review the running configuration of a Kubelet via the /configz endpoint on the Kubelet API port (typically 10250/TCP). Accessing these with appropriate credentials will provide details of the Kubelet's configuration.

Unauthorized requests will be denied.

Automated

Kubelets, by default, allow all authenticated requests (even anonymous ones) without needing explicit authorization checks from the apiserver. You should restrict this behavior and only allow explicitly authorized requests.

If using a Kubelet config file, edit the file to set authorization: mode to Webhook. If using executable arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_AUTHZ_ARGS variable. --authorization-mode=Webhook Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Enable Kubelet authentication using certificates.

By default, --client-ca-file argument is not set.

Run the following command on each node: ps -ef | grep kubelet Verify that the --client-ca-file argument exists and is set to the location of the client certificate authority file. If the --client-ca-file argument is not present, check that there is a Kubelet config file specified by --config, and that the file sets authentication: x509: clientCAFile to the location of the client certificate authority file.

You require TLS to be configured on apiserver as well as kubelets.

Automated

The connections from the apiserver to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the apiserver does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks. Enabling Kubelet certificate authentication ensures that the apiserver could authenticate the Kubelet before submitting any requests.

If using a Kubelet config file, edit the file to set authentication: x509: clientCAFile to the location of the client CA file. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_AUTHZ_ARGS variable. --client-ca-file=<path/to/client-ca-file> Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Disable the read-only port.

Run the following command on each node: ps -ef | grep kubelet Verify that the --read-only-port argument exists and is set to 0. If the --read-only-port argument is not present, check that there is a Kubelet config file specified by --config. Check that if there is a readOnlyPort entry in the file, it is set to 0.

Removal of the read-only port will require that any service which made use of it will need to be re-configured to use the main Kubelet API.

Manual

The Kubelet process provides a read-only API in addition to the main Kubelet API. Unauthenticated access is provided to this read-only API which could possibly retrieve potentially sensitive information about the cluster.

If using a Kubelet config file, edit the file to set readOnlyPort to 0. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --read-only-port=0 Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Do not disable timeouts on streaming connections.

By default, --streaming-connection-idle-timeout is set to 4 hours.

Run the following command on each node: ps -ef | grep kubelet Verify that the --streaming-connection-idle-timeout argument is not set to 0. If the argument is not present, and there is a Kubelet config file specified by --config, check that it does not set streamingConnectionIdleTimeout to 0.

Long-lived connections could be interrupted.

Manual

Setting idle timeouts ensures that you are protected against Denial-of-Service attacks, inactive connections and running out of ephemeral ports. Note: By default, --streaming-connection-idle-timeout is set to 4 hours which might be too high for your environment. Setting this as appropriate would additionally ensure that such streaming connections are timed out after serving legitimate use cases.

If using a Kubelet config file, edit the file to set streamingConnectionIdleTimeout to a value other than 0. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. --streaming-connection-idle-timeout=5m Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Allow Kubelet to manage iptables.

By default, --make-iptables-util-chains argument is set to true.

Run the following command on each node: ps -ef | grep kubelet Verify that if the --make-iptables-util-chains argument exists then it is set to true. If the --make-iptables-util-chains argument does not exist, and there is a Kubelet config file specified by --config, verify that the file does not set makeIPTablesUtilChains to false.

Kubelet would manage the iptables on the system and keep it in sync. If you are using any other iptables management solution, then there might be some conflicts.

Automated

Kubelets can automatically manage the required changes to iptables based on how you choose your networking options for the pods. It is recommended to let kubelets manage the changes to iptables. This ensures that the iptables configuration remains in sync with pods networking configuration. Manually configuring iptables with dynamic pod network configuration changes might hamper the communication between pods/containers and to the outside world. You might have iptables rules too restrictive or too open.

If using a Kubelet config file, edit the file to set makeIPTablesUtilChains: true. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and remove the --make-iptables-util-chains argument from the KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Do not override node hostnames.

By default, --hostname-override argument is not set.

Run the following command on each node: ps -ef | grep kubelet Verify that --hostname-override argument does not exist. Note This setting is not configurable via the Kubelet config file.

Some cloud providers may require this flag to ensure that hostname matches names issued by the cloud provider. In these environments, this recommendation should not apply.

Manual

Overriding hostnames could potentially break TLS setup between the kubelet and the apiserver. Additionally, with overridden hostnames, it becomes increasingly difficult to associate logs with a particular node and process them for security analytics. Hence, you should setup your kubelet nodes with resolvable FQDNs and avoid overriding the hostnames with IPs.

Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and remove the --hostname-override argument from the KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 2

4 Worker Nodes

4.2 Kubelet

Security relevant information should be captured. The eventRecordQPS on the Kubelet configuration can be used to limit the rate at which events are gathered and sets the maximum event creations per second. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet.

By default, eventRecordQPS argument is set to 5.

Run the following command on each node: sudo grep eventRecordQPS /etc/systemd/system/kubelet.service.d/10-kubeadm.conf or If using command line arguments, kubelet service file is located /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf sudo grep eventRecordQPS /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf Review the value set for the argument and determine whether this has been set to an appropriate level for the cluster. If the argument does not exist, check that there is a Kubelet config file specified by --config and review the value in this location. If using command line arguments

Setting this parameter to 0 could result in a denial of service condition due to excessive events being created. The cluster's event processing and storage systems should be scaled to handle expected event loads.

Manual

It is important to capture all events and not restrict event creation. Events are an important source of security information and analytics that ensure that your environment is consistently monitored using the event data.

If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Setup TLS connection on the Kubelets.

Run the following command on each node: ps -ef | grep kubelet Verify that the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate. If these arguments are not present, check that there is a Kubelet config specified by --config and that it contains appropriate settings for tlsCertFile and tlsPrivateKeyFile.

Manual

The connections from the apiserver to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the apiserver does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks.

If using a Kubelet config file, edit the file to set tlsCertFile to the location of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile to the location of the corresponding private key file. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameters in KUBELET_CERTIFICATE_ARGS variable. --tls-cert-file=<path/to/tls-certificate-file> --tls-private-key-file=<path/to/tls-key-file> Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Enable kubelet client certificate rotation.

By default, kubelet client certificate rotation is enabled.

Run the following command on each node: ps -ef | grep kubelet Verify that the RotateKubeletServerCertificate argument is not present, or is set to true. If the RotateKubeletServerCertificate argument is not present, verify that if there is a Kubelet config file specified by --config, that file does not contain RotateKubeletServerCertificate: false.

None

Automated

The --rotate-certificates setting causes the kubelet to rotate its client certificates by creating new CSRs as its existing credentials expire. This automated periodic rotation ensures that the there is no downtime due to expired certificates and thus addressing availability in the CIA security triad. Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself. Note: This feature also require the RotateKubeletClientCertificate feature gate to be enabled (which is the default since Kubernetes v1.7)

If using a Kubelet config file, edit the file to add the line rotateCertificates: true or remove it altogether to use the default value. If using command line arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS variable or set --rotate-certificates=true . Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Enable kubelet server certificate rotation.

By default, kubelet server certificate rotation is enabled.

Ignore this check if serverTLSBootstrap is true in the kubelet config file or if the --rotate-server-certificates parameter is set on kubelet Run the following command on each node: ps -ef | grep kubelet Verify that RotateKubeletServerCertificate argument exists and is set to true.

None

Manual

RotateKubeletServerCertificate causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that the there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad. Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself.

Edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. --feature-gates=RotateKubeletServerCertificate=true Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

Level 1

4 Worker Nodes

4.2 Kubelet

Ensure that the Kubelet is configured to only use strong cryptographic ciphers.

By default the Kubernetes API server supports a wide range of TLS ciphers

The set of cryptographic ciphers currently considered secure is the following: - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 - TLS_RSA_WITH_AES_256_GCM_SHA384 - TLS_RSA_WITH_AES_128_GCM_SHA256 Run the following command on each node: ps -ef | grep kubelet If the --tls-cipher-suites argument is present, ensure it only contains values included in this set. If it is not present check that there is a Kubelet config file specified by --config, and that file sets tlsCipherSuites: to only include values from this set.

Kubelet clients that cannot support modern cryptographic ciphers will not be able to make connections to the Kubelet API.

Manual

TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS ciphersuites including some that have security concerns, weakening the protection provided.

If using a Kubelet config file, edit the file to set tlsCipherSuites: to TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 or to a subset of these values. If using executable arguments, edit the kubelet service file /etc/kubernetes/kubelet.conf on each worker node and set the --tls-cipher-suites parameter as follows, or to a subset of these values. --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service

The list chosen above should be fine for modern clients. It's essentially the list from the Mozilla Modern cipher option with the ciphersuites supporting CBC mode removed, as CBC has traditionally had a lot of issues

Level 1

4 Worker Nodes

4.2 Kubelet

Ensure that the Kubelet sets limits on the number of PIDs that can be created by pods running on the node.

By default the number of PIDs is not limited.

Review the Kubelet's start-up parameters for the value of --pod-max-pids, and check the Kubelet configuration file for the PodPidsLimit . If neither of these values is set, then there is no limit in place.

Setting this value will restrict the number of processes per pod. If this limit is lower than the number of PIDs required by a pod it will not operate.

Manual

By default pods running in a cluster can consume any number of PIDs, potentially exhausting the resources available on the node. Setting an appropriate limit reduces the risk of a denial of service attack on cluster nodes.

Decide on an appropriate level for this parameter and set it, either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting.

Level 1

4 Worker Nodes

4.2 Kubelet

Ensure that the Kubelet enforces the use of the RuntimeDefault seccomp profile

By default the seccomp profile is not enabled.

Review the Kubelet's start-up parameters for the value of --seccomp-default, and check the Kubelet configuration file for the seccompDefault . If neither of these values is set, then the seccomp profile is not in use.

Setting this will remove some rights from pods running on the node.

Manual

By default, Kubernetes disables the seccomp profile which ships with most container runtimes. Setting this parameter will ensure workloads running on the node are protected by the runtime's seccomp profile.

Set the parameter, either via the --seccomp-default command line parameter or the seccompDefault configuration file setting.

Level 2

4 Worker Nodes

4.2 Kubelet

Ensuring that --IPAddressDeny is set to Any will facilitate allowlisting of only IP addresses that are explicitly set with the --IPAddressAllow parameter which will block unspecified IP addresses from communicating with the kubelet component.

By default IPAddressDeny is not enabled.

Review the Kubelet's start-up parameters for the value of --IPAddressDeny, and check the Kubelet configuration file for IPAddressDeny=any. If this entry is present it should be accompanied by IPAddressAllow={{ kubelet_secure_addresses }} to allow the control plane to communicate with the component.

Configuring the setting IPAddressDeny=any will deny service to any IP address not specified in the complimentary setting IPAddressDeny=any configuration parameter. Applying IPAddressDeny=any alone will completely disable communication with the component.

Manual

By default, Kubernetes allows any IP address to communicate with the kubelet component IP restrictions and IP whitelisting are security best practices and reduce the attack surface of the kubelet.

IPAddressDeny=any IPAddressAllow={{ kubelet_secure_addresses }} *Note kubelet_secure_addresses: localhost link-local {{ kube_pods_subnets | regex_replace(',', ' ') }} {{ kube_node_addresses }} {{ loadbalancer_apiserver.address | default('')

Level 1

4 Worker Nodes

4.3 kube-proxy

Do not bind the kube-proxy metrics port to non-loopback addresses.

The default value is 127.0.0.1:10249

review the start-up flags provided to kube proxy Run the following command on each node: ps -ef | grep -i kube-proxy Ensure that the --metrics-bind-address parameter is not set to a value other than 127.0.0.1. From the output of this command gather the location specified in the --config parameter. Review any file stored at that location and ensure that it does not specify a value other than 127.0.0.1 for metricsBindAddress.

3rd party services which try to access metrics or configuration information related to kube-proxy will require access to the localhost interface of the node.

Manual

kube-proxy has two APIs which provided access to information about the service and can be bound to network ports. The metrics API service includes endpoints (/metrics and /configz) which disclose information about the configuration and operation of kube-proxy. These endpoints should not be exposed to untrusted networks as they do not support encryption or authentication to restrict access to the data they provide.

Modify or remove any values which bind the metrics service to a non-localhost address

Level 1

5 Policies

5.1 RBAC and Service Accounts

The RBAC role cluster-admin provides wide-ranging powers over the environment and should be used only where and when needed.

By default a single clusterrolebinding called cluster-admin is provided with the system:masters group as its principal.

Obtain a list of the principals who have access to the cluster-admin role by reviewing the clusterrolebinding output for each role binding that has access to the cluster-admin role. kubectl get clusterrolebindings -o=custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[*].name Review each principal listed and ensure that cluster-admin privilege is required for it.

Care should be taken before removing any clusterrolebindings from the environment to ensure they were not required for operation of the cluster. Specifically, modifications should not be made to clusterrolebindings with the system: prefix as they are required for the operation of system components.

Manual

Kubernetes provides a set of default roles where RBAC is used. Some of these roles such as cluster-admin provide wide-ranging privileges which should only be applied where absolutely necessary. Roles such as cluster-admin allow super-user access to perform any action on any resource. When used in a ClusterRoleBinding, it gives full control over every resource in the cluster and in all namespaces. When used in a RoleBinding, it gives full control over every resource in the rolebinding's namespace, including the namespace itself.

Identify all clusterrolebindings to the cluster-admin role. Check if they are used and if they need this role or if they could use a role with fewer privileges. Where possible, first bind users to a lower privileged role and then remove the clusterrolebinding to the cluster-admin role : kubectl delete clusterrolebinding [name]

Level 1

5 Policies

5.1 RBAC and Service Accounts

The Kubernetes API stores secrets, which may be service account tokens for the Kubernetes API or credentials used by workloads in the cluster. Access to these secrets should be restricted to the smallest possible group of users to reduce the risk of privilege escalation.

By default in a kubeadm cluster the following list of principals have get privileges on secret objects CLUSTERROLEBINDING SUBJECT TYPE SA-NAMESPACE cluster-admin system:masters Group system:controller:clusterrole-aggregation-controller clusterrole-aggregation-controller ServiceAccount kube-system system:controller:expand-controller expand-controller ServiceAccount kube-system system:controller:generic-garbage-collector generic-garbage-collector ServiceAccount kube-system system:controller:namespace-controller namespace-controller ServiceAccount kube-system system:controller:persistent-volume-binder persistent-volume-binder ServiceAccount kube-system system:kube-controller-manager system:kube-controller-manager User

Review the users who have get, list, or watch access to secrets objects in the Kubernetes API.