My road to Certified Kubernetes Security Specialist (CKS)

WARNING: Before reading this doc 🙂 🙂

1: As everyone needs to sign NDA with CNCF, I can’t tell you the exact question asked during the exam neither I have GB of memory, but I can give you the pointer what to expect in the exam

2: As we all know, Kubernetes world updates everyday, so some of the stuff might not be relevant after a few days/weeks/month.

3: Please don’t ask for any exam dumps or questions; that defeats the whole purpose of the exam.

I have been preparing for the Kubernetes exam for the last two years (~1 year for CKS). I purchased the exam voucher, then canceled it, re-purchased it again, booked the exam, and then withdrew it, till I reached the point where I only had a few months left either to write the exam or lose $$$.

Exam Preparation

• KodeKloud is the best platform to learn Kubernetes not only for CKS exam but as a whole. They not only give you enough knowledge to clear exam but give you enough pointer to get going on any concept https://kodekloud.com/courses/certified-kubernetes-security-specialist-cks/
• Kim Wustkamp excellent udemy course. Kim cover all the concepts in-depth. I highly recommend Kim course for anyone preparing for this exam https://www.udemy.com/course/certified-kubernetes-security-specialist/
• CNCF youtube channel: The best place to learn any Kubernetes concept in-depth https://www.youtube.com/c/cloudnativefdn

Topics

You must need to know these 6 concepts to clear the exam

• Admission Controller
• AppArmor
• NetworkPolicy
• Falco
• PSP
• Auditing

Some low hanging fruits are

• Secrets
• Trivy
• RBAC
• CIS Benchmarks
• Runtimeclasses(gvisor)

I am sharing few of the tricks to make you life easier. These involved some of mistake that I have done during preparation, as most of these concepts are new to me

1. AppArmor

• Filename and the profile name may be different.

# cat /etc/apparmor.d/backend

#include <tunables/global>

profile my-backend

• As you can see in the above case filename is backend but the profile name is my-backend. So make sure in the annotation you need to specify profile name(my-backend) not the filename.

annotations:

container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend

• Similarly the key you are specifying for annotation took container name not the pod name

key: container.apparmor.security.beta.kubernetes.io/<container_name>

So in the below pod definition file,container name is nginx but the pod name is my-site. Make sure as an annotation specify the container name not the pod name.

apiVersion: v1

kind: Pod

metadata:

labels:

run: nginx

name: my-site

namespace: omni

annotations:

container.apparmor.security.beta.kubernetes.io/nginx: localhost/restricted-frontend

spec:

containers:

- image: nginx:alpine

imagePullPolicy: IfNotPresent

name: nginx

• Annotation will look like this

annotations:

container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend

For more info check the Apparmor doc https://kubernetes.io/docs/tutorials/clusters/apparmor/

2. Secrets

• Remember the command to decode the password

echo "cGFzc3dvcmQK" |base64 -d

OR if you are comfortable with jsonpath you can directly extract the secret from the secret itself 

kubectl -n demo get secrets my-super-secret -o jsonpath='{.data.SUPER_PASSWORD}' | base64 --decode

3. Extracting the image name

• If you have multiple pod in namespace and you want to get the image name

kubectl describe pod -n demo |grep "Image:" |awk '{print $2}’

Similarly if you are comfortable jsonpath

kubectl -n demo get pods -o json | jq -r '.items[].spec.containers[].image'

4. With trivy you can pass multiple severity level like HIGH and CRITICAL

trivy image --severity HIGH,CRITICAL

• You can write a simple for loop to go over the image with severity HIGH and CRITICAL

for image in `kubectl describe pod -n demo |grep "Image:" |awk '{print $2}'`; do trivy image --severity HIGH,CRITICAL $image; done

5. Location of seccomp profiles

ps aux|grep -i kubelet

root 3099 5.6 3.6 1959932 73840 ? Ssl 13:51 5:34 /usr/bin/kubelet — bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf — kubeconfig=/etc/kubernetes/kubelet.conf — config=/var/lib/kubelet/config.yaml — network-plugin=cni — pod-infra-container-image=k8s.gcr.io/pause:3.2

# ls -l /var/lib/kubelet/seccomp/profiles/

total 4

-rw-r — r — 1 root root 40 Sep 19 15:29 audit.json

• To verify if the seccomp profile is loaded

# kubectl describe pod audit-nginx |grep -i seccomp
Annotations: seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/audit.json

For more info check https://kubernetes.io/docs/tutorials/clusters/seccomp/

6. Before making any changes to static pod, don’t forget to take the backup

cp /etc/kubernetes/manifests/kube-apiserver.yaml /root

In case if something got messed, you have an option to revert it or check the logs under

# cd /var/log/pods/

root@controlplane:/var/log/pods# ls -ltr

total 60

drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-controller-manager-controlplane_a875134e700993a22f67999011829566

drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_etcd-controlplane_2f8bb5f0185e15d99a68ed2d44d2f886

drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-scheduler-controlplane_81d2d21449d64d5e6d5e9069a7ca99ed

drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_kube-proxy-bqjr9_a144abe0-9cf4-44f0-97cb-d55799e0fecb

drwxr-xr-x 5 root root 4096 Sep 19 13:52 kube-system_weave-net-wztjk_a57bcb51-b36d-4f14-8a26-f7821b8e768b

drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-24q65_978b7833-c1fb-4a15-82ce-33d4a20c5425

drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-76rf4_2c66c2b7-bf13-4fbf-8489-ba98356d9063

drwxr-xr-x 3 root root 4096 Sep 19 15:01 delta_simple-webapp-2_ef6cebd9-55fe-4ed7-bebd-224fb8840934

drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_app-1401_51453c92-cc07-45d1-a8e7-091dd4022a70

drwxr-xr-x 3 root root 4096 Sep 19 15:03 magnum_app-0403_b8742e0f-2daf-4b8f-8bef-b2f9b3f82336

drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_image-bouncer-webhook-f95dfcbdd-9rgh9_e251d15d-05c1-452f-9af6-d1c37b2155c5

drwxr-xr-x 3 root root 4096 Sep 19 15:09 omni_frontend-site_0d920b3b-35e5-40a4-a02e-9b2f4ade53e1

drwxr-xr-x 3 root root 4096 Sep 19 15:21 orion_app-xyz_0b795e17-51d8-44ed-b2b3-f18c7759b8c4

drwxr-xr-x 3 root root 4096 Sep 19 15:30 default_audit-nginx_5aeaf92c-5ec3-4607-abe4-09692896fcf0

drwxr-xr-x 3 root root 4096 Sep 19 15:32 kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744

After making any changes, always verify if the pod is up in kube-system namespace

# kubectl get pod -n kube-system

You can run it in loop either using -w flag or using watch command

watch kubectl get pod -n kube-system

7. Falco

• Always check if falco service is up and running(In exam you don’t need to install, it come pre-installed)

systemctl status falco

• To get the log for falco service

journalctl -fu falco

• To list all syscall

falco --list=syscall

This is important if you don’t want to go to falco official doc during the exam to look for events https://falco.org/docs/rules/supported-fields/

Remember file_output is disabled by default, don’t forget to enable it

file_output:

enabled: false → true

keep_alive: false

filename: ./events.txt

8. For gvisor the name of the class is in small

spec:
  runtimeClassName: gvisor

As it mentioned in camelcase with V capital at most if the places sometime its confusing(gVisor)GitHub – google/gvisor: Application Kernel for Containers
gVisor is an application kernel, written in Go, that implements a substantial portion of the Linux system surface. It…github.com

9. When you want to perform a rolebinding using service account remember the format is namespace:service account

--serviceaccount=namespace:serviceaccountname

So your rolebinding argument accept both namespace and then service account

kubectl create myrolebinding my-role-binding --role=my-write --serviceaccount=demo:developer

10. Don’t Forget to restart the daemon For e.g. If using a Kubelet config file, edit the file to set authorization: mode to Webhook

systemctl daemon-reload
systemctl restart kubelet.service

11. Auditing

• You can’t enable/create auditing by using kubectl

kubectl create -f /etc/kubernetes/my-audit.yaml

error: unable to recognize "/etc/kubernetes/prod-audit.yaml": no matches for kind "Policy" in version "audit.k8s.io/v1”

• Sample my-audit.yaml file look like this

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
  resources: ["pods"]

• To enable it

--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
    --audit-log-path=/var/log/audit.log

• and don’t forget to mount the volume

volumeMounts:
  - mountPath: /etc/kubernetes/audit-policy.yaml
    name: audit
    readOnly: true
  - mountPath: /var/log/audit.log
    name: audit-log
    readOnly: false

• and finally configure the hostPath:

...
- name: audit
  hostPath:
    path: /etc/kubernetes/audit-policy.yaml
    type: File

- name: audit-log
  hostPath:
    path: /var/log/audit.log
    type: FileOrCreate

Reference: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

In case something got messed up, check Kube-api server logs

# cd /var/log/pods/kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744
# tail -f kube-apiserver/0.log

12. Don’t forget to enable PodSecurity Policy

- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

13. You can’t create admission controller using kubectl

kubectl create -f /etc/kubernetes/pki/admission_configuration.yaml

error: unable to recognize "/etc/kubernetes/pki/admission_configuration.yaml": no matches for kind "AdmissionConfiguration" in version "apiserver.config.k8s.io/v1”

Sample admission controller config file look like this

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
  configuration:
    imagePolicy:
      kubeConfigFile: <path-to-kubeconfig-file>
      allowTTL: 50
      denyTTL: 50
      retryBackoff: 500
      defaultAllow: true

• To enable it

- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ImagePolicyWebhook

- --admission-control-config-file=/etc/kubernetes/pki/admission_configuration.yaml

For more info

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Some of the tips I mentioned here is too basic for few people but as I am new to this, these are some of the mistakes I have done while preparing for this exam. Hopefully you will find these tips useful but if you have any other tips you have encountered please leave it in the comment section.

In the end I will say this exam is challenging, so stay calm and give your best shot.

The best way to connect with me is via any of the below mediums

• Website: https://101daysofdevops.com/
• Linkedin: https://www.linkedin.com/in/prashant-lakhera-696119b/
• Twitter: @100daysofdevops OR @lakhera2015
• Facebook: https://www.facebook.com/groups/795382630808645/
• Medium: https://medium.com/@devopslearning
• GitHub: https://github.com/100daysofdevops/100daysofdevops
• YouTube Channel: https://www.youtube.com/user/laprashant/videos
• Slack: https://join.slack.com/t/100daysofdevops/shared_invite/zt-au03logz-YfDUp_FJF4rAUeDEbgWmsg
• Reddit: r/101DaysofDevops
• Meetup: https://www.meetup.com/100daysofdevops/