• About WordPress
    • WordPress.org
    • Documentation
    • Support
    • Feedback
  • Log In
  • Register
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Have any question?

101daysofdevops@gmail.com
RegisterLogin
101DaysofDevops
  • Home
  • About Us
  • Blog
  • Courses
  • Contact Us

Blog

  • Home
  • Blog
  • Blog
  • My road to Certified Kubernetes Security Specialist (CKS)

My road to Certified Kubernetes Security Specialist (CKS)

  • Posted by lakhera2020
  • Date September 21, 2021
  • Comments 0 comment

WARNING: Before reading this doc đꙂ đꙂ

1: As everyone needs to sign NDA with CNCF, I can’t tell you the exact question asked during the exam neither I have GB of memory, but I can give you the pointer what to expect in the exam

2: As we all know, Kubernetes world updates everyday, so some of the stuff might not be relevant after a few days/weeks/month.

3: Please don’t ask for any exam dumps or questions; that defeats the whole purpose of the exam.

I have been preparing for the Kubernetes exam for the last two years (~1 year for CKS). I purchased the exam voucher, then canceled it, re-purchased it again, booked the exam, and then withdrew it, till I reached the point where I only had a few months left either to write the exam or lose $$$.

Exam Preparation

  • KodeKloud is the best platform to learn Kubernetes not only for CKS exam but as a whole. They not only give you enough knowledge to clear exam but give you enough pointer to get going on any concept https://kodekloud.com/courses/certified-kubernetes-security-specialist-cks/
  • Kim Wustkamp excellent udemy course. Kim cover all the concepts in-depth. I highly recommend Kim course for anyone preparing for this exam https://www.udemy.com/course/certified-kubernetes-security-specialist/
  • CNCF youtube channel: The best place to learn any Kubernetes concept in-depth https://www.youtube.com/c/cloudnativefdn

Topics

You must need to know these 6 concepts to clear the exam

  • Admission Controller
  • AppArmor
  • NetworkPolicy
  • Falco
  • PSP
  • Auditing

Some low hanging fruits are

  • Secrets
  • Trivy
  • RBAC
  • CIS Benchmarks
  • Runtimeclasses(gvisor)

I am sharing few of the tricks to make you life easier. These involved some of mistake that I have done during preparation, as most of these concepts are new to me

1. AppArmor

  • Filename and the profile name may be different.
# cat /etc/apparmor.d/backend
#include <tunables/global>
profile my-backend
  • As you can see in the above case filename is backend but the profile name is my-backend. So make sure in the annotation you need to specify profile name(my-backend) not the filename.
annotations:
container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend
  • Similarly the key you are specifying for annotation took container name not the pod name
key: container.apparmor.security.beta.kubernetes.io/<container_name>

So in the below pod definition file,container name is nginx but the pod name is my-site. Make sure as an annotation specify the container name not the pod name.

apiVersion: v1
kind: Pod
metadata:
labels:
run: nginx
name: my-site
namespace: omni
annotations:
container.apparmor.security.beta.kubernetes.io/nginx: localhost/restricted-frontend
spec:
containers:
- image: nginx:alpine
imagePullPolicy: IfNotPresent
name: nginx
  • Annotation will look like this
annotations:
container.apparmor.security.beta.kubernetes.io/nginx: localhost/my-backend

For more info check the Apparmor doc https://kubernetes.io/docs/tutorials/clusters/apparmor/

2. Secrets

  • Remember the command to decode the password
echo "cGFzc3dvcmQK" |base64 -d

OR if you are comfortable with jsonpath you can directly extract the secret from the secret itself 

kubectl -n demo get secrets my-super-secret -o jsonpath='{.data.SUPER_PASSWORD}' | base64 --decode

3. Extracting the image name

  • If you have multiple pod in namespace and you want to get the image name
kubectl describe pod -n demo |grep "Image:" |awk '{print $2}’

Similarly if you are comfortable jsonpath

kubectl -n demo get pods -o json | jq -r '.items[].spec.containers[].image'

4. With trivy you can pass multiple severity level like HIGH and CRITICAL

trivy image --severity HIGH,CRITICAL
  • You can write a simple for loop to go over the image with severity HIGH and CRITICAL
for image in `kubectl describe pod -n demo |grep "Image:" |awk '{print $2}'`; do trivy image --severity HIGH,CRITICAL $image; done

5. Location of seccomp profiles

ps aux|grep -i kubelet
root 3099 5.6 3.6 1959932 73840 ? Ssl 13:51 5:34 /usr/bin/kubelet — bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf — kubeconfig=/etc/kubernetes/kubelet.conf — config=/var/lib/kubelet/config.yaml — network-plugin=cni — pod-infra-container-image=k8s.gcr.io/pause:3.2
# ls -l /var/lib/kubelet/seccomp/profiles/
total 4
-rw-r — r — 1 root root 40 Sep 19 15:29 audit.json
  • To verify if the seccomp profile is loaded
# kubectl describe pod audit-nginx |grep -i seccomp
Annotations: seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/audit.json

For more info check https://kubernetes.io/docs/tutorials/clusters/seccomp/

6. Before making any changes to static pod, don’t forget to take the backup

cp /etc/kubernetes/manifests/kube-apiserver.yaml /root

In case if something got messed, you have an option to revert it or check the logs under

# cd /var/log/pods/
root@controlplane:/var/log/pods# ls -ltr
total 60
drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-controller-manager-controlplane_a875134e700993a22f67999011829566
drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_etcd-controlplane_2f8bb5f0185e15d99a68ed2d44d2f886
drwxr-xr-x 3 root root 4096 Sep 19 13:51 kube-system_kube-scheduler-controlplane_81d2d21449d64d5e6d5e9069a7ca99ed
drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_kube-proxy-bqjr9_a144abe0-9cf4-44f0-97cb-d55799e0fecb
drwxr-xr-x 5 root root 4096 Sep 19 13:52 kube-system_weave-net-wztjk_a57bcb51-b36d-4f14-8a26-f7821b8e768b
drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-24q65_978b7833-c1fb-4a15-82ce-33d4a20c5425
drwxr-xr-x 3 root root 4096 Sep 19 13:52 kube-system_coredns-74ff55c5b-76rf4_2c66c2b7-bf13-4fbf-8489-ba98356d9063
drwxr-xr-x 3 root root 4096 Sep 19 15:01 delta_simple-webapp-2_ef6cebd9-55fe-4ed7-bebd-224fb8840934
drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_app-1401_51453c92-cc07-45d1-a8e7-091dd4022a70
drwxr-xr-x 3 root root 4096 Sep 19 15:03 magnum_app-0403_b8742e0f-2daf-4b8f-8bef-b2f9b3f82336
drwxr-xr-x 3 root root 4096 Sep 19 15:03 default_image-bouncer-webhook-f95dfcbdd-9rgh9_e251d15d-05c1-452f-9af6-d1c37b2155c5
drwxr-xr-x 3 root root 4096 Sep 19 15:09 omni_frontend-site_0d920b3b-35e5-40a4-a02e-9b2f4ade53e1
drwxr-xr-x 3 root root 4096 Sep 19 15:21 orion_app-xyz_0b795e17-51d8-44ed-b2b3-f18c7759b8c4
drwxr-xr-x 3 root root 4096 Sep 19 15:30 default_audit-nginx_5aeaf92c-5ec3-4607-abe4-09692896fcf0
drwxr-xr-x 3 root root 4096 Sep 19 15:32 kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744

After making any changes, always verify if the pod is up in kube-system namespace

# kubectl get pod -n kube-system

You can run it in loop either using -w flag or using watch command

watch kubectl get pod -n kube-system

7. Falco

  • Always check if falco service is up and running(In exam you don’t need to install, it come pre-installed)
systemctl status falco
  • To get the log for falco service
journalctl -fu falco
  • To list all syscall
falco --list=syscall

This is important if you don’t want to go to falco official doc during the exam to look for events https://falco.org/docs/rules/supported-fields/





Remember file_output is disabled by default, don’t forget to enable it

file_output:
enabled: false → true
keep_alive: false
filename: ./events.txt

8. For gvisor the name of the class is in small

spec:
runtimeClassName: gvisor

As it mentioned in camelcase with V capital at most if the places sometime its confusing(gVisor)GitHub – google/gvisor: Application Kernel for Containers
gVisor is an application kernel, written in Go, that implements a substantial portion of the Linux system surface. It…github.com

9. When you want to perform a rolebinding using service account remember the format is namespace:service account

--serviceaccount=namespace:serviceaccountname

So your rolebinding argument accept both namespace and then service account

kubectl create myrolebinding my-role-binding --role=my-write --serviceaccount=demo:developer

10. Don’t Forget to restart the daemon For e.g. If using a Kubelet config file, edit the file to set authorization: mode to Webhook

systemctl daemon-reload
systemctl restart kubelet.service

11. Auditing

  • You can’t enable/create auditing by using kubectl
kubectl create -f /etc/kubernetes/my-audit.yaml
error: unable to recognize "/etc/kubernetes/prod-audit.yaml": no matches for kind "Policy" in version "audit.k8s.io/v1”
  • Sample my-audit.yaml file look like this
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
-
level: RequestResponse
resources:
-
group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
  • To enable it
--audit-policy-file=/etc/kubernetes/audit-policy.yaml \
--audit-log-path=/var/log/audit.log




  • and don’t forget to mount the volume
volumeMounts:
-
mountPath: /etc/kubernetes/audit-policy.yaml
name: audit
readOnly: true
-
mountPath: /var/log/audit.log
name: audit-log
readOnly: false




  • and finally configure the hostPath:
...
-
name: audit
hostPath:
path: /etc/kubernetes/audit-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/audit.log
type: FileOrCreate




Reference: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

In case something got messed up, check Kube-api server logs

# cd /var/log/pods/kube-system_kube-apiserver-controlplane_815c2fc080069ad9aee6a41d37ae1744
# tail -f kube-apiserver/0.log

12. Don’t forget to enable PodSecurity Policy

- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

13. You can’t create admission controller using kubectl

kubectl create -f /etc/kubernetes/pki/admission_configuration.yaml
error: unable to recognize "/etc/kubernetes/pki/admission_configuration.yaml": no matches for kind "AdmissionConfiguration" in version "apiserver.config.k8s.io/v1”

Sample admission controller config file look like this

apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
-
name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: <path-to-kubeconfig-file>
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: true
  • To enable it
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ImagePolicyWebhook
- --admission-control-config-file=/etc/kubernetes/pki/admission_configuration.yaml

For more info

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

Some of the tips I mentioned here is too basic for few people but as I am new to this, these are some of the mistakes I have done while preparing for this exam. Hopefully you will find these tips useful but if you have any other tips you have encountered please leave it in the comment section.

In the end I will say this exam is challenging, so stay calm and give your best shot.

The best way to connect with me is via any of the below mediums

  • Website: https://101daysofdevops.com/
  • Linkedin: https://www.linkedin.com/in/prashant-lakhera-696119b/
  • Twitter: @100daysofdevops OR @lakhera2015
  • Facebook: https://www.facebook.com/groups/795382630808645/
  • Medium: https://medium.com/@devopslearning
  • GitHub: https://github.com/100daysofdevops/100daysofdevops
  • YouTube Channel: https://www.youtube.com/user/laprashant/videos
  • Slack: https://join.slack.com/t/100daysofdevops/shared_invite/zt-au03logz-YfDUp_FJF4rAUeDEbgWmsg
  • Reddit: r/101DaysofDevops
  • Meetup: https://www.meetup.com/100daysofdevops/

Tag:automation, cloud, cncf, devops, docker, kubernetes

  • Share:
author avatar
lakhera2020

Previous post

How to stop/start EC2 instance on a scheduled basis to save cost by using Boto3 and Lambda
September 21, 2021

Next post

My road to Gremlin Chaos Engineering Practitioner Certificate
October 16, 2021

You may also like

Am I reading the iostat command output correctly?
25 April, 2022

Iostat command came from the same sysstat family package # rpm -qf `which iostat` sysstat-11.7.3-6.el8.x86_64 It mainly read data from /proc/diskstats # cat /proc/diskstats 259 0 nvme1n1 147 0 6536 …

Debugging Performance Issue using SAR
21 April, 2022

What is SAR? SAR is a utility used to collect and report system activity. It collects data relating to most core system functions and writes those metrics to binary data …

4 common Kubernetes Pods Error and Debugging
20 April, 2022

Why do Kubernetes Pods fail? The two most common reasons for Kubernetes pod failure is The container inside the pod doesn’t start, which we also call a startup failure. The …

Leave A Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Am I reading the iostat command output correctly?
  • Debugging Performance Issue using SAR
  • 4 common Kubernetes Pods Error and Debugging
  • My road to Gremlin Chaos Engineering Practitioner Certificate
  • My road to Certified Kubernetes Security Specialist (CKS)

Recent Comments

  • lakhera2020 on Debugging Performance Issue using SAR
  • Anonymous on Debugging Performance Issue using SAR
  • Pety on Day 2 – MetalLB Load Balancer for Bare Metal Kubernetes
  • akashambasta on Day 1 – AWS IAM User
  • rd on 100 Days of AWS

 

101daysofdevops@gmail.com

  • Home
  • About Us
  • Courses
  • Blog

© 101daysofdevops. All rights reserved.

Login with your site account

Lost your password?

Not a member yet? Register now

Register a new account

Are you a member? Login now