Day 5: Kubewatch — A watcher for Kubernetes
As per the official github link kubewatch is a Kubernetes watcher that publishes notifications to Slack/hipchat/mattermost/flock channels. It watches the cluster for resource changes and notifies them through webhooks.
Basically kubewatch looks for events like pod/deployment/confimap creation/deletion and sends notification to selected channels like
In this blog, we will use helm to install kubewatch and use slack as the notification medium. But before setting up helm and slack, you must need to have your running Kubernetes cluster. Please check the link on how to configure the Kubernetes cluster using kind.
- Go to your slack workspace, and you have the option to create a new channel, or you can use an existing channel. For this demo, I am going to use an existing channel slacktest(but creating a new channel is pretty easy, you need to click on Create a new channel).
- Once you have a slack channel, the next step is to get a slack token to integrate with Kubewatch. To create a slack token, go to https://api.slack.com/apps/new and create a new app. Give your App Name, for eg: kubewatch, and select your Development Slack Workspace for eg.100daysofdevops in this case.
- In the next step we need to define our token scope. In this case we are using chat:write as scope which gives permission to post messages in approved channels & conversations.
- Now scroll down and click on Oauth & Permissions and note Bot User OAuth Access token.
- One more step you need to perform is to invite the Bot to the channel
Once you have helm installed in your cluster(if you want more information about helm and install it, please follow this link).
- Add the bitnami repo.
helm repo add bitnami https://charts.bitnami.com/bitnami "bitnami" has been added to your repositories
- To verify it
helm repo list NAME URL bitnami https://charts.bitnami.com/bitnami
- Next, we will run a repo update to make sure all the charts are updated and in sync.
helm repo update Update Complete. ⎈Happy Helming!⎈
- Search for kubewatch chart
helm search repo kubewatc NAME CHART VERSION APP VERSION DESCRIPTION bitnami/kubewatch 3.2.17 0.1.0 Kubewatch is a Kubernetes watcher that currentl...h
- As we know all helm chart comes with a default set of values but in this case, we want to modify these values according to our requirement.
helm show values bitnami/kubewatch > ~/kubewatch.yaml
- Now open this file and modify few parameters. First, let start with slack and add the channel to send notification and add token we have created during the slack section.
slack: enabled: true # Slack channel to notify channel: "slacktest" # Slack bots token. Create using: https://my.slack.com/services/new/bot # and invite the bot to your channel using: /join @botname token: "xoxb-XXXXXXX"
- In the next step, we need to define what I want to monitor. So, in this case, I want to monitor only the events from the default namespace(namespaceToWatch: “default”) and the resources to watch(deployment, services, pod). If you want to monitor any other resources, changes the parameter to true(for eg. replicaset: false to replicaset: true).
# namespace to watch, leave it empty for watching all. namespaceToWatch: "default" # Resources to watch resourcesToWatch: deployment: true replicationcontroller: false replicaset: false daemonset: false services: true pod: true job: false persistentvolume: false
- One important parameter you need to set is rbac to true. If you set it to a default value to false, then the service account this helm chart creates doesn’t have access to list Kubernetes resources such as pod, deployments, services, etc.
rbac: create: true
- With all the configuration in place, its time to deploy the helm chart with our customized values.
helm install my-kubewatch bitnami/kubewatch --values ~/kubewatch.yam NAME: my-kubewatch LAST DEPLOYED: Sat Oct 23 08:35:34 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: kubewatch CHART VERSION: 3.2.17 APP VERSION: 0.1.0 ** Please be patient while the chart is being deployed ** To verify that kubewatch has started, run:l
- If you now execute the command shown at the last command’s output, you will see the my-kubewatch pod created in the default namespace.
kubectl get deploy -w --namespace default my-kubewatch NAME READY UP-TO-DATE AVAILABLE AGE my-kubewatch 0/1 1 0 11s my-kubewatch 1/1 1 1 16s
- Try to create any pod in the default namespace
kubectl run nginx2 --image=nginx pod/nginx2 created
- If you go to the slack channel, you will see a notification like this
- You can also tail the logs of your pod to verify these notifications
kubectl logs my-kubewatch-5fd7dbf69d-8hjs ==> Config file exists... time="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-pod time="2021-10-23T15:35:50Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-deployment time="2021-10-23T15:35:50Z" level=info msg="Processing add to deployment: default/my-kubewatch" pkg=kubewatch-deployment time="2021-10-23T15:35:50Z" level=info msg="Processing add to pod: default/my-kubewatch-5fd7dbf69d-8hjsk" pkg=kubewatch-pod time="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-deployment time="2021-10-23T15:35:50Z" level=info msg="Kubewatch controller synced and ready" pkg=kubewatch-pod time="2021-10-23T15:36:03Z" level=info msg="Processing add to pod: default/nginx2" pkg=kubewatch-pod time="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod time="2021-10-23T15:36:03Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod 2021/10/23 15:36:03 Message successfully sent to channel C02JMBPQZP0 at 1635003363.000700 2021/10/23 15:36:04 Message successfully sent to channel C02JMBPQZP0 at 1635003363.000800 time="2021-10-23T15:36:16Z" level=info msg="Processing update to pod: default/nginx2" pkg=kubewatch-pod 2021/10/23 15:36:16 Message successfully sent to channel C02JMBPQZP0 at 1635003376.000900k
Kubewatch is a powerful tool and sends a notification for any events happening in your Kubernetes cluster. But please configure it wisely, as too many notifications in the busy cluster might lead to notification fatigue, and you will start missing the important ones.