I have setup K0smotron Cluster on my Kubernetes cluster. When I try to get the worker to join the K0smotron Cluster using the token generated from K0smotron cluster, I get the following status from systemd.
● k0sworker.service - k0s - Zero Friction Kubernetes
Loaded: loaded (/etc/systemd/system/k0sworker.service; enabled; preset: enabled)
Active: active (running) since Sat 2024-03-16 12:27:51 UTC; 3s ago
Docs: https://docs.k0sproject.io
Main PID: 6034 (k0s)
Tasks: 7
Memory: 22.8M
CPU: 114ms
CGroup: /system.slice/k0sworker.service
└─6034 /usr/local/bin/k0s worker --token-file=/etc/k0s/token/token-file
Mar 16 12:27:51 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 16 12:27:51 k8manw1 k0s[6034]: time="2024-03-16 12:27:51" level=info msg="No cached API server addresses found"
I am not sure what I am missing or how to go about tracing the issue.
Hi @faisal.aziz
Most certainly it’s because the file /etc/k0s/token/token-file doesn’t exist or is empty.
Can you please verify if it exists and that it’s not empty? If it doesn’t exist or it’s empty generate a new one following this guide.
If not please describe the steps you followed to generate the token file (or if you used something else such as k0sctl to deploy the node).
Thank you for getting back.
I used following command on my Kubernetes cluster to generate the token.
kubectl get secret --namespace k0smotron-demo k0smotron-demo-token -o jsonpath='{.data.token}' | base64 -d
which generated the following token
H4sIAAAAAAAC/2xVXY+rOBJ971+RP3DvGEhaN5H2oSGYhARnbOxy8BtgbvNhCCF0h7Da/77qTI+0K81buU7pHMtynZP2FRTDrbp0m8Wn9ZKbj9tYDLfNy4/Fd715WSwWi7wYxup3ladj8SP9GMvLUI2PHzod083iGKPxGFseEzrklbtlEIpYqJAiLNgTQ6PXWGEsmMt8sqUSeoWwE4vQVcgArUZytNme1nAihq5UQ1DcrCsqpkH7KyTAdBoxJkV/zKw+THeq5g1mTNwP1GBXI+0z0IQavKMcAxN4ycWqjCzVS1MeNPSJaKe64PgqgblKKCfamRPz8SsHpjRaC85dwvywZF1+L977vzH2vxgXYVf46koBPIVWkjaYgM/OeotVBmqVtJaMcJ/wM1NRA46yNU6F5Ua+3goH7zxDwqQmvhDgUcBAhY9iEfoa4edsjPDn15sIQVZZYE3UKU3sgypa7WkU3lXNas0ji9Q6YGIVkR2+6vPbQZtwJobdThge6hzeWa2mBMoPjn2HCbYTc4mEoTOtTZ8Fpc8M9g62YsDDQJ0ZMAlt1sEp9tmDILVVUgfsfkFcWp6CcoqsfgZLICqZF9llqGcVcAdcyt8dOof3E4cjt/WZYXXmHbuSGh8Bl6+JBeHR2g8Hy50PVoJUtTZeDUDQ1BFMQmLv7WjrJnFXKpjLNGqYiTuz5TPZpU4+5Sh8sN0eHWciRLu0U/t2B9kn9DGyROTTKZh6bUpPof1BBissjXtW3dsyCuCiziTMdlAVO0wjUbJImmPcvSFmwom35JEHqqaNmajNkH6MoOXtnkoYUr88KFTOCSrd+K0PSecuebM+Zna4PVo9i6rx9QSwk9t8UP7aVVBGGfTbog4VF2sB7S9btvrPrFkfik5R5d8mKjWhDXaZ3bO0CV2vsVziazcCaNhOY1qtHSHA1ch4tO1d4YcdN2ovGsujMlkyEQoqmCvE+ycX+ztD/gSS+crCZ9qEPOPiruu3w2lLEiHKnturk5TkJlo85HNuRztswMcn2uohl+WQinylZ/frD86swS5HxP3SF05up7y85zN4smM8M+GFvPXbvDEJFxhnnHxkGHzlA0prY2cBO1AT9oVYT8QO56zWaeSrR9Gya3HuZ9aMUyyn2wmz0wFFg/THhjTCSg2zvaY0RWvCogOfxSOGILIFZ5e0MRY/wzHD6pOfyzSr80F/7UbLHixYl6LR16y7zGTLmApMwnfgEFxeeTtWKW8OIigjXZMhcS52bsOOS+MdZ9jSM/lIm7VKbHKn9hKxxgoSx8Wk6V36GFeq+jVF57eZbKHTDguTVjykVA5070tJexuEqaO2V3wO96l5s5TVd9K35jhY0dTBtxira277iEjA3PY/Ve0aht27lupAhFlGbX8XwcrKqtEBgZvECjuvNjityTE/mwfp3M+iK4VAuFROX6bb0pFgILXRzGcFIK2UtrqnjrYVGDvavVlk1rPe6mXkACUOjNQe91mzPBxRGTOkuj85Onx7M+YNfacIYvBNEAvigi+evnykl389Df5WDJ/FsFmU49jfNn/8Ya3tn9brr58W+mkhZ+Og5dJ5WSy6tC02iwbdXvJLNxbT+FdU/FV/R8V3bjynvhoft+fpIytMMf7ILpfxNg5p//9sH8NQdOOPv5mezabq9GbhXbrf1ftLPxS/i6Ho8uK2Wfz7Py9frE/xb5J/oH8KP68wXpqi2yxaZI+P4qdtyub6esur67WultP65b8BAAD//+EjmoEGBwAA
And I just checked that’s what I have in /etc/k0s/token/token-file.
Hi @faisal.aziz
After having another look your token seems fine and that log line doesn’t mean what I understood initially). You should get the whole k0s worker logs to see what’s going on and see where does it crash.
Try getting the output of journalctl -u k0sworker | grep 'Started k0sworker.service' -B200
This gets the logs, looks for the Started k0sworker.service and acquires the 200 previous lines.
Maybe there we can find the issue.
It seems I can’t upload a text file here.
Following few line is what I get from journalctl command you gave me, and it’s replicated all over.
Mar 20 11:27:47 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:27:47 k8manw1 k0s[19445]: time="2024-03-20 11:27:47" level=info msg="No cached API server addresses found"
Mar 20 11:32:03 k8manw1 k0s[19445]: Error: configmaps "worker-config-default-1.29" is forbidden: User "system:node:k8manw1" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k8manw1' and this object
Mar 20 11:32:03 k8manw1 systemd[1]: k0sworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 11:32:03 k8manw1 systemd[1]: k0sworker.service: Failed with result 'exit-code'.
Mar 20 11:34:03 k8manw1 systemd[1]: k0sworker.service: Scheduled restart job, restart counter is at 910.
Mar 20 11:34:03 k8manw1 systemd[1]: Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:34:03 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:34:03 k8manw1 k0s[19458]: time="2024-03-20 11:34:03" level=info msg="No cached API server addresses found"
Mar 20 11:38:20 k8manw1 k0s[19458]: Error: configmaps "worker-config-default-1.29" is forbidden: User "system:node:k8manw1" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k8manw1' and this object
Mar 20 11:38:20 k8manw1 systemd[1]: k0sworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 11:38:20 k8manw1 systemd[1]: k0sworker.service: Failed with result 'exit-code'.
Mar 20 11:40:20 k8manw1 systemd[1]: k0sworker.service: Scheduled restart job, restart counter is at 911.
Mar 20 11:40:20 k8manw1 systemd[1]: Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:40:20 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:40:20 k8manw1 k0s[19470]: time="2024-03-20 11:40:20" level=info msg="No cached API server addresses found"
Mar 20 11:44:36 k8manw1 k0s[19470]: Error: configmaps "worker-config-default-1.29" is forbidden: User "system:node:k8manw1" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k8manw1' and this object
Mar 20 11:44:36 k8manw1 systemd[1]: k0sworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 11:44:36 k8manw1 systemd[1]: k0sworker.service: Failed with result 'exit-code'.
Mar 20 11:46:36 k8manw1 systemd[1]: k0sworker.service: Scheduled restart job, restart counter is at 912.
Mar 20 11:46:36 k8manw1 systemd[1]: Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:46:36 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:46:36 k8manw1 k0s[19481]: time="2024-03-20 11:46:36" level=info msg="No cached API server addresses found"
Mar 20 11:50:52 k8manw1 k0s[19481]: Error: configmaps "worker-config-default-1.29" is forbidden: User "system:node:k8manw1" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k8manw1' and this object
Mar 20 11:50:52 k8manw1 systemd[1]: k0sworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 11:50:52 k8manw1 systemd[1]: k0sworker.service: Failed with result 'exit-code'.
Mar 20 11:52:52 k8manw1 systemd[1]: k0sworker.service: Scheduled restart job, restart counter is at 913.
Mar 20 11:52:52 k8manw1 systemd[1]: Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:52:52 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:52:52 k8manw1 k0s[19492]: time="2024-03-20 11:52:52" level=info msg="No cached API server addresses found"
Mar 20 11:57:08 k8manw1 k0s[19492]: Error: configmaps "worker-config-default-1.29" is forbidden: User "system:node:k8manw1" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k8manw1' and this object
Mar 20 11:57:08 k8manw1 systemd[1]: k0sworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 20 11:57:08 k8manw1 systemd[1]: k0sworker.service: Failed with result 'exit-code'.
Mar 20 11:59:08 k8manw1 systemd[1]: k0sworker.service: Scheduled restart job, restart counter is at 914.
Mar 20 11:59:08 k8manw1 systemd[1]: Stopped k0sworker.service - k0s - Zero Friction Kubernetes.
Mar 20 11:59:08 k8manw1 systemd[1]: Started k0sworker.service - k0s - Zero Friction Kubernetes.
It seems like the token is missing the group? perhaps the token was not generated correctly but since the join token request is now gone we can’t know for sure.
Can you request a new token doing:
1- Go to the worker and stop k0s: sudo systemctl stop k0sworker
2- Uinstall k0s in this worker: sudo k0s reset
3- Create a new token request
apiVersion: k0smotron.io/v1beta1
kind: JoinTokenRequest
metadata:
name: new-test-token
namespace: default
spec:
clusterRef:
name: <k0smotron cluster name>
namespace: <k0smotron cluster namespace>
4- Gather the contents of the secret here: kubectl get secret -n <k0smotron cluster namespace> new-test-token -o yaml. You can remove the token itself. I just want to see the labels.
5- Start the node with the new token.
My token request looks something like,
apiVersion: k0smotron.io/v1beta1
kind: JoinTokenRequest
metadata:
name: k0smotron-new-token
namespace: k0smotron-demo
spec:
clusterRef:
name: k0smotron-demo
namespace: k0smotron-demo
Running,
kubectl get secret -n k0smotron-demo k0smotron-demo-token -o yaml
Gives me,
apiVersion: v1
data:
token: ...
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"k0smotron.io/v1beta1","kind":"JoinTokenRequest","metadata":{"annotations":{},"name":"k0smotron-demo-token","namespace":"k0smotron-demo"},"spec":{"clusterRef":{"name":"k0smotron-demo","namespace":"k0smotron-demo"}}}
creationTimestamp: "2024-03-16T09:39:44Z"
labels:
k0smotron.io/cluster: k0smotron-demo
k0smotron.io/role: worker
k0smotron.io/token-request: k0smotron-demo-token
name: k0smotron-demo-token
namespace: k0smotron-demo
ownerReferences:
- apiVersion: k0smotron.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: JoinTokenRequest
name: k0smotron-demo-token
uid: d6b0f19e-c8f1-40f2-86ed-7875a058b678
resourceVersion: "44748"
uid: 02676b72-4805-442b-848e-e30c8bc63b06
type: Opaque
Getting the same error with new token.