K0s Getting Started Example not working (Traefik Ingress Controller)

All about k0s
#1

I’ve settle on K0s, as K3s documentation and forum was not productive in getting a full working setup with Traefik.

Hopefully this forum has users with enough Traefik and K0s experience.

I hope my efforts will cause some positive changes on k0s Getting started documentation.
As any reader can imagine, wanting a k8s distribution that actually has “Zero Friction” is much needed.

I am trying to do something quite simple and I am sure I am not the only one.
Set up a 3 node cluster in HA with bare minimum dependencies and get metrics showing up in Lens.

For sake of simplicity I have 3 nodes, setup as controller+worker (k0sctl would constantly hang waiting for a Ready state when adding worker nodes to the k0sctl yaml)

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - role: controller+worker
    noTaints: true
    ssh:
      address: 192.168.1.110 
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: controller+worker
    noTaints: true
    ssh:
      address: 192.168.1.114  
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: controller+worker
    noTaints: true
    ssh:
      address: 192.168.1.112  
      user: root
      keyPath: ~/.ssh/id_rsa                  
  k0s:
    version: v1.28.4+k0s.0
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
       name: k0s-cluster
      spec:        
        network:
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy    
        extensions:
          helm:
            repositories:
            - name: traefik
              url: https://traefik.github.io/charts
            - name: bitnami
              url: https://charts.bitnami.com/bitnami
            charts:
            - name: traefik
              chartname: traefik/traefik
              version: "20.5.3"
              namespace: default
            - name: metallb
              chartname: bitnami/metallb
              version: "2.5.4"
              namespace: default
              values: |
                configInline:
                  address-pools:
                  - name: generic-cluster-pool
                    protocol: layer2
                    addresses:
                    - 192.168.1.150-192.168.9.155

The above yaml uses k0sctl, and follows the Traefik Ingress example on k0s official docs.

the yaml applies with no error bu when running k0s kubectl get all traefik does not get an external IP assigned.

service/traefik LoadBalancer 10.96.226.129 <pending> 80:30193/TCP,443:31634/TCP 74s

and I don’t see the daemonset.apps/metallb listed in my pods.

The example listed on k0s is using an outdated example from 2020 linked here From Zero to Hero: Getting Started with k0s and Traefik

Extremely High CPU usage with simple Multi Node cluster. (kube-apis)
#2

k0s yes, Traefik I’m not sure.

Just tried your config locally, of course with different addresses etc., and I see this:

% kubectl get chart -n kube-system k0s-addon-chart-metallb -o json | jq .status
{
  "error": "can't locate chart `bitnami/metallb-2.5.4`: chart \"metallb\" matching 2.5.4 not found in bitnami index. (try 'helm repo update'): no chart version found for metallb-2.5.4",
  "updated": "2023-12-14 10:35:12.974067775 +0200 EET m=+12.165992392",
  "valuesHash": "1820b3cd446cfe1543ebc968e4ddd61d44ad2646e7c79ed5f296413fce9ff7ba"
}

Looks like the version on the examples is bit old and bitnami has purged that from the repo. Check for newer versions and their configuration at metallb 4.8.0 · bitnami/bitnami

#3

@rohanrehman
Yes, this does not work anymore.
The oldes Version for bitnami/metallb is 3.0.1
to fix your issue follow my instructions:

create k0s.yaml
I use every time this
k0s default-config > /var/lib/k0s/k0s.yaml
If you do the same you need to create the folder first then create the template:

mkdir -p /var/lib/k0s && chmod 755 /var/lib/k0s
k0s default-config > /var/lib/k0s/k0s.yaml

under the extensions you can do something like this:

extensions:
    helm:
      concurrencyLevel: 5
      repositories:
      - name: traefik
        url: https://traefik.github.io/charts
      - name: bitnami
        url: https://charts.bitnami.com/bitnami
      charts:
      - name: traefik
        chartname: traefik/traefik
        version: "26.0.0"
        namespace: default
        values: |
         "some values you like to use...."
      - name: metallb
        chartname: bitnami/metallb
        version: "4.8.0"
        namespace: default

you can let the rest as default…

then you need to create a file let us say in a folder
/var/lib/k0s/metallb/ —>
cd /var/lib/k0s && mkdir -p metallb

then you need to configure the IPAddressPool like this

I named this file address-pool.yaml (you can choose the name as your choice)

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: default
spec:
  addresses:
  - 10.0.202.25-10.0.202.30

and 2nd file

i named l2advertisment.yaml (same here name as your choice)

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertisment
  namespace: default

and fire up the node

fire up the controller
k0s install controller -c /var/lib/k0s/k0s.yaml {--single}
or if you like to see what happens try this

for example
k0s server --config=/var/lib/k0s/k0s.yaml --enable-worker --single --debug=true

and after the cluster is running you can apply the address pool and l2advertisment. Important to wait enough time because metallb must run first (delay 1 or 2 Minutes depending of your system)

k0s kubectl apply -f /var/lib/k0s/metallb/address-pool.yaml
k0s kubectl apply -f /var/lib/k0s/metallb/l2advertisment.yaml

after this IP is added to traefik and you can access it by following the next steps

hope this helps

1 Like
#4

more details on why you need to use CRD

how metallb CRD works:
https://metallb.universe.tf/configuration/