Microk8sの証明書の更新

Tags:

踏み台SSHサーバにログインできない事象が発生した．

原因

原因に気づいたきっかけ

実行したコマンド

microk8s kubectl get pod

コマンドの結果

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-01-16T04:20:06Z is after 2023-01-14T14:12:29Z

TLS証明書のエラーでkubectlが使用できなくなった - DENET 技術ブログ

上記の記事よりクライアント証明書の期限が切れると上記のエラーが起きることが分かる．

根本原因

Microk8sのサーバ証明書とフロントプロキシクライアント証明書が期限切れしていた．以下のコマンドで確かめられる．

sudo microk8s.refresh-certs -c

コマンドの結果

CA certificate will expire in 3063 days.
The server certificate will expire in -2 days.
The front proxy client certificate will expire in -2 days.

CA証明書は3063日後に失効する．サーバ証明書は2日前に失効している．フロントプロキシクライアント証明書は2日前に失効している．

対処

以下の記事を参考に証明書を更新する．

Microk8s: Unable to connect to the server: x509: certificate has expired or is not yet valid - DEV Community 👩‍💻👨‍💻

まず，サーバ証明書を以下のコマンドで更新する．

sudo microk8s.refresh-certs -e server.crt

コマンドの結果

Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Restarting service kubelite.
Restarting service cluster-agent.

次にフロントプロキシクライアントの証明書を以下のコマンドで更新する．

sudo microk8s.refresh-certs -e ca.crt

コマンドの結果

Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Signature ok
subject=CN = front-proxy-client
Getting CA Private Key
1
Creating new kubeconfig file
2023-01-16T04:26:32Z INFO Waiting for "snap.microk8s.daemon-kubelite.service" to stop.
Stopped.
Started.

The CA certificates have been replaced. Kubernetes will restart the pods of your workloads.
Any worker nodes you may have in your cluster need to be removed and re-joined to become aware of the new CA.

証明書の期限を以下のコマンドでチェックする．

sudo microk8s.refresh-certs -c

コマンドの結果

The CA certificate will expire in 3649 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in 364 days.

上記の結果から証明書が更新されたことが確認できた．なお，1年後に同じ事象が再発する可能性が高い．

Kubernetesクラスタを以下のコマンドで確認してみる．

microk8s kubectl get pod -n prod-ssh-jump

コマンドの結果

NAME                                 READY   STATUS        RESTARTS   AGE
jumpsv-deployment-56fcf7d64d-grhr5   1/1     Terminating   10         397d
jumpsv-deployment-56fcf7d64d-dz47h   1/1     Terminating   2          298d
jumpsv-deployment-78cd66d785-4g2xk   0/1     Unknown       10         209d
jumpsv-deployment-78cd66d785-bvl2k   0/1     Unknown       10         209d
jumpsv-deployment-78cd66d785-6pm8g   1/1     Running       11         209d

Podが動いていることがわかる．

以下のコマンドでノードの状況を確認する．

microk8s kubectl get node

コマンドの結果

NAME     STATUS     ROLES    AGE    VERSION
rock-j   NotReady   <none>   586d   v1.21.10-3+456ebd2bfd3b5d
rock-r   NotReady   <none>   586d   v1.21.12-3+6937f71915b56b
rock-m   Ready      <none>   586d   v1.21.13-3+cbc10c94808907

rock-jはすでにストレージから消失している．rock-rは存在するため，rock-rでも同様のコマンドを実行する必要がある．

koyama@rock-r:~$ sudo microk8s.refresh-certs -c
The CA certificate will expire in 3063 days.
The server certificate will expire in -222 days.
The front proxy client certificate will expire in -222 days.

koyama@rock-r:~$ sudo microk8s.refresh-certs
Please use the '--cert' flag to select the certificate you need refreshed.

Available certificate options:
'server.crt': refreshes the server certificate
'front-proxy-client.crt': refreshes the front proxy client certificate
'ca.crt': refreshes the root CA and all certificates created from it.
            Warning: refreshing the root CA requires nodes to leave and re-join the cluster
            
koyama@rock-r:~$ sudo microk8s.refresh-certs -e server.crt
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Restarting service kubelite.
Restarting service cluster-agent.
koyama@rock-r:~$ sudo microk8s.refresh-certs -e front-proxy-client.crt
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=CN = front-proxy-client
Getting CA Private Key
Restarting service kubelite.

koyama@rock-r:~$ sudo microk8s.refresh-certs -c
The CA certificate will expire in 3063 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in 364 days.

回復した．

koyama@rock-r:~$ microk8s kubectl get node
NAME     STATUS     ROLES    AGE    VERSION
rock-j   NotReady   <none>   586d   v1.21.10-3+456ebd2bfd3b5d
rock-r   Ready      <none>   586d   v1.21.13-3+cbc10c94808907
rock-m   Ready      <none>   586d   v1.21.13-3+cbc10c94808907