Microk8sの証明書の更新
踏み台SSHサーバにログインできない事象が発生した．
原因
原因に気づいたきっかけ
実行したコマンド
microk8s kubectl get pod
コマンドの結果
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-01-16T04:20:06Z is after 2023-01-14T14:12:29Z
TLS証明書のエラーでkubectlが使用できなくなった - DENET 技術ブログ
上記の記事よりクライアント証明書の期限が切れると上記のエラーが起きることが分かる．
根本原因
Microk8sのサーバ証明書とフロントプロキシクライアント証明書が期限切れしていた．以下のコマンドで確かめられる．
sudo microk8s.refresh-certs -c
コマンドの結果
CA certificate will expire in 3063 days.
The server certificate will expire in -2 days.
The front proxy client certificate will expire in -2 days.
CA証明書は3063日後に失効する． サーバ証明書は2日前に失効している． フロントプロキシクライアント証明書は2日前に失効している．
対処
以下の記事を参考に証明書を更新する．
まず，サーバ証明書を以下のコマンドで更新する．
sudo microk8s.refresh-certs -e server.crt
コマンドの結果
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Restarting service kubelite.
Restarting service cluster-agent.
次にフロントプロキシクライアントの証明書を以下のコマンドで更新する．
sudo microk8s.refresh-certs -e ca.crt
コマンドの結果
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Signature ok
subject=CN = front-proxy-client
Getting CA Private Key
1
Creating new kubeconfig file
2023-01-16T04:26:32Z INFO Waiting for "snap.microk8s.daemon-kubelite.service" to stop.
Stopped.
Started.
The CA certificates have been replaced. Kubernetes will restart the pods of your workloads.
Any worker nodes you may have in your cluster need to be removed and re-joined to become aware of the new CA.
証明書の期限を以下のコマンドでチェックする．
sudo microk8s.refresh-certs -c
コマンドの結果
The CA certificate will expire in 3649 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in 364 days.
上記の結果から証明書が更新されたことが確認できた．なお，1年後に同じ事象が再発する可能性が高い．
Kubernetesクラスタを以下のコマンドで確認してみる．
microk8s kubectl get pod -n prod-ssh-jump
コマンドの結果
NAME READY STATUS RESTARTS AGE
jumpsv-deployment-56fcf7d64d-grhr5 1/1 Terminating 10 397d
jumpsv-deployment-56fcf7d64d-dz47h 1/1 Terminating 2 298d
jumpsv-deployment-78cd66d785-4g2xk 0/1 Unknown 10 209d
jumpsv-deployment-78cd66d785-bvl2k 0/1 Unknown 10 209d
jumpsv-deployment-78cd66d785-6pm8g 1/1 Running 11 209d
Podが動いていることがわかる．
以下のコマンドでノードの状況を確認する．
microk8s kubectl get node
コマンドの結果
NAME STATUS ROLES AGE VERSION
rock-j NotReady <none> 586d v1.21.10-3+456ebd2bfd3b5d
rock-r NotReady <none> 586d v1.21.12-3+6937f71915b56b
rock-m Ready <none> 586d v1.21.13-3+cbc10c94808907
rock-jはすでにストレージから消失している．rock-rは存在するため，rock-rでも同様のコマンドを実行する必要がある．
koyama@rock-r:~$ sudo microk8s.refresh-certs -c
The CA certificate will expire in 3063 days.
The server certificate will expire in -222 days.
The front proxy client certificate will expire in -222 days.
koyama@rock-r:~$ sudo microk8s.refresh-certs
Please use the '--cert' flag to select the certificate you need refreshed.
Available certificate options:
'server.crt': refreshes the server certificate
'front-proxy-client.crt': refreshes the front proxy client certificate
'ca.crt': refreshes the root CA and all certificates created from it.
Warning: refreshing the root CA requires nodes to leave and re-join the cluster
koyama@rock-r:~$ sudo microk8s.refresh-certs -e server.crt
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Restarting service kubelite.
Restarting service cluster-agent.
koyama@rock-r:~$ sudo microk8s.refresh-certs -e front-proxy-client.crt
Taking a backup of the current certificates under /var/snap/microk8s/3410/certs-backup/
Creating new certificates
Signature ok
subject=CN = front-proxy-client
Getting CA Private Key
Restarting service kubelite.
koyama@rock-r:~$ sudo microk8s.refresh-certs -c
The CA certificate will expire in 3063 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in 364 days.
回復した．
koyama@rock-r:~$ microk8s kubectl get node
NAME STATUS ROLES AGE VERSION
rock-j NotReady <none> 586d v1.21.10-3+456ebd2bfd3b5d
rock-r Ready <none> 586d v1.21.13-3+cbc10c94808907
rock-m Ready <none> 586d v1.21.13-3+cbc10c94808907