어느날 kubernetes 상태 확인을 위해 kubectl을 치자 발생하는 에러 메시지…
E1218 05:21:48.113070 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z E1218 05:21:48.115822 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z E1218 05:21:48.118161 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z E1218 05:21:48.121162 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z E1218 05:21:48.124428 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
x509 같은 키워드를 보아 인증서로 추측 되므로 빠른 구글링으로 조치 방법 확인…
kubeadm certs check-expiration 로 인증서 만료 여부를 확인해보겠습니다…
test@test-master-01:~$ sudo kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [check-expiration] Error reading configuration from the Cluster. Falling back to default configuration CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 05, 2024 15:09 UTC <invalid> ca no apiserver Dec 05, 2024 15:09 UTC <invalid> ca no apiserver-etcd-client Dec 05, 2024 15:09 UTC <invalid> etcd-ca no apiserver-kubelet-client Dec 05, 2024 15:09 UTC <invalid> ca no controller-manager.conf Dec 05, 2024 15:09 UTC <invalid> ca no etcd-healthcheck-client Dec 05, 2024 15:09 UTC <invalid> etcd-ca no etcd-peer Dec 05, 2024 15:09 UTC <invalid> etcd-ca no etcd-server Dec 05, 2024 15:09 UTC <invalid> etcd-ca no front-proxy-client Dec 05, 2024 15:09 UTC <invalid> front-proxy-ca no scheduler.conf Dec 05, 2024 15:09 UTC <invalid> ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Dec 03, 2033 15:09 UTC 8y no etcd-ca Dec 03, 2033 15:09 UTC 8y no front-proxy-ca Dec 03, 2033 15:09 UTC 8y no
기존 인증서 백업 처리
sudo cp -pr /etc/kubernetes/ /etc/kubernetes_backup
인증서 갱신 및 확인
test@test-master-01:~$ sudo kubeadm certs renew all [renew] Reading configuration from the cluster... [renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [renew] Error reading configuration from the Cluster. Falling back to default configuration certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed certificate embedded in the kubeconfig file for the controller manager to use renewed certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed certificate embedded in the kubeconfig file for the scheduler manager to use renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates. test@test-master-01:~$ sudo kubeadm certs check-expiration [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 17, 2025 20:29 UTC 364d ca no apiserver Dec 17, 2025 20:29 UTC 364d ca no apiserver-etcd-client Dec 17, 2025 20:29 UTC 364d etcd-ca no apiserver-kubelet-client Dec 17, 2025 20:29 UTC 364d ca no controller-manager.conf Dec 17, 2025 20:29 UTC 364d ca no etcd-healthcheck-client Dec 17, 2025 20:29 UTC 364d etcd-ca no etcd-peer Dec 17, 2025 20:29 UTC 364d etcd-ca no etcd-server Dec 17, 2025 20:29 UTC 364d etcd-ca no front-proxy-client Dec 17, 2025 20:29 UTC 364d front-proxy-ca no scheduler.conf Dec 17, 2025 20:29 UTC 364d ca no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Dec 03, 2033 15:09 UTC 8y no etcd-ca Dec 03, 2033 15:09 UTC 8y no front-proxy-ca Dec 03, 2033 15:09 UTC 8y no
kubectl을 실행 해도 kube-apiserver, kube-controller-manager, kube-scheduler and etcd 을 재시작 하기 전까지는 계속 오류 발생…
kubectl을 사용하는 계정의 홈디렉토리에도 config에 인증서가 포함되어 있어 해당 파일도 덮어 씌워 줍니다. (root로 실행할 경우 /roo/.kube/config)
test@test-master-01:~$ sudo cp /etc/kubernetes/admin.conf /home/test/.kube/config test@test-master-01:~$ chown test:test /home/test/.kube/config test@test-master-01:~$ ll /home/test/.kube/config -rw------- 1 test test 5650 Dec 18 05:42 /home/test/.kube/config
프로세스 중지 및 서비스 재시작
test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-apiserver) test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-controller-manager) test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-scheduler) test@test-master-01:~$ sudo systemctl restart kubelet test@test-master-01:~$ sudo systemctl daemon-reload
kubectl을 쳐보겠습니다…
test@test-master-01:~$ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx ingress-nginx-controller-6dfcb8658d-8rhbq 1/1 Running 1 (72d ago) 172d kube-system calico-kube-controllers-7ddc4f45bc-d8259 1/1 Running 1 (72d ago) 147d kube-system calico-node-5mk6f 1/1 Running 11 (72d ago) 376d kube-system calico-node-b6jxh 1/1 Running 16 (72d ago) 376d kube-system calico-node-qqmt4 1/1 Running 14 (72d ago) 376d kube-system calico-node-xjhg4 1/1 Running 10 (72d ago) 192d kube-system coredns-5dd5756b68-t2pq8 1/1 Running 11 (72d ago) 377d kube-system coredns-5dd5756b68-tx2xj 1/1 Running 11 (72d ago) 377d kube-system etcd-k8s-master-01 1/1 Running 13 (72d ago) 377d kube-system kube-apiserver-k8s-master-01 1/1 Running 19 (68s ago) 377d kube-system kube-controller-manager-k8s-master-01 1/1 Running 15 (60s ago) 377d kube-system kube-proxy-22kqn 1/1 Running 7 (72d ago) 173d kube-system kube-proxy-b9mbg 1/1 Running 2 (72d ago) 173d kube-system kube-proxy-n4q6t 1/1 Running 7 (72d ago) 173d kube-system kube-proxy-x649t 1/1 Running 6 (72d ago) 173d kube-system kube-scheduler-k8s-master-01 1/1 Running 15 (54s ago) 377d kube-system metrics-server-777dff589b-hmdhl 1/1 Running 2 (72d ago) 169d mariadb-system mariadb-79d8f666bc-jnzlm 1/1 Running 1 (72d ago) 170d metallb-system controller-686877b9fc-9x9mh 1/1 Running 2 (72d ago) 172d metallb-system speaker-7kn6b 1/1 Running 1 (72d ago) 172d metallb-system speaker-c7rrx 1/1 Running 1 (72d ago) 172d metallb-system speaker-hcvlw 1/1 Running 1 (72d ago) 172d
잘 됩니다! 완료!