Container | IT Garage

어느날 kubernetes 상태 확인을 위해 kubectl을 치자 발생하는 에러 메시지…

E1218 05:21:48.113070 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
E1218 05:21:48.115822 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
E1218 05:21:48.118161 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
E1218 05:21:48.121162 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
E1218 05:21:48.124428 1685746 memcache.go:265] couldn't get current server API group list: Get "https://192.168.26.101:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-12-18T05:21:48+09:00 is after 2024-12-05T15:09:04Z

x509 같은 키워드를 보아 인증서로 추측 되므로 빠른 구글링으로 조치 방법 확인…

kubeadm certs check-expiration 로 인증서 만료 여부를 확인해보겠습니다…

test@test-master-01:~$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 05, 2024 15:09 UTC   <invalid>       ca                      no
apiserver                  Dec 05, 2024 15:09 UTC   <invalid>       ca                      no
apiserver-etcd-client      Dec 05, 2024 15:09 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Dec 05, 2024 15:09 UTC   <invalid>       ca                      no
controller-manager.conf    Dec 05, 2024 15:09 UTC   <invalid>       ca                      no
etcd-healthcheck-client    Dec 05, 2024 15:09 UTC   <invalid>       etcd-ca                 no
etcd-peer                  Dec 05, 2024 15:09 UTC   <invalid>       etcd-ca                 no
etcd-server                Dec 05, 2024 15:09 UTC   <invalid>       etcd-ca                 no
front-proxy-client         Dec 05, 2024 15:09 UTC   <invalid>       front-proxy-ca          no
scheduler.conf             Dec 05, 2024 15:09 UTC   <invalid>       ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 03, 2033 15:09 UTC   8y              no
etcd-ca                 Dec 03, 2033 15:09 UTC   8y              no
front-proxy-ca          Dec 03, 2033 15:09 UTC   8y              no

기존 인증서 백업 처리

sudo cp -pr /etc/kubernetes/ /etc/kubernetes_backup

인증서 갱신 및 확인

test@test-master-01:~$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
test@test-master-01:~$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 17, 2025 20:29 UTC   364d            ca                      no
apiserver                  Dec 17, 2025 20:29 UTC   364d            ca                      no
apiserver-etcd-client      Dec 17, 2025 20:29 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Dec 17, 2025 20:29 UTC   364d            ca                      no
controller-manager.conf    Dec 17, 2025 20:29 UTC   364d            ca                      no
etcd-healthcheck-client    Dec 17, 2025 20:29 UTC   364d            etcd-ca                 no
etcd-peer                  Dec 17, 2025 20:29 UTC   364d            etcd-ca                 no
etcd-server                Dec 17, 2025 20:29 UTC   364d            etcd-ca                 no
front-proxy-client         Dec 17, 2025 20:29 UTC   364d            front-proxy-ca          no
scheduler.conf             Dec 17, 2025 20:29 UTC   364d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 03, 2033 15:09 UTC   8y              no
etcd-ca                 Dec 03, 2033 15:09 UTC   8y              no
front-proxy-ca          Dec 03, 2033 15:09 UTC   8y              no

kubectl을 실행 해도 kube-apiserver, kube-controller-manager, kube-scheduler and etcd 을 재시작 하기 전까지는 계속 오류 발생…

kubectl을 사용하는 계정의 홈디렉토리에도 config에 인증서가 포함되어 있어 해당 파일도 덮어 씌워 줍니다. (root로 실행할 경우 /roo/.kube/config)

test@test-master-01:~$ sudo cp /etc/kubernetes/admin.conf /home/test/.kube/config
test@test-master-01:~$ chown test:test /home/test/.kube/config
test@test-master-01:~$ ll /home/test/.kube/config
-rw------- 1 test test 5650 Dec 18 05:42 /home/test/.kube/config

프로세스 중지 및 서비스 재시작

test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-apiserver)
test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-controller-manager)
test@test-master-01:~$ sudo kill -s SIGHUP $(pidof kube-scheduler)
test@test-master-01:~$ sudo systemctl restart kubelet
test@test-master-01:~$ sudo systemctl daemon-reload

kubectl을 쳐보겠습니다…

test@test-master-01:~$ kubectl get po -A
NAMESPACE          NAME                                                      READY   STATUS      RESTARTS       AGE
ingress-nginx      ingress-nginx-controller-6dfcb8658d-8rhbq                 1/1     Running     1 (72d ago)    172d
kube-system        calico-kube-controllers-7ddc4f45bc-d8259                  1/1     Running     1 (72d ago)    147d
kube-system        calico-node-5mk6f                                         1/1     Running     11 (72d ago)   376d
kube-system        calico-node-b6jxh                                         1/1     Running     16 (72d ago)   376d
kube-system        calico-node-qqmt4                                         1/1     Running     14 (72d ago)   376d
kube-system        calico-node-xjhg4                                         1/1     Running     10 (72d ago)   192d
kube-system        coredns-5dd5756b68-t2pq8                                  1/1     Running     11 (72d ago)   377d
kube-system        coredns-5dd5756b68-tx2xj                                  1/1     Running     11 (72d ago)   377d
kube-system        etcd-k8s-master-01                                        1/1     Running     13 (72d ago)   377d
kube-system        kube-apiserver-k8s-master-01                              1/1     Running     19 (68s ago)   377d
kube-system        kube-controller-manager-k8s-master-01                     1/1     Running     15 (60s ago)   377d
kube-system        kube-proxy-22kqn                                          1/1     Running     7 (72d ago)    173d
kube-system        kube-proxy-b9mbg                                          1/1     Running     2 (72d ago)    173d
kube-system        kube-proxy-n4q6t                                          1/1     Running     7 (72d ago)    173d
kube-system        kube-proxy-x649t                                          1/1     Running     6 (72d ago)    173d
kube-system        kube-scheduler-k8s-master-01                              1/1     Running     15 (54s ago)   377d
kube-system        metrics-server-777dff589b-hmdhl                           1/1     Running     2 (72d ago)    169d
mariadb-system     mariadb-79d8f666bc-jnzlm                                  1/1     Running     1 (72d ago)    170d
metallb-system     controller-686877b9fc-9x9mh                               1/1     Running     2 (72d ago)    172d
metallb-system     speaker-7kn6b                                             1/1     Running     1 (72d ago)    172d
metallb-system     speaker-c7rrx                                             1/1     Running     1 (72d ago)    172d
metallb-system     speaker-hcvlw                                             1/1     Running     1 (72d ago)    172d

잘 됩니다! 완료!

구성 내역

Kubernetes 1.28.2
Ubuntu 22.04.3 LTS
Container : cri-dockerd
CNI : calico
구성용 계정 : ubuntuadm
Master node : k8s-master-01
Worker node : k8s-worker-01 ~ 02

구성전 모든 노드의 swap은 off 처리…

[공통]

cri-docker 설치 시작

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo systemctl enable –now docker && sudo systemctl status docker –no-pager
sudo usermod -aG docker ubuntuadm
sudo docker container ls

cri-docker Install
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d ‘”‘ -f 4|sed ‘s/v//g’)
echo $VER
wget https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd-${VER}.amd64.tgz
tar xvf cri-dockerd-${VER}.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/

cri-docker Version Check
cri-dockerd –version

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e ‘s,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,’ /etc/systemd/system/cri-docker.service

sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable –now cri-docker.socket

cri-docker Active Check
sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo systemctl status cri-docker.socket –no-pager

Docker cgroup Change Require to Systemd
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
“exec-opts”: [“native.cgroupdriver=systemd”],
“log-driver”: “json-file”,
“log-opts”: {
“max-size”: “100m”
},
“storage-driver”: “overlay2”
}
EOF

sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo docker info | grep Cgroup

환경 설정

Kernel Forwarding
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl –system

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

필요한 sysctl 파라미터를 설정하면, 재부팅 후에도 값이 유지된다.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

재부팅하지 않고 sysctl 파라미터 적용하기
sudo sysctl –system

Package 설치

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg && \
echo “deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

[Master node]

sudo kubeadm config images pull –cri-socket unix:///run/cri-dockerd.sock
sudo kubeadm init –cri-socket /var/run/cri-dockerd.sock

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes -o wide
kubectl get pods -A
kubectl describe node k8s-master-01

Calico CNI 설치
curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O

curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -O
kubectl apply -f calico.yaml

kubectl get nodes
kubectl get pod –all-namespaces

[worker node]

kubeadm init 실행시 마지막 출력되는 명령어 사용
sudo kubeadm join 192.168.0.101:6443 –token xxxxxxxxxxxx \
–discovery-token-ca-cert-hash sha256:xxxxxxxxxxxx \
–cri-socket /var/run/cri-dockerd.sock

kubectl get nodes

kubectl 명령어 자동 완성

https://kubernetes.io/ko/docs/tasks/tools/included/optional-kubectl-configs-bash-linux/

echo ‘source <(kubectl completion bash)’ >>~/.bashrc
echo ‘alias k=kubectl’ >>~/.bashrc
echo ‘complete -o default -F __start_kubectl k’ >>~/.bashrc

IT Garage

카테고리 보관물: Container

kubectl : certificate has expired or is not yet valid

Kubernetes 기본 설치(Version 1.28)