카테고리 보관물: IT

Kubernetes cluster upgrade

2026년 1월 15일 Jason 댓글 남기기

Kubernetes는 기능적/보안적인 이유로 주기적인 Cluster upgrade가 필연적입니다.

현재 버전인 1.32.*에서 1.33.* 버전으로 Upgrade가 목표이며 만일 1.35.* 같은 몇 단계 상위 버전 Upgrade를 계획하더라도 1.33, 1.34, 1.35 버전을 단계별로 순차 진행해야 되는 것은 동일합니다.

Master node가 이중화된 환경은 추가 과정이 있지만 과정 숙달을 목표로 메뉴얼을 기반하여 기본적인 순서로 진행해보겠습니다.

# Kubernetes 메뉴얼
https://v1-33.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/change-package-repository/#verifying-if-the-kubernetes-package-repositories-are-used

[Master Node(Control plane)]

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

sudo kubeadm upgrade plan

sudo kubeadm upgrade apply v1.33.7

kubectl drain test-dev-master-01 --ignore-daemonsets

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon test-dev-master-01

# Log
test@test-dev-master-01:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@test-dev-master-01:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@test-dev-master-01:~$ sudo apt update
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
230 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@test-dev-master-01:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@test-dev-master-01:~$ sudo apt-mark unhold kubeadm && \
> sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
> sudo apt-mark hold kubeadm
kubeadm was already not on hold.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 229 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (33.8 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@test-dev-master-01:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}
test@test-dev-master-01:~$ sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.7
[upgrade/versions] kubeadm version: v1.33.7
I0113 17:54:44.730671 2116008 version.go:261] remote version is much newer: v1.35.0; falling back to: stable-1.33
[upgrade/versions] Target version: v1.33.7
[upgrade/versions] Latest version in the v1.32 series: v1.32.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet     test-dev-master-01   v1.32.7   v1.32.11
kubelet     test-dev-worker-01   v1.32.7   v1.32.11
kubelet     test-dev-worker-02   v1.32.7   v1.32.11

Upgrade to the latest version in the v1.32 series:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver            test-dev-master-01   v1.32.7    v1.32.11
kube-controller-manager   test-dev-master-01   v1.32.7    v1.32.11
kube-scheduler            test-dev-master-01   v1.32.7    v1.32.11
kube-proxy                                    1.32.7     v1.32.11
CoreDNS                                       v1.11.3    v1.12.0
etcd                      test-dev-master-01   3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.32.11

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet     test-dev-master-01   v1.32.7   v1.33.7
kubelet     test-dev-worker-01   v1.32.7   v1.33.7
kubelet     test-dev-worker-02   v1.32.7   v1.33.7

Upgrade to the latest stable version:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver            test-dev-master-01   v1.32.7    v1.33.7
kube-controller-manager   test-dev-master-01   v1.32.7    v1.33.7
kube-scheduler            test-dev-master-01   v1.32.7    v1.33.7
kube-proxy                                    1.32.7     v1.33.7
CoreDNS                                       v1.11.3    v1.12.0
etcd                      test-dev-master-01   3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.7

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

test@test-dev-master-01:~$ sudo kubeadm upgrade apply v1.33.7
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.7"
[upgrade/versions] Cluster version: v1.32.7
[upgrade/versions] kubeadm version: v1.33.7
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.33.7" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3418553132"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0113 18:01:48.219276 2117131 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.7".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
test@test-dev-master-01:~$ kubectl drain test-dev-master-01 --ignore-daemonsets
node/test-dev-master-01 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-km28t, kube-system/kube-proxy-ghl52
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-vv654
pod/calico-kube-controllers-6d5bc68bd-vv654 evicted
node/test-dev-master-01 drained
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready,SchedulingDisabled   control-plane   166d   v1.32.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready                      <none>          166d   v1.32.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready                      <none>          166d   v1.32.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test@test-dev-master-01:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 227 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (51.6 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@test-dev-master-01:~$ sudo systemctl daemon-reload
test@test-dev-master-01:~$ sudo systemctl restart kubelet
test@test-dev-master-01:~$ kubectl uncordon test-dev-master-01
node/test-dev-master-01 uncordoned
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready    control-plane   166d   v1.33.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready    <none>          166d   v1.32.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready    <none>          166d   v1.32.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

[Worker Node #1]

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain test-dev-worker-01 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon test-dev-worker-01

# Log
test@test-dev-worker-01:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@test-dev-worker-01:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@test-dev-worker-01:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:3 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:6 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:7 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 3s (3,513 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
241 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@test-dev-worker-01:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@test-dev-worker-01:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 240 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (28.4 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@test-dev-worker-01:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@test-dev-master-01:~$ kubectl drain test-dev-worker-01 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/test-dev-worker-01 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-fdff8, kube-system/kube-proxy-2rz9d
evicting pod kube-system/coredns-674b8bbfcf-7jxkk
pod/coredns-674b8bbfcf-7jxkk evicted
node/test-dev-worker-01 drained
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready                      control-plane   166d   v1.33.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready,SchedulingDisabled   <none>          166d   v1.32.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready                      <none>          166d   v1.32.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@test-dev-worker-01:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 238 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (49.7 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@test-dev-worker-01:~$ sudo systemctl daemon-reload
test@test-dev-worker-01:~$ sudo systemctl restart kubelet
test@test-dev-worker-01:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@test-dev-master-01:~$ kubectl uncordon test-dev-worker-01
node/test-dev-worker-01 uncordoned
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready    control-plane   166d   v1.33.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready    <none>          166d   v1.33.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready    <none>          166d   v1.32.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

[Worker Node #2]

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain test-dev-worker-02 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon test-dev-worker-02

# Log
test@test-dev-worker-02:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@test-dev-worker-02:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@test-dev-worker-02:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:7 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [19.7 kB]
Get:31 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 4s (3,410 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
229 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@test-dev-worker-02:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@test-dev-worker-02:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 228 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 1s (23.5 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@test-dev-worker-02:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@test-dev-master-01:~$ kubectl drain test-dev-worker-02 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/test-dev-worker-02 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-drhtt, kube-system/kube-proxy-hpkd8
evicting pod kube-system/coredns-674b8bbfcf-f6txf
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-7mppz
pod/calico-kube-controllers-6d5bc68bd-7mppz evicted
pod/coredns-674b8bbfcf-f6txf evicted
node/test-dev-worker-02 drained
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready                      control-plane   166d   v1.33.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready                      <none>          166d   v1.33.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready,SchedulingDisabled   <none>          166d   v1.32.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@test-dev-worker-02:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 226 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (47.2 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@test-dev-worker-02:~$ sudo systemctl daemon-reload
test@test-dev-worker-02:~$ sudo systemctl restart kubelet
test@test-dev-worker-02:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@test-dev-master-01:~$ kubectl uncordon test-dev-worker-02
node/test-dev-worker-02 uncordoned
test@test-dev-master-01:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
test-dev-master-01   Ready    control-plane   166d   v1.33.7   192.168.0.131   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-01   Ready    <none>          166d   v1.33.7   192.168.0.135   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test-dev-worker-02   Ready    <none>          166d   v1.33.7   192.168.0.136   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

K8S, Object

Ceph MDS Pod Anti-Affinity Troubleshoot

2025년 12월 29일 Jason 댓글 남기기

kubernetes node upgrade등의 진행시 Ceph status에서 알람이 확인 되는 경우가 있습니다.
해당 현상 확인 및 처리 방법에 대한 정리내용 입니다.

work node 1번을 drain 시킨 후 확인된 알림

# ceph 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_WARN
            4 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 10h)
    mgr: b(active, since 5M), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 10h), 3 in (since 18M)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 20.17k objects, 882 MiB
    usage:   7.6 GiB used, 82 GiB / 90 GiB avail
    pgs:     113 active+clean

  io:
    client:   852 B/s rd, 2 op/s rd, 0 op/s wr

# ceph pod 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mgr|mds|mon|osd' 
rook-ceph-mds-myfs-a-77d484dc4-jddf9 2/2 Running 4 537d 172.16.118.75 test-worker-02 <none> <none> 
rook-ceph-mds-myfs-b-bd6ddc59b-l2b4t 2/2 Running 4 537d 172.16.118.72 test-worker-02 <none> <none> 
rook-ceph-mgr-a-7595f6b7d8-v2ww6 3/3 Running 8 546d 172.16.7.148 test-worker-03 <none> <none> 
rook-ceph-mgr-b-7cdf75cdb6-bmmgq 3/3 Running 0 171d 172.16.36.215 test-worker-01 <none> <none> 
rook-ceph-mon-a-54db4674f4-9z847 2/2 Running 6 546d 172.16.118.101 test-worker-02 <none> <none> 
rook-ceph-mon-b-54788d658b-wd658 2/2 Running 4 546d 172.16.36.230 test-worker-01 <none> <none> 
rook-ceph-mon-c-84f87b7c5-9z6ck 2/2 Running 4 546d 172.16.7.153 test-worker-03 <none> <none> 
rook-ceph-osd-0-788c4889ff-5gvcm 2/2 Running 0 171d 172.16.36.226 test-worker-01 <none> <none> 
rook-ceph-osd-1-7795c9dc4c-hzvqv 2/2 Running 0 171d 172.16.118.106 test-worker-02 <none> <none> 
rook-ceph-osd-2-6db8dc77dc-f8ct9 2/2 Running 0 171d 172.16.7.160 test-worker-03 <none> <none> 
rook-ceph-osd-prepare-test-worker-01-hldb7 0/1 Completed 0 314d <none> test-worker-01 <none> <none> 
rook-ceph-osd-prepare-test-worker-02-wv5rc 0/1 Completed 0 314d 172.16.118.84 test-worker-02 <none> <none> 
rook-ceph-osd-prepare-test-worker-03-pbnbb 0/1 Completed 0 314d 172.16.7.172 test-worker-03 <none> <none>

# MDS pod node 위치 확인
test@test-master-01:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-77d484dc4-jddf9                      2/2     Running     0                18s    172.16.118.75    test-worker-02   <none>           <none>
rook-ceph-mds-myfs-b-bd6ddc59b-l2b4t                      2/2     Running     0                18s    172.16.118.72    test-worker-02   <none>           <none>

# cephFS 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE        MDS       ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active      myfs-b  Reqs:    0 /s  35.9k  18.0k  4301      2
0-s   standby-replay  myfs-a  Evts:    0 /s  35.9k  18.0k  4301      0
      POOL         TYPE     USED  AVAIL
 myfs-metadata   metadata   629M  25.8G
myfs-replicated    data    12.0k  25.8G
   myfs-data0      data     982M  25.8G
MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)

2개의 MDS pod가 기동중이며 active 상태이지만 2개 모두 woker node 2번에 집중되어 기동되고 있는 상태 입니다.
cephFS 서비스는 현재 이상이 없지만 향후 서비스 성능 리스크가 있을 것으로 예상 되므로 pod Anti-affinity 를 구성하여 MDS pod가 하나의 work node에 집중되는 것을 방지하도록 하겠습니다.

# pod Anti-affinity 반영
test@test-master-01:~$ kubectl -n rook-ceph patch cephfilesystem myfs --type='merge' -p '
> spec:
  metadataServer:
    placement:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ["rook-ceph-mds"]
            - key: rook_file_system
              operator: In
              values: ["myfs"]
          topologyKey: kubernetes.io/hostname
'
cephfilesystem.ceph.rook.io/myfs patched

# 적용 확인
test@test-master-01:~$ kubectl -n rook-ceph get cephfilesystem myfs -o yaml | sed -n '1,260p'
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"ceph.rook.io/v1","kind":"CephFilesystem","metadata":{"annotations":{},"name":"myfs","namespace":"rook-ceph"},"spec":{"dataPools":[{"replicated":{"size":3}}],"metadataPool":{"replicated":{"size":3}},"metadataServer":{"activeCount":1,"activeStandby":true},"preserveFilesystemOnDelete":true}}
  creationTimestamp: "2024-07-07T15:23:38Z"
  finalizers:
  - cephfilesystem.ceph.rook.io
  generation: 3
  name: myfs
  namespace: rook-ceph
  resourceVersion: "115648910"
  uid: 92cd0904-f6e1-4b15-853d-165b87be04d5
spec:
  dataPools:
  - application: ""
    erasureCoded:
      codingChunks: 0
      dataChunks: 0
    mirroring: {}
    quotas: {}
    replicated:
      size: 3
    statusCheck:
      mirror: {}
  metadataPool:
    application: ""
    erasureCoded:
      codingChunks: 0
      dataChunks: 0
    mirroring: {}
    quotas: {}
    replicated:
      size: 3
    statusCheck:
      mirror: {}
  metadataServer:
    activeCount: 1
    activeStandby: true
    placement:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - rook-ceph-mds
            - key: rook_file_system
              operator: In
              values:
              - myfs
          topologyKey: kubernetes.io/hostname
    resources: {}
  preserveFilesystemOnDelete: true
  statusCheck:
    mirror: {}
status:
  observedGeneration: 2
  phase: Ready

# worker node 1번 uncordon 후 worker node 2번을 drain 처리시 MDS pod 위치 worker node 1, 3번 확인
test@test-master-01:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-58846844d6-nd5mk                     2/2     Running     0             53s     172.16.36.216    test-worker-01   <none>           <none>
rook-ceph-mds-myfs-b-6b4d9476cb-q6b6p                     2/2     Running     0             38s     172.16.7.190     test-worker-03   <none>           <none>

# cephFS 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE        MDS       ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active      myfs-a  Reqs:    0 /s  35.9k  18.0k  4301      2
0-s   standby-replay  myfs-b  Evts:    0 /s  35.9k  18.0k  4301      0
      POOL         TYPE     USED  AVAIL
 myfs-metadata   metadata   621M  25.8G
myfs-replicated    data    12.0k  25.8G
   myfs-data0      data     982M  25.8G
MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)

MDS pod의 위치는 의도대로 분산 deploy 되었지만 ceph status 에서는 여전히 HEALTH_WARN 상태 입니다.
해당 알람은 mgr pod event에 대한 내역으로 MDS 이슈 처리사항과는 무관하지만 event 확인 후 초기화 처리 하도록 하겠습니다.
mgr available status가 true 이면 상태는 정상입니다.

# ceph health의 mgr alarm 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_WARN
            4 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 2m)
    mgr: b(active, since 10m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 110s), 3 in (since 18M)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 20.17k objects, 881 MiB
    usage:   6.3 GiB used, 84 GiB / 90 GiB avail
    pgs:     113 active+clean

  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

# mgr 상태 정상 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph mgr stat
{
    "epoch": 476,
    "available": true,
    "active_name": "b",
    "num_standby": 1
}

# mgr crash 목록 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash ls
ID                                                                ENTITY  NEW
2025-12-26T09:01:17.354121Z_c76c6eaf-4bf7-4cf9-a9ec-f646fe857b76  mgr.b    *
2025-12-26T09:01:32.345473Z_4dfd271c-3d5b-4c89-88cf-13ba096f327b  mgr.b    *
2025-12-26T09:01:47.357321Z_0f938fb6-4c50-4b58-815d-5990fbe4bbb7  mgr.b    *
2025-12-26T09:02:02.329492Z_43d344a7-b71f-442e-a664-1852dda3a3f3  mgr.b    *
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash stat
4 crashes recorded

# mgr crash 이력 정리 후 health alarm 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash archive-all
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 6m)
    mgr: b(active, since 14m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 5m), 3 in (since 18M)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 20.17k objects, 881 MiB
    usage:   6.3 GiB used, 84 GiB / 90 GiB avail
    pgs:     113 active+clean

  io:
    client:   922 B/s rd, 1 op/s rd, 0 op/s wr

추가로 node drain, uncordon 등의 action 시 rebalancing 이 발생할 수 있으나 일정 시간 후에 재확인 하시면 HEALTH_OK 상태로 변경 확인할 수 있습니다.

test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_WARN
            1/3 mons down, quorum a,c
            4 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,c (age 0.275988s), out of quorum: b
    mgr: b(active, since 7m), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 8m), 3 in (since 18M)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 20.17k objects, 881 MiB
    usage:   6.3 GiB used, 84 GiB / 90 GiB avail
    pgs:     113 active+clean

  io:
    client:   852 B/s rd, 1 op/s rd, 0 op/s wr

Object

ceph dashboard OSD alarm troubleshoot

2025년 12월 23일 Jason 댓글 남기기

# 대시보드의 OSD 알람 처리

bash-4.4$ ceph status
  cluster:
    id: d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_WARN
            mon b is low on available space


  services:
    mon: 3 daemons, quorum a,b,c (age 4M)
    mgr: b(active, since 4M), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 4 osds: 3 up (since 4M), 3 in (since 13M)

  data:
    volumes: 1/1 healthy
    pools: 5 pools, 113 pgs
    objects: 20.17k objects, 871 MiB
    usage: 7.4 GiB used, 83 GiB / 90 GiB avail
    pgs: 113 active+clean

  io:
    client: 853 B/s rd, 2 op/s rd, 0 op/s wr

bash-4.4$ ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 90 GiB 83 GiB 7.4 GiB 7.4 GiB 8.22
TOTAL 90 GiB 83 GiB 7.4 GiB 7.4 GiB 8.22

--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 449 KiB 2 1.3 MiB 0 26 GiB
replicapool 3 32 338 MiB 126 1013 MiB 1.25 26 GiB
myfs-metadata 4 16 209 MiB 390 628 MiB 0.78 26 GiB
myfs-replicated 5 32 158 B 3.00k 12 KiB 0 26 GiB
myfs-data0 6 32 286 MiB 16.65k 983 MiB 1.21 26 GiB
bash-4.4$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
 0 k8s-worker-01 2515M 27.5G 0 0 1 90 exists,up
 1 k8s-worker-02 2511M 27.5G 0 0 0 0 exists,up
 2 k8s-worker-03 2544M 27.5G 0 0 1 16 exists,up
 3 0 0 0 0 0 0 autoout,exists,new

bash-4.4$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08789 root default
-3 0.02930 host k8s-worker-01
 0 ssd 0.02930 osd.0 up 1.00000 1.00000
-5 0.02930 host k8s-worker-02
 1 ssd 0.02930 osd.1 up 1.00000 1.00000
-7 0.02930 host k8s-worker-03
 2 ssd 0.02930 osd.2 up 1.00000 1.00000
 3 0 osd.3 down 0 1.00000

# 미사용 osd 제거
bash-4.4$ ceph osd crush remove osd.3
device 'osd.3' does not appear in the crush map
bash-4.4$ ceph auth del osd.3
bash-4.4$ ceph osd rm 3
removed osd.3
bash-4.4$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.08789 root default
-3 0.02930 host k8s-worker-01
 0 ssd 0.02930 osd.0 up 1.00000 1.00000
-5 0.02930 host k8s-worker-02
 1 ssd 0.02930 osd.1 up 1.00000 1.00000
-7 0.02930 host k8s-worker-03
 2 ssd 0.02930 osd.2 up 1.00000 1.00000

IT Garage