kubernetes 无法删除 pod 问题的解决
Mar272018
[摘要] kubernetes 可能会产生垃圾或者僵尸pod，在删除rc的时候，相应的pod没有被删除，手动删除pod后会自动重新创建，这时一般需要先删除掉相关联的resources，实际中还要具体情况具体分析。
在使用Helm部署gitlab的时候发现有垃圾的pod存在，一直删除不掉
|
1 2 3 4 5 |
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE gitlab-migrations.1-2km-4rwnm 1/1 Running 158 21h gitlab-migrations.1-30d-gx9nz 1/1 Running 164 21h gitlab-migrations.1-dsw-m5qbc 1/1 Running 167 21h |
一般来说是要删除pod关联的rc，rs以及deployment等等，可以执行如下命令查询
|
1 2 3 4 5 6 7 8 9 |
kubectl get ClusterRole -n kube-system kubectl get ClusterRoleBinding -n kube-system kubectl get Deployments -n kube-system kubectl get Pods -n kube-system kubectl get ServiceAccounts -n kube-system kubectl get CustomResourceDefinition -n kube-system kubectl get replicasets -n kube-system kubectl get replicationcontrollers -n kube-system kubectl get services -n kube-system |
找到对应的标签进行删除，对应的pod一般就会被删除掉，如果pod还在的话再手动删除pod。
实际中我这三个pod还是无法删除掉，删除之后仍然会重新启动，所以仔细看了下pod的具体描述内容。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
# kubectl describe pod -n kube-system gitlab-migrations.1-30d-gx9nz Name: gitlab-migrations.1-30d-gx9nz Namespace: kube-system Node: 172.30.0.169/172.30.0.169 Start Time: Mon, 26 Mar 2018 11:53:45 +0800 Labels: app=migrations controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209 job-name=gitlab-migrations.1-30d release=gitlab Annotations: <none> Status: Running IP: 172.100.14.7 Controlled By: Job/gitlab-migrations.1-30d Init Containers: configure: Container ID: docker://a3051d24e65267489aa97ae368bcbe5819e4ae4f3b5ddc4d277b0b25ee585716 Image: busybox Image ID: docker-pullable://docker.io/busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21 Port: <none> Command: sh /config/configure State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 26 Mar 2018 11:54:24 +0800 Finished: Mon, 26 Mar 2018 11:54:24 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /config from migrations-config (ro) /init-config from init-migrations-secrets (ro) /init-secrets from migrations-secrets (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro) Containers: migrations: Container ID: docker://c33e809de76ba44da330063d9e3c6f58631b30e3c805ea561dc68af38ffe3926 Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-rails:latest Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-rails@sha256:7dca5eb3b686342a9cff798f8349ad07459a1ef1e9d47a23b787b284d874cc17 Port: <none> Args: /scripts/wait-for-deps /scripts/db-migrate State: Running Started: Tue, 27 Mar 2018 09:50:09 +0800 Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 27 Mar 2018 09:41:52 +0800 Finished: Tue, 27 Mar 2018 09:45:02 +0800 Ready: True Restart Count: 164 Environment: CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates CONFIG_DIRECTORY: /var/opt/gitlab/config/gitlab/ GITLAB_ROOT_PASSWORD: Vi4ree2R Mounts: /etc/gitlab from migrations-secrets (ro) /home/git/gitlab/config/secrets.yml from migrations-secrets (rw) /var/opt/gitlab/templates from migrations-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-6rdp2 (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: migrations-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: gitlab-migrations Optional: false init-migrations-secrets: <unknown> migrations-secrets: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory default-token-6rdp2: Type: Secret (a volume populated by a Secret) SecretName: default-token-6rdp2 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 6m (x655 over 21h) kubelet, 172.30.0.169 MountVolume.SetUp failed for volume "migrations-config" : configmaps "gitlab-migrations" not found Warning BackOff 3m (x3516 over 21h) kubelet, 172.30.0.169 Back-off restarting failed container |
需要关注的地方是Labels标签
|
1 2 3 4 |
Labels: app=migrations controller-uid=d0fd1214-30a8-11e8-89a7-525400d81209 job-name=gitlab-migrations.1-30d release=gitlab |
发现有个job-name=gitlab-migrations.xxx
于是继续查看kubernetes是否有job存在
|
1 2 3 4 5 |
# kubectl get jobs -n kube-system NAME DESIRED SUCCESSFUL AGE gitlab-migrations.1-2km 1 0 23h gitlab-migrations.1-30d 1 0 23h gitlab-migrations.1-dsw 1 0 23h |
先执行删除job看下：
|
1 2 3 4 5 6 |
# kubectl delete job gitlab-migrations.1-2km -n kube-system job "gitlab-migrations.1-2km" deleted # kubectl delete job gitlab-migrations.1-30d -n kube-system job "gitlab-migrations.1-30d" deleted # kubectl delete job gitlab-migrations.1-dsw -n kube-system job "gitlab-migrations.1-dsw" deleted |
再看pod情况：
|
1 2 3 4 |
# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE gitlab-migrations.1-30d-gx9nz 1/1 Terminating 173 23h gitlab-migrations.1-dsw-m5qbc 0/1 Terminating 176 23h |
附图为kubernets 的一些对象资源：
再看pod已经被删除了，还有处于Terminating状态的，一会就会消失掉。至此pod删除成功，所以删除与pods相关联的依赖资源是解决问题的最终方法！
本文固定链接: https://www.sudops.com/kubernetes-cannot-delete-pod.html | 运维·速度
kubernetes 无法删除 pod 问题的解决：目前有1 条留言
发表评论
欢迎关注 运维·速度 微信公众号
唉，我这边的pod怎么删除都不行，deployment也删了，services也删除了，上面列的资源也删除了试了，好顽固的pod
2021-10-19 14:47