Namespace Terminating

Namespace Terminating

Posted by BlueFat on Sunday, November 27, 2022

问题

删除 namespace 失败,一直处于 terminating 状态。

[root@ha ~]# kubectl get ns 
NAME                          STATUS        AGE
default                       Active        4d
kube-system                   Active        4d
local                         Active        3d6h
monitoring                    Terminating   19m

排查

检查该 namespace 中是否还存在未被清理的资源

namespace=monitoring
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n ${namespace}

检查是否存在未 Ready 的 APIService

kubectl get APIService
kubectl api-resources
# 报错
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
[root@ha ~]# kubectl get apiservice | grep metrics
NAME                                   SERVICE                         AVAILABLE                 AGE
v1beta1.metrics.k8s.io                 monitoring/prometheus-adapter   False (ServiceNotFound)   4d

[root@ha ~]# kubectl delete apiservice v1beta1.metrics.k8s.io 
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted

解决

尝试使用强制删除

kubectl delete ns monitoring --force --grace-period=0

若不行再使用原生接口删除

本地通过 kubectl proxy 开启代理,便于无认证访问 apiserver api

kubectl proxy
curl -H "Content-Type: application/json" -XPUT -d '{"apiVersion":"v1","kind":"Namespace","metadata":{"name":"DELETE_NAMESPACE"},"spec":{"finalizers":[]}}' http://localhost:8001/api/v1/namespaces/DELETE_NAMESPACE/finalize

DELETE_NAMESPACE 换成要删除的名称空间

或者

NAMESPACE=delete-me
kubectl get ns $NAMESPACE -o json | jq '.spec.finalizers=[]' > ns.json
kubectl proxy --port=8899 &
PID=$!
curl -X PUT http://localhost:8899/api/v1/namespaces/$NAMESPACE/finalize -H "Content-Type: application/json" --data-binary @ns.json
kill $PID