Kubernetes 删除 istio-system namesapce 时候，出现 Terminating 解决办法
当你在 Kubernetes 中遇到无法删除处于 Terminating 状态的命名空间时，可能是由于该命名空间中仍有活跃的资源或服务。要解决这个问题，你可以尝试以下几个步骤：
一、常规方法
-
检查命名空间中的活跃资源：使用以下命令列出命名空间中的所有活跃资源:
[root@k8s-master-node1 zuoyang]# kubectl get all --all-namespaces -o custom-columns=KIND:.kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name KIND NAMESPACE NAME Pod argocd devops-argocd-application-controller-0 Pod argocd devops-argocd-applicationset-controller-84b686cc4b-lxx69 Pod argocd devops-argocd-dex-server-76b589998c-mfjst Pod argocd devops-argocd-notifications-controller-6f469bd5c6-7mc4f Pod argocd devops-argocd-redis-584dfc65c5-22hcq Pod argocd devops-argocd-repo-server-87748fdcf-dnxmd Pod argocd devops-argocd-server-b9fc8f9d8-hqzc9 Pod cert-manager cert-manager-56886c8df6-8zsl5 Pod cert-manager cert-manager-cainjector-845f47b7cb-nk6wt Pod cert-manager cert-manager-webhook-6976d886fd-v4twp Pod kube-system calico-kube-controllers-7f94f687d6-scd25 Pod kube-system calico-node-24266 Pod kube-system calico-node-pqpmb Pod kube-system calico-node-qhwpz Pod kube-system coredns-67ddbf998c-cxbq4 Pod kube-system coredns-67ddbf998c-wv2mg Pod kube-system haproxy-k8s-worker-node1 Pod kube-system haproxy-k8s-worker-node2 Pod kube-system kube-apiserver-k8s-master-node1 Pod kube-system kube-controller-manager-k8s-master-node1 Pod kube-system kube-proxy-5zprb Pod kube-system kube-proxy-jbd8z Pod kube-system kube-proxy-q97v4 Pod kube-system kube-scheduler-k8s-master-node1 Pod kube-system metrics-server-754d6db9f5-lntvs Pod kube-system nodelocaldns-d4kxr Pod kube-system nodelocaldns-hmzhw Pod kube-system nodelocaldns-n9dtn Pod kube-system openebs-localpv-provisioner-5677c78f85-bm67w Pod kube-system snapshot-controller-0 Pod kubesphere-controls-system default-http-backend-7c6855956d-rwc2d Pod kubesphere-controls-system kubectl-admin-549d5cb676-pvdp7 Pod kubesphere-devops-system devops-28424640-g5z2m Pod kubesphere-devops-system devops-28424670-v6lqw Pod kubesphere-devops-system devops-28424700-vvbkt Pod kubesphere-devops-system devops-28424730-b68sq Pod kubesphere-devops-system devops-apiserver-5856cbdb8-b9qg7 Pod kubesphere-devops-system devops-controller-78f4547f7-mp7dh Pod kubesphere-devops-system devops-jenkins-6db57dd86d-nqjwn Pod kubesphere-devops-system s2ioperator-0 Pod kubesphere-logging-system fluentbit-operator-7d6889669d-dthkw Pod kubesphere-logging-system ks-events-exporter-74b55d4995-ztqg4 Pod kubesphere-logging-system ks-events-operator-7f5d5698-8fcc5 Pod kubesphere-logging-system ks-events-ruler-568ddf48f8-n66nh Pod kubesphere-logging-system ks-events-ruler-568ddf48f8-ntd4x Pod kubesphere-logging-system kube-auditing-operator-549b9f58b8-5fvg5 Pod kubesphere-logging-system kube-auditing-webhook-deploy-65fdb7b7c-2pfn7 Pod kubesphere-logging-system kube-auditing-webhook-deploy-65fdb7b7c-4t8km Pod kubesphere-logging-system logsidecar-injector-deploy-79cd76bf67-dlgwq Pod kubesphere-logging-system logsidecar-injector-deploy-79cd76bf67-zrnw2 Pod kubesphere-monitoring-system alertmanager-main-0 Pod kubesphere-monitoring-system kube-state-metrics-69df444d46-4nzl9 Pod kubesphere-monitoring-system node-exporter-5znjv Pod kubesphere-monitoring-system node-exporter-g6mkb Pod kubesphere-monitoring-system node-exporter-pgm4t Pod kubesphere-monitoring-system notification-manager-deployment-8c998c4fb-cs99p Pod kubesphere-monitoring-system notification-manager-deployment-8c998c4fb-hj725 Pod kubesphere-monitoring-system notification-manager-operator-6979d79df7-q2gm8 Pod kubesphere-monitoring-system prometheus-k8s-0 Pod kubesphere-monitoring-system prometheus-operator-66cfbfdd45-pflth Pod kubesphere-monitoring-system thanos-ruler-kubesphere-0 Pod kubesphere-system ks-apiserver-7f94cb7968-4gf5k Pod kubesphere-system ks-console-59d55d7fc6-dz8zg Pod kubesphere-system ks-controller-manager-645fbfd86d-kngjh Pod kubesphere-system ks-installer-6fb6fc8dd4-c7svh Pod kubesphere-system minio-778b64d7bb-jkvzv Pod kubesphere-system openldap-0 Pod kubesphere-system redis-8c57d9bd9-q7gf8 Service argocd devops-argocd-application-controller Service argocd devops-argocd-applicationset-controller Service argocd devops-argocd-dex-server Service argocd devops-argocd-redis Service argocd devops-argocd-repo-server Service argocd devops-argocd-server Service cert-manager cert-manager Service cert-manager cert-manager-webhook Service default kubernetes Service kube-system coredns Service kube-system etcd Service kube-system kube-controller-manager-svc Service kube-system kube-scheduler-svc Service kube-system kubelet Service kube-system metrics-server Service kubesphere-controls-system default-http-backend Service kubesphere-devops-system devops-apiserver Service kubesphere-devops-system devops-jenkins Service kubesphere-devops-system devops-jenkins-agent Service kubesphere-devops-system s2ioperator-metrics-service Service kubesphere-devops-system s2ioperator-trigger-service Service kubesphere-devops-system webhook-server-service Service kubesphere-logging-system ks-events-admission Service kubesphere-logging-system ks-events-ruler Service kubesphere-logging-system kube-auditing-webhook-svc Service kubesphere-logging-system logsidecar-injector-admission Service kubesphere-monitoring-system alertmanager-main Service kubesphere-monitoring-system alertmanager-operated Service kubesphere-monitoring-system kube-state-metrics Service kubesphere-monitoring-system node-exporter Service kubesphere-monitoring-system notification-manager-controller-metrics Service kubesphere-monitoring-system notification-manager-svc Service kubesphere-monitoring-system notification-manager-webhook Service kubesphere-monitoring-system prometheus-k8s Service kubesphere-monitoring-system prometheus-operated Service kubesphere-monitoring-system prometheus-operator Service kubesphere-monitoring-system thanos-ruler-kubesphere Service kubesphere-monitoring-system thanos-ruler-operated Service kubesphere-system ks-apiserver Service kubesphere-system ks-console Service kubesphere-system ks-controller-manager Service kubesphere-system minio Service kubesphere-system openldap Service kubesphere-system redis DaemonSet kube-system calico-node DaemonSet kube-system kube-proxy DaemonSet kube-system nodelocaldns DaemonSet kubesphere-monitoring-system node-exporter Deployment argocd devops-argocd-applicationset-controller Deployment argocd devops-argocd-dex-server Deployment argocd devops-argocd-notifications-controller Deployment argocd devops-argocd-redis Deployment argocd devops-argocd-repo-server Deployment argocd devops-argocd-server Deployment cert-manager cert-manager Deployment cert-manager cert-manager-cainjector Deployment cert-manager cert-manager-webhook Deployment kube-system calico-kube-controllers Deployment kube-system coredns Deployment kube-system metrics-server Deployment kube-system openebs-localpv-provisioner Deployment kubesphere-controls-system default-http-backend Deployment kubesphere-controls-system kubectl-admin Deployment kubesphere-devops-system devops-apiserver Deployment kubesphere-devops-system devops-controller Deployment kubesphere-devops-system devops-jenkins Deployment kubesphere-logging-system fluentbit-operator Deployment kubesphere-logging-system ks-events-exporter Deployment kubesphere-logging-system ks-events-operator Deployment kubesphere-logging-system ks-events-ruler Deployment kubesphere-logging-system kube-auditing-operator Deployment kubesphere-logging-system kube-auditing-webhook-deploy Deployment kubesphere-logging-system logsidecar-injector-deploy Deployment kubesphere-monitoring-system kube-state-metrics Deployment kubesphere-monitoring-system notification-manager-deployment Deployment kubesphere-monitoring-system notification-manager-operator Deployment kubesphere-monitoring-system prometheus-operator Deployment kubesphere-system ks-apiserver Deployment kubesphere-system ks-console Deployment kubesphere-system ks-controller-manager Deployment kubesphere-system ks-installer Deployment kubesphere-system minio Deployment kubesphere-system redis ReplicaSet argocd devops-argocd-applicationset-controller-84b686cc4b ReplicaSet argocd devops-argocd-dex-server-76b589998c ReplicaSet argocd devops-argocd-notifications-controller-6f469bd5c6 ReplicaSet argocd devops-argocd-redis-584dfc65c5 ReplicaSet argocd devops-argocd-repo-server-87748fdcf ReplicaSet argocd devops-argocd-server-b9fc8f9d8 ReplicaSet cert-manager cert-manager-56886c8df6 ReplicaSet cert-manager cert-manager-cainjector-845f47b7cb ReplicaSet cert-manager cert-manager-webhook-6976d886fd ReplicaSet kube-system calico-kube-controllers-7f94f687d6 ReplicaSet kube-system calico-kube-controllers-b46dcdc76 ReplicaSet kube-system coredns-67ddbf998c ReplicaSet kube-system metrics-server-754d6db9f5 ReplicaSet kube-system openebs-localpv-provisioner-5677c78f85 ReplicaSet kubesphere-controls-system default-http-backend-7c6855956d ReplicaSet kubesphere-controls-system kubectl-admin-549d5cb676 ReplicaSet kubesphere-devops-system devops-apiserver-5856cbdb8 ReplicaSet kubesphere-devops-system devops-controller-78f4547f7 ReplicaSet kubesphere-devops-system devops-jenkins-6db57dd86d ReplicaSet kubesphere-logging-system fluentbit-operator-7d6889669d ReplicaSet kubesphere-logging-system ks-events-exporter-74b55d4995 ReplicaSet kubesphere-logging-system ks-events-operator-7f5d5698 ReplicaSet kubesphere-logging-system ks-events-ruler-568ddf48f8 ReplicaSet kubesphere-logging-system kube-auditing-operator-549b9f58b8 ReplicaSet kubesphere-logging-system kube-auditing-webhook-deploy-65fdb7b7c ReplicaSet kubesphere-logging-system logsidecar-injector-deploy-79cd76bf67 ReplicaSet kubesphere-monitoring-system kube-state-metrics-69df444d46 ReplicaSet kubesphere-monitoring-system notification-manager-deployment-8c998c4fb ReplicaSet kubesphere-monitoring-system notification-manager-operator-6979d79df7 ReplicaSet kubesphere-monitoring-system prometheus-operator-66cfbfdd45 ReplicaSet kubesphere-system ks-apiserver-7f94cb7968 ReplicaSet kubesphere-system ks-console-59d55d7fc6 ReplicaSet kubesphere-system ks-controller-manager-645fbfd86d ReplicaSet kubesphere-system ks-installer-6fb6fc8dd4 ReplicaSet kubesphere-system ks-installer-c549d79dc ReplicaSet kubesphere-system minio-778b64d7bb ReplicaSet kubesphere-system redis-8c57d9bd9 StatefulSet argocd devops-argocd-application-controller StatefulSet kube-system snapshot-controller StatefulSet kubesphere-devops-system s2ioperator StatefulSet kubesphere-monitoring-system alertmanager-main StatefulSet kubesphere-monitoring-system prometheus-k8s StatefulSet kubesphere-monitoring-system thanos-ruler-kubesphere StatefulSet kubesphere-system openldap CronJob kubesphere-devops-system devops Job kubesphere-devops-system devops-28424640 Job kubesphere-devops-system devops-28424670 Job kubesphere-devops-system devops-28424700 Job kubesphere-devops-system devops-28424730
这将显示所有命名空间中的资源，包括活跃的和已删除的。查找处于 Terminating 状态的命名空间下的资源，并确认它们是否正在被删除。
- 删除命名空间中的服务：如果你发现有活跃的服务在命名空间中，你可以使用以下命令删除它们：
kubectl delete service <service-name> -n <namespace>
将
<service-name>替换为要删除的服务的名称，<namespace>替换为命名空间的名称。重复此步骤，直到删除所有活跃的服务。 - 等待资源释放：在删除活跃的服务后，Kubernetes 将开始释放相关的资源。你需要等待一段时间，以确保所有资源都被正确释放。
-
尝试再次删除命名空间：一旦所有活跃的资源都被释放，你可以尝试再次删除命名空间。使用以下命令：
kubectl delete namespace <namespace>
将
<namespace>替换为要删除的命名空间的名称。执行命令后，等待一段时间，直到命名空间被成功删除。
如果以上步骤仍然无法解决问题，建议通过 Kubernetes 原生 API 删除试试，多次尝试过，均有效。
二、直接通过原生API删除
通过 Kubernetes 原生 API 删除处于 Terminating 状态的命名空间是一种可行的方法。你可以使用 Kubernetes 的 REST API 来执行删除操作。以下是一个示例的步骤：
- 获取 Kubernetes API 的地址和身份验证凭据。你需要具有足够的权限来执行删除操作。
- 使用适当的编程语言或工具，例如 Python 的 requests 库，编写一个脚本或应用程序来调用 Kubernetes API。
- 构建 DELETE 请求，指定要删除的命名空间的名称。请求的 URL 格式如下：https://<Kubernetes-API-address>/api/v1/namespaces/<namespace-name>将 <Kubernetes-API-address> 替换为 Kubernetes API 的地址，<namespace-name> 替换为要删除的命名空间的名称。
- 在请求中设置适当的身份验证凭据，例如使用基本身份验证或令牌认证。
- 发送 DELETE 请求到 Kubernetes API，并等待响应。如果删除成功，你将收到一个 HTTP 200 响应。
请注意，通过原生 API 删除命名空间仍然需要遵循 Kubernetes 的安全和权限要求。确保你具有足够的权限来执行删除操作，并遵循最佳实践来处理敏感凭据和身份验证信息。
三、通过API代理，执行JSON 访问API删除
通过启动一个 API 代理来删除处于 Terminating 状态的命名空间的方法并不是 Kubernetes 原生 API 的标准用法。直接编辑 JSON 文件和通过 API 代理删除命名空间的方法涉及多个步骤，并且需要小心处理。
3.1、使用kubectl get namespace istio-system -o json > tmp.json获取istio-system命名空间的 JSON 表示，并将其保存到tmp.json文件中。
[root@k8s-master-node1 zuoyang]# kubectl get namespace istio-system -o json > tmp.json -rw-r--r--. 1 root root 0 Jan 17 20:41 tmp.json
3.2、使用kubectl proxy --port=8088启动一个 API 代理服务器。
[root@k8s-master-node1 zuoyang]# kubectl proxy --port=8088 Starting to serve on 127.0.0.1:8088
3.3、在 tmp.json文件中删除与 finalizers相关的部分。
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "istio-system",
"uid": "ecce5153-9888-48f5-aa57-67b020965739",
"resourceVersion": "3446075",
"creationTimestamp": "2024-01-16T06:37:03Z",
"deletionTimestamp": "2024-01-18T01:36:28Z",
"labels": {
"istio-injection": "disabled",
"kubernetes.io/metadata.name": "istio-system",
"kubesphere.io/namespace": "istio-system",
"kubesphere.io/workspace": "system-workspace"
},
"ownerReferences": [
{
"apiVersion": "tenant.kubesphere.io/v1alpha1",
"kind": "Workspace",
"name": "system-workspace",
"uid": "a620f1f0-8633-4359-beff-c46b97948deb",
"controller": true,
"blockOwnerDeletion": true
}
],
"managedFields": [
{
"manager": "istioctl",
"operation": "Update",
"apiVersion": "v1",
"time": "2024-01-16T06:37:03Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
".": {},
"f:kubernetes.io/metadata.name": {}
}
}
}
},
{
"manager": "kubectl-label",
"operation": "Update",
"apiVersion": "v1",
"time": "2024-01-17T01:21:40Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
"f:istio-injection": {},
"f:kubesphere.io/namespace": {},
"f:kubesphere.io/workspace": {}
}
}
}
},
{
"manager": "controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2024-01-17T01:27:30Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"a620f1f0-8633-4359-beff-c46b97948deb\"}": {}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2024-01-18T01:36:33Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"NamespaceContentRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
}
}
},
"subresource": "status"
}
]
},
####井号的部分需要删除掉
####"spec": {
#### "finalizers": [
#### "kubernetes"
#### ]
####},
"status": {
"phase": "Terminating",
"conditions": [
{
"type": "NamespaceDeletionDiscoveryFailure",
"status": "False",
"lastTransitionTime": "2024-01-18T01:36:33Z",
"reason": "ResourcesDiscovered",
"message": "All resources successfully discovered"
},
{
"type": "NamespaceDeletionGroupVersionParsingFailure",
"status": "False",
"lastTransitionTime": "2024-01-18T01:36:33Z",
"reason": "ParsedGroupVersions",
"message": "All legacy kube types successfully parsed"
},
{
"type": "NamespaceDeletionContentFailure",
"status": "False",
"lastTransitionTime": "2024-01-18T01:36:33Z",
"reason": "ContentDeleted",
"message": "All content successfully deleted, may be waiting on finalization"
},
{
"type": "NamespaceContentRemaining",
"status": "True",
"lastTransitionTime": "2024-01-18T01:36:33Z",
"reason": "SomeResourcesRemain",
"message": "Some resources are remaining: kialis.kiali.io has 1 resource instances"
},
{
"type": "NamespaceFinalizersRemaining",
"status": "True",
"lastTransitionTime": "2024-01-18T01:36:33Z",
"reason": "SomeFinalizersRemain",
"message": "Some content in the namespace has finalizers remaining: kiali.io/finalizer in 1 resource instances"
}
]
}
}
3.3、使用curl命令将修改后的 JSON 数据发送到 API 代理服务器，以执行删除操作,请注意，我给的例子里是 namespace 是 istio-system/finalize，你要根据实际的 namespace 来 。
curl -k -H "Content-Type:application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8088/api/v1/namespaces/istio-system/finalize标签：Istio,Kubernetes,kubesphere,argocd,devops,system,Pod,kube From： https://www.cnblogs.com/zuoyang/p/17971835