When working with Kubernetes, you may encounter a situation where a namespace gets stuck in the Terminating
state. This usually happens because Kubernetes is waiting for certain resources to be cleaned up, but due to issues with finalizers or dangling resources, the namespace remains in a perpetual terminating state. If you’ve already tried deleting resources manually and they are stuck, you can use the method below to force-delete the namespace.
In this guide, we’ll show how to force-delete a namespace using kubectl proxy
and a manual JSON edit.
Symptoms
You might encounter the following symptoms:
- Running
kubectl get namespace
shows the namespace in aTerminating
state for a long time.microk8s kubectl get namespace NAME STATUS AGE metallb-system Terminating 119m kube-system Active 132m default Active 132m
- Deleting the namespace using
kubectl delete namespace <namespace-name>
does not work, and the namespace remains stuck.microk8s kubectl delete namespace metallb-system namespace "metallb-system" deleted
However, checking the status again still shows it inTerminating
.
Steps to Force-Delete the Namespace
Follow these steps to force Kubernetes to remove the stuck namespace.
Step 1: Start kubectl proxy
Start the kubectl proxy
process, which allows us to interact with the Kubernetes API directly:
microk8s kubectl proxy --port=8001 &
The &
symbol runs the process in the background.
Step 2: Fetch and Edit the Namespace JSON
Set the namespace variable for convenience:
NAMESPACE=metallb-system
Use the following command to fetch the namespace definition in JSON format and remove the finalizers
field, which is causing the termination to be stuck:
microk8s kubectl get namespace $NAMESPACE -o json | jq '.spec = {"finalizers":[]}' > temp.json
This command:
- Retrieves the namespace definition in JSON format.
- Uses
jq
to set thefinalizers
field to an empty array ([]
). - Saves the modified JSON to a file named
temp.json
.
Step 3: Send the Modified JSON to the Kubernetes API
Use curl
to send the modified JSON back to Kubernetes, instructing it to finalize and delete the namespace:
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
This sends a PUT request to the Kubernetes API server, bypassing the stuck finalizers and forcing Kubernetes to finalize the deletion of the namespace.
Step 4: Verify the Deletion
Check if the namespace has been deleted:
microk8s kubectl get namespace
You should no longer see the $NAMESPACE
namespace in the list.
Step 5: Cleanup
Stop the kubectl proxy
process:
kill %1
This command stops the background kubectl proxy
process that was started in Step 1.
Conclusion
By following these steps, you can force-delete a stuck Kubernetes namespace that remains in a Terminating
state due to finalizers or dangling resources. This method is especially useful when dealing with namespaces that have custom resources or controllers (like MetalLB) that fail to clean up properly.
Always ensure that you understand the implications of force-deleting a namespace, as it may leave dangling resources behind. After deleting, it’s a good idea to verify that all critical components have been redeployed correctly.
If you found this guide helpful or have any questions, feel free to leave a comment or share your experience!