Connecting shell to a Node in Kubernetes

As Developers, when working with Kubernetes usually we focus on the creation of pods, deployments, and some other resources (services, ingress, secrets, config maps) required for our application and we delegate the orchestration of the applications to Kubernetes. Rarely, we must interact with nodes directly, and it’s more strange to access via ssh.

One reason to access a Kubernetes node by SSH might be to verify the existence or the content of a file or configuration directly. On my case, I had to mount a volume of type hostPath and I needed to verify that some files were really created in the node.

So, how to access the now by SSH and interact with the node in order to execute any command from the terminal if necessary. We might do it with a simple SSH, however it requires in most of the cases a secure connection between our machine and the node machine instances. For example, if we are using EKS (Kubernetes on AWS) we would need SSH keys or some kind of infrastructure like Bastion for that purpose. That’s why we might want to use a less complex alternative.

NSenter script

This a small script that installs a pod in the kubernetes cluster, the docker image uses nsenter which is a small program of the util-linux package. That way we can access the kubernetes node.

#!/bin/sh
node=${1}
nodeName=$(kubectl get node ${node} -o template --template='{{index .metadata.labels "kubernetes.io/hostname"}}') 
nodeSelector='"nodeSelector": { "kubernetes.io/hostname": "'${nodeName:?}'" },'
podName=${USER}-nsenter-${node}
kubectl run ${podName:?} --restart=Never -it --rm --image overriden --overrides '
{
  "spec": {
    "hostPID": true,
    "hostNetwork": true,
    '"${nodeSelector?}"'
    "tolerations": [{
        "operator": "Exists"
    }],
    "containers": [
      {
        "name": "nsenter",
        "image": "alexeiled/nsenter:2.34",
        "command": [
          "/nsenter", "--all", "--target=1", "--", "su", "-"
        ],
        "stdin": true,
        "tty": true,
        "securityContext": {
          "privileged": true
        }
      }
    ]
  }
}' --attach "$@"

Just download or copy the previous script and execute it on your local indicating the node name to access.

# get cluster nodes
kubectl get nodes

# output
NAME                                            STATUS   ROLES    AGE     VERSION
ip-192-168-151-104.us-west-2.compute.internal   Ready    <none>   8d      v1.13.7-eks-c57ff8
ip-192-168-171-140.us-west-2.compute.internal   Ready    <none>   7d11h   v1.13.7-eks-c57ff8

# open superuser shell on specified node
./nsenter-node.sh ip-192-168-151-104.us-west-2.compute.internal

Once the script runs, we have access to the prompt from the node, and we can visualize and execute command lines from it.

# If you don't see a command prompt, try pressing enter.
# prompt
[root@ip-192-168-151-104 ~] ls 

Kubectl-ssh plugin

Provider-agnostic way of opening a remote shell to a Kubernetes node. Enables you to access a node even when it doesn’t run an SSH server or when you don’t have the required credentials. Also, the way you log in is always the same, regardless of what provides the Kubernetes cluster (e.g. Minikube, Kind, Docker Desktop, GKE, AKS, EKS, …)

https://github.com/luksa/kubectl-plugins#kubectl-ssh-node

Basically, this alternatives does the same as the nsenter script previously mentioned, it creates a pod in the Kubernetes cluster and access the node from that pod using some linux utilities packages. The difference is that the plugin is install on kubectl so the usage is by that tool.

kubectl ssh node my-node     # access a node in a multi-node cluster

Conclusions

IMHO, I prefer the alternative of the script nsenter-node.sh because it doesn’t require any installation, y easiness of usage. Both options allow me to access the shell of the indicated node without SSH Keys, or VPN, or Bastion, or any other infraestructure to access those servers as kubernetes nodes.

References:

• https://alexei-led.github.io/post/k8s_node_shell/
• https://medium.com/@marko.luksa/a-consistent-provider-agnostic-way-to-ssh-into-any-kubernetes-node-99328372d09f
• https://github.com/luksa/kubectl-plugins#kubectl-ssh-node