As of OpenShift 4 release, Red Hat Enterprise Linux CoreOS (RHCOS) is the the recommended and supported operating system running on all OpenShift Container Platform machines. RHCOS combines the quality standards of Red Hat Enterprise Linux (RHEL) with the automated, remote upgrade features from Container Linux.

RHCOS does not ship with package managers such as yum or dnf. This OS features transactional upgrades using the rpm-ostree system where updates are delivered by means of container images and are part of the OpenShift Container Platform update process.

Without OS package manager and support for direct download and installation of RPM packages the only way to run tools not pre-packaged with the OS is through containers. The good thing is CoreOS comes with a script called toolbox that launches a container to let you bring in your favorite debugging or admin tools.

Running toolbox container in OpenShift 4 RHCOS machines

You can start the toolbox container using the toolbox script provided. But first access the OpenShift node from where you want to run admin tools in the container.

You can use oc debug command or SSH.

--- Access node with SSH ---
$ ssh [email protected]

--- Access node with oc debug command ---
$ oc debug node/

Access with oc debug example.

$ oc debug node/node01.ocp.computingforgeeks.com
Starting pod/node01ocpcomputingforgeekscom-debug ...
To use host binaries, run `chroot /host`

The prompt comes from a special-purpose tools container that mounts the node root file system at the /host folder, and allows yoy to inspect the files from the node.

You need to start a chroot shell in the /host folder as shown in the command output. This will enable you to use host binaries in the shell.

chroot /host

You’ll see output like below:

chroot /host
Pod IP: 10.10.30.235
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# 

To start the toolbox container use the following command.

$ /usr/bin/toolbox

The first time you run the script it will download the toolbox container image to your node.

Trying to pull registry.redhat.io/rhel8/support-tools...
Getting image source signatures
Copying blob ec1681b6a383 done  
Copying blob c4d668e229cd done  
Copying blob 6b1688d3542f done  
Copying config 50b63c2aff done  
Writing manifest to image destination
Storing signatures
50b63c2aff8c13f9f8594c9eaf5fc961f39c74df6d9c6ddde8ca705f78f3c14d

Then it spins the container with Podman.

Spawning a container 'toolbox-core' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
command: podman run -it --name toolbox-core --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=toolbox-core -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest

Run telnet / tcpdump in OpenShift v4 CoreOS Servers

Once you’re in the container shell, you can use yum package manager to install debug and administration tools you want.

--- Install network tools ---
# yum -y install iproute net-tools

--- Install telnet ---
# yum -y install telnet

--- Install tcpdump ---
# yum -y install tcpdump

--- Install any other tool ---
# yum -y install 

Using telnet:

# telnet  

Using tcpdump:

Identify interface name – You need network tools installed.

# ip link show | head 
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3:  mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:1a:4a:16:01:73 brd ff:ff:ff:ff:ff:ff
7: ovs-system:  mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 4e:66:b9:32:0d:26 brd ff:ff:ff:ff:ff:ff
8: br0:  mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 72:d6:df:e8:13:48 brd ff:ff:ff:ff:ff:ff
9: vxlan_sys_4789:  mtu 65000 qdisc noqueue master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 4a:c4:7f:c1:85:f7 brd ff:ff:ff:ff:ff:ff

Using tcpdump.

# tcpdump 

This example is for capturing packets from any interface destined to port 443. We’re saving the output to a path in Node file system, accessible on /host directory.

# tcpdump -i any port 443 -s 0 -vv -s 0 -w  /host/tmp/testpacketname.pcap

You can replace any with interface name, e.g:

-i ens3

To end the capture press Control-C.

Running tcpdump capture from a container

Open a debug shell or SSH to a Node where the target Pod is running:

$ oc debug node/

--- OR ---
$ ssh [email protected]

Identify the ID of the target Pod’s Process ID using the crictl ps command:

# crictl ps

My container ID is 51a17d9a4b376. Let’s save this as variable.

container_id="51a17d9a4b376"

Get container PID:

container_pid=$(crictl inspect --output yaml $container_id  | grep 'pid:' | awk '{print $2}')

Confirm value:

# echo $container_pid
1124033
# ps 1124033
    PID TTY      STAT   TIME COMMAND
1124033 ?        Ss     0:00 /bin/sleep 3650d

You can the start the tcpdump in the container’s network namespace using the command below.

# nsenter -n -t $container_pid -- tcpdump 

--- Example ---
# tcpdump -i any port 443 -s 0 -vv -s 0 -w  /host/tmp/testpacketname.pcap

Please note that tcpdump needs to be installed in the container before running the commands.

More articles on OpenShift.

How To Install ArgoCD on OpenShift Cluster

How To Install Istio Service Mesh on OpenShift 4.x

Run Ceph toolbox for Rook on Kubernetes / OpenShift