How to do network sniff for Kubernetes pod running on GKE?
Recently, when we ran into an intermittent network slowdown for a service deployed on GKE, we thought it would be nice if we could capture the TCP packets for analysis to gain some insights on what is going on from the service side. Tim Downey described how to capture network traffic by using Ephemeral Containers in capturing network traffic from a kubernetes pod with ephemeral debug containers. Unfortunately, we don’t have Ephemeral Containers enabled in our GKE cluster. Luckily, Tim gave us an alternative method to capture network packets even without Ephemeral Containers — using ksniff. Today, I will walk you through how to set up ksniff to capture network packets in a K8S pod running on GKE.
ksniff is a kubectl plugin that utilizes tcpdump and Wireshark to start a remote capture on any pod in your Kubernetes cluster. You get the full power of Wireshark with minimal impact on your running pods.
⚠️ Ksniff isn’t production ready yet, running ksniff for production workloads isn’t recommended at this point.
We will walk through the steps to install the ksniff, wireshark, and X Server on WSL running on Windows 10.
Tools used:
- Windows 10 WSL
- Wireshark + X Server (VcXsrv Windows X Server)
- kubectl
- ksniff
Steps:
Install the latest kubectl
To use ksniff, you need to have the latest kubectl installed.
Build and Install ksniff from source
Normally, installation of ksniff is very straightforward. However, I ran into an error when I ran ksniff in privileged mode on GKE, because the support of the socket option is not available in any of the release builds yet. I had to build and install ksniff from the source code.
Install newer version of Wireshark
You may see all your captured packets showing “Unknown” in Protocol column. It is caused by the fact that Wireshark does not support the tcpdump command captured packet format. You will need to install newer version of Wireshark.
Setup X Server (VcXsrv Windows X Server)
To run the Wireshark in WSL, we need to have an X Server setup. Here we are using the VcXSrv Windows X Server.
After you installed the VcXsrv on Windows 10, you can set your DISPLAY environment variable in WSL by;
- Find out the Windows ip by using
ifconfigcommand. - Set the DISPLAY environment variable by using
export DISPLAY=”x.x.x.x:0"
Run ksniff in privileged mode to capture network traffic on GKE
Now you can start to sniff the network traffic from the K8S pod on GKE and do your packet analysis;
- Start the X Server on your Windows 10. Otherwise, you may not be able to start the Wireshark, which causes the temporary ksniff pod got killed. You may see the following error:
failed to start sniffing using privileged pod, exit code: ‘137’ error=”command terminated with exit code 137" - Start sniffing by issuing the command:
sudo kubectl sniff <your-pod-name-here> -f “port 8080” -n default -p — socket /var/run/docker.sock - When you are done with the capturing, you may need to manually delete the ksniff temporary pod from your GKE cluster.
Hope this is helpful.
