Skip to main content

New Whisperers optimisation reduces CPU usage by 30%!

· 4 min read
Thibaut Raballand
Creator of Spider

I got the idea to remove the processing of empty packets from the Whisperers, when you DON'T WANT / NEED to store those packets in the server.

And saved us 30% in CPU usage!

Idea​

After all recent optimisations, Whisperers CPU usage is a downside of the global Spider architecture.
I have several ideas in the roadmap to reduce this load, but they involved a major investment:

  • moving to a compiled language (Rust or Go)
  • using eBPF instead of libpcap

Meanwhile, I got the idea to remove the processing of empty packets from the Whisperers, when you DON'T WANT / NEED to store those packets in the server.

This is only valid for TCP sessions.
Those sessions require many 'session management' packets to ensure the delivery of packets over the network:

  • [SYN] packet to establish connection
  • [SYN, ACK] packet to confirm connection
  • [ACK] packets to confirm reception of data, or connection termination
  • [RST] packets to clear a connection on error
  • [FIN] packet to end connection
  • [FIN, ACK] packet to confirm end reception

Some of those packets are useful to capture to track TCP state: SYN, RST and ACK. But ACKs may be skipped.

This could lead to a good share of reduced capture and processing, and thus reducing the Whisperers load.

Implementation​

Research​

After some research, I found the libpcap filter below that allows to:

  • Avoid capturing TCP packets without data
  • Except for SYN, RST and FIN
(tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0)  
or (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)

Yesp, I know, not easy to invent that one!

GUI​

I introduced a new setting in Whisperer configuration to add this filter when capturing only TCP and not wanting to save empty packets:

New config option

This option is inactive by default.

Whisperers behavior change​

When the above setting is set, then whisperers:

  1. Automatically add the extra filter to the libpcap filter
  2. Adjust TCP tracking algorithm to account the closing of TCP session even without the last ACK (as they are not captured)
  3. Send only packets with TCP payload to the server

Monitoring​

The better track the change, I added 2 new graphics to Whisperers tab in monitoring:

  1. Whisperers total CPU usage
    1. To show the sum of CPU usage of all replicas of the same Whisperers.
    2. It gives a good information of the overhead of Spider monitoring on the client system.
  2. Whisperers average used RAM
    1. To show the average RAM usage by Whisperers across all their replicas.
    2. As often Whisperers are spread across the client cluster nodes, an average RAM usage is a better indicator that the sum.

New monitoring graphics

As you may see, the bottom level charts show CPU and RAM for ALL replicas, which is quite a mess, when top level charts aggregate information.

The tooltips show the count of replicas in the timeframe:

Tooltips

Results​

Activating the option introduced:

  • A drop of more than 30% of Whisperers CPU usage
  • But the new Whisperer release (Node 16) introduces an increase of RAM usage, especially for STOPPED Whisperers.. (?!)

Whisperers effect

It also revealed, as expected:

  • A drop of packets reception (and processing) -40%
  • While the transmitted data is nearly the same -7%

Uploaded effect

From services side,

  • 41% decrease of Pack-write total CPU usage, which dropped from 41% to 24%.

Pack write effect In green on the chart.

Thus reducing the overall servers load of 7%:

Servers load effect

Conclusion​

Very nice results for a small and easy improvement!

Of course, quality of parsing hasn't decreased ;)