Skip to main content

Reducing the number of pods

A default Spider installation runs about 95 pods in the spider-system namespace, most of them small and idle. That is a lot for a single-node cluster, a lab, or a cluster with a low pod-per-node cap (for example the Kubernetes default of 110).

Availability

The consolidated poller and the shared Redis store need Helm chart 8.1.0 or later. Turning off unused protocols works with any version.

This page lists the options to run fewer pods, from the safest to the most invasive. They are independent: use any combination.

OptionPods savedData loss when switchingIsolation lost
Turn off the protocols you do not useup to ~4 per protocolNo (nothing is parsed)None
One poller for everything14NoA slow job can starve the others
One Redis for the protocol storesup to 7Yes (cached data of these protocols)One memory flood can evict the others

With the three combined, a full installation drops from about 95 to about 74 pods; turning off unused protocols saves more.

Turn off the protocols you do not use​

Each optional protocol is a stack of its own: a parser, a read service, a poller and a Redis store (about 4 pods). Protocol parsing is switched on or off with the global.flips of your values.yaml:

global:
flips:
postgresqlParsing: true
redisParsing: false
grpcParsing: false
kafkaParsing: false
mysqlParsing: false
dnsParsing: false
mqttParsing: false
mongoParsing: false
webMessageStreamParsing: false # SSE and WebSocket messages

A protocol whose flip is false deploys none of its pods, and its jobs are left out of the consolidated poller below. See the feature flags for the complete list.

This is the cheapest option and the only one that also saves CPU and memory: do it first.

One poller for everything​

By default each protocol has its own poller deployment (poller-web, poller-pg, poller-kafka, ...): 15 pods draining Redis queues into Elasticsearch. The pollers are all the same binary running different jobs, so they can run together in one deployment, poller-all.

Add this to your values.yaml:

services:
pollerAll:
enabled: true
autoscaling:
min: 1
max: 5
cpuThreshold: 500m
resources:
requests:
cpu: 100m
memory: 300Mi
pollerWeb: { enabled: false }
pollerPg: { enabled: false }
pollerTcp: { enabled: false }
pollerCiphers: { enabled: false }
pollerWhisperers: { enabled: false }
pollerPackets: { enabled: false }
pollerHosts: { enabled: false }
pollerRedis: { enabled: false }
pollerGrpc: { enabled: false }
pollerKafka: { enabled: false }
pollerMysql: { enabled: false }
pollerDns: { enabled: false }
pollerMqtt: { enabled: false }
pollerMongo: { enabled: false }
pollerWebMessageStream: { enabled: false }

Then helm upgrade as usual.

  • The jobs of poller-all follow your feature flags: enabling a protocol adds its jobs, with no other step.
  • The per-job tuning you already have (services.pollerKafka.threadsCount, services.pollerKafka.jobs.<job>.threadsCount...) keeps applying.
  • The pod scales as a whole with its autoscaling settings. Size its resources for the sum of the jobs you run.
  • No data is lost when switching in either direction: the queues live in Redis and the new poller drains them where the old ones stopped.
  • Trade-off: a stuck or very busy job now competes with all the others for the same CPU, and you can no longer scale one protocol alone.

One Redis for the protocol stores​

Each of the eight protocols PostgreSQL, Redis, gRPC, Kafka, MySQL, DNS, MQTT and MongoDB has a Redis store of its own (redis-postgresql, redis-kafka, ...). They can share a single instance, redis-protocols. redis-pack, redis-tcp, redis-web and redis-shared stay separate: they are the hot or memory-heavy ones.

Add this to your values.yaml:

redis:
redisProtocols:
enabled: true
maxmemory: 4Gb # size it for the protocols you run
redisPostgresql: { enabled: false }
redisRedis: { enabled: false }
redisGrpc: { enabled: false }
redisKafka: { enabled: false }
redisMysql: { enabled: false }
redisDns: { enabled: false }
redisMqtt: { enabled: false }
redisMongo: { enabled: false }

cacheNodes:
pgComContents: redis-protocols
pgComs: redis-protocols
pgParsing: redis-protocols
redisComs: redis-protocols
redisComContents: redis-protocols
redisParsing: redis-protocols
grpcComs: redis-protocols
grpcParsing: redis-protocols
grpcComContents: redis-protocols
kafkaComs: redis-protocols
kafkaParsing: redis-protocols
kafkaComContents: redis-protocols
mysqlComs: redis-protocols
mysqlParsing: redis-protocols
mysqlComContents: redis-protocols
dnsComs: redis-protocols
dnsParsing: redis-protocols
mqttComs: redis-protocols
mqttParsing: redis-protocols
mqttComContents: redis-protocols
mongoComs: redis-protocols
mongoParsing: redis-protocols
mongoComContents: redis-protocols
mongoRecords: redis-protocols

Then helm upgrade as usual.

Data loss when switching

There is no migration. Switching to or from redis-protocols drops the data these Redis stores hold at that moment: the communications, parsing logs and queued items not yet written to Elasticsearch, for the eight protocols above. Data already in Elasticsearch is not touched. Do it on a new installation, or when losing a few minutes of freshly captured traffic is acceptable.

What you give up:

  • The instance is single-threaded: it becomes the throughput ceiling for all the protocols it carries.
  • A memory flood on one protocol evicts the keys of the others (Redis uses an allkeys-lru policy), and a single out-of-memory event affects all of them.
  • Memory monitoring reports one store: the per-protocol Redis screens of the monitoring UI show nothing for the protocols folded in. redis-protocols has its own memory gauge.

Keep the same choice across upgrades: the parsers and pollers must point at the same instance that holds their queues.

Which combination?​

SituationSuggested setup
Production, all protocols, plenty of nodesDefault
Production, a few protocolsTurn off unused protocols only
Single-node cluster, lab, demo, low pod capUnused protocols off + one poller + one Redis
Constrained and the traffic is heavyUnused protocols off + one poller; keep separate Redis stores

Check the result with kubectl get pods -n spider-system and the monitoring UI: the polling queues of the jobs must keep draining (near zero), and the parsing delay must stay low.