Skip to main content

System architecture

The system is built around several architecture concepts:

  • Microservices

    • The system is composed of tens of microservices, with independent storage and functional segregation.
    • They communicate together using HTTP REST APIs with JSON payloads, and Protobuf over gRPC on the high-frequency parsing path, all with JWT signed authentication & authorization.

  • CQRS - Command & Query Responsibility Segregation

    • Capture / Analysis and Search / Representation flows are decoupled and operated with independent services, optimized for their own processing.

High level system view​

High level view

Technologies​

Spider is built using modern technologies and architecture patterns:

  • Agents built in Go
    • Packets capture with AF_PACKET or LIBPCAP
    • Tls keys & network usage capture with eBPF
  • Microservices in Go, Node.js and Lua
    • The hot data path (capture, parsing, polling) is written in Go; Node.js serves the management and UI-support APIs
  • Open API with REST APIs
  • GRPC on critical paths, with Protobuf serialization of packets, TCP sessions and communications
  • Security with JSON Web Tokens
  • Clustering with Kubernetes
  • Web application using React.js
  • Distributed memory with Redis
  • Search engine with Elasticsearch

Technologies

Parsing process​

Capture / Analysis and Search flows are decoupled and operated with independent services, optimized for their own processing.

Parsing Architecture

Data storage​

Spider uses Redis as a distributed memory to have optimal processing speed.

Then Spider serializes captured and monitoring data in Elasticsearch search engine.

  • Its open API exposes Elasticsearch query and aggregation features.
  • Elasticsearch is configured for both speed of ingestion and search optimisation, with time based rolling indices, and progressive asynchronous loading of data on the UI.
  • Spider is designed to store billions of communications efficiently.

Low level network data​

  • Packets
  • Tcp sessions
  • UDP sessions - datagrams grouped into flows, or into request/response conversations

Ciphering protocols​

  • TLS
  • QUIC - for DNS over QUIC and HTTP/3

Higher protocols​

  • HTTP communications (HTTP/1.x and HTTP/2) and gRPC - parsed together by a single web parser, since a connection is told apart from another by what is sent over it, not by the port it uses
  • HTTP/3 - HTTP, gRPC and SSE over QUIC (UDP flows), decrypted and parsed by the same web parser
  • GraphQL and MCP (Model Context Protocol) - read by the same web parser, as an enrichment of the HTTP communications (and of the stream messages for subscriptions and streamed answers)
  • DNS - over UDP, TCP, DNS over TLS and DNS over QUIC
  • Redis
  • MQTT - 3.1, 3.1.1 and 5.0, over TCP and TLS
  • MongoDB - OP_MSG, compressed and legacy messages, over TCP and TLS
  • PostgreSQL
  • MySQL
  • Kafka
  • WebSocket and Server-Sent Events
  • More will come...

Configuration data​

  • Users
  • Teams
  • Whisperers
  • Gociphers
  • Controllers

Monitoring data​

  • Whisperer status
  • Capture status
  • …

AI agents access​

Spider is also an OAuth 2.1 authorization server, and exposes a remote MCP server so that AI agents can query captured traffic.

  • The Customers service is the authorization server: it registers or recognises the agent (dynamic registration or a Client ID Metadata Document), shows the consent screen, and issues opaque access and refresh tokens. The token is meaningless to any other service.
  • The MCP server (/mcp) is a resource server: it never issues tokens. It resolves the agent's token back into the approving user's identity through Customers' introspection endpoint, authenticating itself with its own application identity.
  • Every tool call then goes through the same read services as the UI and the spider CLI, with the user's own rights. Ownership, access filters and payload restrictions are enforced there, not in the MCP server.
  • The MCP server is stateless: any replica serves any request, and the selected team lives in Redis.

Both are off by default, gated by the MCP licence capability, customers.oauth.enabled and the mcp feature flag. See Connect an AI agent and Connecting agents with OAuth.

Complete system view​

Spider set of microservices and related storage is visible in real time in its monitoring UI, with network flows and quality.

Microservices

Scalability​

Spider system architecture has been proven to scale very well, with a track record of parsing more than 1GB/min with only 7 cores of CPU used.

Perf