Skip to main content

Configuring a Whisperer

Configuring a Whisperer is made through two different tabs grouping:

  • Capture config, for the agent configuration
  • Parsing config, for the server parsing configuration

Settings.png

Whisperer Type​

Whisperers exist in different types.
The type is defined at the creation or just after, and cannot be changed afterward.

Available types:

Whisperer typeDescription
UPLOADWhisperers dedicated to manual uploaded data (from the UI). They have higher retention duration.
INTERFACEWhisperers dedicated to real time streaming capture of data.
FILEWhisperers acting like real time but reading data from local pcap files. Used mostly for debugging.

The Whisperer type is set in the Capture Config tab in Source / Mode field:

WhispererTypeConfig.png

Import / Export configuration​

You may EXPORT the configuration of a Whisperer to IMPORT it when editing the configuration.

The export is in JSON, and may also be edited manually.

Copy configuration between Whisperers​

When editing you may click on COPY button to copy the configuration of another Whisperer unto yours.

It's even faster than export/import!

Capture Config​

CaptureConfig.png

Source​

Configure the source for packets capture.

Source.png

SettingDescriptionTypeComment
ModeThe type of the WhispererListINTERFACE | FILE | UPLOAD
Network interfaceThe network interface used for captureListany is a virtual interface on Linux that aggregates all others
lo is localhost
ethx are often the interfaces mounted by Kube
Pcap filterThe pcap filter to filter what network traffic you want to captureStringVery important to set right.
You often limit to tcp, or tcp port 80, or tcp portrange 3000-3010
MethodMethod used to capture the Network.
By default AFPACKET_THEN_LIBPCAP is used, which tries AFNET with a fallback to LIBPCAP. AFNET is slightly faster, but may not be always available, depending on your kernel.
ListLIBPCAP | AFPACKET | AFPACKET_THEN_LIBPCAP
Server hostTells what is the host of Spider, not to capture communications going to it.
To avoid infinite loops!
Fixed
Capture bufferCapture buffer of pcap in the Kernel, in kBIntegerShould often be several thousands.

BPF / pcap filter editor​

The pcap filter field has a dedicated editor:

PcapFilterEditor.png

Syntax highlighting - BPF keywords (tcp, udp, host, port, and, or, not), operators, and address literals are colored. A dense filter like tcp and (host 10.0.0.1 or port 5432) and not port 80 is readable at a glance.

Live parser - The filter is parsed as you type. Syntax errors are flagged inline before you submit - no more rejected captures because of a typo.

PcapFilterEditorError.png

The parser also detects common mistakes, like selecting many ports with and.

PcapFilterEditorWarning.png

Snippet picker - A snippet picker sits on top of the input. Pick a snippet - "port 443", "portrange 8000-9000", "host" - and it drops into the editor with the port number or hostname preselected, ready to edit or use as-is.

PcapFilterEditorEdit.png

Attachments​

Configure attachments made with this Whisperer by a Spider Controller.

Attachments.png

SettingDescriptionTypeComment
Time to liveWhen set, defines the maximum time an attachment of this Whisperer may be active.Duration in Sec or Min

Circuit breakers​

Circuit breakers settings allow to limit Cpu usage and Ram usage of Whisperers on the host they are capturing.
Thus limiting footprint in case of network traffic burst.

CircuitBreakers.png

SettingDescriptionTypeComment
CPU circuit breakerTells if the Whisperer should stop the capture when its CPU usage is too high.BooleanCircuit breakers lasts the duration of stats collection.
CPU limitSet the threshold for the circuit breaker. In percentage.Float
RAM circuit breakerTells if the Whisperer should stop the capture when its RAM usage is too high.Boolean
RAM limitSet the threshold for the circuit breaker. In MB.Integer
Throughput limiterTells if the Whisperer should stop the capture when captured traffic is too high.Boolean
Throughput limitSet the threshold for the throughput. In MB/min.Integer
info

For LocalAgents Whisperers, only admins may change the Throughput limit, set to 5MB/min by default.

Packets sending​

Settings to control the emission of packets.

PcapSending.png

SettingDescriptionTypeComment
BO time outTime out threshold when calling POST /packets API.Duration in S
Size before sendingSize threshold in kB of the buffer of packets before sending to back end.Integer
Sending frequencyDuration threshold after which the buffer of packets is sent even if not full.Duration in S
Parallel sendingHow many parallel sending of buffers are authorizedInteger
Sending queue sizeIf not slot for parallel sending is available, the Whisperer will queue the message.
When queue is full, new are discarded.
Integer

When server throughput may not be fast enough for capture load, you should increase Parallel sending setting.

caution

Those values have a direct effect on maximum Whisperer memory size:

  • queue sizexsize before sending = maximum queue size
  • Default to 500kB * 100 = 50 MB

DNS​

Configures the IP reverse resolving feature.

DNS.png

SettingDescriptionTypeComment
Resolve IPsWhether the Whisperer should reverse resolve IP addresses to host names.Boolean
Custom DNS ServerWhether the Whisperer uses the system resolution or a custom DNS server.Boolean
- DNS hostHost of the custom DNS server.String
- DNS portPort of the custom DNS server.Integer
TTLHow long hostname resolution is kept in cache.Duration in H
Refresh rateHow often a cached hostname resolution is refreshed.Duration in S
Send update delayHow often an update of the list of hostnames and IPs is sent to backend.Duration in S
BO time outTime out threshold when calling POST /hosts API.Duration in S
Purge delayHow often do the Whisperer clean the cache for TTL passed resolutions.Duration in H

Tcp Sessions​

Configures the tracking and sending of Tcp sessions.

Tcp Sessions.png

SettingDescriptionTypeComment
Track TCP sessionsWhether the Whisperer should track TCP sessions. If not, it will only send raw packets.BooleanUsually, you want this 😊
BO timeoutTime out threshold when calling POST /sessions API.Duration in S
Sending frequencyDuration threshold after which the buffer of sessions is sent.Duration in S
Tcp sessions TTLTime to Live after which an inactive - but not closed - Tcp session is removed from memory.Duration in M
Send only data packetsWhen set, a specific - complex - pcap filter is added to capture only data packets & packets required for Tcp sessions tracking. Only packets with data are sent. Reduces Spider CPU usage.
Please note that TCP sessions statistics will then be erroneous (IP payload etc.).
Boolean
Sessions sent at onceMax count of Tcp sessions sent by API request.Integer
Parallel sendingHow many parallel sending of buffers are authorized.Integer
Sending queue sizeIf not slot for parallel sending is available, the Whisperer will queue the message.
When queue is full, new are discarded.
Integer
Max packetLot sizeMaximum size of contingent Tcp payload authorized. Usually means a maximum size of request or response.
Packets above are sent to Spider but are not taken into account when parsing.
Avoids blowing up the servers memory. In Bytes.
Integer

UDP sessions​

Configures the grouping of UDP datagrams into transport sessions, so they can be parsed on the server side. DNS over UDP, DNS over QUIC and HTTP/3 produce nothing unless this is on. See UDP sessions for what a UDP flow and a UDP conversation are.

SettingDescriptionTypeComment
Track UDP sessionsWhether the Whisperer groups UDP datagrams into sessions. client.udpSessions.trackBooleanOff by default
Flow timeoutA UDP flow idle for longer is closed. flowTimeOutDuration in SDefault: PT30S
Conversation timeoutA UDP conversation idle for longer is closed. conversationTimeOutDuration in SDefault: PT60S
QUIC flow timeoutTimeout of the flows on QUIC ports. quicFlowTimeOutDuration in SDefault: PT120S. Advanced
Extra server portsDatagrams to or from these ports are grouped into flows. serverPortsPorts or rangesDefault: 53. Advanced
Extra conversation portsDatagrams to or from these ports are grouped into conversations. conversationPortsPorts or rangesDefault: 53. Advanced
  • When DNS parsing is on, the ports are derived from the DNS parsing settings: the DNS Ports to parse become server and conversation ports, the DoQ ports become server ports only (a QUIC connection is a flow, not a conversation). Likewise, with HTTP/3 parsing on, the HTTP/3 UDP ports become server ports. Only add extra ports that no parser list covers.
  • The Whisperer adds a UDP branch to its capture filter, built from the server ports, next to your own filter. Your pcap filter restricts the TCP branch only, and TCP capture is unchanged.
  • Timeouts are whole seconds, zero is refused.
  • Only IPv4 datagrams are tracked.
QUIC flows and the backend cache TTLs

A QUIC flow can only be decrypted while its TLS keys are still in the backend cache. Raising quicFlowTimeOut above PT120S needs the TCP session cache TTL of Tcp-Update (PT3M30S) and the DNS parsing log TTL (PT3M) raised above it too. For HTTP/3, the web parser keeps the parsing log of a QUIC flow at least 5 minutes, which covers quicFlowTimeOut up to about 4 minutes.

Filtering​

Configures filtering on hostnames to capture packets (from and to) or to avoid.

Filtering.png

SettingDescriptionTypeComment
Track by defaultWhether the Whisperer tracks (all) IP addresses by default.BooleanSet to false when you want to only track certain hostnames
Wait for resolvingWhether the Whisperer waits for having resolve the IP address to track its packets.BooleanOff by default: packets seen while resolving are dropped, TLS and QUIC handshakes included
Track unresolved IPWhether the Whisperer tracks IP addresses it could not resolve.Boolean
Hosts to trackList of regular expressions (or string) for valid hostnames to track.Array of PatternsA whitelist
Host to ignoreList of regular expressions (or string) for hostnames to avoid tracking.Array of PatternsA blacklist

An IP resolved to a hostname to track is not re-resolved until the host is not seen for long time (DNS.ttl).
An IP resolved to a hostname not to track is re-resolved every DNS.refreshRate.

When tracking by default, as soon as a host is to ignore, don't track.
When not tracking by default, as soon as a host is to track, do track.

tip

Hostnames filtering is a very powerful feature to limit the volume of data and the CPU usage.

  • When you don't need to track packets from or to a hostname, use this.
  • When you know the IP of the hosts to ignore, rather use a pcap filter.

Duplicates​

Duplicates.png

When capturing with 2 instances of the same whisperer from both side of the same communication, Spider captures twice packets and TCP sessions.
The following options allow to avoid duplicated communications resources by creating common ids and adding unique checks on server side.

To avoid duplicated packets (when you save them), choose the relevant option on the parsing part of configuration.

SettingDescriptionTypeComment
Avoid duplicated communicationsWhether the Whisperer should generate unique Ids to identify duplicates of packetsBoolean

Tls Keys​

TrackTLSKeys.png

Whisperers can track TLS keys together with a Gocipher agent.
The Whisperer will look for TLS information in the TCP session it captures, and the Gocipher will capture the session TLS secrets.

Due to technical constraints, TLS keys are captured on ALL PORTS. Pcap filter has no effect on TLS capture.
However, you may limit what port to link or not by:

  • Not capturing their packets
  • Listing the ports to link in the Parsing Config
SettingDescriptionTypeComment
Track TLS KeysWhether the Whisperer & Gociphers should track and send TLS secrets to the back office.Boolean
Key log fallback (/dev/null)Also read the TLS key log of workloads whose TLS library Gocipher cannot hook (Envoy / Istio sidecars, rustls, static BoringSSL). The workload must write its key log to /dev/null: see TLS key log fallback.
client.tlsKeys.keylogFallback
BooleanOff by default. Disabled when Track TLS Keys is off. Toggling restarts the captures of the target (a gap of a few seconds). Needs Linux kernel 5.8+

Status sending​

Configures the monitoring.

StatusSending.png

SettingDescriptionTypeComment
Send statusWhether to send Whisperer statistics to the back end.BooleanCrucial for monitoring.
BO timeoutTime out threshold when calling POST /status API.Duration in S
Sending frequencyFrequency to send statuses.Duration in S10 / 20s

Dump packets on client​

You may ask the Whisperer to dump .pcap files of packets when capturing.
Used only for internal debugging.

DumpPackets.png

SettingDescriptionTypeComment
Dump packetsWhether the Whisperer should dump packets to filesBoolean
Buffer sizeSize in kB of the files to dump (approx)Integer
Output pathOutput path to store the dumpsString

Configuration polling​

ConfigurationPolling.png

SettingDescriptionTypeComment
FrequencyTells how often the Whisperer calls to check a configuration changeDuration in sec.

Common parsing settings​

ParsingConfig.png

caution

Parsing options are cached for 5 minutes for INTERFACE Whisperers and 1 minute for UPLOAD Whisperers.

Automatic correlation tags​

Following a request across services needs an identifier that is the same on every hop. Most stacks already emit one - a W3C traceparent from an OpenTelemetry SDK, an X-Request-ID minted by an edge proxy, an X-Correlation-ID set by an API gateway - but until this release Spider extracted none of them until somebody wrote the rules by hand.

Spider now ships those rules pre-written, under a single tag name: correlationId.

Seeding a new Whisperer​

The Whisperer creation form carries an Extract correlation ids switch in the Parsing & saving section, on by default. Leave it on and the new Whisperer starts extracting correlationId from its first captured packet, with nothing else configured.

Adding them to an existing Whisperer​

Each protocol's tag section carries an Add default correlation tags button. It appends any seeded rule for that protocol that is not already present, comparing on name, path and pattern - so pressing it twice cannot create duplicates, which would otherwise inflate the tag's count and cardinality.

AddDefaultCorrelationTags.png

There is deliberately no bulk backfill of existing Whisperers: that would edit your configuration unasked, and could collide with a correlationId rule you wrote yourself.

What gets seeded​

ProtocolSideSourceExtracted
HTTPRequesttraceparent headerthe 32-hex trace id only
HTTPRequestX-Request-ID headerthe whole value
HTTPRequestX-Correlation-ID headerthe whole value
HTTPResponseX-Request-ID, X-Correlation-ID headersthe whole value
gRPCRequesttraceparent metadatathe 32-hex trace id only
gRPCRequestx-request-id, x-correlation-id metadatathe whole value
KafkaRecordstraceparent record headerthe 32-hex trace id only
KafkaRecordsx-request-id record headerthe whole value
PostgreSQLRequesttraceparent in a sqlcommenter query commentthe 32-hex trace id only
MySQLRequesttraceparent in a query commentthe 32-hex trace id only
MQTTPUBLISHtraceparent MQTT 5 user propertythe 32-hex trace id only
MQTTPUBLISHx-request-id, x-correlation-id MQTT 5 user propertiesthe whole value

Rules sharing one name accumulate into a single deduplicated value list, so one filter selection correlates a chain whichever convention each hop happened to use.

GrpcCorrelationTags.png

The gRPC request tags of a seeded Whisperer. The three correlationId rules accumulate into one tag; only the traceparent one needs a Pattern, to capture the stable trace id out of the composite value. orderId alongside them is a hand-written business tag - the two kinds live side by side and are configured identically.

Only the trace id, never the whole traceparent

A traceparent is 00-<32-hex trace id>-<16-hex span id>-<flags>, and the span id changes at every hop - that is the point of the format. Matching the whole header value would match exactly one hop and never the chain, which is precisely why the seeded rules capture the trace id segment alone.

HTTP works even with headers turned off

The HTTP rules match the raw wire header block, which reaches extraction independently of Save raw headers and Headers to filter - those govern only what is stored. So the seeded rules work on a Whisperer that saves no headers at all.

Known gaps​

  • Redis is not correlated and will not be: RESP has nowhere to carry a propagation header, and no instrumentation puts one there.
  • MySQL is not seeded. Add a query text tag named correlationId with a pattern such as traceparent='00-([0-9a-f]{32}) to correlate statements carrying a sqlcommenter-style comment.
  • MQTT 3.1.1 has no user properties: only MQTT 5 messages are correlated.
  • PostgreSQL prepared statements. Extraction is deliberately local to the message that carried the query text, so a cached prepared statement executed many times only carries the identifier on the request that actually prepared it. Every other execution is simply untagged on the PostgreSQL hop - a missing correlation rather than a wrong one.

Once extracted, tell the UI which tag names identify a chain in the Correlation setting.

The same business id (an order id, say) often travels through several protocols: in an HTTP URI, a Redis key, a Kafka record, an SQL statement, a MongoDB command. It also travels between services over a single protocol: /orders/{id} in the URI at one service, an X-Order-Id header at a second, an orderId JSON field at a third. The Link ids pass of the Capture Advisor mines the whisperer's captured traffic for ids that recur across protocols, or across the locations of one protocol (e.g. between microservices), then proposes one concept name with the tag rules that extract it in every place where it appears. The rules are ordinary tag rules; they differ from the automatic correlation tags in that they target business ids, not technical correlation ids.

  • What is examined. HTTP, Redis, MQTT, Kafka, PostgreSQL, MySQL, MongoDB and gRPC, on the request side and on the reply side (SQL result columns, MongoDB reply documents, decoded record and message fields). Each proposed rule is checked against the parser playground before it is offered.
  • Not proposed. An id found in one place only (every service calling /orders/{id} is one rule), or echoed within a single call (in the request URI and the response body of the same communication). Correlation ids (trace ids, request ids) are never offered as a concept. Sensitive fields are skipped. WebSocket and SSE messages are examined (connection URI, message data, SSE event id) and need the same payload permission as HTTP; an id seen only within one WebSocket/SSE connection is not a link; compressed (context-takeover) and binary messages are not read; when WebSocket or SSE parsing is off (per whisperer or cluster-wide) a note says so. DNS stays out.
  • Where. The Link ids button of the whisperer Parsing tab (see Capture Advisor), a pass of the Capture Advisor setup wizard, spider templates link-ids and spider templates apply-link-ids in the Spider CLI, and the spider_link_ids and spider_apply_link_ids MCP tools.
  • Payload access. Values and reply-side sources are examined only for callers allowed to see payloads. For anyone else nothing is mined from content and a note says so.
  • Settings it depends on. A protocol whose content is not stored cannot be examined: save the content of the communications, bind the content schemas that decode Kafka, MQTT and gRPC payloads, and keep the data rows saved (saveDataRows for SQL and MongoDB replies, saveWriteDocs for MongoDB written documents). When one of them is off, a note on the result says which part was skipped.
  • Saving. In the whisperer configuration editor, Add merges the chosen rules into the unsaved draft; nothing is stored until you press Save. From the wizard, the CLI and the MCP tools, applying is all-or-nothing: the rules of every protocol and location go in one configuration write, and if one rule is refused (HTTP 422) nothing is written.

Rules apply to traffic parsed afterwards, not to communications already stored.

Timezone​

Timezone.png

You may associate a timezone to each Whisperer.

When the Whisperer is selected on the UI, the times of the captured communications are displayed according to this time zone - depending on the settings.

Data storage policy​

In Spider configuration, you may define Customisable Data Store Policies.
These defines different available retention periods for Whisperers data.

DSP.png

The Data storage policy of the Whisperer defines how long you want to keep its data.

  • The default value is default (definitive lack of imagination here)
  • It may be changed at any time in the lifetime of a Whisperer, but existing data will still be associated and managed by the policy active at the time of indexing
  • When the setting is set to a Policy that do not exist (any more), the default one is used.

Packets saving​

Configures if packets should be saved for analysis.

SavePackets.png

Spider is built to parse packets in streaming.
You do not need to save packets unless you want to analyze at low packet level.

  • When saving packets,

    • Packets are saved to Elasticsearch.
    • It requires much CPU and space.
    • You may choose to avoid duplicated packets when capturing the two side of the same communication.
      • This is costly for ES as it then has to check for if the packet exists when saving. Activate it only when needed.
  • When not saving packets,

    • Packets are only kept in memory until they are parsed.
    • This prevents reconstructing payload data for TCP or HTTP once the parsing is finished.
    • You should definitely activate 'Save content' HTTP option, or else, you won't see the data ;)

Do not save packets:

  • when you filter out part of the parsed data, and you don't want the data to be reconstructed,
  • or when you don't need to keep packets, to save space and ES resources.

You may activate saving packets only temporarily when needed.

SettingDescriptionTypeComment
Save PacketsWhether Spider should save packets from this Whisperer.Boolean
- Avoid duplicated packetsWhether Spider should check for duplicated packets to avoid saving twice the same.Boolean

Tcp sessions saving​

You may wish not to save TCP sessions in Spider (once the parsing is done). Use this:

  • when your parsing quality is good,
  • when you don't care of TCP sessions.

TcpSave.png

SettingDescriptionTypeComment
Save TCP sessionsWhether Spider should save Tcp sessions from this Whisperer.Boolean
tip

When not saving packets, saving Tcp sessions has little interest.

TLS Keys​

TLSKeysParsing.png

You may wish to link or not captured TLS secrets and keys to TCP sessions.
You may define if link should be done by default, or if it should be restricted to certain ports.

SettingDescriptionTypeComment
Link TLS Keys to TCP sessionsWhether this TLS keys captured for this Whisperer should be associated to its TCP sessions.Boolean
Link by defaultWhether Spider should link on all ports by default.Boolean
Ports to linkDefine server ports using TLS encryption.
The TCP sessions on those ports will be linked to TLS secrets captured.
You may define lists of ports or ranges: 80, 8080, 9200, 3000-3100.
Array of Integers or Integers ranges
Ports to ignoreDefine server ports not to link.Array of Integers or Integers ranges

Host names​

When Whisperer resolves IP addresses to capture hostnames, it can determine automatically a shortname to display based on the FQDN provided by the DNS.

For this, specify one or several regular expression(s) that are used to extract parts of the FQDN, which are then concatenated by '.'. Patterns are executed in order. First one that matches is selected.

HostNames.png

SettingDescriptionTypeComment
Name patternsList of patterns to transform a FQDN into something shorter, but meaningful.Array of Patterns
tip

This is particularly useful to have shorter names on the UI.

  • You don't need to see: myservice.mynamespace.svc.cluster.local
  • Seeing myservice is enough.

The FQDN is kept along the short name, don't worry:

HostId.png

Web traffic parsing​

HTTP/1.x, HTTP/2 and gRPC all travel as ordinary TCP connections, and Spider tells them apart on the connection itself rather than by the port they arrive on. HTTP/3 is the exception: it travels over QUIC, on UDP, and is turned on separately, see HTTP/3 parsing. Configuring which traffic gets parsed is therefore unified across the three: one list of ports selects which TCP sessions the web parser looks at, and a switch per protocol chooses which of what it detects is saved as a communication.

Web parsing​

Configures which TCP sessions should be parsed for web traffic, and which detected protocols are saved.

WebParsing.png

SettingDescriptionTypeComment
Parse web trafficWhether this Whisperer is capturing web traffic (HTTP/1.x, HTTP/2, gRPC) that should be parsed.Boolean
Parse by defaultWhether Spider should parse all ports by default.Boolean
Ports to parseDefine server ports exposing web traffic.
The TCP sessions on those ports will be parsed.
You may define lists of ports or ranges: 80, 8080, 50051, 3000-3100.
Array of Integers or Integers ranges
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Save HTTP communicationsWhether the HTTP/1.x and HTTP/2 communications detected on the parsed ports should be saved.Boolean
Save gRPC communicationsWhether the gRPC communications detected on the parsed ports should be saved.BooleanHidden when gRPC is not enabled by your licence
Save SSE communicationsWhether a text/event-stream response detected on the parsed ports is split into individual stream messages.BooleanHidden when stream message parsing is not enabled
Parse HTTP/3 (QUIC)Whether HTTP, gRPC and SSE carried over QUIC are parsed, on the HTTP/3 UDP ports. server.http3.parseHttp3BooleanOff by default. See HTTP/3 parsing
- HTTP/3 UDP portsUDP ports carrying HTTP/3. server.http3.ports
Single ports only, no ranges.
Array of IntegersDefault: 443. Advanced

The parsing log settings are not here: they are per protocol, and stay in the HTTP Parsing and gRPC Parsing sections below.

info

Ports select sessions, protocols select output. A port no longer belongs to a single protocol: HTTP/1.x, HTTP/2 and gRPC are all detected on the connection itself, so one port may carry any of them. Ports to parse / Ports to ignore decide which TCP sessions the web parser looks at; Save HTTP communications, Save gRPC communications and Save SSE communications decide which of what it detects is kept as a communication.

tip

HTTP/2 needs no configuration of its own. It is detected automatically on the same ports as HTTP/1.x, including over TLS, with no extra setting to turn on.

caution

Behaviour change. Before this release, a port declared for gRPC only that carried plain HTTP/1.1 traffic produced nothing - that traffic was invisible. After this release it produces an HTTP communication, because the port is shared and the protocol is no longer inferred from it. This is the correct outcome, but it changes what an existing Whisperer records: if you see new HTTP volume on a port previously configured as gRPC-only, this is why.

tip

Migration is automatic. Existing Whisperer configurations are migrated when read: the old HTTP and gRPC port lists are merged, Parse web traffic turns on if either the old Parse for HTTP or Parse for gRPC was on, and Save HTTP communications / Save gRPC communications follow the old protocol switches. No operator action is required.

HTTP/3 parsing​

HTTP/3 runs over QUIC, so it is captured as UDP sessions and decrypted with the TLS keys Gocipher captures, then parsed by the same web parser as HTTP/1.x and HTTP/2. Its communications are ordinary HTTP communications with httpVersion 3.0, and they follow the same settings: Save HTTP, Save gRPC and Save SSE communications, filters, templates and tags.

It needs, on the same Whisperer:

  • Parse web traffic and Parse HTTP/3 (QUIC) on, and the server port in HTTP/3 UDP ports. The ports are added to the captured UDP ports automatically.
  • Track UDP sessions. Without it, nothing is captured, and the UI says so.
  • Tls Keys tracked, with a Gocipher on the client or the server process. QUIC is always encrypted: a flow without its keys is skipped, never decoded.
  • Save content, or Save only TLS decrypted content, in HTTP filtering (server.webStreams.saveContent / saveTlsDecryptedContent) to keep the bodies. QUIC packets are not decrypted again at read time, so without either an HTTP/3 body shows Body not stored.
caution

A QUIC connection whose handshake was not captured cannot be decrypted: its flows are skipped. After attaching a Whisperer or turning HTTP/3 on, restart the client or wait for it to open new connections. A connection that moves to a new UDP flow later (NAT rebinding, client migration, an idle gap) keeps being decoded.

info

A port listed both in the DNS over QUIC ports and in the HTTP/3 UDP ports is parsed as DNS.

HTTP protocol parsing settings​

HTTP parser provides the useful, and complex parsing options.
They allow filtering, but most importantly extracting business knowledge to show it on Spider.

HTTP Parsing​

Which TCP sessions are parsed, and whether HTTP output is saved at all, is configured once for all web traffic under Web traffic parsing. This panel shows that the ports are shared, and holds the HTTP parsing log settings.

HttpParsing.png

SettingDescriptionTypeComment
PortsRead-only reminder that the ports are shared with gRPC, under Web traffic parsing.-Not editable here
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE)BooleanFor Spider troubleshooting only

HTTP filtering​

Configures filtering to remove headers, content or whole communication for certain URIs:

  • When it is sensitive information (and in production)
  • When it is not required to save them (like monitoring requests, healthcheck requests etc.), to save place.

Configures if Spider should save reassembled communication content.

HttpFiltering.png

SettingDescriptionTypeComment
Headers to filterDefine headers to remove in parsed object (like Basic auth header 😉).
You may filter by header name or value content.
It expects patterns to apply on the HTTP header line.
Array of Patterns
URIs to filterFilter out communications by URIs.
For sensitive communications like auth calls.
Array of Patterns
Save contentDefine if the content should be saved within the HTTP resource.
Important when not saving packets.
Boolean
Save only TLS decrypted contentSave the content of TLS-encrypted communications only, when Save content is off.
Required for HTTP/3 bodies, which cannot be decrypted again from the packets. saveTlsDecryptedContent
BooleanShown when TLS keys are linked to sessions and Save content is off
- URIs to filter request bodyFilter out request body contents by URIs patterns.Array of Patterns
- URIs to filter response bodyFilter out response body contents by URIs patterns.Array of Patterns
Save raw headersDefine if the raw headers should be saved within the HTTP resource.
- URIs to filter request raw headersFilter out request raw headers by URIs patterns.Array of Patterns
- URIs to filter response raw headersFilter out response raw headers by URIs patterns.Array of Patterns
tip

Regexp are matched case insensitive.

tip

Save content when not saving packets.

tip

Save raw headers when you need to see what was really exchanged (and you don't filter any headers).

tip

Think about filtering ^Authorization: Basic headers to avoid saving passwords in Spider on production.

Request templates​

On the parsed HTTP communications, you may define regular expression patterns to name the request.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

  • The name of the request:
    Ex: Creation of order, createAnOrder, POST /orders/{id}, depending on your style.
  • The parts of the request to parse to identify it. They will be concatenated, as in their raw format.
    • Method
    • URI, mandatory, with querystring
    • Headers
    • Request body
  • The regular expression to execute, with /ms flags.

Many regular expressions may be defined. They are matched in order, and the first match is kept.

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

RequestsTemplates.png

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups 💪.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
MethodShould method be matched.Boolean
URIShould URI be matched.Boolean
HeadersShould headers be matched.Boolean
BodyShould body be reassembled, transfer decoded, decompressed and matched.Boolean
tip

This is one of the most useful feature of parsing.

Obviously this requires CPU, as for the next two ones.
But it is quite reasonable and optimised.
And the value is so much worth it!!

Request tags​

On the parsed HTTP communications, you may define regular expression patterns to tag the request.

The tag name and value(s) are saved in the communication metadata and may be used when searching.
You define:

  • The name of the tag: Ex: 'ClientName', 'ApiVersion', 'Authenticated'...
  • The parts of the request to parse to tag it. They will be concatenated, as in their raw format.
    • Method
    • URI with querystring
    • Headers
    • Request body
  • The regular expression to execute, with /msg flags.

If the regular expression includes capture group(s), the captured data will be stored as tag value(s).
Without capture group, the tag value will be 'true' (text).
Many regular expressions may be defined. They are all executed, and all the tags are kept.

Alongside the tag value, are also saved:

  • The tag count
    • How often it matched
  • The tag cardinality
    • How often it matched with different values

This allows counting (and searching) the number of items in a response, the number of different customers...

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

RequestsTags.png

SettingDescriptionTypeComment
NameName of the tag.String
PatternRegular expression to capture the tag.Pattern
MethodShould method be matched.Boolean
URIShould URI be matched.Boolean
HeadersShould headers be matched.Boolean
BodyShould body be reassembled, transfer decoded, decompressed and matched.Boolean
tip

Together with response tags, these are the most value-adding features of parsing!

Response tags​

On the parsed HTTP communications, you may define regular expression patterns to tag the response.

It is the same process as for the request, but on the response.

ResponseTags.png

SettingDescriptionTypeComment
NameName of the tag.String
PatternRegular expression to capture the tag.Pattern
StatusShould status line be matched.Boolean
HeadersShould headers be matched.Boolean
BodyShould body be reassembled, transfer decoded, decompressed and matched.Boolean
tip

When defining the same tag in request or response, or many times the same tag in either, all extracted values will be associated to the same tag and counted for tag count and cardinality.

Schema bindings​

Schema bindings let Spider decode HTTP request and response bodies into structured JSON for display and extraction.

Each binding maps an HTTP method + URI template + Content-Type to a serialization format (Protobuf, MessagePack, or JSON), so Spider can transcode the raw bytes into a readable format.

HttpSchemaBindings.png HttpSchemaBindingsEdit.png

SettingDescriptionTypeComment
NameDisplay name for the binding.String
MethodHTTP method to match (e.g. GET, POST). Leave empty to match all methods.ListGET | POST | PUT | DELETE | PATCH | HEAD | OPTIONS | (any)
URI templateURI pattern in RFC 6570 format (e.g. /api/users/{id}). Path parameters are treated as wildcards.String
Content-TypeContent-Type header value to match (e.g. application/x-protobuf). Trailing * matches vendor families.String
Request formatSerialization format of the request body.Listprotobuf | msgpack | json
Request schemaSchema ID for the request body.String
Request typeProtobuf message type name for the request body.StringRequired when format is protobuf
Response formatSerialization format of the response body.Listprotobuf | msgpack | json
Response schemaSchema ID for the response body.String
Response typeProtobuf message type name for the response body.StringRequired when format is protobuf
tip

Schema bindings are useful when your HTTP API exchanges binary-encoded bodies (protobuf, msgpack) instead of plain JSON. Once bound, Spider decodes the body and displays it as readable JSON, and request/response tags can extract fields from it.

SSE and WebSocket message parsing settings​

A Server-Sent Events response and a WebSocket connection are not request/response exchanges - each is one HTTP request that stays open and then carries a long series of messages. Spider parses both into one communication per message, so a feed that runs for hours becomes a searchable series instead of a single communication whose body keeps growing.

Both live in the SSE & WebSocket parser configuration panel, and each has its own switch, its own content flag and its own templates and tags. What differs is the shape of a message:

SSEWebSocket
DirectionServer to client onlyBoth directions
TransportHTTP/1.1 and HTTP/2HTTP/1.1 (101 upgrade) and HTTP/2 (RFC 8441 extended CONNECT)
Recognised byContent-Type: text/event-stream on the responseThe handshake that opened the connection
Matchable partsURI, event type, Last-Event-ID, dataURI, direction, opcode, data
info

There is no port list here. SSE and WebSocket both ride the HTTP/web engine: the same ports, the same TCP sessions, the same parser. Which sessions are looked at is decided once under Web traffic parsing; this panel only adds rules for naming and tagging the individual messages.

Turning message parsing on​

Neither switch is in this panel - both are in the Web parsing panel, next to their HTTP and gRPC siblings, because all of them gate the same engine.

Save SSE communications

StateWhat Spider records
OnOne Stream message communication per SSE event, plus the HTTP communication for the request that opened the stream (the handshake).
OffOne ordinary HTTP communication whose response body is the whole accumulated stream. No per-message split.

Save WebSocket communications

StateWhat Spider records
OnOne Stream message communication per data frame and for the close frame, in both directions, plus the HTTP communication for the handshake that opened the connection.
OffOnly the opening handshake, as an ordinary HTTP communication. The frames are still captured as packets, but are not decoded into messages.
caution

Turning either on affects connections opened afterwards. A stream already open when you change the setting keeps the behaviour it started with, so a long-lived feed may take a while to switch over.

For WebSocket this is stricter than it sounds: a connection whose handshake was not captured cannot be parsed at all. The frame stream cannot be joined mid-flight, and there is no retroactive fix - restart the client after attaching the agent.

SSE message content​

SettingDescriptionTypeComment
Save SSE contentWhether each message's payload is saved.BooleanOn by default. Hidden when message parsing is off.
info

A flag of its own, not HTTP's Save content. Unlike an ordinary HTTP body, a suppressed SSE payload cannot be rebuilt from packets afterwards - there is no per-event boundary left in the raw bytes once it is gone. So turning off HTTP's Save content does not also have to lose every SSE message payload: Save SSE content governs SSE independently, and defaults to on.

SSE message templates​

On the parsed stream messages, you may define regular expression patterns to name the message - the same idea as HTTP request templates, applied per message instead of per request.

The name is saved in the message metadata as req.template, and is used to aggregate and filter messages.

You define:

  • The name of the message. Ex: Order tick, orderCreated, depending on your style.
  • The parts of the message to match. They are concatenated, in their raw format.
  • The regular expression to execute, with /ms flags.
SettingDescriptionTypeComment
NameName of the template.
May include placeholders for captured groups, and $t.
String
PatternRegular expression. Must start with ^ and end with $.Pattern
URIShould the stream URI be matched.BooleanOn by default
EventShould the SSE event type be matched.Boolean
Last-Event-IDShould the SSE id: field be matched.Boolean
DataShould the message payload be matched.Boolean

Rules are matched in order and the first match wins.

info

A message is never left without a name. When no rule matches, the message inherits the template of the HTTP request that opened the stream. It is never a bare dash - so a stream whose handshake is named GET /orders/feed produces messages named GET /orders/feed until you write a rule that says otherwise.

tip

$t refines rather than replaces. Writing $t / tick as the name inserts the inherited handshake template in place of $t, so one rule can specialise the stream's name instead of overwriting it. To match a literal $t, escape it as $$t.

note

The matchable parts are not HTTP's method, headers, status and body. A stream message has no method and no headers of its own - the URI is the identity of the connection, shared by both directions - so the parts offered are the SSE event fields. A message with no event: field matches as the SSE default event type, message.

SSE message tags​

On the parsed stream messages, you may define regular expression patterns to tag the message - the same idea as HTTP request tags, applied per message.

The tag name and value(s) are saved in the message metadata as stats.tags, and may be used when searching.

SettingDescriptionTypeComment
NameName of the tag.
Ex: OrderId, Region.
String
PatternRegular expression.Pattern
URIShould the stream URI be matched.BooleanOn by default
EventShould the SSE event type be matched.Boolean
Last-Event-IDShould the SSE id: field be matched.Boolean
DataShould the message payload be matched.Boolean

Every rule runs, not just the first match. With capture group(s), the captured data becomes the tag value(s); without a capture group, the tag value is the text true.

Each message starts from the tags of the HTTP request that opened the stream - its own request tags and any correlation tag - and layers its rules on top:

SituationResult
A tag name only the handshake carriesInherited unchanged.
A tag name only a message rule producesAdded to that message.
Both, same shape (captured values on each side, or a flag on each side)Merged - captured values union with captured values.
Both, different shape (a flag against captured values)The message's rule replaces the inherited value.
tip

Inheritance is what makes a stream message correlate with everything around it. A traceparent picked up on the opening request flows onto every message of that stream, so the messages appear in the same end-to-end trace as the HTTP, PostgreSQL, Redis, gRPC and Kafka activity they belong to - with no rule to write.

tip

Test both rule sets before applying them with the Templates & Tags playground, which has a stream-message mode alongside the HTTP, PostgreSQL, Redis, gRPC and Kafka ones.

WebSocket message content​

SettingDescriptionTypeComment
Save WebSocket contentWhether each message's payload is saved.BooleanOn by default. Hidden when WebSocket parsing is off.
Save ping/pong framesWhether control frames become communications of their own.BooleanOff by default.
info

A flag of its own, not HTTP's Save content. As with SSE, a suppressed WebSocket payload cannot be rebuilt from packets afterwards, so Save WebSocket content governs it independently of HTTP's Save content.

caution

Watch the volume on both of these. A compressed message is stored decompressed, and an inflated payload can be an order of magnitude larger than the bytes on the wire - Save WebSocket content is the only control over that. And a heartbeat every ten seconds is roughly 17 000 documents per day per connection, which is why Save ping/pong frames is off by default; the packets view already shows them.

note

What is decoded, and what is not. A permessage-deflate message is inflated and readable only when that direction negotiated no_context_takeover - a permanent limit, not a gap, because the compression window cannot be carried across a parsing pass. Without it the message keeps its raw payload and a 200 Undecoded label, which is a truthful record rather than a failure, and one direction may decode while the other does not on the same connection. socket.io v4 envelopes on text messages are decoded into their packet type, namespace, event name and ack id. A close frame with no status code is recorded as 1005, never 0 - 0 is the Incomplete status and the two must stay distinguishable.

WebSocket message templates​

On the parsed WebSocket messages, you may define regular expression patterns to name the message - the same idea as SSE message templates, matched against the parts a WebSocket message actually has.

SettingDescriptionTypeComment
NameName of the template.
May include placeholders for captured groups, and $t.
String
PatternRegular expression. Must start with ^ and end with $.Pattern
URIShould the connection URI be matched.BooleanOn by default
DirectionShould the message direction be matched.Boolean
OpcodeShould the frame opcode be matched.Boolean
DataShould the message payload be matched.Boolean

Rules are matched in order and the first match wins. As with SSE, a message that matches no rule inherits the template of the HTTP request that opened the connection rather than being left blank, and $t inserts that inherited template in place of overwriting it.

note

Direction is the part SSE does not have. An SSE message is always server to client, so the part would be a constant; a WebSocket conversation is genuinely bidirectional, and direction is usually the first thing worth matching on. Opcode takes the slot the SSE event type occupies - it is the message's shape discriminator.

There is no Event and no Last-Event-ID here, and they are not silently ignored: a rule copied from the SSE list is rejected on save rather than stored and left matching nothing.

WebSocket message tags​

On the parsed WebSocket messages, you may define regular expression patterns to tag the message - the same idea as SSE message tags, against the same parts as the templates above.

SettingDescriptionTypeComment
NameName of the tag.
Ex: OrderId, Region.
String
PatternRegular expression.Pattern
URIShould the connection URI be matched.BooleanOn by default
DirectionShould the message direction be matched.Boolean
OpcodeShould the frame opcode be matched.Boolean
DataShould the message payload be matched.Boolean

Every rule runs, not just the first match, and inheritance from the opening request works exactly as it does for SSE message tags - including the traceparent that puts every message of the connection in the same end-to-end trace as the HTTP, PostgreSQL, Redis, gRPC and Kafka activity around it.

GraphQL parsing​

GraphQL rides the same web engine as HTTP, HTTP/2, SSE and WebSocket - there is no separate port list and, unlike the settings above, no dedicated panel yet in this UI. It is configured through server.graphql, shown here at its default:

{
"parseGraphql": true,
"paths": ["/graphql"],
"heuristic": false,
"maxBodySize": 65536
}
KeyMeaning
parseGraphqlWhether HTTP communications on paths are enriched with req.graphql / res.graphql. On by default.
pathsExact request paths to treat as GraphQL endpoints.
heuristicDetect GraphQL bodies off the configured paths too. Off by default - a REST body that happens to carry a query string key would otherwise be mislabelled.
maxBodySizeBody size cap, in bytes, for GraphQL parsing. A request over this size is not enriched; a response over it gets res.graphql.status: UNKNOWN.

See GraphQL for what gets extracted and how it is templated, and HTTP GraphQL tab for where it shows up in the Analysis UI. This also needs the cluster-wide global.flips.graphqlParsing flip.

MCP parsing​

MCP (Model Context Protocol) rides the same web engine - no separate port list and no dedicated panel in this UI. It is configured through server.mcp, shown here at its default:

{
"parseMcp": true,
"paths": ["/mcp"],
"contentFallback": false,
"maxBodySize": 262144
}
KeyMeaning
parseMcpWhether HTTP communications on paths are enriched with req.mcp / res.mcp. On by default.
pathsExact request paths to treat as MCP endpoints.
contentFallbackAlso detect MCP on other paths, from the request's headers and JSON-RPC method. Off by default - it costs a look at more POST bodies.
maxBodySizeBody size cap, in bytes. A request over this size is not enriched; a response over it gets res.mcp.status: UNKNOWN. Tool results can be large, hence the higher default than GraphQL.

See MCP for what gets extracted. This also needs the cluster-wide global.flips.mcpParsing flip - not global.flips.mcp, which enables Spider's own MCP server.

PostgreSQL protocol parsing settings​

PostgreSQL parser provides similar parsing options as HTTP, but simpler.
They allow some filtering for safety, but most also, as for HTTP, they allow extracting business knowledge to show it on Spider.

PostgreSQL Parsing​

Configures which TCP sessions should be parsed for PostgreSQL communications.

PsqlParsing.png

SettingDescriptionTypeComment
Parse for PostgreSQLWhether this Whisperer is capturing PostgreSQL communications that should be parsed.Boolean
Parse by defaultWhether Spider should parse all ports by default.Boolean
Ports to parseDefine server ports exposing PostgreSQL API.
The TCP sessions on those ports will be parsed for HTTP communications.
You may define lists of ports or ranges: 5432, 5430-5440.
Array of Integers or Integers ranges
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debugBooleanFor Spider troubleshooting only

PostgreSQL filtering​

Configures filtering to some specific messages:

  • When it is sensitive information (like connections and in production)
  • When it is not required to save them (like data rows etc.), to save place.

PsqlFiltering.png

SettingDescriptionTypeComment
Save connectionsDefine if we should save Startup requests/responsesBooleanSaves space when not saved. Connection metadata is included in all coms.
Save password messagesWhen saving Startup, you may want not to save password messages. Saving those could be unsafe if PostgreSQL is not using crypted password handshake.Boolean
Save disconnectionsNot very useful messages ;)Boolean
Save SELECT data rowsTo save space, you may want NOT TO save data received in response of SQL queries.Boolean
Save INSERT rowsTo save space, you may want NOT TO save data sent to be saved in PostgreSQLBoolean

Query templates​

On the parsed PostgreSQL communications, you may define regular expression patterns to give a name to the query.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

  • The name of the request:
    Ex: Creation of order, createAnOrder, POST /orders/{id}, depending on your style.
  • The regular expression to execute, with /ms flags over the SQL query.

Many regular expressions may be defined. They are matched in order, and the first match is kept.

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

PsqlQueryTemplates.png

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups 💪.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
tip

This is one of the most useful feature of parsing.

Obviously this requires CPU, as for the next two ones.
But it is quite reasonable and optimized.
And the value is so much worth it!!

Request tags​

On the parsed PostgreSQL communications, you may define fields to extract from the query or result set.

The tag name and value(s) are saved in the communication metadata and may be used when searching.
You define:

  • The name of the tag: Ex: 'ClientName', 'OrderId'...
  • The name of the fields to find.
    • Format: [table].[column]->>[jsonpath] or [table].[column].[jsonpath] or [table].[column] (no jsonpath)
    • If the field is already extracted from the JSON object in the field to check, use dot notation.
    • If Spider has to extract the field from the JSON object, use ->>.
  • The fields of the request to fetch values from.
    • Checked fields, used in WHERE or JOIN conditions
    • Insert fields, used to insert data in
    • Update fields, updated by the UPDATE command
    • Conflict update fields, updated when a conflict happens

The values associated to theses fields will be extracted from the queries, or find the bound values of prepared statements.
Once extracted, they will be saved as tag value(s) for searches, stats etc.

Note that fields computed based on original ones like sum, format, conversions are not supported.
Only original table fields are supported.

Alongside the tag value, are also saved:

  • The tag count
    • How often it matched
  • The tag cardinality
    • How often it matched with different values

This allows counting (and searching) the number of items in a response, the number of different customers...

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

PsqlRequestsTags.png

In this example, we want to extract the customer id from any query to Orders, the customer id being stored in a JSONB field data as:

{
"customer": {
"id": 123
}
}

Customer id is used:

  • In where conditions, as data->'customer'->>'id' = $1
  • In INSERT or UPDATE statements, as part of the json inserted to data jsonb column.

The above setting extracts customer id from all queries to Orders table, as a tag to be able to search PostgreSQL communications addressing this specific customer.

SettingDescriptionTypeComment
Tag nameName of the tag.String
Table fieldField to extract the value from.StringLeave empty when extracting from the query text
PatternOptional regular expression narrowing the extracted value.PatternRequired when Query is on
CheckedShould we extract from checked fields values.Boolean
InsertShould we extract from inserted fields values.Boolean
UpdateShould we extract from updated fields values.Boolean
ConflictShould we extract from updated fields values when conflict.Boolean
QueryShould we match the raw SQL query text of the request.BooleanRuns Pattern over the query text instead of a table field

Extracting from the query text​

The Query option changes what the rule reads: instead of resolving a table field, it runs Pattern over the raw SQL text of the request and stores capture group 1. This is what lets Spider pull an identifier out of a comment rather than out of the data, and it is how the seeded sqlcommenter correlation rule works.

Prepared statements

Extraction is deliberately local to the message that carried the query text. An execution of a cached prepared statement inherits no query text, so it produces no tag on that hop rather than a value borrowed from an unrelated request. On a driver that prepares once and executes many times, only the request that actually prepared the statement carries the tag.

tip

Together with response tags, these are the most value-adding features of parsing!

Response tags​

On the parsed PostgreSQL communications, you may define what fields you want to extract from the response.

It is the same process as for the request, but on the response.

PsqlResponseTags.png

SettingDescriptionTypeComment
Tag nameName of the tag.String
Table fieldField to extract the value from.String
tip

When defining the same tag in request or response, or many times the same tag in either, all extracted values will be associated to the same tag and counted for tag count and cardinality.

Redis protocol parsing settings​

Redis parser provides parsing options to decode Redis commands and responses.
They allow filtering, and as for HTTP, they allow extracting business knowledge to show it on Spider.

Redis Parsing​

Configures which TCP sessions should be parsed for Redis communications.

RedisParsing.png

SettingDescriptionTypeComment
Parse for RedisWhether this Whisperer is capturing Redis communications that should be parsed.Boolean
Parse by defaultWhether Spider should parse all ports by default.Boolean
Ports to parseDefine server ports exposing Redis API.
The TCP sessions on those ports will be parsed for Redis communications.
You may define lists of ports or ranges: 6379, 6380-6389.
Array of Integers or Integers rangesDefault: 6379
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debugBooleanFor Spider troubleshooting only

Redis content settings​

Configures whether Redis command values should be saved.

RedisContent.png

SettingDescriptionTypeComment
Save content for writesSave values in write commands (SET, HSET, LPUSH, etc.)Boolean
Save content for readsSave values in read commands (GET, HGET, LRANGE, etc.)Boolean
Save content for pub/subSave content in Pub/Sub messages (PUBLISH, SUBSCRIBE, etc.)Boolean

Request templates​

On the parsed Redis communications, you may define regular expression patterns to name the request.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

  • The name of the request:
    Ex: Get user, Set session, depending on your style.
  • The parts of the request to parse to identify it. They will be concatenated.
    • Command (e.g. GET, SET, HSET)
    • Key
    • Args

Many regular expressions may be defined. They are matched in order, and the first match is kept.

RedisRequestTemplates.png

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups 💪.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
CommandShould command be matched.Boolean
KeyShould key be matched.Boolean
ArgsShould additional arguments be matched.Boolean

Request tags​

On the parsed Redis communications, you may define regular expression patterns to tag the request.

The tag name and value(s) are saved in the communication metadata and may be used when searching.

RedisRequestTags.png

SettingDescriptionTypeComment
NameName of the tag.String
PatternRegular expression to capture the tag.Pattern
CommandShould command be matched.Boolean
KeyShould key be matched.Boolean
ArgsShould additional arguments be matched.Boolean
ValueShould the value being written be matched.Boolean

Response tags​

On the parsed Redis communications, you may define regular expression patterns to tag the response.

RedisResponseTags.png

SettingDescriptionTypeComment
NameName of the tag.String
PatternRegular expression to capture the tag.Pattern
Reply typeShould the reply type be matched.Boolean
ErrorShould error messages be matched.Boolean
ValueShould the returned value be matched.Boolean

Schema bindings​

Schema bindings let Spider decode raw Redis values into structured JSON for display and extraction.

Each binding maps a Redis key pattern to a serialization format (Protobuf, MessagePack, or JSON), so Spider can transcoding the raw bytes into a readable format.

RedisSchemaBindings.png

SettingDescriptionTypeComment
NameDisplay name for the binding.String
DBRedis database number to match (e.g. 0, 1). Leave empty to match all databases.String
Key globGlob pattern to match Redis keys (e.g. user:*, session:*:data).String
FormatSerialization format of the value.Listprotobuf | msgpack | json
Schema IDSchema identifier used for protobuf (matches a schema registered in Spider).String
Message typeProtobuf message type name within the schema.StringRequired when format is protobuf
tip

Schema bindings are required for Request and Response tags that use field paths - Spider needs to decode the value before it can extract fields from it.

gRPC protocol parsing settings​

gRPC parser provides parsing options to decode gRPC messages using protobuf schemas.
They allow filtering, and they enable extracting business knowledge from structured messages.

gRPC Parsing​

Which TCP sessions are parsed, and whether gRPC output is saved at all, is configured once for all web traffic under Web traffic parsing - there is no separate gRPC port list. This panel shows that the ports are shared, and holds the gRPC parsing log settings.

grpcParsing.png

SettingDescriptionTypeComment
PortsRead-only reminder that the ports are shared with HTTP, under Web traffic parsing.-Not editable here
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE)BooleanFor Spider troubleshooting only

gRPC content settings​

Configures whether gRPC message bodies should be saved.

GrpcContent.png

SettingDescriptionTypeComment
Save content for requestsSave request message bodies.Boolean
Save content for responsesSave response message bodies.Boolean

Request templates​

On the parsed gRPC communications, you may define regular expression patterns to name the request.

GrpcRequestTemplates.png

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups 💪.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
Full methodShould the full method path be matched (e.g. /package.Service/Method).BooleanDefault: true
MetadataShould gRPC metadata (headers) be matched.Boolean
MessageShould the request message body (decoded) be matched.BooleanRequires a schema binding for the method

Request tags​

On the parsed gRPC communications, you may extract fields from the request using gjson paths.

Tags use gjson paths to navigate a view object assembled for each communication.

GrpcRequestTags.png

The path navigates a view object with these top-level fields:

FieldContent
serviceService name (e.g. UserService)
methodMethod name (e.g. GetUser)
fullMethodFull gRPC path (e.g. /pkg.UserService/GetUser)
metadata.<key>gRPC request metadata (headers), e.g. metadata.x-tenant
message.<field>Decoded request body (requires a schema binding)
SettingDescriptionTypeComment
NameName of the tag.String
Pathgjson path into the request view (e.g. service, metadata.x-tenant, message.user.id, message.items.#.id).Stringmessage.* requires a schema binding for the method
PatternOptional regular expression narrowing the selected value.PatternSee Narrowing a selected value

Narrowing a selected value​

A gjson path selects a value whole, which is not always what you want: a W3C traceparent carries a stable trace id and a per-hop span id in one string, an Authorization metadata entry carries a scheme and a token, a compound key carries a tenant and an id. The optional Pattern column narrows what the path selected.

PatternEffect
EmptyThe selected value is stored whole. This is the historical behaviour, unchanged.
With a capture groupCapture group 1 replaces the value. A value the pattern does not match is dropped, not stored raw.
Without a capture groupActs as a filter: the selected value is kept when it matches and dropped otherwise.

A pattern that does not compile makes the rule inert, and is reported as an error in the playground.

The seeded correlation rules are exactly this: a path selecting metadata.traceparent, narrowed by ^[\da-f]{2}-([\da-f]{32})- to the trace id.

Response tags​

On the parsed gRPC communications, you may extract fields from the response using gjson paths.

The path navigates a view object with these top-level fields:

FieldContent
grpcStatusgRPC status code integer (0 = OK)
grpcStatusNamegRPC status name (e.g. OK, NOT_FOUND, INTERNAL)
grpcMessagegRPC error message string (when status ≠ OK)
metadata.<key>Trailing gRPC metadata (headers)
message.<field>Decoded response body (requires a schema binding)
SettingDescriptionTypeComment
NameName of the tag.String
Pathgjson path into the response view (e.g. grpcStatusName, grpcMessage, message.items.#, message.status.code).Stringmessage.* requires a schema binding for the method
PatternOptional regular expression narrowing the selected value.PatternSee Narrowing a selected value

Schema bindings​

Schema bindings let Spider decode gRPC message bytes into structured JSON for display and extraction.

Each binding maps a service/method pattern to request and response schemas (typically protobuf).

GrpcSchemaBindings.png

SettingDescriptionTypeComment
NameDisplay name for the binding.String
Service patternRegex pattern to match the gRPC service name (e.g. .*\.UserService). Leave empty to match all.String
Method patternRegex pattern to match the gRPC method name (e.g. Get.*). Leave empty to match all.String
Request formatSerialization format of the request message.Listprotobuf | msgpack | json
Request schemaSchema ID for the request.String
Request typeProtobuf message type for the request.String
Response formatSerialization format of the response message.Listprotobuf | msgpack | json
Response schemaSchema ID for the response.String
Response typeProtobuf message type for the response.String
tip

Schema bindings are required for Request and Response tags - Spider needs to decode the protobuf message before it can extract fields from it.

Kafka protocol parsing settings​

Kafka parser provides parsing options to decode Kafka Produce and Fetch operations.
They allow filtering, and they enable extracting business knowledge from record payloads.

Kafka Parsing​

Configures which TCP sessions should be parsed for Kafka communications.

KafkaParsing.png

SettingDescriptionTypeComment
Parse for KafkaWhether this Whisperer is capturing Kafka communications that should be parsed.Boolean
Parse by defaultWhether Spider should parse all ports by default.Boolean
Ports to parseDefine server ports exposing Kafka brokers.
The TCP sessions on those ports will be parsed for Kafka communications.
You may define lists of ports or ranges: 9092, 9093.
Array of Integers or Integers rangesDefault: 9092
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debugBooleanFor Spider troubleshooting only

Kafka content settings​

Configures whether Kafka record key/value bytes should be saved.

KafkaContent.png

SettingDescriptionTypeComment
Save content for producesSave record content (key/value) from Produce requests.Boolean
Save content for fetchesSave record content (key/value) from Fetch responses.Boolean

Request templates​

On the parsed Kafka communications, you may define regular expression patterns to name the request.

KafkaRequestTemplates.png

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups 💪.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
CommandShould the Kafka command (e.g. Produce, Fetch) be matched.BooleanDefault: true
TopicShould the topic name be matched.BooleanDefault: true
Client IDShould the client ID be matched.Boolean
MessageShould the decoded record value be matched.BooleanRequires a schema binding for the topic

Request tags​

On the parsed Kafka communications, you may extract fields from the Produce request using gjson paths.

Tags use gjson paths to navigate a view object assembled for each communication.

KafkaRequestTags.png

The path navigates a view object with these top-level fields:

FieldContent
commandKafka command (e.g. Produce)
clientIdKafka client ID
topic.0First topic name (e.g. orders)
records.N.keyDecoded key of record N (requires a schema binding)
records.N.headers.<key>Record header of record N (e.g. records.#.headers.traceparent). No schema binding needed - headers are plain wire bytes.
records.N.value.<field>Field from decoded value of record N (requires a schema binding)

Use gjson array queries to span all records: records.#.value.customerId extracts customerId from every record.

SettingDescriptionTypeComment
NameName of the tag.String
Pathgjson path into the request view (e.g. command, topic.0, records.0.value.order.id, records.#.value.customerId).Stringrecords.*.value.* requires a schema binding for the topic
PatternOptional regular expression narrowing the selected value.PatternSee Narrowing a selected value

Response tags​

On the parsed Kafka communications, you may extract fields from the Fetch response using gjson paths.

The path navigates a view object with these top-level fields:

FieldContent
statusCodeHTTP-equivalent status code (200 = OK, 400 = error)
statusTextStatus description
errorCodeKafka error code (0 = success)
topic.0First topic name
records.N.value.<field>Field from decoded value of record N (requires a schema binding)
SettingDescriptionTypeComment
NameName of the tag.String
Pathgjson path into the response view (e.g. statusCode, errorCode, records.#, records.0.value.status).Stringrecords.*.value.* requires a schema binding for the topic
PatternOptional regular expression narrowing the selected value.PatternSee Narrowing a selected value

Schema bindings​

Schema bindings let Spider decode Kafka record bytes into structured JSON for display and extraction.

Each binding maps a topic pattern to a serialization format. Spider supports Protobuf, MessagePack, JSON, and Avro - including Confluent Schema Registry wire-format detection.

KafkaSchemaBindings.png

SettingDescriptionTypeComment
NameDisplay name for the binding.String
Topic globGlob pattern to match Kafka topic names (e.g. orders-*, user.events). Leave empty for all.String
SideWhich part of the record to decode.Listkey | value | (both)
Registry IDConfluent Schema Registry ID embedded in the record wire format. Overrides format/schema ID.IntegerOptional - only needed for Confluent wire format
FormatSerialization format of the record.Listprotobuf | msgpack | json | avro
Schema IDSchema identifier registered in Spider.String
Message typeProtobuf or Avro message type name within the schema.StringRequired when format is protobuf
tip

Schema bindings unlock templates and tags on record content - Spider must decode the bytes before it can match patterns or extract field values.

note

When records carry the Confluent wire-format prefix (magic byte 0x00 + 4-byte schema ID), Spider auto-detects the schema ID embedded in the record. You may define a binding with a matching Registry ID to explicitly map that ID to a schema registered in Spider.

MySQL protocol parsing settings​

MySQL parser provides similar parsing options as PostgreSQL.
They allow filtering for safety and to save space, and they allow extracting business knowledge from the statements and their results.

MySQL Parsing​

Configures which TCP sessions should be parsed for MySQL communications.

SettingDescriptionTypeComment
Parse for MySQLWhether this Whisperer is capturing MySQL communications that should be parsed.BooleanOn by default
Parse by defaultWhether Spider should parse all ports by default.BooleanOff by default
Ports to parseDefine server ports exposing MySQL.
The TCP sessions on those ports will be parsed for MySQL communications.
You may define lists of ports or ranges: 3306, 3300-3310.
Array of Integers or Integers rangesDefault: 3306
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debugBooleanFor Spider troubleshooting only
note

MySQL parsing also needs the mysqlParsing feature flag to be on in the Helm chart. Encrypted MySQL sessions are decrypted with the TLS keys captured by Gocipher.

MySQL filtering and content​

Configures what is saved from the statements and their results, to protect sensitive data or to save space. The SQL text, the bound parameters and the result rows are saved in a separate content document per communication.

SettingDescriptionTypeComment
Save disconnectionsSave the COM_QUIT communications.BooleanOn by default
Save SELECT data rowsTo save space, you may want NOT TO save the rows received in response of SQL queries.BooleanOn by default
Save INSERT rowsTo save space or protect data, you may want NOT TO save the values sent by INSERT (and REPLACE) statements. When off, the literals and parameters are replaced by a visible <redacted> marker.BooleanOn by default
Max content bytesMaximum size of the content (parameters and rows) saved for one communication. Rows past the limit are counted but not saved.IntegerDefault: 524288 (512 KiB). Maximum: 16777216 (16 MiB).
note

These settings govern what is saved. Tags are extracted from the live traffic, including result rows tagged by response tags, even when the rows are not saved.

MySQL query templates​

On the parsed MySQL communications, you may define regular expression patterns to give a name to the statement.

The name is saved in the communication metadata (req.template) and is used to aggregate and filter requests.
It works as for PostgreSQL query templates: a name, and a regular expression matched over the SQL text. Rules are matched in order, and the first match is kept.

SettingDescriptionTypeComment
NameName of the template.
May include placeholder for captured groups.
String
PatternRegular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.Pattern
No rule needed to start

When no rule matches, the template is the normalised SQL: literals replaced by ?, lists collapsed, comments removed. Statements differing only by their values already share a template. Write rules only to give them a business name, or to group several statements.

For a prepared statement, the pattern is matched over the SQL text of its PREPARE.

MySQL request tags​

On the parsed MySQL communications, you may define fields to extract from the statement.

The tag name and value(s) are saved in the communication metadata and may be used when searching.
The rules are the same as for PostgreSQL request tags, on the analysis of the statement:

  • a table field, table.column, optionally with a JSON path (table.column->>path),
  • looked up in the checked, inserted, updated or conflict updated fields,
  • or a regular expression over the raw SQL text, with the Query option.
SettingDescriptionTypeComment
Tag nameName of the tag.String
Table fieldField to extract the value from.StringLeave empty when extracting from the query text
PatternRegular expression, run over the query text.PatternRequired when Query is on
CheckedShould we extract from checked fields values.Boolean
InsertShould we extract from inserted fields values.Boolean
UpdateShould we extract from updated fields values.Boolean
ConflictShould we extract from updated fields values when conflict.Boolean
QueryShould we match the raw SQL query text of the request.BooleanRuns Pattern over the query text instead of a table field

Two differences from PostgreSQL:

  • Bound parameters are resolved. When the statement is a prepared one, a ? placeholder in a checked, inserted or updated field is replaced by the value bound at execution. A tag on customers.email extracts grace@example.com, not ?. A NULL or undecodable parameter yields no value.
  • Query text extraction reaches comments. The Query option runs over the raw SQL including comments, so a /*traceparent='00-...'*/ comment can be captured. Every capture group of every match contributes a value; a pattern with no group contributes the whole match.

Every rule runs, and rules sharing a name accumulate their values under the same tag, with count and cardinality.

Prepared statements

An execution of a prepared statement carries the SQL text of its PREPARE, so Query rules and table fields work on every execution. An execution whose PREPARE was not captured has no SQL text and no analysis: it gets no tag.

INSERT rows not saved

Tags are computed before INSERT values are redacted. A tag on an inserted column carries the real value even when Save INSERT rows is off. Your tag rule is the consent to extract it.

MySQL response tags​

On the parsed MySQL communications, you may define which result columns you want to extract values from.

SettingDescriptionTypeComment
Tag nameName of the tag.String
ColumnName of the result column. Use column->>path to extract a field of a JSON column value.String

The column is looked up by name in every result set of the response, exact match first, then case-insensitive. NULL values are ignored. Rows are read even when Save SELECT data rows is off, and beyond Max content bytes.

Choose low-cardinality columns

The values of a tag are not capped: tagging a unique column of a large SELECT stores every distinct value.

DNS protocol parsing settings​

DNS is parsed over UDP, TCP, DNS over TLS and DNS over QUIC, into DNS communications. DNS is metadata only: there is no content to save or filter.

DNS Parsing​

Configures which transport sessions should be parsed for DNS communications.

SettingDescriptionTypeComment
Parse for DNSWhether this Whisperer is capturing DNS communications that should be parsed. server.dns.parseDnsBooleanOff by default
Parse by defaultWhether Spider should parse all ports by default.BooleanOff by default
Ports to parseServer ports carrying DNS, over UDP, TCP or TLS.
You may define lists of ports or ranges: 53, 853, 5300-5310.
On UDP, they are captured automatically, see UDP sessions.
Array of Integers or Integers rangesDefault: 53, 853
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
DoQ (QUIC) portsUDP ports carrying DNS over QUIC. server.dns.quicPorts
They are captured automatically as UDP flows, and need TLS keys.
Array of IntegersDefault: 853. Advanced
Save parsing logDefine if the parsing tracking resource should be saved.BooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debugBooleanFor Spider troubleshooting only
note

DNS parsing also needs the dnsParsing feature flag to be on in the Helm chart.

  • DNS over UDP needs UDP sessions to be tracked.
  • DNS over TLS and DNS over QUIC are decrypted with the TLS keys captured by Gocipher: they need Tls Keys to be tracked. A flow on a DoQ port that gets no TLS key is skipped, never read as plain DNS.

DNS filtering​

SettingDescriptionTypeComment
Response timeoutHow long a query waits for its response before being saved without one (status 1000). Measured on the capture timeline, not wall-clock. server.dns.responseTimeoutDuration in SDefault: PT5S

DNS name templates​

On the parsed DNS communications, you may define regular expression patterns to give a name to the question.

The name is saved in the communication metadata (req.template) and is used to aggregate and filter queries. The patterns are matched against the query name, lower-cased and without its trailing dot. Rules are matched in order, and the first match wins.

SettingDescriptionTypeComment
NameName of the template.
A literal, or $1, $2... back-references to the pattern's capture groups.
String
PatternRegular expression.Pattern
No rule needed to start

When no rule matches, the template is the normalised name: numbers, UUIDs, long hexadecimal strings and encoded IP addresses replaced by ?, and every reverse lookup collapsed to ?.in-addr.arpa. See Templates.

DNS request tags​

On the parsed DNS communications, you may define values to extract from the question.

SettingDescriptionTypeComment
Tag nameName of the tag. No dot.String
Query fieldqname, qtype or qclass.StringEmpty means qname when a pattern is given
PatternOptional regular expression applied to the field. The first capture group is the value, else the whole match.PatternA field or a pattern is required

DNS response tags​

On the parsed DNS communications, you may define values to extract from the response.

SettingDescriptionTypeComment
Tag nameName of the tag. No dot.String
Response fieldrcode, answers or min_ttl.String
RR typeFor answers only: keep the answers of this record type (A, AAAA, CNAME...).StringEmpty means all answers

Tags are extracted from the full answer set, before the 64 records per section are capped for storage. A tag on answers holds every matching record's data.

MQTT protocol parsing settings​

MQTT 3.1, 3.1.1 and 5.0 are parsed over TCP, plain or decrypted from TLS, into MQTT communications. The settings are in the server.mqtt block of the Whisperer configuration.

MQTT Parsing​

Configures which TCP sessions should be parsed for MQTT communications.

SettingDescriptionTypeComment
Parse for MQTTWhether this Whisperer is capturing MQTT communications that should be parsed. server.mqtt.parseMqttBooleanOn by default
Parse by defaultWhether Spider should parse all ports by default.BooleanOff by default
Ports to parseServer ports of the MQTT brokers.
The TCP sessions on those ports will be parsed for MQTT communications.
You may define lists of ports or ranges: 1883, 8883, 1880-1890.
Array of Integers or Integers rangesDefault: 1883, 8883
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Idle timeoutHow long a quiet MQTT connection stays one TCP session. It also sets the Gossiper's idle timeout for these ports only. server.mqtt.idleTimeoutDuration in whole minutesDefault: PT45M. Maximum: PT48M
Save parsing logDefine if the parsing tracking resource should be saved. server.mqtt.saveMqttParsingBooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debug. server.mqtt.keepWholeParsingLogBooleanFor Spider troubleshooting only
note

MQTT parsing also needs the mqttParsing feature flag to be on in the Helm chart. MQTT over TLS is decrypted with the TLS keys captured by Gocipher: it needs Tls Keys to be tracked.

Ports and connection start
  • A broker on a non-standard port must be listed in Ports to parse, otherwise its sessions are not parsed and no MQTT communication appears.
  • Spider must see the CONNECT of a connection to parse it: the protocol version, and so the packet layout, is only known from it. A client connected before the Whisperer was attached is not parsed: its session is skipped. Restart the client, or attach first.
Why an idle timeout per port

IoT clients often stay connected for hours with a long keepalive. The Whisperer splits a TCP session idle for more than 2 minutes, and the second half has no CONNECT (and, over TLS, no handshake to link keys to), so it cannot be parsed. The MQTT idle timeout keeps such connections in one session, on the MQTT ports only. A client whose keepalive is longer than the idle timeout is still split.

MQTT content​

Configures what is saved from the PUBLISH payloads, to protect sensitive data or to save space. Payloads are saved in a separate content document per communication, and are only shown to users with payload access.

SettingDescriptionTypeComment
Save payloadsSave the PUBLISH payloads. When off, every topic is metadata only. server.mqtt.savePayloadsBooleanOn by default
Max payload bytesMaximum size of a saved payload. Larger payloads are truncated, and their original size is kept. server.mqtt.maxPayloadBytesIntegerDefault: 65536 (64 KiB). Maximum: 262144 (256 KiB)
Excluded topicsTopic filters whose PUBLISH messages are not saved at all: no communication, no payload. server.mqtt.topics.excludeArray of topic filtersDefault: $SYS/#
Metadata only topicsTopic filters whose communications are saved without their payload. server.mqtt.topics.metadataOnlyArray of topic filters
Save CONNECT passwordSave the password of the CONNECT packets in the content store. server.mqtt.savePasswordBooleanOff by default. Advanced. Needs Save payloads

Topic filters use the MQTT wildcards: + for one level, # for all the levels below, as in sensors/+/temp or $SYS/#.

note
  • $SYS/# is excluded by default, since brokers publish it continuously. Remove it to see the broker statistics.
  • The payload hash and size are computed even when the payload is not saved.
  • The user name of a connection is always kept. The password is never stored on the communication: when Save CONNECT password is on, it goes to the content store, and only users with payload access to the Whisperer can read it. It is never exported.
Volume

There is no sampling. To reduce the volume of a busy broker, exclude the noisy topics, make them metadata only, turn Save payloads off, or put the Whisperer on a shortLife data storage policy.

MQTT content schemas​

Content schemas tell Spider how to decode the binary payloads of a topic, to show them as JSON and to extract tags from them.

SettingDescriptionTypeComment
NameDisplay name of the binding.String
Topic filterMQTT topic filter the binding applies to (+ and # allowed).String
Content typeMQTT 5 content type the binding applies to. Empty for any.String
FormatSerialization format of the payload.Listprotobuf | avro | cbor | msgpack | json
Schema IDSchema registered in Spider.StringRequired for protobuf and avro
Message typeProtobuf message type name within the schema.StringRequired for protobuf

The first matching binding wins. Without a binding, Spider still decodes:

  • Sparkplug B payloads, recognised from their spBv1.0/... topic;
  • MQTT 5 payloads declaring a JSON, CBOR or MessagePack content type.

MQTT 3.1.1 has no content type: a binding on the topic filter is then the only way to decode a binary payload.

MQTT topic templates​

On the parsed MQTT communications, you may define regular expression patterns to give a name to the topics.

The name is saved in the communication metadata (req.template) and is used to aggregate and filter messages, and to group the topics of the Topic Map. The patterns are matched against the topic of the PUBLISH messages. Rules are matched in order, and the first match wins.

SettingDescriptionTypeComment
NameName of the template.
May use the groups of the pattern: plant-${1}.
String
PatternRegular expression (RE2) matched on the topic.Pattern

At most 50 rules.

No rule needed to start

When no rule matches, the template is the normalised topic: numeric levels, UUIDs and long hexadecimal levels are replaced by ?, so plant/7/temp becomes plant/?/temp. The other packets (CONNECT, SUBSCRIBE...) are named by their type. See Templates.

MQTT tags​

On the parsed MQTT communications, you may define values to extract from each PUBLISH message.

SettingDescriptionTypeComment
NameName of the tag. Letters, digits, _ and -, no dot.String
PathA gjson path over the view of the message described below.String
PatternOptional regular expression. Its first group replaces the value; with no group, it only keeps the values that match.Pattern

The path reads a view of the message with:

PathValue
topicThe topic
topicLevels.NThe level N of the topic, from 0: topicLevels.1 of plant/7/temp is 7
clientId, usernameThe client of the connection
qos, retain, dupThe PUBLISH flags
directionreq for a client publish, res for a delivery to a subscriber
contentType, payloadFormat, responseTopicMQTT 5 properties
userProperties.<name>An MQTT 5 user property
payload.<path>A field of the payload, when it is JSON, Sparkplug B, or decoded by a content schema

At most 50 tags. Rules sharing a name accumulate their values under the same tag, with count and cardinality.

Tags are not protected like payloads

Tags are extracted even when the payload is not saved (metadata only topics, Save payloads off), and they are visible to every user who can read the Whisperer, including users without payload access. Your tag rule is the consent to extract the value. Do not tag secrets.

The Add default correlation tags button adds the correlation tags of MQTT 5: the traceparent, x-request-id and x-correlation-id user properties.

MongoDB protocol parsing settings​

MongoDB is parsed over TCP, plain or decrypted from TLS, into MongoDB communications. The settings are in the server.mongo block of the Whisperer configuration.

MongoDB Parsing​

Configures which TCP sessions should be parsed for MongoDB communications.

SettingDescriptionTypeComment
Parse for MongoDBWhether this Whisperer is capturing MongoDB communications that should be parsed. server.mongo.parseMongoBooleanOn by default
Parse by defaultWhether Spider should parse all ports by default.BooleanOff by default
Ports to parseServer ports of the mongod and mongos processes.
The TCP sessions on those ports will be parsed for MongoDB communications.
You may define lists of ports or ranges: 27017-27019, 28000.
Array of Integers or Integers rangesDefault: 27017, 27018, 27019
Ports to ignoreDefine server ports not to parse.Array of Integers or Integers ranges
Parse monitoringSave the driver and server housekeeping commands: hello, isMaster, ping, replSetHeartbeat, buildInfo. server.mongo.parseMonitoringBooleanOff by default
Parse replicationSave one communication per batch of the replica set members' oplog tailing cursors. server.mongo.parseReplicationBooleanOff by default
Save parsing logDefine if the parsing tracking resource should be saved. server.mongo.saveMongoParsingBooleanFor Spider troubleshooting only
- Keep whole logDefine if we should keep the whole log (may be HUGE) - only for temporary debug. server.mongo.keepWholeParsingLogBooleanFor Spider troubleshooting only
note

MongoDB parsing also needs the mongoParsing feature flag to be on in the Helm chart. MongoDB over TLS is decrypted with the TLS keys captured by Gocipher: it needs Tls Keys to be tracked.

Monitoring noise is off, except the first hello

Drivers send a hello every ~10 seconds per pool, and replica set members every ~2 seconds per pair. With Parse monitoring off these are dropped, except the first hello of each connection, which is always parsed: it carries the driver, the application name and the compressors shown in the connection metadata.

Connections started before the capture

Spider recovers a MongoDB connection joined in the middle: it scans for the next valid message and carries on. But the driver, application name and user come from the handshake. A connection opened before the Whisperer was attached is flagged handshake not captured until the pool recycles that connection (a driver maxIdleTimeMS, or a restart of the client). When Spider has seen the same connection before, for example after an idle split, it restores the metadata from a connection record.

MongoDB content​

Configures what is saved from the commands and their replies, to protect sensitive data or to save space. Content is saved in a separate document per communication, and only shown to users with payload access.

SettingDescriptionTypeComment
Save write documentsSave the documents sent by insert, update, findAndModify and bulkWrite in the content store. When off, they are never kept. server.mongo.saveWriteDocsBooleanOn by default
Save reply documentsSave the documents received in reply to find, aggregate, getMore... server.mongo.saveDataRowsBooleanOn by default
Command text lengthLength after which the command shown on the communication itself is cut. The content store keeps the whole command. server.mongo.commandDocMaxRunesIntegerDefault: 1000
Max content bytesMaximum size of the command, and of the reply documents, saved for one communication. Larger content is cut and its original size kept. server.mongo.maxContentBytesIntegerDefault: 1048576 (1 MiB). Maximum: 16777216 (16 MiB)
note
  • On the communication itself, write payloads (insert documents, update values, findAndModify replacement, bulkWrite operations) are replaced by a visible <redacted> marker. The values of a find filter are kept.
  • The content store holds the commands and documents as they are on the wire. A user with payload access can read what the application wrote, for example a createUser password or a stored secret. Do not give payload access you do not intend to give, or turn off Save write documents and Save reply documents.
  • The authentication exchange (saslStart, saslContinue, speculativeAuthenticate) is always replaced by <redacted>, in the content store too: only the user name is read from it.

MongoDB request templates​

On the parsed MongoDB communications, you may define regular expression patterns to give a name to, or to merge, query shapes.

Without any rule, the template is the automatic query shape, see Templates. A rule is useful to give a shape a business name, or to merge the many shapes of dynamically built filters into one. The name is saved in the communication metadata (req.template) and used to aggregate and filter requests. Rules are matched in order, and the first match wins.

SettingDescriptionTypeComment
NameName of the template.
May include placeholders for captured groups: ${1}.
String
PatternRegular expression (RE2) matched on the text described below.Pattern
Add databasePut the database name in front of the matched text.BooleantoParse.db
Add collectionPut the collection name in front of the matched text.BooleantoParse.collection

The pattern is matched against the query shape (find orders {status: ?}), preceded by the database and/or the collection when selected. With both selected, find orders {status: ?} is matched as shop orders find {status: ?}: the collection appears once. At most 50 rules.

MongoDB request tags​

On the parsed MongoDB communications, you may define values to extract from each command.

SettingDescriptionTypeComment
NameName of the tag. Letters, digits, _ and -, no dot.String
PathA gjson path over the view of the command described below.String
PatternOptional regular expression. Its first group replaces the value; with no group, it only keeps the values that match.Pattern

The path reads a view of the command with:

PathValue
command, db, collectionThe command name, the database and the collection
commentThe $comment of the command, for example a trace parent put there by the application
appName, readPreference, writeConcernConnection and request options
commandDoc.<path>A field of the whole command document, for example commandDoc.filter.tenant or commandDoc.documents.0.sku

MongoDB response tags​

The same rules, run over the reply of the command.

PathValue
ok, code, codeName, errmsgThe outcome of the command
n, nModified, docCountDocuments matched, modified and returned
documents.<index>.<path>A field of a reply document - the first 20 documents, 256 KiB decoded at most

At most 50 request tags, 50 response tags. Rules sharing a name accumulate their values under the same tag, with count and cardinality. Arrays give one value per element; objects are skipped.

Tags are not protected like payloads

Tags are extracted from the command and the reply even when Save write documents and Save reply documents are off, and they are visible to every user who can read the Whisperer, including users without payload access. Your tag rule is the consent to extract the value. Do not tag secrets.

Trace the application's queries

Most drivers let the application set a comment on a command. Put the trace parent there and add a request tag on comment with the pattern traceparent=[0-9a-f]{2}-([0-9a-f]{32})-: the MongoDB commands then join the HTTP and gRPC calls of the same request.

Templates & Tags playground​

The Whisperer configuration panel has a Playground section that lets you immediately test your Templates and Tags configuration on historical data, without waiting for the Whisperer to reload.

It takes a configurable number of the first items (1–50, default 10) from the current filtered list and runs Templates and Tags extraction on them. It shows immediately which Templates and Tags match, what values they extract, and for each tag the match count and cardinality (number of distinct values) - allowing you to validate or adjust your configuration on the spot.

WhispererPlayground.png

Before this feature, testing a new template required:

  • pushing the config,
  • waiting for the Whisperer to reload,
  • capturing some traffic,
  • waiting for the configuration to apply on the backend,
  • then checking if the template or tags matched.

Now you click Run over history data and see the result immediately.

The playground works for HTTP, PostgreSQL, MySQL, Redis, gRPC, Kafka, DNS and MQTT parsing configurations.

When schema bindings are configured, the playground also runs the transcoding step - record bytes are decoded before templates and tags are evaluated. This lets you validate that your bindings decode correctly and that your tags extract the right fields, all from one place.

Show binding details​

The playground result table has a Show binding details toggle.

WhispererPlaygroundBindingDetails.png

When enabled, each result row expands to reveal:

  • Schema resolution - which binding matched the item (by name), or a clear error if no binding matched or the schema was invalid. This is shown per direction (request and response independently).
  • Transcoded body - the decoded request and/or response payload rendered as a JSON tree, exactly as Spider stores and displays it.

Binding errors are always visible, even with the toggle off - you will never silently miss a misconfigured binding.

tip

Use this toggle to debug schema bindings: if the transcoded body is missing or wrong, the resolution line tells you whether the binding matched at all and which one was selected.

note

This feature requires that the Whisperer is configured to save its complete data:

  • HTTP: Raw headers and body
  • PostgreSQL: INSERT and SELECT data rows
  • Redis / gRPC / Kafka: Content saving enabled (so raw bytes are available to decode)