Configuring a Whisperer

Configuring a Whisperer is made through two different tabs grouping:

• Capture config, for the agent configuration
• Parsing config, for the server parsing configuration

Whisperer Type

Whisperers exist in different types.
The type is defined at the creation or just after, and cannot be changed afterward.

Available types:

Whisperer type	Description
UPLOAD	Whisperers dedicated to manual uploaded data (from the UI). They have higher retention duration.
INTERFACE	Whisperers dedicated to real time streaming capture of data.
FILE	Whisperers acting like real time but reading data from local pcap files. Used mostly for debugging.

The Whisperer type is set in the Capture Config tab in Source / Mode field:

Import / Export configuration

You may EXPORT the configuration of a Whisperer to IMPORT it when editing the configuration.

The export is in JSON, and may also be edited manually.

Copy configuration between Whisperers

When editing you may click on COPY button to copy the configuration of another Whisperer unto yours.

It's even faster than export/import!

Capture Config

Source

Configure the source for packets capture.

Setting	Description	Type	Comment
Mode	The type of the Whisperer	List	INTERFACE | FILE | UPLOAD
Network interface	The network interface used for capture	List	any is a virtual interface on Linux that aggregates all others lo is localhost ethx are often the interfaces mounted by Kube
Pcap filter	The pcap filter to filter what network traffic you want to capture	String	Very important to set right. You often limit to tcp, or tcp port 80, or tcp portrange 3000-3010
Method	Method used to capture the Network. By default AFPACKET_THEN_LIBPCAP is used, which tries AFNET with a fallback to LIBPCAP. AFNET is slightly faster, but may not be always available, depending on your kernel.	List	LIBPCAP | AFPACKET | AFPACKET_THEN_LIBPCAP
Server host	Tells what is the host of Spider, not to capture communications going to it. To avoid infinite loops!	Fixed
Capture buffer	Capture buffer of pcap in the Kernel, in kB	Integer	Should often be several thousands.

BPF / pcap filter editor

The pcap filter field has a dedicated editor:

Syntax highlighting — BPF keywords (tcp, udp, host, port, and, or, not), operators, and address literals are colored. A dense filter like tcp and (host 10.0.0.1 or port 5432) and not port 80 is readable at a glance.

Live parser — The filter is parsed as you type. Syntax errors are flagged inline before you submit — no more rejected captures because of a typo.

The parser also detects common mistakes, like selecting many ports with and.

Snippet picker — A snippet picker sits on top of the input. Pick a snippet — "port 443", "portrange 8000-9000", "host" — and it drops into the editor with the port number or hostname preselected, ready to edit or use as-is.

Attachments

Configure attachments made with this Whisperer by a Spider Controller.

Setting	Description	Type	Comment
Time to live	When set, defines the maximum time an attachment of this Whisperer may be active.	Duration in Sec or Min

Circuit breakers

Circuit breakers settings allow to limit Cpu usage and Ram usage of Whisperers on the host they are capturing.
Thus limiting footprint in case of network traffic burst.

Setting	Description	Type	Comment
CPU circuit breaker	Tells if the Whisperer should stop the capture when its CPU usage is too high.	Boolean	Circuit breakers lasts the duration of stats collection.
CPU limit	Set the threshold for the circuit breaker. In percentage.	Float
RAM circuit breaker	Tells if the Whisperer should stop the capture when its RAM usage is too high.	Boolean
RAM limit	Set the threshold for the circuit breaker. In MB.	Integer
Throughput limiter	Tells if the Whisperer should stop the capture when captured traffic is too high.	Boolean
Throughput limit	Set the threshold for the throughput. In MB/min.	Integer

info

For LocalAgents Whisperers, only admins may change the Throughput limit, set to 5MB/min by default.

Packets sending

Settings to control the emission of packets.

Setting	Description	Type	Comment
BO time out	Time out threshold when calling POST /packets API.	Duration in S
Size before sending	Size threshold in kB of the buffer of packets before sending to back end.	Integer
Sending frequency	Duration threshold after which the buffer of packets is sent even if not full.	Duration in S
Parallel sending	How many parallel sending of buffers are authorized	Integer
Sending queue size	If not slot for parallel sending is available, the Whisperer will queue the message. When queue is full, new are discarded.	Integer

When server throughput may not be fast enough for capture load, you should increase Parallel sending setting.

caution

Those values have a direct effect on maximum Whisperer memory size:

• queue sizexsize before sending = maximum queue size
• Default to 500kB * 100 = 50 MB

DNS

Configures the IP reverse resolving feature.

Setting	Description	Type	Comment
Resolve IPs	Whether the Whisperer should reverse resolve IP addresses to host names.	Boolean
Custom DNS Server	Whether the Whisperer uses the system resolution or a custom DNS server.	Boolean
- DNS host	Host of the custom DNS server.	String
- DNS port	Port of the custom DNS server.	Integer
TTL	How long hostname resolution is kept in cache.	Duration in H
Refresh rate	How often a cached hostname resolution is refreshed.	Duration in S
Send update delay	How often an update of the list of hostnames and IPs is sent to backend.	Duration in S
BO time out	Time out threshold when calling POST /hosts API.	Duration in S
Purge delay	How often do the Whisperer clean the cache for TTL passed resolutions.	Duration in H

Tcp Sessions

Configures the tracking and sending of Tcp sessions.

Setting	Description	Type	Comment
Track TCP sessions	Whether the Whisperer should track TCP sessions. If not, it will only send raw packets.	Boolean	Usually, you want this 😊
BO timeout	Time out threshold when calling POST /sessions API.	Duration in S
Sending frequency	Duration threshold after which the buffer of sessions is sent.	Duration in S
Tcp sessions TTL	Time to Live after which an inactive - but not closed - Tcp session is removed from memory.	Duration in M
Send only data packets	When set, a specific - complex - pcap filter is added to capture only data packets & packets required for Tcp sessions tracking. Only packets with data are sent. Reduces Spider CPU usage. Please note that TCP sessions statistics will then be erroneous (IP payload etc.).	Boolean
Sessions sent at once	Max count of Tcp sessions sent by API request.	Integer
Parallel sending	How many parallel sending of buffers are authorized.	Integer
Sending queue size	If not slot for parallel sending is available, the Whisperer will queue the message. When queue is full, new are discarded.	Integer
Max packetLot size	Maximum size of contingent Tcp payload authorized. Usually means a maximum size of request or response. Packets above are sent to Spider but are not taken into account when parsing. Avoids blowing up the servers memory. In Bytes.	Integer

Filtering

Configures filtering on hostnames to capture packets (from and to) or to avoid.

Setting	Description	Type	Comment
Track by default	Whether the Whisperer tracks (all) IP addresses by default.	Boolean	Set to false when you want to only track certain hostnames
Wait for resolving	Whether the Whisperer waits for having resolve the IP address to track its packets.	Boolean
Track unresolved IP	Whether the Whisperer tracks IP addresses it could not resolve.	Boolean
Hosts to track	List of regular expressions (or string) for valid hostnames to track.	Array of Patterns	A whitelist
Host to ignore	List of regular expressions (or string) for hostnames to avoid tracking.	Array of Patterns	A blacklist

An IP resolved to a hostname to track is not re-resolved until the host is not seen for long time (DNS.ttl).
An IP resolved to a hostname not to track is re-resolved every DNS.refreshRate.

When tracking by default, as soon as a host is to ignore, don't track.
When not tracking by default, as soon as a host is to track, do track.

tip

Hostnames filtering is a very powerful feature to limit the volume of data and the CPU usage.

• When you don't need to track packets from or to a hostname, use this.
• When you know the IP of the hosts to ignore, rather use a pcap filter.

Duplicates

When capturing with 2 instances of the same whisperer from both side of the same communication, Spider captures twice packets and TCP sessions.
The following options allow to avoid duplicated communications resources by creating common ids and adding unique checks on server side.

To avoid duplicated packets (when you save them), choose the relevant option on the parsing part of configuration.

Setting	Description	Type	Comment
Avoid duplicated communications	Whether the Whisperer should generate unique Ids to identify duplicates of packets	Boolean

Tls Keys

Whisperers can track TLS keys together with a Gocipher agent.
The Whisperer will look for TLS information in the TCP session it captures, and the Gocipher will capture the session TLS secrets.

Due to technical constraints, TLS keys are captured on ALL PORTS. Pcap filter has no effect on TLS capture.
However, you may limit what port to link or not by:

• Not capturing their packets
• Listing the ports to link in the Parsing Config

Setting	Description	Type	Comment
Track TLS Keys	Whether the Whisperer & Gociphers should track and send TLS secrets to the back office.	Boolean

Status sending

Configures the monitoring.

Setting	Description	Type	Comment
Send status	Whether to send Whisperer statistics to the back end.	Boolean	Crucial for monitoring.
BO timeout	Time out threshold when calling POST /status API.	Duration in S
Sending frequency	Frequency to send statuses.	Duration in S	10 / 20s

Dump packets on client

You may ask the Whisperer to dump .pcap files of packets when capturing.
Used only for internal debugging.

Setting	Description	Type	Comment
Dump packets	Whether the Whisperer should dump packets to files	Boolean
Buffer size	Size in kB of the files to dump (approx)	Integer
Output path	Output path to store the dumps	String

Configuration polling

Setting	Description	Type	Comment
Frequency	Tells how often the Whisperer calls to check a configuration change	Duration in sec.

Common parsing settings

caution

Parsing options are cached for 5 minutes for INTERFACE Whisperers and 1 minute for UPLOAD Whisperers.

Timezone

You may associate a timezone to each Whisperer.

When the Whisperer is selected on the UI, the times of the captured communications are displayed according to this time zone - depending on the settings.

Data storage policy

In Spider configuration, you may define Customisable Data Store Policies.
These defines different available retention periods for Whisperers data.

The Data storage policy of the Whisperer defines how long you want to keep its data.

• The default value is default (definitive lack of imagination here)
• It may be changed at any time in the lifetime of a Whisperer, but existing data will still be associated and managed by the policy active at the time of indexing
• When the setting is set to a Policy that do not exist (any more), the default one is used.

Packets saving

Configures if packets should be saved for analysis.

Spider is built to parse packets in streaming.
You do not need to save packets unless you want to analyze at low packet level.

• When saving packets,

  • Packets are saved to Elasticsearch.
  • It requires much CPU and space.
  • You may choose to avoid duplicated packets when capturing the two side of the same communication.
    • This is costly for ES as it then has to check for if the packet exists when saving. Activate it only when needed.

• When not saving packets,

  • Packets are only kept in memory until they are parsed.
  • This prevents reconstructing payload data for TCP or HTTP once the parsing is finished.
  • You should definitely activate 'Save content' HTTP option, or else, you won't see the data ;)

Do not save packets:

• when you filter out part of the parsed data, and you don't want the data to be reconstructed,
• or when you don't need to keep packets, to save space and ES resources.

You may activate saving packets only temporarily when needed.

Setting	Description	Type	Comment
Save Packets	Whether Spider should save packets from this Whisperer.	Boolean
- Avoid duplicated packets	Whether Spider should check for duplicated packets to avoid saving twice the same.	Boolean

Tcp sessions saving

You may wish not to save TCP sessions in Spider (once the parsing is done). Use this:

• when your parsing quality is good,
• when you don't care of TCP sessions.

Setting	Description	Type	Comment
Save TCP sessions	Whether Spider should save Tcp sessions from this Whisperer.	Boolean

tip

When not saving packets, saving Tcp sessions has little interest.

TLS Keys

You may wish to link or not captured TLS secrets and keys to TCP sessions.
You may define if link should be done by default, or if it should be restricted to certain ports.

Setting	Description	Type	Comment
Link TLS Keys to TCP sessions	Whether this TLS keys captured for this Whisperer should be associated to its TCP sessions.	Boolean
Link by default	Whether Spider should link on all ports by default.	Boolean
Ports to link	Define server ports using TLS encryption. The TCP sessions on those ports will be linked to TLS secrets captured. You may define lists of ports or ranges: 80, 8080, 9200, 3000-3100.	Array of Integers or Integers ranges
Ports to ignore	Define server ports not to link.	Array of Integers or Integers ranges

Host names

When Whisperer resolves IP addresses to capture hostnames, it can determine automatically a shortname to display based on the FQDN provided by the DNS.

For this, specify one or several regular expression(s) that are used to extract parts of the FQDN, which are then concatenated by '.'. Patterns are executed in order. First one that matches is selected.

Setting	Description	Type	Comment
Name patterns	List of patterns to transform a FQDN into something shorter, but meaningful.	Array of Patterns

tip

This is particularly useful to have shorter names on the UI.

• You don't need to see: myservice.mynamespace.svc.cluster.local
• Seeing myservice is enough.

The FQDN is kept along the short name, don't worry:

HTTP protocol parsing settings

HTTP parser provides the useful, and complex parsing options.
They allow filtering, but most importantly extracting business knowledge to show it on Spider.

HTTP Parsing

Configures which TCP sessions should be parsed for Http communications.

Setting	Description	Type	Comment
Parse for HTTP	Whether this Whisperer is capturing HTTP communications that should be parsed.	Boolean
Parse by default	Whether Spider should parse all ports by default.	Boolean
Ports to parse	Define server ports exposing HTTP API. The TCP sessions on those ports will be parsed for HTTP communications. You may define lists of ports or ranges: 80, 8080, 9200, 3000-3100.	Array of Integers or Integers ranges
Ports to ignore	Define server ports not to parse.	Array of Integers or Integers ranges
Save parsing log	Define if the parsing tracking resource should be saved.	Boolean	For Spider troubleshooting only
- Keep whole log	Define if we should keep the whole log (may be HUGE)	Boolean	For Spider troubleshooting only

HTTP filtering

Configures filtering to remove headers, content or whole communication for certain URIs:

• When it is sensitive information (and in production)
• When it is not required to save them (like monitoring requests, healthcheck requests etc.), to save place.

Configures if Spider should save reassembled communication content.

Setting	Description	Type	Comment
Headers to filter	Define headers to remove in parsed object (like Basic auth header 😉). You may filter by header name or value content. It expects patterns to apply on the HTTP header line.	Array of Patterns
URIs to filter	Filter out communications by URIs. For sensitive communications like auth calls.	Array of Patterns
Save content	Define if the content should be saved within the HTTP resource. Important when not saving packets.	Boolean
- URIs to filter request body	Filter out request body contents by URIs patterns.	Array of Patterns
- URIs to filter response body	Filter out response body contents by URIs patterns.	Array of Patterns
Save raw headers	Define if the raw headers should be saved within the HTTP resource.
- URIs to filter request raw headers	Filter out request raw headers by URIs patterns.	Array of Patterns
- URIs to filter response raw headers	Filter out response raw headers by URIs patterns.	Array of Patterns

tip

Regexp are matched case insensitive.

tip

Save content when not saving packets.

tip

Save raw headers when you need to see what was really exchanged (and you don't filter any headers).

tip

Think about filtering ^Authorization: Basic headers to avoid saving passwords in Spider on production.

Request templates

On the parsed HTTP communications, you may define regular expression patterns to name the request.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

• The name of the request:
  Ex: Creation of order, createAnOrder, POST /orders/{id}, depending on your style.
• The parts of the request to parse to identify it. They will be concatenated, as in their raw format.
  • Method
  • URI, mandatory, with querystring
  • Headers
  • Request body
• The regular expression to execute, with /ms flags.

Many regular expressions may be defined. They are matched in order, and the first match is kept.

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

Setting	Description	Type	Comment
Name	Name of the template. May include placeholder for captured groups 💪.	String
Pattern	Regular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.	Pattern
Method	Should method be matched.	Boolean
URI	Should URI be matched.	Boolean
Headers	Should headers be matched.	Boolean
Body	Should body be reassembled, transfer decoded, decompressed and matched.	Boolean

tip

This is one of the most useful feature of parsing.

Obviously this requires CPU, as for the next two ones.
But it is quite reasonable and optimised.
And the value is so much worth it!!

Request tags

On the parsed HTTP communications, you may define regular expression patterns to tag the request.

The tag name and value(s) are saved in the communication metadata and may be used when searching.
You define:

• The name of the tag: Ex: 'ClientName', 'ApiVersion', 'Authenticated'...
• The parts of the request to parse to tag it. They will be concatenated, as in their raw format.
  • Method
  • URI with querystring
  • Headers
  • Request body
• The regular expression to execute, with /msg flags.

If the regular expression includes capture group(s), the captured data will be stored as tag value(s).
Without capture group, the tag value will be 'true' (text).
Many regular expressions may be defined. They are all executed, and all the tags are kept.

Alongside the tag value, are also saved:

• The tag count
  • How often it matched
• The tag cardinality
  • How often it matched with different values

This allows counting (and searching) the number of items in a response, the number of different customers...

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

Setting	Description	Type	Comment
Name	Name of the tag.	String
Pattern	Regular expression to capture the tag.	Pattern
Method	Should method be matched.	Boolean
URI	Should URI be matched.	Boolean
Headers	Should headers be matched.	Boolean
Body	Should body be reassembled, transfer decoded, decompressed and matched.	Boolean

tip

Together with response tags, these are the most value-adding features of parsing!

Response tags

On the parsed HTTP communications, you may define regular expression patterns to tag the response.

It is the same process as for the request, but on the response.

Setting	Description	Type	Comment
Name	Name of the tag.	String
Pattern	Regular expression to capture the tag.	Pattern
Status	Should status line be matched.	Boolean
Headers	Should headers be matched.	Boolean
Body	Should body be reassembled, transfer decoded, decompressed and matched.	Boolean

tip

When defining the same tag in request or response, or many times the same tag in either, all extracted values will be associated to the same tag and counted for tag count and cardinality.

Schema bindings

Schema bindings let Spider decode HTTP request and response bodies into structured JSON for display and extraction.

Each binding maps an HTTP method + URI template + Content-Type to a serialization format (Protobuf, MessagePack, or JSON), so Spider can transcode the raw bytes into a readable format.

Setting	Description	Type	Comment
Name	Display name for the binding.	String
Method	HTTP method to match (e.g. GET, POST). Leave empty to match all methods.	List	GET | POST | PUT | DELETE | PATCH | HEAD | OPTIONS | (any)
URI template	URI pattern in RFC 6570 format (e.g. /api/users/{id}). Path parameters are treated as wildcards.	String
Content-Type	Content-Type header value to match (e.g. application/x-protobuf). Trailing * matches vendor families.	String
Request format	Serialization format of the request body.	List	protobuf | msgpack | json
Request schema	Schema ID for the request body.	String
Request type	Protobuf message type name for the request body.	String	Required when format is protobuf
Response format	Serialization format of the response body.	List	protobuf | msgpack | json
Response schema	Schema ID for the response body.	String
Response type	Protobuf message type name for the response body.	String	Required when format is protobuf

tip

Schema bindings are useful when your HTTP API exchanges binary-encoded bodies (protobuf, msgpack) instead of plain JSON. Once bound, Spider decodes the body and displays it as readable JSON, and request/response tags can extract fields from it.

PostgreSQL protocol parsing settings

PostgreSQL parser provides similar parsing options as HTTP, but simpler.
They allow some filtering for safety, but most also, as for HTTP, they allow extracting business knowledge to show it on Spider.

PostgreSQL Parsing

Configures which TCP sessions should be parsed for PostgreSQL communications.

Setting	Description	Type	Comment
Parse for PostgreSQL	Whether this Whisperer is capturing PostgreSQL communications that should be parsed.	Boolean
Parse by default	Whether Spider should parse all ports by default.	Boolean
Ports to parse	Define server ports exposing PostgreSQL API. The TCP sessions on those ports will be parsed for HTTP communications. You may define lists of ports or ranges: 5432, 5430-5440.	Array of Integers or Integers ranges
Ports to ignore	Define server ports not to parse.	Array of Integers or Integers ranges
Save parsing log	Define if the parsing tracking resource should be saved.	Boolean	For Spider troubleshooting only
- Keep whole log	Define if we should keep the whole log (may be HUGE) - only for temporary debug	Boolean	For Spider troubleshooting only

PostgreSQL filtering

Configures filtering to some specific messages:

• When it is sensitive information (like connections and in production)
• When it is not required to save them (like data rows etc.), to save place.

Setting	Description	Type	Comment
Save connections	Define if we should save Startup requests/responses	Boolean	Saves space when not saved. Connection metadata is included in all coms.
Save password messages	When saving Startup, you may want not to save password messages. Saving those could be unsafe if PostgreSQL is not using crypted password handshake.	Boolean
Save disconnections	Not very useful messages ;)	Boolean
Save SELECT data rows	To save space, you may want NOT TO save data received in response of SQL queries.	Boolean
Save INSERT rows	To save space, you may want NOT TO save data sent to be saved in PostgreSQL	Boolean

Query templates

On the parsed PostgreSQL communications, you may define regular expression patterns to give a name to the query.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

• The name of the request:
  Ex: Creation of order, createAnOrder, POST /orders/{id}, depending on your style.
• The regular expression to execute, with /ms flags over the SQL query.

Many regular expressions may be defined. They are matched in order, and the first match is kept.

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

Setting	Description	Type	Comment
Name	Name of the template. May include placeholder for captured groups 💪.	String
Pattern	Regular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.	Pattern

tip

This is one of the most useful feature of parsing.

Obviously this requires CPU, as for the next two ones.
But it is quite reasonable and optimized.
And the value is so much worth it!!

Request tags

On the parsed PostgreSQL communications, you may define fields to extract from the query or result set.

The tag name and value(s) are saved in the communication metadata and may be used when searching.
You define:

• The name of the tag: Ex: 'ClientName', 'OrderId'...
• The name of the fields to find.
  • Format: [table].[column]->>[jsonpath] or [table].[column].[jsonpath] or [table].[column] (no jsonpath)
  • If the field is already extracted from the JSON object in the field to check, use dot notation.
  • If Spider has to extract the field from the JSON object, use ->>.
• The fields of the request to fetch values from.
  • Checked fields, used in WHERE or JOIN conditions
  • Insert fields, used to insert data in
  • Update fields, updated by the UPDATE command
  • Conflict update fields, updated when a conflict happens

The values associated to theses fields will be extracted from the queries, or find the bound values of prepared statements.
Once extracted, they will be saved as tag value(s) for searches, stats etc.

Note that fields computed based on original ones like sum, format, conversions are not supported.
Only original table fields are supported.

Alongside the tag value, are also saved:

• The tag count
  • How often it matched
• The tag cardinality
  • How often it matched with different values

This allows counting (and searching) the number of items in a response, the number of different customers...

To create many at once, you may prepare the list in a spreadsheet like Google docs with name, regexp and options in columns.
And then copy paste (without headers)

In this example, we want to extract the customer id from any query to Orders, the customer id being stored in a JSONB field data as:

{
  "customer": {
    "id": 123
  }
}

Customer id is used:

• In where conditions, as data->'customer'->>'id' = $1
• In INSERT or UPDATE statements, as part of the json inserted to data jsonb column.

The above setting extracts customer id from all queries to Orders table, as a tag to be able to search PostgreSQL communications addressing this specific customer.

Setting	Description	Type	Comment
Tag name	Name of the tag.	String
Table field	Field to extract the value from.	String
Checked	Should we extract from checked fields values.	Boolean
Insert	Should we extract from inserted fields values.	Boolean
Update	Should we extract from updated fields values.	Boolean
Conflict	Should we extract from updated fields values when conflict.	Boolean

tip

Together with response tags, these are the most value-adding features of parsing!

Response tags

On the parsed PostgreSQL communications, you may define what fields you want to extract from the response.

It is the same process as for the request, but on the response.

Setting	Description	Type	Comment
Tag name	Name of the tag.	String
Table field	Field to extract the value from.	String

tip

When defining the same tag in request or response, or many times the same tag in either, all extracted values will be associated to the same tag and counted for tag count and cardinality.

Redis protocol parsing settings

Redis parser provides parsing options to decode Redis commands and responses.
They allow filtering, and as for HTTP, they allow extracting business knowledge to show it on Spider.

Redis Parsing

Configures which TCP sessions should be parsed for Redis communications.

Setting	Description	Type	Comment
Parse for Redis	Whether this Whisperer is capturing Redis communications that should be parsed.	Boolean
Parse by default	Whether Spider should parse all ports by default.	Boolean
Ports to parse	Define server ports exposing Redis API. The TCP sessions on those ports will be parsed for Redis communications. You may define lists of ports or ranges: 6379, 6380-6389.	Array of Integers or Integers ranges	Default: 6379
Ports to ignore	Define server ports not to parse.	Array of Integers or Integers ranges
Save parsing log	Define if the parsing tracking resource should be saved.	Boolean	For Spider troubleshooting only
- Keep whole log	Define if we should keep the whole log (may be HUGE) — only for temporary debug	Boolean	For Spider troubleshooting only

Redis content settings

Configures whether Redis command values should be saved.

Setting	Description	Type	Comment
Save content for writes	Save values in write commands (SET, HSET, LPUSH, etc.)	Boolean
Save content for reads	Save values in read commands (GET, HGET, LRANGE, etc.)	Boolean
Save content for pub/sub	Save content in Pub/Sub messages (PUBLISH, SUBSCRIBE, etc.)	Boolean

Request templates

On the parsed Redis communications, you may define regular expression patterns to name the request.

The name is saved in the communication metadata and is used to aggregate and filter requests.
You define:

• The name of the request:
  Ex: Get user, Set session, depending on your style.
• The parts of the request to parse to identify it. They will be concatenated.
  • Command (e.g. GET, SET, HSET)
  • Key
  • Args

Many regular expressions may be defined. They are matched in order, and the first match is kept.

Setting	Description	Type	Comment
Name	Name of the template. May include placeholder for captured groups 💪.	String
Pattern	Regular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.	Pattern
Command	Should command be matched.	Boolean
Key	Should key be matched.	Boolean
Args	Should additional arguments be matched.	Boolean

Request tags

On the parsed Redis communications, you may define regular expression patterns to tag the request.

The tag name and value(s) are saved in the communication metadata and may be used when searching.

Setting	Description	Type	Comment
Name	Name of the tag.	String
Pattern	Regular expression to capture the tag.	Pattern
Command	Should command be matched.	Boolean
Key	Should key be matched.	Boolean
Args	Should additional arguments be matched.	Boolean
Value	Should the value being written be matched.	Boolean

Response tags

On the parsed Redis communications, you may define regular expression patterns to tag the response.

Setting	Description	Type	Comment
Name	Name of the tag.	String
Pattern	Regular expression to capture the tag.	Pattern
Reply type	Should the reply type be matched.	Boolean
Error	Should error messages be matched.	Boolean
Value	Should the returned value be matched.	Boolean

Schema bindings

Schema bindings let Spider decode raw Redis values into structured JSON for display and extraction.

Each binding maps a Redis key pattern to a serialization format (Protobuf, MessagePack, or JSON), so Spider can transcoding the raw bytes into a readable format.

Setting	Description	Type	Comment
Name	Display name for the binding.	String
DB	Redis database number to match (e.g. 0, 1). Leave empty to match all databases.	String
Key glob	Glob pattern to match Redis keys (e.g. user:*, session:*:data).	String
Format	Serialization format of the value.	List	protobuf | msgpack | json
Schema ID	Schema identifier used for protobuf (matches a schema registered in Spider).	String
Message type	Protobuf message type name within the schema.	String	Required when format is protobuf

tip

Schema bindings are required for Request and Response tags that use field paths — Spider needs to decode the value before it can extract fields from it.

gRPC protocol parsing settings

gRPC parser provides parsing options to decode gRPC messages using protobuf schemas.
They allow filtering, and they enable extracting business knowledge from structured messages.

gRPC Parsing

Configures which TCP sessions should be parsed for gRPC communications.

Setting	Description	Type	Comment
Parse for gRPC	Whether this Whisperer is capturing gRPC communications that should be parsed.	Boolean
Parse by default	Whether Spider should parse all ports by default.	Boolean
Ports to parse	Define server ports exposing gRPC API. The TCP sessions on those ports will be parsed for gRPC communications. You may define lists of ports or ranges: 50051, 50051-50060.	Array of Integers or Integers ranges	Default: 50051
Ports to ignore	Define server ports not to parse.	Array of Integers or Integers ranges
Save parsing log	Define if the parsing tracking resource should be saved.	Boolean	For Spider troubleshooting only
- Keep whole log	Define if we should keep the whole log (may be HUGE) — only for temporary debug	Boolean	For Spider troubleshooting only

gRPC content settings

Configures whether gRPC message bodies should be saved.

Setting	Description	Type	Comment
Save content for requests	Save request message bodies.	Boolean
Save content for responses	Save response message bodies.	Boolean

Request templates

On the parsed gRPC communications, you may define regular expression patterns to name the request.

Setting	Description	Type	Comment
Name	Name of the template. May include placeholder for captured groups 💪.	String
Pattern	Regular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.	Pattern
Full method	Should the full method path be matched (e.g. /package.Service/Method).	Boolean	Default: true
Metadata	Should gRPC metadata (headers) be matched.	Boolean
Message	Should the request message body (decoded) be matched.	Boolean	Requires a schema binding for the method

Request tags

On the parsed gRPC communications, you may extract fields from the request using gjson paths.

Tags use gjson paths to navigate a view object assembled for each communication.

The path navigates a view object with these top-level fields:

Field	Content
service	Service name (e.g. UserService)
method	Method name (e.g. GetUser)
fullMethod	Full gRPC path (e.g. /pkg.UserService/GetUser)
metadata.<key>	gRPC request metadata (headers), e.g. metadata.x-tenant
message.<field>	Decoded request body (requires a schema binding)

Setting	Description	Type	Comment
Name	Name of the tag.	String
Path	gjson path into the request view (e.g. service, metadata.x-tenant, message.user.id, message.items.#.id).	String	message.* requires a schema binding for the method

Response tags

On the parsed gRPC communications, you may extract fields from the response using gjson paths.

The path navigates a view object with these top-level fields:

Field	Content
grpcStatus	gRPC status code integer (0 = OK)
grpcStatusName	gRPC status name (e.g. OK, NOT_FOUND, INTERNAL)
grpcMessage	gRPC error message string (when status ≠ OK)
metadata.<key>	Trailing gRPC metadata (headers)
message.<field>	Decoded response body (requires a schema binding)

Setting	Description	Type	Comment
Name	Name of the tag.	String
Path	gjson path into the response view (e.g. grpcStatusName, grpcMessage, message.items.#, message.status.code).	String	message.* requires a schema binding for the method

Schema bindings

Schema bindings let Spider decode gRPC message bytes into structured JSON for display and extraction.

Each binding maps a service/method pattern to request and response schemas (typically protobuf).

Setting	Description	Type	Comment
Name	Display name for the binding.	String
Service pattern	Regex pattern to match the gRPC service name (e.g. .*\.UserService). Leave empty to match all.	String
Method pattern	Regex pattern to match the gRPC method name (e.g. Get.*). Leave empty to match all.	String
Request format	Serialization format of the request message.	List	protobuf | msgpack | json
Request schema	Schema ID for the request.	String
Request type	Protobuf message type for the request.	String
Response format	Serialization format of the response message.	List	protobuf | msgpack | json
Response schema	Schema ID for the response.	String
Response type	Protobuf message type for the response.	String

tip

Schema bindings are required for Request and Response tags — Spider needs to decode the protobuf message before it can extract fields from it.

Kafka protocol parsing settings

Kafka parser provides parsing options to decode Kafka Produce and Fetch operations.
They allow filtering, and they enable extracting business knowledge from record payloads.

Kafka Parsing

Configures which TCP sessions should be parsed for Kafka communications.

Setting	Description	Type	Comment
Parse for Kafka	Whether this Whisperer is capturing Kafka communications that should be parsed.	Boolean
Parse by default	Whether Spider should parse all ports by default.	Boolean
Ports to parse	Define server ports exposing Kafka brokers. The TCP sessions on those ports will be parsed for Kafka communications. You may define lists of ports or ranges: 9092, 9093.	Array of Integers or Integers ranges	Default: 9092
Ports to ignore	Define server ports not to parse.	Array of Integers or Integers ranges
Save parsing log	Define if the parsing tracking resource should be saved.	Boolean	For Spider troubleshooting only
- Keep whole log	Define if we should keep the whole log (may be HUGE) — only for temporary debug	Boolean	For Spider troubleshooting only

Kafka content settings

Configures whether Kafka record key/value bytes should be saved.

Setting	Description	Type	Comment
Save content for produces	Save record content (key/value) from Produce requests.	Boolean
Save content for fetches	Save record content (key/value) from Fetch responses.	Boolean

Request templates

On the parsed Kafka communications, you may define regular expression patterns to name the request.

Setting	Description	Type	Comment
Name	Name of the template. May include placeholder for captured groups 💪.	String
Pattern	Regular expression. When using placeholders, use ^ and $ to enclose the pattern and avoid surprises.	Pattern
Command	Should the Kafka command (e.g. Produce, Fetch) be matched.	Boolean	Default: true
Topic	Should the topic name be matched.	Boolean	Default: true
Client ID	Should the client ID be matched.	Boolean
Message	Should the decoded record value be matched.	Boolean	Requires a schema binding for the topic

Request tags

On the parsed Kafka communications, you may extract fields from the Produce request using gjson paths.

Tags use gjson paths to navigate a view object assembled for each communication.

The path navigates a view object with these top-level fields:

Field	Content
command	Kafka command (e.g. Produce)
clientId	Kafka client ID
topic.0	First topic name (e.g. orders)
records.N.key	Decoded key of record N (requires a schema binding)
records.N.value.<field>	Field from decoded value of record N (requires a schema binding)

Use gjson array queries to span all records: records.#.value.customerId extracts customerId from every record.

Setting	Description	Type	Comment
Name	Name of the tag.	String
Path	gjson path into the request view (e.g. command, topic.0, records.0.value.order.id, records.#.value.customerId).	String	records.*.value.* requires a schema binding for the topic

Response tags

On the parsed Kafka communications, you may extract fields from the Fetch response using gjson paths.

The path navigates a view object with these top-level fields:

Field	Content
statusCode	HTTP-equivalent status code (200 = OK, 400 = error)
statusText	Status description
errorCode	Kafka error code (0 = success)
topic.0	First topic name
records.N.value.<field>	Field from decoded value of record N (requires a schema binding)

Setting	Description	Type	Comment
Name	Name of the tag.	String
Path	gjson path into the response view (e.g. statusCode, errorCode, records.#, records.0.value.status).	String	records.*.value.* requires a schema binding for the topic

Schema bindings

Schema bindings let Spider decode Kafka record bytes into structured JSON for display and extraction.

Each binding maps a topic pattern to a serialization format. Spider supports Protobuf, MessagePack, JSON, and Avro — including Confluent Schema Registry wire-format detection.

Setting	Description	Type	Comment
Name	Display name for the binding.	String
Topic glob	Glob pattern to match Kafka topic names (e.g. orders-*, user.events). Leave empty for all.	String
Side	Which part of the record to decode.	List	key | value | (both)
Registry ID	Confluent Schema Registry ID embedded in the record wire format. Overrides format/schema ID.	Integer	Optional — only needed for Confluent wire format
Format	Serialization format of the record.	List	protobuf | msgpack | json | avro
Schema ID	Schema identifier registered in Spider.	String
Message type	Protobuf or Avro message type name within the schema.	String	Required when format is protobuf

tip

Schema bindings unlock templates and tags on record content — Spider must decode the bytes before it can match patterns or extract field values.

note

When records carry the Confluent wire-format prefix (magic byte 0x00 + 4-byte schema ID), Spider auto-detects the schema ID embedded in the record. You may define a binding with a matching Registry ID to explicitly map that ID to a schema registered in Spider.

Templates & Tags playground

The Whisperer configuration panel has a Playground section that lets you immediately test your Templates and Tags configuration on historical data, without waiting for the Whisperer to reload.

It takes a configurable number of the first items (1–50, default 10) from the current filtered list and runs Templates and Tags extraction on them. It shows immediately which Templates and Tags match, what values they extract, and for each tag the match count and cardinality (number of distinct values) — allowing you to validate or adjust your configuration on the spot.

Before this feature, testing a new template required:

• pushing the config,
• waiting for the Whisperer to reload,
• capturing some traffic,
• waiting for the configuration to apply on the backend,
• then checking if the template or tags matched.

Now you click Run over history data and see the result immediately.

The playground works for HTTP, PostgreSQL, Redis, gRPC, and Kafka parsing configurations.

When schema bindings are configured, the playground also runs the transcoding step — record bytes are decoded before templates and tags are evaluated. This lets you validate that your bindings decode correctly and that your tags extract the right fields, all from one place.

Show binding details

The playground result table has a Show binding details toggle.

When enabled, each result row expands to reveal:

• Schema resolution — which binding matched the item (by name), or a clear error if no binding matched or the schema was invalid. This is shown per direction (request and response independently).
• Transcoded body — the decoded request and/or response payload rendered as a JSON tree, exactly as Spider stores and displays it.

Binding errors are always visible, even with the toggle off — you will never silently miss a misconfigured binding.

tip

Use this toggle to debug schema bindings: if the transcoded body is missing or wrong, the resolution line tells you whether the binding matched at all and which one was selected.

note

This feature requires that the Whisperer is configured to save its complete data:

• HTTP: Raw headers and body
• PostgreSQL: INSERT and SELECT data rows
• Redis / gRPC / Kafka: Content saving enabled (so raw bytes are available to decode)