Kafka communications

Concept

A Kafka communication is a single Kafka protocol request/response exchange decoded from the TCP frames of a broker session, keyed by its Kafka ApiKey (rendered as command). Each communication pairs one client request with one broker response — similar to HTTP and Redis in shape, with no streaming transaction or per-message split.

Each Kafka communication is associated with:

• Source and destination IP address
• Source and destination TCP port
• The Kafka protocol command (e.g. Produce, Fetch, Metadata, ApiVersions, SaslHandshake, SaslAuthenticate), its numeric apiKey and apiVersion
• The clientId and correlationId set by the Kafka client
• The topic[], partition[] and offset[] touched — multi-valued when the request spans several topics or partitions
• The record key and value payloads carried in the request or the response, optionally transcoded to JSON via a schema binding
• The Kafka errorCode returned by the broker, mapped onto an HTTP-like status

Kafka communications are extracted while parsing the TCP sessions in streaming, on the Spider backend. Kafka-over-TLS is decrypted with the master secrets captured by Gocipher, in the same way as gRPC and Redis.

note

A Kafka communication is a paired request + response (like HTTP and Redis). A request spanning multiple topics or partitions is still a single com — topic, partition and offset are multi-valued arrays. There is no streaming transaction or per-message split.

Kafka communications appear in the dedicated Kafka view — grid, timeline, dashboard, sequence diagram, waterfall and statistics.

Global Tab

The Global tab summarises the exchange: timing, identification, network, request, response and Spider info.

Timing

The timing diagram shows the client and broker hosts, the request (green) and response (orange) shapes scaled by the packet timestamps with per-packet markers, the identity line (command - size), and the response summary (status + size). The Kafka broker port (9092 by convention) is labelled with the command.

• The client and broker hosts are shown, by order of preference:
  • The short name of the Host, as extracted during integration in the backend, or as renamed by a user
  • The FQDN of the Host
  • The IP address, if no Host found

General

The General section shows identification information:

Field	Description
Command	The Kafka protocol command (e.g. Produce, Fetch, Metadata, ApiVersions)
ApiKey	The numeric Kafka ApiKey identifying the command (e.g. 0 for Produce, 1 for Fetch)
ApiVersion	The version of the Kafka request/response schema used
Client id	The clientId set by the Kafka client in the request header
Correlation id	The correlationId used by the client to match this response to its request
Kind	Request-class grouping (e.g. Produce, Fetch, metadata/admin, SASL)
Status	A computed status, made to match HTTP semantics

Spider maps the Kafka errorCode onto HTTP-like status codes so Kafka communications are comparable with HTTP, PostgreSQL, Redis and gRPC ones (e.g. errorCode 0 → 200 OK, Kafka error codes → 4xx / 5xx). The original Kafka errorCode and errorMessage are kept on the response.

Request

The Request section shows summary information of the request part:

Field	Description
Start	Date the request started, user's local time
End	Date the request ended
Topics	Topics touched by the request
Partitions	Partition numbers addressed
Records	Number of records sent
Size	Total bytes sent to the broker

Response

The Response section shows summary information of the response part:

Field	Description
Start	Date the response started
End	Date the response ended
Duration	The difference between request start and response end
Error code	The Kafka error code returned by the broker (0 = success)
Error message	The Kafka error description when the error code is non-zero
Records	Number of records received (for Fetch responses)
Size	Total bytes received from the broker

Network

The Network section shows the connection endpoints (From / To) and the Whisperer that captured the exchange.

The request Template (a stable name for recurrent commands, defaulting to command + topics) and any extracted Tags are configured at Whisperer level and are searchable.

Tags

Tags are a very powerful feature of Spider's parsing process.

They are configured at Whisperer level, and allow custom extraction of data from the records exchanged. They allow searching and aggregating by the extracted values.

Unlike HTTP / Redis tags — which match a regular expression over the exchanged bytes — Kafka tags are field paths. Each tag names a path into the decoded record view:

• command, clientId, topic, partition — scalar fields on the communication
• value.<field.path> or key.<field.path> — paths into the decoded record JSON (requires a schema binding to transcode the binary record to JSON)

This makes record-field extraction exact (no regex over binary), at the cost of needing a schema binding to reach value.* or key.* paths.

tip

Tags may be enriched / decoded by a plugin. For instance:

• To translate internal identifiers from a record field into their business names
• With a link to open them in the application under observation

Spider info

This section shows Spider's own information:

Field	Description
Whisperer	Name and link to the Whisperer
Instance	Instance of the Whisperer that captured the communication
Tcp session	Tcp session id and link
Request parsing	Parsing status of the request
Response parsing	Parsing status of the response

Request tab

The Request tab shows the records sent by the client (for Produce commands) or the fetch parameters (for Fetch commands).

Each record is shown in an indexed record table — one row per record. The table shows the topic, partition, key and value for each record. A toggle switches between the raw bytes and the transcoded JSON form when a schema binding matches.

Response tab

The Response tab shows the records returned by the broker (for Fetch responses) and the per-partition acknowledgment or error (for Produce responses).

The same indexed record table is used — one row per record — with the same raw ↔ transcoded toggle.

Transcoded view

When a content schema is bound to a Kafka topic or command, Spider decodes the matching record keys and values from their binary form (Avro, or Confluent-framed Avro / Protobuf resolved by the embedded Schema Registry id) to searchable JSON. The record table then defaults to the Transcoded view, and falls back to Raw with a banner when the binding misses or decoding fails. The request and response sides are bound independently. Tags and templates can extract from the JSON form of transcoded records.

Packets tab

The Packets tab shows the network packets composing the Kafka communication. You may open them by clicking on the rows.

Field	Description
Dir	Direction of the packet
Time	Time the packet was captured
Payload	Size of the TCP payload
Info	Flags and beginning of TCP payload, in ASCII

::: note Kafka is almost always carried over TLS, so the packets are encrypted. The only ways to see them decrypted is to access the TCP session, and:

• to read the packets from the beginning, in the content tab,
• or to export the session in .pcapng format (for Wireshark or alike) :::

Source

The Source tab shows the Kafka communication in JSON format. Below is a sample Produce Kafka communication as it is available through the API:

{
    "@id": "xK3mPqRsT9BuYvWoNa8czQ.kafka-broker-0-7f8b9d-nx4vp.382941.10.42.0.91-54210-10.42.0.12-9092.712345678.KafkaCom.1",
    "@type": "KafkaCom",
    "dateModified": "2026-06-26T10:23:41.157+00:00",
    "tcpSession": "xK3mPqRsT9BuYvWoNa8czQ.kafka-broker-0-7f8b9d-nx4vp.382941.10.42.0.91-54210-10.42.0.12-9092.712345678",
    "instanceId": "kafka-broker-0-7f8b9d-nx4vp",
    "whisperer": "xK3mPqRsT9BuYvWoNa8czQ",
    "kind": "Produce",
    "req": {
        "status": "COMPLETE",
        "command": "Produce",
        "apiKey": 0,
        "apiVersion": 9,
        "clientId": "order-service-producer",
        "correlationId": 42,
        "flexible": true,
        "topic": ["orders"],
        "partition": [0, 1],
        "offset": [],
        "recordCount": 3,
        "batchCount": 2,
        "compression": "snappy",
        "messageFormat": 2,
        "keySchemaId": [],
        "valueSchemaId": [1001],
        "template": "Produce orders",
        "size": 284,
        "start": 1782553421.089345,
        "startDate": "2026-06-26T10:23:41.089Z",
        "end": 1782553421.091872,
        "endDate": "2026-06-26T10:23:41.091Z"
    },
    "res": {
        "status": "COMPLETE",
        "command": "Produce",
        "apiKey": 0,
        "apiVersion": 9,
        "clientId": "order-service-producer",
        "correlationId": 42,
        "flexible": true,
        "topic": ["orders"],
        "partition": [0, 1],
        "offset": [100, 42],
        "recordCount": 0,
        "errorCode": 0,
        "errorMessage": "",
        "size": 47,
        "start": 1782553421.093614,
        "startDate": "2026-06-26T10:23:41.093Z",
        "end": 1782553421.094021,
        "endDate": "2026-06-26T10:23:41.094Z"
    },
    "stats": {
        "withTls": true,
        "statusCode": 200,
        "statusText": "OK",
        "duration": 0.004676055908203125,
        "dst": {
            "ip": "10.42.0.12",
            "name": "kafka-broker-0.kafka.svc.cluster.local.",
            "port": 9092,
            "socket": "10.42.0.12:9092"
        },
        "src": {
            "ip": "10.42.0.91",
            "name": "order-service-7f9d4b-kx2rm.shop.pod.cluster.local.",
            "port": 54210,
            "socket": "10.42.0.91:54210"
        },
        "timespan": {
            "gte": "2026-06-26T10:23:41.089Z",
            "lte": "2026-06-26T10:23:41.094Z"
        },
        "schemaIds": [1001],
        "confluentFramed": true,
        "tags": {
            "values": {},
            "count": {},
            "cardinality": {}
        }
    },
    "kafkaParsing": "xK3mPqRsT9BuYvWoNa8czQ.kafka-broker-0-7f8b9d-nx4vp.382941.10.42.0.91-54210-10.42.0.12-9092.712345678.KafkaParsingLog"
}

Most fields are self-explanatory. Most fields are indexed and searchable.

Some extra explanation:

Field	Description
kind	Request-class grouping: Produce, Fetch, Metadata, ApiVersions, SaslHandshake, SaslAuthenticate or other admin commands
req|res.command	The Kafka protocol command name (e.g. Produce, Fetch, Metadata)
req|res.apiKey	The numeric Kafka ApiKey identifying the command
req|res.apiVersion	The version of the Kafka request/response schema
req|res.clientId	The clientId set by the Kafka client
req|res.correlationId	Client-assigned integer used to match a response to its request
req|res.topic	Topics touched by this exchange (multi-valued array)
req|res.partition	Partition numbers addressed (multi-valued array, same order as topic)
req|res.offset	For Fetch requests: the start partition offset to read from; for Produce responses: the base offsets assigned by the broker. Multi-valued array, same order as partition
req.recordCount	Number of records sent in the request
req.compression	Compression codec applied to the RecordBatch (none, gzip, snappy, lz4, zstd)
res.errorCode	Kafka error code returned by the broker (0 = success)
res.errorMessage	Human-readable Kafka error description when errorCode is non-zero
req.template	Computed command name (see Templates) — defaults to command + topics
req|res.keySchemaId	Confluent Schema-Registry ids detected in the record key Confluent framing (multi-valued)
req|res.valueSchemaId	Confluent Schema-Registry ids detected in the record value Confluent framing (multi-valued)
stats.withTls	true when the exchange was carried over TLS
stats.statusCode	The Kafka errorCode mapped onto HTTP semantics (0 → 200, error → 4xx / 5xx)
stats.timespan	Indexed as range, to allow range searching and aggregations
stats.schemaIds	Union of all keySchemaId and valueSchemaId values across request and response
stats.confluentFramed	true when at least one record carries a Confluent Schema-Registry framing header
stats.tags.*	Extracted tag values, count and cardinality
kafkaParsing	Id of the parsing log document for this communication

note

The full record key and value bytes are stored separately (in a dedicated per-communication KafkaComContent store) and fetched on demand from the Request / Response tabs, so the communication document itself stays small and fast to search.