Stream messages

Concept

A stream message is one message pushed on a long-lived HTTP stream — today, one Server-Sent Events event. It is not a request/response exchange: a single HTTP request opens the stream and stays open, and the server then pushes a series of messages back over it, sometimes for hours.

Spider captures that as one communication per message, rather than as a single HTTP communication whose response body keeps growing. A feed that emits ten thousand events becomes ten thousand searchable entries, each with its own timing, size, payload, template and tags.

Two kinds of document describe one stream:

• The handshake — an ordinary HTTP communication for the request that opened the stream. It carries the method, the URI, the request and response headers, the status, and the workload identity of both ends.
• The messages — one Stream message communication per event pushed on that stream, each linked back to the handshake by its transactionId.

Each stream message is associated with:

• Source and destination IP address, TCP port and workload identity
• The kind of stream: SSE today, WEBSOCKET when that engine lands
• Its direction (req for client→server, res for server→client — every SSE message is res)
• Its index within that direction, and its txSeq — the global position across both directions
• Whether it is the terminal message of the stream
• The SSE event fields: event type, last-event-id and retry
• The message payload, inlined or stored in the content sidecar like an HTTP body
• Its own template and tags

Stream messages are extracted while parsing the TCP sessions in streaming, on the Spider backend, by the same web parser that handles HTTP/1.x, HTTP/2 and gRPC. SSE over TLS is decrypted with the master secrets captured by Gocipher, exactly as for the other web protocols.

note

Identity fields describe the connection, not one direction. method, uri, hash and template live on the req side of every message regardless of its direction, because they identify the stream. The payload — status, size, packets, opcode, SSE fields and body — lives on the side named by the message's direction. For SSE that is always res, so a server-pushed byte is counted in res.size, not req.size.

info

method is synthesized: an SSE event has no HTTP method, so Spider labels it SRV_PUSH (server→client) or CLT_PUSH (client→server). uri is the handshake's URI with the event type appended as a fragment (/orders/feed#tick) — a fragment is never sent on the wire, so it cannot be mistaken for a routable path segment — and hash carries that event type on its own so it stays separately filterable.

Where stream messages appear

Stream messages are members of the same indices the HTTP view reads, so they appear alongside HTTP communications rather than in a silo of their own.

View	Behaviour
All (multi-communications)	Stream messages are listed with every other protocol, with their own Stream message type.
HTTP	Hidden by default, shown when Show stream messages in HTTP search is on.

The HTTP view hides them by default deliberately: a single long-lived stream can produce thousands of messages, and they would otherwise flood a grid you opened to look at requests. The toggle lives in Settings → Display:

Setting	Description
Show stream messages in HTTP search	Show individual stream messages alongside regular HTTP communications in the HTTP view. Off by default.

The setting is persisted per user, and it drives the navigation timeline as well as the grid, so the counts above the grid always match what the grid lists.

Reaching the messages of one stream

From a handshake HTTP communication, the Stream Messages tab lists every message pushed on that stream, in order, with its event type, timing and size. It is the natural entry point: you find the request that opened the feed in the HTTP view, then open its messages.

Selecting the stream itself opens the stream transaction view, which summarises the whole connection rather than one message.

Stream message detail

A single stream message opens in a four-tab detail view.

Overview Tab

The Overview tab summarises the message: timing, general identification, the message fields, the network endpoints and the tags.

Section	Content
Timing	When the message was pushed, and its duration.
General	@id, Whisperer, TCP session, Parsing log, Kind, Transaction id, Direction, Index, Sequence (txSeq), Terminal.
Message	Method, URI, Template, Opcode, SSE event type, SSE last-event-id, SSE retry, Status, Size.
Network	From / To, with identification and workload identity.
Tags	The message's tags — its own, unioned with the ones inherited from the opening request.

Payload Tab

The Payload tab renders the message body. It is stored and rendered like an HTTP body: inlined below the size threshold, and otherwise held in the stream-message content store and fetched on demand.

Packets Tab

The Packets tab lists the network packets the message was reconstructed from, as on every other protocol.

Source Tab

The Source tab shows the raw stored document, for troubleshooting.

Stream transaction detail

The stream transaction view summarises one connection and all the messages on it.

Global Tab

Section	Content
Stream	Whisperer, TCP session, Transaction id, Kind.
Timing	Start, Stop, Duration.
Messages	Total messages, Request messages, Response messages, Request size, Response size.
Status	Status, Last event type, Last event id.

Flow Tab

The Flow tab plots the messages of the stream on a compact timeline, in order, with the handshake drawn at its head. It scrolls as you reach the end, so a stream with thousands of messages stays usable, and payloads are truncated in the timeline rows — open a message to see it in full.

note

The handshake is drawn but not counted: it appears on the timeline so you can see when the stream opened, but it is not one of the stream's messages and is excluded from the message counts.

Source Tab

The raw stored document for the transaction.

Templates and tags

Stream messages carry their own template and tags, configured per Whisperer under SSE stream message parsing settings.

Two behaviours are worth knowing when reading the grid:

• A message is never left without a template. When no message rule matches, it inherits the template of the HTTP request that opened the stream, so the column is never a bare dash.
• Tags are inherited then layered. A message starts from the opening request's tags — including its correlation tag — and adds its own. That is what places a stream message in the same end-to-end trace as the HTTP, PostgreSQL, Redis, gRPC and Kafka activity around it, with no rule to write.

Limitations

• WebSocket is not parsed yet. The kind field and the req/res direction split exist for it, but only SSE has an engine today. A WebSocket connection is still recorded as its handshake HTTP communication.
• A stream opened before the Whisperer attached is not split. Spider recognises a stream from the Content-Type: text/event-stream header of the response that opens it. On a connection captured mid-flight those headers were never seen, so the traffic is not recognised as a stream — the same limitation that already applies to TLS and to HTTP/2. In practice, deploy the Whisperer before the workload it observes.
• Turning Save SSE communications on or off affects new connections only. A stream already open keeps the behaviour it started with.