Http Parsing
The following services are managing HTTP communications parsing.
Below is their default configuration.
webWrite
webWrite is the Helm values key for the web parser (web-parser, renamed from web-write /
web-streams-write). Since the HTTP/2 work, it parses HTTP/1.x, HTTP/2 and gRPC together - there is
no separate gRPC parser service. It also parses HTTP/3: QUIC UDP flows reach it on the same queue, and its
quicContinuation block (enabled true, ttl PT4M, barrierWait PT30S, packetTtl PT45S) governs how a QUIC
connection is followed when it moves to a new UDP flow. packetTtl is Pack-Write's packet cache TTL: keep the two in step. Its configuration reflects that: one parsingJob reading a single
merged queue, and a protocols block carrying one store/cache set per protocol it may output.
"@id": web-parser
"@type": ServerConfiguration
version: '0.1'
protocols:
http:
comStore:
node: http://elasticsearch:9200
indexPurge: spider-search-httpcom-upload
indexGet: spider-search-httpcom
getTimeout: PT10S
purgeTimeout: PT10S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
contentStore:
node: http://elasticsearch:9200
indexGet: spider-search-httpcomcontent
indexPurge: spider-active-httpcomcontent-upload-default
getTimeout: PT15S
purgeTimeout: PT15S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT15S
circuitThreshold: 0.8
comsCache:
server: redis
port: 6379
db: 2
timeOut: PT10S
ttl: PT25S
circuitDuration: PT30S
circuitThreshold: 1
compressed: false
comsContentCache:
server: redis
port: 6379
db: 12
timeOut: PT10S
ttl: PT25S
circuitDuration: PT30S
circuitThreshold: 1
compressed: false
grpc: # present only when the grpcParsing feature flip is on
comStore:
node: http://elasticsearch:9200
indexPurge: spider-search-grpccom-upload
indexGet: spider-search-grpccom
getTimeout: PT10S
purgeTimeout: PT10S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
contentStore:
node: http://elasticsearch:9200
indexPurge: spider-search-grpccomcontent-upload
indexGet: spider-search-grpccomcontent
getTimeout: PT10S
purgeTimeout: PT10S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
comsCache: # a distinct Redis instance (redis-grpc), not the HTTP cache's db
server: redis-grpc
port: 6379
db: 2
timeOut: PT10S
ttl: PT25S
circuitDuration: PT30S
circuitThreshold: 1
compressed: false
comsContentCache:
server: redis-grpc
port: 6379
db: 12
timeOut: PT10S
ttl: PT25S
circuitDuration: PT30S
circuitThreshold: 1
compressed: false
parsingLog:
store:
node: http://elasticsearch:9200
indexPurge: spider-search-httppers-upload
indexGet: spider-search-httppers
getTimeout: PT10S
purgeTimeout: PT10S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
cache:
server: redis
port: 6379
db: 3
timeOut: PT10S
ttl: PT2M
circuitDuration: PT30S
circuitThreshold: 1
compressed: false
parsingJob:
pollingDelay: PT1S
jobRunners: 1
uri: http://tcp-update/v1/parsing-jobs/web
label: "/tcp-update/parsing-jobs/web"
timeout: PT10S
parsingDelay: PT10S
queues: ["web", "http", "grpc"] # "grpc" dropped from the list when grpcParsing is off
purgeJob:
sizeLimitKB: 100
parsingStatusSynchro:
delay: PT5S
queue: statsSynchro
maxBucketSize: 100
tcpSessions:
uri: http://tcp-update/v1/tcp-sessions
label: "/tcp-update/tcp-sessions"
timeout: PT10S
sizeLimitKB: 100
compressed: true
packetsByIndex:
uri: http://pack-read/v1/packets/of/tcpsession
label: "/pack-read/packets/of/tcpsession"
timeout: PT10S
whisps:
get:
uri: http://whisp/v1/whisperers/{id}
label: "/whisp/whisperers"
timeout: PT10S
config:
uri: http://whisp/v1/whisperers/{id}/config?view=full
label: "/whisp/whisperers/config"
timeout: PT10S
There is one parsingLog (parsing state / tracking resource), shared across protocols - a TCP session
being parsed for web traffic has a single parsing log regardless of whether it turns out to carry HTTP
or gRPC.
Two related settings live in the shared common configuration, not in webWrite itself:
parserProtocols- the list of protocols with a dedicated parsing queue (drives queue naming, e.g. theweb/http/grpcqueues above).outputProtocols- the list of protocols the web parser may emit a communication for (http,grpc,websocket,sse). Bothsseandwebsockethave an engine and appear in data - see SSE and WebSocket message parsing settings.
webRead
"@id": web-streams-read
"@type": ServerConfiguration
version: '0.1'
httpComStore:
node: http://elasticsearch:9200
indexGet: spider-search-httpcom
getTimeout: PT15S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT15S
circuitThreshold: 0.8
httpComContentStore:
node: http://elasticsearch:9200
indexGet: spider-search-httpcomcontent
getTimeout: PT15S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT15S
circuitThreshold: 0.8
httpPersStore:
node: http://elasticsearch:9200
indexGet: spider-search-httppers
indexParsingStatus: spider-search-parsing-status-httppers
getTimeout: PT10S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
packets:
uri: http://pack-read/v1/packets/payload/tcp/?force=true
label: "/packets/payload/tcp"
timeout: PT10S
searchRequest:
sizeLimitKB: 100
whisps:
uri: http://whisp/v1/whisperers/{id}/config?view=server
label: "/whisp/whisperers/config"
timeout: PT10S
webHttpComPoller
"@id": web-httpcom-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpCom
itemStore:
node: http://elasticsearch:9200
useDataStoragePolicies: true
useDistinctPolicyForUpload: true
streamedDataStoragePolicies:
- name: default
indexSave: spider-active-httpcom-streaming-default
uploadedDataStoragePolicies:
- name: default
indexSave: spider-active-httpcom-upload-default
saveTimeout: PT5S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
itemCache:
server: redis
port: 6379
db: 2
timeOut: PT2S
ttl: PT45S
circuitDuration: PT15S
circuitThreshold: 1
compressed: false
polling:
queue: httpComToSynchronize
queueType: SORTED_SET
scoreAttribute: _update
size: 175
jobRunners: 1
delay: PT2S
automaticESIds: false
updateCache:
keepInCache: false
removeFromCacheCondition: {}
noTTLCondition: {}
whisps:
config:
uri: http://whisp/v1/whisperers/{id}/config?view=full
label: "/whisp/whisperers/config"
timeout: PT10S
webHttpComContentPoller
"@id": web-httpcom-content-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpCom
itemStore:
node: http://elasticsearch:9200
useDataStoragePolicies: true
useDistinctPolicyForUpload: true
streamedDataStoragePolicies:
- name: default
indexSave: spider-active-httpcomcontent-streaming-default
uploadedDataStoragePolicies:
- name: default
indexSave: spider-active-httpcomcontent-upload-default
saveTimeout: PT5S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
itemCache:
server: redis
port: 6379
db: 12
timeOut: PT2S
ttl: PT45S
circuitDuration: PT15S
circuitThreshold: 1
compressed: false
polling:
queue: httpComContentToSynchronize
queueType: SORTED_SET
scoreAttribute: _update
size: 175
jobRunners: 1
delay: PT2S
automaticESIds: false
updateCache:
keepInCache: false
removeFromCacheCondition: {}
noTTLCondition: {}
whisps:
config:
uri: http://whisp/v1/whisperers/{id}/config?view=full
label: "/whisp/whisperers/config"
timeout: PT10S
webHttpPersPoller
"@id": web-httppers-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpPers
itemStore:
node: http://elasticsearch:9200
useDataStoragePolicies: true
useDistinctPolicyForUpload: true
streamedDataStoragePolicies:
- name: default
indexSave: spider-active-httppers-streaming-default
uploadedDataStoragePolicies:
- name: default
indexSave: spider-active-httppers-upload-default
saveTimeout: PT5S
connectTimeout: PT2S
connectRetryDelay: PT15S
connectRetryTimes: 15
circuitDuration: PT30S
circuitThreshold: 1
itemCache:
server: redis
port: 6379
db: 3
timeOut: PT2S
ttl: PT45S
longTtl: PT3M30S
circuitDuration: PT15S
circuitThreshold: 1
compressed: false
polling:
queue: httpPersToSynchronize
queueType: SORTED_SET_HMAP
scoreAttribute: first
size: 175
jobRunners: 1
delay: PT5S
automaticESIds: false
updateCache:
keepInCache: true
useLongTtlAsSafety: true
removeFromCacheCondition:
property: status
values:
- COMPLETED
- ERROR
noTTLCondition: {}
saveInESConditions:
- property: state
values:
- CLOSED
- CLOSE_WAIT
- ESTABLISHED
- LAST_ACK
whisps:
config:
uri: http://whisp/v1/whisperers/{id}/config?view=full
label: "/whisp/whisperers/config"
timeout: PT10S
GraphQL
GraphQL is not a separate parser - it is an enrichment the web parser applies to the HTTP/1.x and
HTTP/2 communications it already produces on the paths configured in server.graphql.paths (default
["/graphql"]). Off the configured paths, nothing is detected unless heuristic is turned on, which
widens detection at the risk of mislabelling a REST body that happens to carry a query string key -
it is off by default for that reason.
Detected shapes:
- POST JSON -
content-type: application/json, body an object with a stringquery, or an object carryingextensions.persistedQuery.sha256Hash(a persisted / APQ hash-only request), or a non-empty array of such objects (a batch). - POST raw -
content-type: application/graphql, body is the document itself. - GET - the query string carries
query=(and optionallyoperationName=,variables=,extensions=).
The body used is the decoded one (gzip/deflate/br undone), capped at server.graphql.maxBodySize
(default 65536 bytes). Over the cap, an unknown content-encoding, or a request whose body never
completed capture, gets no GraphQL enrichment at all - not an error, not a parsing-log failure.
A matching user webStreams.reqTemplates rule always wins: GraphQL only fills the template when no
user rule matched the request.
Request fields (req.graphql.*)
| Field | Content |
|---|---|
operationType | query / mutation / subscription, or batch for a batched request; empty for a hash-only persisted query |
operationName | Selected operation name: the body's operationName, else the document's single named operation, else empty |
rootFields | Top-level selection field names of the selected operation, sorted, deduped, capped at 20 |
queryHash | Normalised document hash (literals to placeholders, whitespace/comments stripped, aliases dropped, arguments/selections sorted), first 16 hex characters |
persistedQueryHash | The full Automatic Persisted Query (APQ) sha256Hash, when the request carries one |
variableNames | Names of the declared variables of the selected operation - never their values |
batchSize | Number of operations in the request (1 when not batched) |
operations | Batch only: one identity string per member, capped at 20 |
status | Request-side only value: INVALID_DOCUMENT when the body matched a GraphQL shape but did not parse, or parsed with no operation to select |
The operation template (visible as req.template, and in the grid) takes one of these forms:
| Case | Template |
|---|---|
| Named operation | mutation CreateOrder |
| Anonymous operation | query order,customer (root fields, comma-joined, capped) |
Persisted, operationName sent | persisted GetOrder |
| Persisted, no name | persisted ecf4a1b2 (first 8 hex characters of the hash) |
| Batch | batch[3] query GetA + mutation SetB + … (first members, then + …) |
INVALID_DOCUMENT | Unchanged - the HTTP template still applies |
Response fields (res.graphql.*)
Parsed from the JSON response body, under the same decode and size rules, whatever the HTTP status
- servers disagree on it: a validation error can come back
200,400or404depending on the GraphQL server. Neitherstats.statusCodenor the communication's own status is ever changed by this.
| Value | Meaning |
|---|---|
OK | No errors in the response |
PARTIAL | errors present, and at least one non-null root field in data |
ERROR | errors present, and data is null, absent, or every root field in it is null |
UNKNOWN | The body could not be parsed as a GraphQL result, or was over the size cap |
Alongside status: errorCount (total errors[] length, summed across a batch response), errorCode
(the first errors[].extensions.code that carries one, e.g. UNAUTHENTICATED), and errorMessage
(the first error message that is a string - an error whose message is missing or not a string is
counted in errorCount but does not fill errorMessage), capped at 256 bytes after masking.
Variables
Only variable names ever reach req.graphql.variableNames or a template - values stay in the
request body, behind the same content permission as any other payload. errorMessage goes through a
value-marker mask: GraphQL servers routinely echo the offending value next to a fixed phrase (got invalid value ..., ... value: ..., with value '...'), so everything from that phrase to the end
of the message is replaced by …, quoted or bare. Outside such a phrase, a GraphQL name, a variable
reference ("$email") or a type reference ("String!") stays readable; any other quoted text is
masked. errorMessage is not stored at all when the whisperer does not save content for that
communication (saveContent, or saveTlsDecryptedContent for decrypted TLS) or its URI matches
urisToFilterResContent; status, errorCount and errorCode are still stored. The
one residual is a custom resolver's own message that echoes a bare value with none of those marker
phrases - it is not masked, so treat errorMessage as untrusted for anything sensitive and prefer
errorCode for grouping/aggregation.
The URL query string (req.query) is stored exactly as it is for any other GET request, and for a
GraphQL GET it contains the operation's variable values in clear. This is existing HTTP behaviour,
not something GraphQL parsing changes - restrict content/query visibility the same way you would for
any other GET request carrying sensitive parameters.
Persisted queries
A hash-only request (Automatic Persisted Queries) carries no document, so it has no operationType
and no rootFields to report - the platform never guesses them from a prior request. Its template is
persisted <name> when the client sent an operationName, else persisted <hash8> (the first 8 hex
characters of persistedQueryHash). There is no name-learning across requests: two hash-only calls to
the same persisted query, one with a name and one without, produce two different templates.
Subscriptions
GraphQL subscriptions carried over WebSocket or SSE are enriched the same way as request/response GraphQL, one document per message of the long-lived stream (see the WebMessageStream reference for the message schema itself).
- WebSocket - the connection is GraphQL when its negotiated subprotocol is
graphql-transport-ws(graphql-ws) orgraphql-ws(legacy subscriptions-transport-ws), on any path. With no subprotocol negotiated, the connection is only enriched on a path configured inserver.graphql.paths(or underheuristic). - SSE, distinct-connections and native - detected the same way as an ordinary GraphQL request (see
Detected shapes above): the opening request itself is a GraphQL request (POST JSON, POST raw, or GET
with
query=). - SSE, single-connection mode - the stream request carries no GraphQL document of its own (the
client POSTs
subscribe/completeoperations on a separate connection); it is recognised only by its path being configured inserver.graphql.paths(or underheuristic). See the caution below for what this costs.
Each message of the operation carries req.graphql.* identity - operationType, operationName,
rootFields, queryHash, persistedQueryHash - on both directions, with one exception: graphql-sse
single-connection mode never has req.graphql on any message, by design - see the caution below,
which covers why and what to do about it instead of relying on req.graphql there. The subscribe/start
message that opens the operation additionally carries variableNames (and batchSize: 1); later
messages of the same operation do not repeat it. res.graphql.* outcome - status, errorCount,
errorCode, errorMessage - is set only on server-to-client result/error messages (WebSocket
next/data/error, SSE next). The split for the rest is binary: complete/stop messages always
carry req.graphql identity and the operation template, never an outcome (again, except
single-connection SSE, which has no identity to carry on any message type); connection_init/
connection_ack, ping/pong and ka messages carry no GraphQL field at all.
The operation template (req.template) is inherited by every message of the operation from its
handshake, but only when the handshake itself has no user template (empty, or -) - a user template
set on the handshake (a matching webStreams.reqTemplates rule, see above) wins instead, exactly as
for an ordinary GraphQL request. For WebSocket, server.webSocket.reqTemplates rules still apply on
top of the inherited value, using the $t token to reference it: a rule named $t / inbound on the
inherited template subscription P9_WatchA yields subscription P9_WatchA / inbound. For SSE, the
handshake already carries the final operation template, so its events simply inherit it the usual way.
When a graphql-sse deployment separates the stream from the operation - the client opens the SSE
stream on one connection and posts subscribe/complete operations on another - the stream's next
events carry res.graphql outcome only, with no req.graphql identity to pair it with, since the POST
that started the operation is a separate HTTP exchange the stream never sees. Native graphql-sse, where
the subscribe request is the SSE handshake itself, does not have this gap.
Only the SSE next and complete event types are read. A result sent under the default message
event type (no event: field naming next) is not enriched.
Not parsed
- Multipart uploads (
multipart/form-data, the GraphQL multipart request spec) - not detected. - Bodies over
server.graphql.maxBodySize- no enrichment, silently.
MCP
MCP (Model Context Protocol, Streamable HTTP transport) is not a separate parser either - the web parser
enriches the HTTP/1.x and HTTP/2 communications it already produces on the paths configured in
server.mcp.paths (default ["/mcp"]). It needs two switches: the cluster-wide flip
global.flips.mcpParsing (off by default) and server.mcp.parseMcp on the whisperer (on by default).
global.flips.mcp is a different flip - it enables Spider's own MCP server - and does not affect parsing.
What is extracted, on req.mcp: the JSON-RPC method (initialize, tools/call, ...), the target (tool or
prompt name, or the resource URI without query string and fragment), the argument names, the message
kind (REQUEST, NOTIFICATION, RESPONSE), the client name and version, and the Mcp-Session-Id. The
session id is issued in the response to initialize, so it is copied onto the request from that response; it is
empty on stateless servers. On res.mcp: the outcome status (OK, TOOL_ERROR for a tool that reported
isError, ERROR for a JSON-RPC error, INPUT_REQUIRED for an unfinished resultType: input_required, UNKNOWN when the body could not be read or was over the size cap) and
the JSON-RPC errorCode. None of this changes stats.statusCode or the communication's own status: a failed tool
call is usually a plain HTTP 200.
Only names are stored. Argument values, tool results and error messages never reach the metadata.
JSON-RPC batches are not enriched. res.mcp is absent for any reply that is not application/json (including
answers streamed as text/event-stream, which keep their req.mcp), and for every request that is not a JSON-RPC
request: notifications, client replies, GET and DELETE. A request body over server.mcp.maxBodySize gets no
req.mcp at all; a response over it gets res.mcp.status: UNKNOWN.