HTTP body transcoding - making protobuf and MessagePack traffic searchable
Spider has been parsing HTTP/1.1 traffic for years - request lines, headers, status codes, response bodies - and let operators write tag and template rules that pull values out of those parts.
This works perfectly when the body is text.
JSON and XML carry their field names in plain sight, so a rule looking for a client id finds the string clientId sitting right there in the bytes.
Protobuf and MessagePack do not work that way: the body is a compact binary blob, and the field names live in a separate schema, not in the payload. The information is there - it just is not readable as text, so the text rules have nothing to grab onto.
The new HTTP body transcoding feature closes that gap.
- The operator uploads the schema once and declares which requests it applies to.
- From then on, Spider quietly converts each matching binary body into the equivalent JSON before the existing tag and template rules run.
- The same rules that already worked for JSON now also work for protobuf and MessagePack - with no change to the rules themselves.