Http Parsing

The following services are managing HTTP communications parsing.
Below is their default configuration.

webWrite

webWrite is the Helm values key for the web parser (web-parser, renamed from web-write / web-streams-write). Since the HTTP/2 work, it parses HTTP/1.x, HTTP/2 and gRPC together — there is no separate gRPC parser service. Its configuration reflects that: one parsingJob reading a single merged queue, and a protocols block carrying one store/cache set per protocol it may output.

"@id": web-parser
"@type": ServerConfiguration
version: '0.1'
protocols:
  http:
    comStore:
      node: http://elasticsearch:9200
      indexPurge: spider-search-httpcom-upload
      indexGet: spider-search-httpcom
      getTimeout: PT10S
      purgeTimeout: PT10S
      connectTimeout: PT2S
      connectRetryDelay: PT15S
      connectRetryTimes: 15
      circuitDuration: PT30S
      circuitThreshold: 1
    contentStore:
      node: http://elasticsearch:9200
      indexGet: spider-search-httpcomcontent
      indexPurge: spider-active-httpcomcontent-upload-default
      getTimeout: PT15S
      purgeTimeout: PT15S
      connectTimeout: PT2S
      connectRetryDelay: PT15S
      connectRetryTimes: 15
      circuitDuration: PT15S
      circuitThreshold: 0.8
    comsCache:
      server: redis
      port: 6379
      db: 2
      timeOut: PT10S
      ttl: PT25S
      circuitDuration: PT30S
      circuitThreshold: 1
      compressed: false
    comsContentCache:
      server: redis
      port: 6379
      db: 12
      timeOut: PT10S
      ttl: PT25S
      circuitDuration: PT30S
      circuitThreshold: 1
      compressed: false
  grpc: # present only when the grpcParsing feature flip is on
    comStore:
      node: http://elasticsearch:9200
      indexPurge: spider-search-grpccom-upload
      indexGet: spider-search-grpccom
      getTimeout: PT10S
      purgeTimeout: PT10S
      connectTimeout: PT2S
      connectRetryDelay: PT15S
      connectRetryTimes: 15
      circuitDuration: PT30S
      circuitThreshold: 1
    contentStore:
      node: http://elasticsearch:9200
      indexPurge: spider-search-grpccomcontent-upload
      indexGet: spider-search-grpccomcontent
      getTimeout: PT10S
      purgeTimeout: PT10S
      connectTimeout: PT2S
      connectRetryDelay: PT15S
      connectRetryTimes: 15
      circuitDuration: PT30S
      circuitThreshold: 1
    comsCache: # a distinct Redis instance (redis-grpc), not the HTTP cache's db
      server: redis-grpc
      port: 6379
      db: 2
      timeOut: PT10S
      ttl: PT25S
      circuitDuration: PT30S
      circuitThreshold: 1
      compressed: false
    comsContentCache:
      server: redis-grpc
      port: 6379
      db: 12
      timeOut: PT10S
      ttl: PT25S
      circuitDuration: PT30S
      circuitThreshold: 1
      compressed: false
parsingLog:
  store:
    node: http://elasticsearch:9200
    indexPurge: spider-search-httppers-upload
    indexGet: spider-search-httppers
    getTimeout: PT10S
    purgeTimeout: PT10S
    connectTimeout: PT2S
    connectRetryDelay: PT15S
    connectRetryTimes: 15
    circuitDuration: PT30S
    circuitThreshold: 1
  cache:
    server: redis
    port: 6379
    db: 3
    timeOut: PT10S
    ttl: PT2M
    circuitDuration: PT30S
    circuitThreshold: 1
    compressed: false
parsingJob:
  pollingDelay: PT1S
  jobRunners: 1
  uri: http://tcp-update/v1/parsing-jobs/web
  label: "/tcp-update/parsing-jobs/web"
  timeout: PT10S
  parsingDelay: PT10S
  queues: ["web", "http", "grpc"] # "grpc" dropped from the list when grpcParsing is off
purgeJob:
  sizeLimitKB: 100
parsingStatusSynchro:
  delay: PT5S
  queue: statsSynchro
  maxBucketSize: 100
tcpSessions:
  uri: http://tcp-update/v1/tcp-sessions
  label: "/tcp-update/tcp-sessions"
  timeout: PT10S
  sizeLimitKB: 100
  compressed: true
packetsByIndex:
  uri: http://pack-read/v1/packets/of/tcpsession
  label: "/pack-read/packets/of/tcpsession"
  timeout: PT10S
whisps:
  get:
    uri: http://whisp/v1/whisperers/{id}
    label: "/whisp/whisperers"
    timeout: PT10S
  config:
    uri: http://whisp/v1/whisperers/{id}/config?view=full
    label: "/whisp/whisperers/config"
    timeout: PT10S

There is one parsingLog (parsing state / tracking resource), shared across protocols — a TCP session being parsed for web traffic has a single parsing log regardless of whether it turns out to carry HTTP or gRPC.

Two related settings live in the shared common configuration, not in webWrite itself:

• parserProtocols — the list of protocols with a dedicated parsing queue (drives queue naming, e.g. the web / http / grpc queues above).
• outputProtocols — the list of protocols the web parser may emit a communication for (http, grpc, websocket, sse). sse has an engine and appears in data — see SSE stream message parsing settings. websocket has no engine yet and never appears in data.

webRead

"@id": web-streams-read
"@type": ServerConfiguration
version: '0.1'
httpComStore:
  node: http://elasticsearch:9200
  indexGet: spider-search-httpcom
  getTimeout: PT15S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT15S
  circuitThreshold: 0.8
httpComContentStore:
  node: http://elasticsearch:9200
  indexGet: spider-search-httpcomcontent
  getTimeout: PT15S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT15S
  circuitThreshold: 0.8
httpPersStore:
  node: http://elasticsearch:9200
  indexGet: spider-search-httppers
  indexParsingStatus: spider-search-parsing-status-httppers
  getTimeout: PT10S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT30S
  circuitThreshold: 1
packets:
  uri: http://pack-read/v1/packets/payload/tcp/?force=true
  label: "/packets/payload/tcp"
  timeout: PT10S
searchRequest:
  sizeLimitKB: 100
whisps:
  uri: http://whisp/v1/whisperers/{id}/config?view=server
  label: "/whisp/whisperers/config"
  timeout: PT10S

webHttpComPoller

"@id": web-httpcom-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpCom
itemStore:
  node: http://elasticsearch:9200
  useDataStoragePolicies: true
  useDistinctPolicyForUpload: true
  streamedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httpcom-streaming-default
  uploadedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httpcom-upload-default
  saveTimeout: PT5S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT30S
  circuitThreshold: 1
itemCache:
  server: redis
  port: 6379
  db: 2
  timeOut: PT2S
  ttl: PT45S
  circuitDuration: PT15S
  circuitThreshold: 1
  compressed:  false
polling:
  queue: httpComToSynchronize
  queueType: SORTED_SET
  scoreAttribute: _update
  size: 175
  jobRunners: 1
  delay: PT2S
  automaticESIds: false
  updateCache:
    keepInCache: false
    removeFromCacheCondition: {}
    noTTLCondition: {}
whisps:
  config:
    uri: http://whisp/v1/whisperers/{id}/config?view=full
    label: "/whisp/whisperers/config"
    timeout: PT10S

webHttpComContentPoller

"@id": web-httpcom-content-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpCom
itemStore:
  node: http://elasticsearch:9200
  useDataStoragePolicies: true
  useDistinctPolicyForUpload: true
  streamedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httpcomcontent-streaming-default
  uploadedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httpcomcontent-upload-default
  saveTimeout: PT5S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT30S
  circuitThreshold: 1
itemCache:
  server: redis
  port: 6379
  db: 12
  timeOut: PT2S
  ttl: PT45S
  circuitDuration: PT15S
  circuitThreshold: 1
  compressed:  false
polling:
  queue: httpComContentToSynchronize
  queueType: SORTED_SET
  scoreAttribute: _update
  size: 175
  jobRunners: 1
  delay: PT2S
  automaticESIds: false
  updateCache:
    keepInCache: false
    removeFromCacheCondition: {}
    noTTLCondition: {}
whisps:
  config:
    uri: http://whisp/v1/whisperers/{id}/config?view=full
    label: "/whisp/whisperers/config"
    timeout: PT10S

webHttpPersPoller

"@id": web-httppers-poller
"@type": ServerConfiguration
version: '0.1'
logField: httpPers
itemStore:
  node: http://elasticsearch:9200
  useDataStoragePolicies: true
  useDistinctPolicyForUpload: true
  streamedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httppers-streaming-default
  uploadedDataStoragePolicies:
    - name: default
      indexSave: spider-active-httppers-upload-default
  saveTimeout: PT5S
  connectTimeout: PT2S
  connectRetryDelay: PT15S
  connectRetryTimes: 15
  circuitDuration: PT30S
  circuitThreshold: 1
itemCache:
  server: redis
  port: 6379
  db: 3
  timeOut: PT2S
  ttl: PT45S
  longTtl: PT3M30S
  circuitDuration: PT15S
  circuitThreshold: 1
  compressed:  false
polling:
  queue: httpPersToSynchronize
  queueType: SORTED_SET_HMAP
  scoreAttribute: first
  size: 175
  jobRunners: 1
  delay: PT5S
  automaticESIds: false
  updateCache:
    keepInCache: true
    useLongTtlAsSafety: true
    removeFromCacheCondition:
      property: status
      values:
      - COMPLETED
      - ERROR
    noTTLCondition: {}
  saveInESConditions:
  - property: state
    values:
    - CLOSED
    - CLOSE_WAIT
    - ESTABLISHED
    - LAST_ACK
whisps:
  config:
    uri: http://whisp/v1/whisperers/{id}/config?view=full
    label: "/whisp/whisperers/config"
    timeout: PT10S