Skip to main content

TLS key log fallback

Gocipher captures TLS secrets by hooking the TLS library of the application: OpenSSL, Go's crypto/tls and the JDK's JSSE.
Some runtimes have no library Gocipher can hook: Envoy (Istio sidecars), rustls, or a statically linked BoringSSL.
For those, the key log fallback reads the TLS key log that the runtime itself can produce, with the key log file set to /dev/null.

When to use it​

RuntimeCaptured without the fallbackUse the fallback
OpenSSL (dynamically linked), Node.js✅No
Go crypto/tls✅No
Java (JSSE)✅No
Envoy / Istio sidecar (expected, not yet validated)NoYes
rustlsNoYes
BoringSSL linked staticallyNoYes
LibreSSL, GnuTLS, NSS, Mbed TLSNoNot validated

The fallback only applies to processes where Gocipher found no TLS library to hook. It never duplicates a secret captured another way.

How it works​

The workload writes its key log lines (CLIENT_RANDOM ..., CLIENT_TRAFFIC_SECRET_0 ..., etc.) to /dev/null. The kernel discards the data, but Gocipher observes the write and keeps the secrets in memory, as for every other mode. Nothing is written to disk.

Gocipher then links the secrets to the TCP sessions, and the sessions are decrypted like any other TLS session (TLS 1.2 and TLS 1.3).

Configure the workload​

The workload must be configured to write its key log to /dev/null.

Envoy / Istio​

Patch the TLS contexts of the sidecar with an EnvoyFilter, scoped to the workload by workloadSelector, to set key_log in the common_tls_context of both the DownstreamTlsContext (inbound) and the UpstreamTlsContext (outbound):

key_log:
path: /dev/null

Notes:

  • Expected to work, not yet validated. Only rustls has been validated end to end. Envoy's flush behaviour has not been measured with Gocipher: if Envoy flushes large batches of keys, or writes them with writev, some keys may be missed (see Limitations).
  • A hot restart of Envoy creates a new process. It is registered only at the next re-initialisation of the Gocipher target (see Limitations).

rustls​

config.key_log = Arc::new(rustls::KeyLogFile::new());

and start the process with SSLKEYLOGFILE=/dev/null.

BoringSSL​

Register a callback with SSL_CTX_set_keylog_callback. The callback must write line + "\n" to a file descriptor opened on /dev/null.

Enable it on the whisperer​

The fallback is off by default and enabled per whisperer.

In the Analysis UI, open the whisperer configuration, section TLS keys of the capture configuration, and check Key log fallback (/dev/null). This sets client.tlsKeys.keylogFallback to true.

  • The change takes effect within a minute, once the whisperer configuration poll picks it up (measured: 20 to 30 seconds). The capture of the target is restarted immediately: expect a gap of a few seconds where new TLS sessions cannot be decrypted. All TLS libraries of the target (OpenSSL, Go and JVM included) are restarted, not only the fallback, and TLS handshakes in flight are lost for a few seconds.
  • Unchecking the option stops the capture for this whisperer.

Check it works​

In the Gocipher details, the discovered process shows an SSL library of mode keylog, named keylog://<target>, with a counter of captured secrets.

Gocipher reports its status every minute, so the library and the hint appear with up to one minute of delay.

If the library is listed, but no key log is received after 5 minutes, the status shows the hint No key log received (it appears 5 to 6 minutes after the attach): Gocipher is ready, but the workload does not write its key log to /dev/null. Check the workload configuration above. This is not an error, and the hint disappears as soon as a secret is captured.

Gocipher logs with the following codes:

CodeLevelMeaning
GCPH-KLOG-000InfoKey log fallback registered for a target (pids: number of processes).
GCPH-KLOG-001ErrorThe kernel probe could not be loaded or attached. The target falls back to its other modes.
GCPH-KLOG-002WarnOne process could not be registered.
GCPH-KLOG-003WarnPartial or malformed key log lines were dropped (rate limited).
GCPH-KLOG-004InfoCapture stopped: normal after disabling the option or when the target is restarted.
GCPH-KLOG-005WarnReading the capture buffer failed.
GCPH-KLOG-006WarnA key log record could not be parsed.

Limitations​

  • Processes are registered when Gocipher starts watching the target. A process started later (Envoy hot restart, a new process in the container) is not captured until the target is re-initialised: restart Gocipher, re-attach the target, or restart the workload.
  • Only plain write() calls to /dev/null are seen. A runtime that writes its key log with writev/pwritev or io_uring is not captured.
  • A single write of more than 4096 bytes is truncated and its trailing partial line is dropped, so the keys of the last handshakes in that write are lost. The drop is counted and logged as a warning (GCPH-KLOG-003, at most once a minute).
  • When Gocipher JVM capture is off, JVM processes are also candidates for the fallback.
  • Needs a Linux kernel with BPF ring buffers (5.8 or later).
  • Only rustls (TLS 1.2 and TLS 1.3) is validated end to end. Envoy / Istio is expected to work but is not yet validated.

Security​

warning

Use /dev/null and nothing else. Any other path makes the workload persist its TLS secrets to disk, which lets anyone reading that file decrypt the traffic. That is not required by Spider, and is entirely the customer's responsibility.