Enable compression in OpenSSL and add opt-in certificate compression support for TLS connections

[pimterry]

EDIT: This diff size looks scary, but the vast majority is autogenerated from the build config update. Changes are now split into separate commits, all the auto-generated output is now in the 3rd commit, so the other parts can be reviewed more easily.

Until now, we've fully disabled all compression features in OpenSSL via no-comp. This PR:

• Removes no-comp from our OpenSSL build, so we can use some compression features. This is required because OPENSSL_NO_COMP implies OPENSSL_NO_COMP_ALG, which disables certificate compression.
• Links in our existing zlib, brotli & zstd implementations for the compression itself.
• Adds a certificateCompression option for TLS contexts, exposing the OpenSSL API for certificate compression, defaulting to disabled (i.e. no visible change). We might want to enable this by default in future but we can debate that separately, no need to do so immediately I think. The new API throws an error if a non-empty option is passed when using a shared OpenSSL without compression support.

There's a lot of changes here but the vast majority (everything in deps/openssl/config/archs) are auto-generated based on the OpenSSL config changes and the actual source changes are quite simple.

This is my first time touching the OpenSSL dep setup directly, so hopefully I've rebuilt it correctly! Extra eyes there very helpful. Notably you can't actually rebuild directly on main via make gen-openssl right now, because /deps/openssl/openssl/test/ doesn't exist (removed since #57835). That's fine for OpenSSL dep updates (which pull the whole tree anyway) but it breaks simple regeneration. I just manually included the test dir locally, which seems to have rebuilt OK without changes. I'll open a separate PR to fix that properly later, but there's too many conflicts with the changes in here to open that at the same time. I've now added a commit to fix this here, adding no-tests to the OpenSSL config, you can now run make gen-openssl to regenerate manually and (nearly, except various timestamps) reproduce this commit yourself.

There's also a small test change required to the config for 3 unusual addon tests: on Mac only, directly linked our internal libopenssl.a into a shared library, without the rest of Node, which no longer works (because OpenSSL now references the compression libs). I don't think this is sensible or supported approach, I can't find any examples of this anywhere else, and these tests cover a feature (OpenSSL custom engines) which is deprecated and will be removed in OpenSSL v4 anyway. A small config change here updates them to instead dynamically look up symbols which should give the same result without issues.

Upsides:

• This reduces certificate transfer size in handshakes significantly. In the example in the tests, total TLS handshake bytes shrinks 50%, even in minimal examples it should still be 15% or so.
• That effect will get more significant in future with larger post-quantum certs and similar.
• We'll probably want to enable this anyway for QUIC eventually, as limits there mean larger handshakes are a significant performance issue. I'm not an expert but my understanding: in TCP servers can generally send a 14KB response before hitting TCP congestion windows and waiting for a client ack. QUIC amplification protection meanwhile effectively gives a 3.6KB window until response, much smaller, and a server handshake message larger than that delays the whole connection setup for an extra RTT. That limit is very tight so compression is necessary to avoid extra RTs for many connections.
• Reduces fingerprintability of Node.js traffic: all browser clients include the certificate compression extension in TLS client hellos, while Node does not, making TLS connections from Node trivially recognizable (so this helps support user-space solutions to Provide APIs to help control TLS fingerprints #41112).

Binary size

Since the compression libraries are already included in Node anyway, this is minimal: a standard Node build on my machine is 131MB, and increases 40KB (0.03%) with this addition.

Compression risks in TLS e.g. CRIME

No-comp was originally here in part to disable TLS record compression (compression during the connection itself, broken by CRIME). In this PR we're not enabling that, we're enabling certificate compression (how certs are transferred during the initial handshake only). There's no known vulnerabilities around certificate compression itself AFAIK, and it's enabled in all modern browsers and user-facing backends like NGINX, HAProxy, Envoy, etc so seems like a safe bet. This PR also keeps it disabled by default - it just becomes optionally available.

Record compression here is not enabled by this change, despite removing no-comp. It's independently disabled in many other ways:

• OpenSSL disables record compression on all new contexts by default, unless explicitly enabled:

  node/deps/openssl/openssl/ssl/ssl_lib.c

  Lines 4207 to 4215 in 2b74812

  	/*
  	* Disable compression by default to prevent CRIME. Applications can
  	* re-enable compression by configuring
  	* SSL_CTX_clear_options(ctx, SSL_OP_NO_COMPRESSION);
  	* or by using the SSL_CONF library. Similarly we also enable TLSv1.3
  	* middlebox compatibility by default. This may be disabled by default in
  	* a later OpenSSL version.
  	*/
  	ret->options |= SSL_OP_NO_COMPRESSION | SSL_OP_ENABLE_MIDDLEBOX_COMPAT;

  (AFAICT we don't expose any capability to reset this option).
• OpenSSL additionally fully disallows all use of record compression with security level 2+ anyway (our default from v24.5/v25+).
• We independently disable all record compression options here:

  node/src/crypto/crypto_util.cc

  Lines 165 to 167 in 2b74812

  	// Turn off compression. Saves memory and protects against CRIME attacks.
  	// No-op with OPENSSL_NO_COMP builds of OpenSSL.
  	sk_SSL_COMP_zero(SSL_COMP_get_compression_methods());

• Record compression was completely removed from the protocol for TLS v1.3, so should be impossible to negotiate there in all scenarios even if we really really wanted to.

Removing it completely from the OpenSSL build as well was a nice extra protection, but it's not strictly required and blocks other unrelated TLS compression features.

[nodejs-github-bot]

Review requested:

• @nodejs/gyp
• @nodejs/security-wg

[pimterry]

Review from @nodejs/crypto also helpful. Although this is a deps change, it's primarily a TLS feature.

As noted in the description: the number of changes files is intimidating, but the vast vast majority are autogenerated updates! It's actually a fairly reasonable changeset 😄

[aduh95]

Could you split the deps/ changes to a separate commit so it's easier to review please?

[pimterry]

@aduh95 Good idea. Now split into three commits: the first reconfigures the OpenSSL build, the 2nd contains all the auto-generated output from that, the 3rd contains the Node changes which use this to enable cert compression.

[pimterry]

It would be great to keep this moving, as it's going to have some large & painful conflicts to resolve if there are any other OpenSSL changes in the meantime. Cert compression has some nice benefits, I'd love to get the option in for v26 next month.

Any other reviews from @nodejs/crypto or maybe @nodejs/net?

[panva]

@pimterry this is not part of crypto that i can provide feedback/review for. Sorry.

[pimterry]

Now updated. I've:

• Added a commit which adds no-tests to the OpenSSL build config. Without this, make gen-openssl fails on main - it only works during the process of a full OpenSSL update, because we git ignore the tests afterwards. No downside to this I think - it's just skipping configuring test code builds that we don't use.
• Rebuilt the OpenSSL sources.
• Rebased & squashed the cert compression commit
• Updated the intermediate change messages to fixup! and added the rebase label, to land the TLS & deps changes as separate lines in the changelog.

That means there's 4 commits to review:

• Enabling compression in OpenSSL build config (50 line change)
• (as a fixup) Adding no-tests to OpenSSL build config (2 line change)
• (as a fixup) Rebuilding OpenSSL (many thousands of lines - but reproducible with make gen-openssl)
• Exposing a new certificateCompression option for TLS (a couple of hundred lines)

@jasnell can you take a look at this? This is useful for TLS generally, but specifically solves the large certificates problem for QUIC you're listing in the docs, by enabling certificate compression so we can avoid handshake round trips. This has been sat open for a while, but worth prioritising for QUIC imo. Disabled by default here for normal TLS so it's optional & definitely non-breaking, but once it's merged we could test out enabling it by default in QUIC contexts.

[jasnell]

Yeah, I've been meaning to come back to this. Record-level compression has long been actively problematic due to CRIME and similar types of exploits. Certificate-compression should be fine but last I checked openssl was gating both behind the same flag. As long as we're certain that enabling this won't accidentally enable record-level compression, then I'm all for it.

[pimterry]

Landed in 21310cb...e52ec44