CVE-2026-27571: nats-server websockets are vulnerable to pre-auth memory DoS

The WebSockets handling of NATS messages handles compressed messages via the WebSockets negotiated compression. The implementation bound the memory size of a NATS message but did not independently bound the memory consumption of the memory stream when constructing a NATS message which might then fail validation for size reasons.

An attacker can use a compression bomb to cause excessive memory consumption, often resulting in the operating system terminating the server process.

The use of compression is negotiated before authentication, so this does not require valid NATS credentials to exploit.

The fix was to bounds the decompression to fail once the message was too large, instead of continuing on.