HollowByte: OpenSSL DoS Vulnerability Exploits Tiny Payloads to Cripple Servers A newly disclosed denial-of-service (DoS) vulnerability, dubbed HollowByte, allows unauthenticated attackers to exhaust server memory with just an 11-byte malicious payload targeting OpenSSL. The flaw, silently patched by the OpenSSL team, affects widely used versions of the library, which underpins secure communications for web servers, databases, and programming languages. ### How HollowByte Works Researchers from Okta’s Red Team revealed that vulnerable OpenSSL versions allocate memory based on a 4-byte header in TLS handshake messages, which declares the size of incoming data before verifying the actual payload. Attackers exploit this by sending an 11-byte input with a falsified header claiming a much larger message will follow. The server reserves the specified memory, then hangs indefinitely waiting for data that never arrives. By repeating this across multiple connections, attackers force the server to allocate excessive memory. While OpenSSL frees buffers when connections drop, the GNU C Library (glibc) retains small-to-medium allocations for reuse, leading to heap fragmentation. The server’s Resident Set Size (RSS) a measure of active memory balloons uncontrollably, even after the attack stops. The only recovery method is a full process restart. ### Impact and Affected Systems OpenSSL is embedded in critical software, including: - Web servers (NGINX, Apache) - Programming runtimes (Node.js, Python, Ruby, PHP) - Databases (MySQL, PostgreSQL) - Most Linux distributions (pre-installed for TLS encryption) Okta’s tests on NGINX demonstrated that low-resource environments can be completely depleted of memory, while high-capacity servers may lose up to 25% of available RAM all while attack traffic remains below typical security alert thresholds. Though DoS flaws are less severe than remote code execution vulnerabilities, they can cause operational disruptions and reputational harm. ### Patch and Mitigation The OpenSSL team addressed HollowByte as a "hardening fix" rather than a formal security vulnerability, but the issue has been resolved in: - OpenSSL 4.0.1 - Backported to 3.6.3, 3.5.7, 3.4.6, and 3.0.21 The fix modifies memory allocation to ignore header claims and expand buffers only when data arrives. Organizations are advised to upgrade OpenSSL packages immediately to prevent exploitation.