pageant: unbounded frame buffering on peer-controlled 4-byte length prefix (memory-exhaustion DoS) (CVE-2026-116775)

Summary

pageant::PageantStream reads a 4-byte big-endian length prefix from its input stream and buffers until len + 4 bytes are available, without ever validating len. The underlying WM_COPYDATA transport physically cannot carry frames larger than 8 KiB (_AGENT_MAX_MSGLEN, the fixed-size shared memory mapping), so an oversized frame is destined to fail — but only after the re-framing task has already buffered the full claimed size. A peer controlling the stream bytes can claim a frame of up to ~4 GiB and either stream that much data (forcing a corresponding allocation) or simply stop mid-frame and keep the partial buffer resident indefinitely.

Vulnerable code

pageant/src/wmmessage.rs, PageantStream::new() (the spawned re-framing task), current main:

let mut buf = BytesMut::new();
while let Ok(n) = two.read_buf(&mut buf).await {
    if n == 0 { break; }
    if buf.len() < 4 { continue; }
    let len = BigEndian::read_u32(&buf) as usize;
    if buf.len() < len + 4 { continue; }   // <-- `len` is trusted; no upper bound
    let msg = buf.split_to(len + 4).freeze();
    // ...
}

len is consumed verbatim from the stream, so buf grows towards len + 4. Even a "complete" oversized frame cannot succeed: query_pageant_direct() writes msg into the _AGENT_MAX_MSGLEN-sized (8192 bytes) shared mapping and fails with Error::Overflow for anything larger — the unbounded buffering is therefore pure attacker-controlled memory growth.

History: introduced in commit 6eac7a157dd2cc81b8fafdb2977f1c388fe9784b ("fix Tabby#10301 Still cannot login with pageant", PR #565, 2025-09-25) together with the length-prefixed framing itself; present in every 0.2.x release (0.1.0 predates the length-prefixed loop and is not affected by this issue).

Attack scenario (SSH agent forwarding)

The crate is documented as a raw transport ("This crate only implements the transport, not the actual SSH agent protocol"). The standard way to service a remote server's forwarded agent channel on Windows is to bridge the SSH channel to the local agent, e.g.:

// inside Handler::server_channel_open_agent_forward (russh client, sketch)
let mut channel = channel.into_stream();          // bytes sent by the SSH server
let mut agent = pageant::PageantStream::new().await?;
tokio::io::copy_bidirectional(&mut channel, &mut agent).await?;

In this (intended) bridging use, the bytes written into PageantStream are the SSH server's channel payload, so the 4-byte length prefix is controlled by a malicious or compromised SSH server. The server only needs to send the four bytes FF FF F0 00, then stream data (or stall). russh's typed AgentClient bounds the response frames it reads with MAX_AGENT_FRAME_LEN (256 KiB) and builds its own request frames, but the raw stream path — the documented transport usage — has no equivalent bound.

Impact: remote memory-exhaustion DoS of the client process. A full-size claim lets the peer force buffering of up to ~4 GiB of memory, which on memory-constrained hosts ends in an allocation-failure abort (uncatchable in Rust). Multiple forwarded channels multiply the effect. No memory-safety violation was observed.

Proof of concept

(Windows; no Pageant instance required — buffering happens before any Pageant interaction)

use tokio::io::AsyncWriteExt;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut stream = pageant::PageantStream::new().await?;
    // Claim a ~4 GiB frame, stream 256 MiB, then stall: the task buffers all of it.
    stream.write_all(&0xFFFF_F000u32.to_be_bytes()).await?;
    let chunk = vec![0u8; 256 * 1024];
    for _ in 0..1024 { stream.write_all(&chunk).await?; } // 256 MiB, frame never completed
    tokio::time::sleep(std::time::Duration::from_secs(30)).await;
    Ok(())
}

Observed on Windows 11, pageant 0.2.4 (source v0.64.1, Rust 1.99): process working set grows from ~7.4 MB to ~264.7 MB after 256 MiB is streamed, and stays there while the peer stalls; no error is ever raised. Memory grows linearly with bytes sent, up to the claimed length. (Allocation behavior additionally verified with a counting global allocator.)

Suggested fix

Reject frames that cannot fit the transport before buffering:

                 let len = BigEndian::read_u32(&buf) as usize;
+                // The WM_COPYDATA transport is backed by a fixed-size shared
+                // memory mapping: larger requests can never fit. Reject them
+                // before buffering. Without this cap a malicious peer could
+                // force up to ~4 GiB of buffering by claiming a huge frame
+                // and never completing it.
+                if len.saturating_add(4) > _AGENT_MAX_MSGLEN {
+                    debug!("dropping oversized Pageant request: {len} bytes");
+                    break;
+                }
                 if buf.len() < len + 4 {
                     continue;
                 }

saturating_add also removes a theoretical 32-bit len + 4 overflow. break shuts down the re-framing task (the writer observes the stream close); returning an error through the stream would work equally well. Happy to submit a PR with this change, or with any alternative you prefer.

Reporter

Tshellauth