CompactHashMap/CompactHashSet/MapMakerInternalMap native deserialization triggers uncontrolled memory allocation

Summary

CompactHashMap, CompactHashSet, and MapMakerInternalMap implement readObject(ObjectInputStream) by reading an attacker-controlled elementCount or initialCapacity from the stream. On the very first insertion during or after deserialization (for the CompactHash* collections, inside allocArrays()) or immediately during deserialization (for MapMakerInternalMap), the implementation eagerly allocates arrays sized to that full, unvalidated value (up to CompactHashing.MAX_SIZE = 2^30-1, ~1.07 billion, for example) before the rest of the stream's actual content has been consumed. A 72-byte malicious serialized stream containing a single tiny element can force a multi-gigabyte allocation and crash a JVM with OutOfMemoryError, dynamically confirmed against the current release.

Details

CompactHashMap.readObject() (guava/src/com/google/common/collect/CompactHashMap.java):

private void readObject(ObjectInputStream stream) throws IOException, ClassNotFoundException {
  stream.defaultReadObject();
  int elementCount = stream.readInt();
  if (elementCount < 0) {
    throw new InvalidObjectException("Invalid size: " + elementCount);
  }
  init(elementCount);
  for (int i = 0; i < elementCount; i++) {
    K key = (K) stream.readObject();
    V value = (V) stream.readObject();
    put(key, value);
  }
}

elementCount is only checked for being non-negative; there is no upper bound and no relationship enforced between the declared count and how much data actually follows in the stream. init(elementCount) (line 251) just stores the value (clamped to [1, CompactHashing.MAX_SIZE], i.e. up to 2^30-1):

void init(int expectedSize) {
  Preconditions.checkArgument(expectedSize >= 0, "Expected size must be >= 0");
  this.metadata = Ints.constrainToRange(expectedSize, 1, CompactHashing.MAX_SIZE);
}

The real allocation is deferred to allocArrays(), which fires lazily on the first put() call — i.e. after only one key/value pair has actually been read off the wire:

int allocArrays() {
  ...
  int expectedSize = metadata;
  int buckets = CompactHashing.tableSize(expectedSize);
  this.table = CompactHashing.createTable(buckets);
  setHashTableMask(buckets - 1);
  this.entries = new int[expectedSize];
  this.keys = new Object[expectedSize];
  this.values = new Object[expectedSize];
  return expectedSize;
}

With expectedSize near CompactHashing.MAX_SIZE (2^30-1), this single call allocates a hash table plus three expectedSize-length arrays — multiple gigabytes — triggered by a stream that has, at that point, only supplied one small real element. CompactHashSet.readObject()/allocArrays() have the identical shape. CompactLinkedHashMap/CompactLinkedHashSet extend these classes and inherit the same unguarded path.

The problem with MapMakerInternalMap is similar, with a size read from the serialized data and stored in a MapMaker instance for later user during deserialization, when the MapMaker is passed back to MapMakerInternalMap and segments are allocated by calling newSegment, whose implementations (example) create Segment instances (example) that allocate arrays of the requested capacity through the newTable helper.

This is the same amplification shape as the already-fixed CVE-2018-10237 (AtomicDoubleArray/CompoundOrdering: attacker declares a large size, library eagerly allocates before validating against actual stream content). That fix rewrote AtomicDoubleArray.readObject() to grow an ImmutableLongArray.Builder incrementally per element actually read, rather than pre-allocating from the declared size — Guava's own HashBiMap.readObject() documents the same defensive principle explicitly in a comment: init(16); // resist hostile attempts to allocate gratuitous heap. CompactHashMap/CompactHashSet do not follow either of these two patterns already present elsewhere in the same codebase.

Reachability: All classes are top-level, implements ... Serializable, and require no special classpath setup beyond Guava being present (near-ubiquitous on JVM backends). A native Java deserialization attacker names the class directly in a hand-crafted stream — no particular Guava factory method needs to have been used by the victim application, exactly as with the precedent CVE.

PoC

Full source and run instructions in the attached poc-guava-compacthash-dos.zip (Maven project, guava:33.7.1-jre — the current latest release — as the only dependency). The PoC:

1. Uses reflection only to legitimately obtain a CompactHashSet instance (it is package-private; a real attacker does not need this step, since they craft the serialized bytes directly rather than calling Guava's own serialization code).
2. Serializes one set with 1 element and one with 2 elements, diffs the two byte streams to locate the exact offset of the elementCount field (byte-precise, not guessed).
3. Patches that one 4-byte field in the 1-element stream from 1 to 0x3FFFFFFF (CompactHashing.MAX_SIZE), leaving the rest of the 72-byte stream (one real "A" element) untouched.
4. Feeds the resulting 72-byte malicious stream into a plain new ObjectInputStream(...).readObject() call against the real, unmodified guava-33.7.1-jre.jar.

Actual run output:

### -Xmx8g (generous heap) ###
Located size field at offset 63, value=1 (expected 1)
Malicious stream size: 72 bytes
Declared elementCount: 1073741823 (~1.07 billion)
Real payload: 1 tiny string element ("A")
=== RESULT ===
Exception: java.lang.OutOfMemoryError: Java heap space
Time to allocate+fail: 4013 ms
Heap used before: 3 MB
Heap used after (at failure point): 4103 MB
JVM max heap (-Xmx): 8192 MB

### -Xmx128m (ordinary service-sized heap) ###
Located size field at offset 63, value=1 (expected 1)
Malicious stream size: 72 bytes
Declared elementCount: 1073741823 (~1.07 billion)
Real payload: 1 tiny string element ("A")
=== RESULT ===
Exception: java.lang.OutOfMemoryError: Java heap space
Time to allocate+fail: 24 ms
Heap used before: 4 MB
Heap used after (at failure point): 2 MB
JVM max heap (-Xmx): 128 MB

A 72-byte input forces ~4.1 GB of real allocation before failing on an 8 GB heap, and crashes a 128 MB heap in 24 ms — over 50,000x amplification between input size and forced allocation.

To reproduce: mvn -q compile dependency:build-classpath -Dmdep.outputFile=cp.txt && java -Xmx128m -cp "target/classes:$(cat cp.txt)" poc.Poc

Impact

Uncontrolled resource consumption (CWE-770 / CWE-400