Zero-sized literal output buffers cause out-of-bounds access

[Sn0wyDay]

Description and impact

The shared implementation behind WriteFloatHex() and WriteDoubleHex(), as well as WriteUint128(), subtracts one from the output size while truncating a formatted value. When a caller supplies size == 0, that subtraction wraps to SIZE_MAX, which leads to an invalid copy length or array index.

With an AddressSanitizer and UndefinedBehaviorSanitizer build, the float and double wrappers abort with AddressSanitizer: negative-size-param: (size=-1), while WriteUint128() aborts on an index of 18446744073709551615. Without sanitizer checks, these accesses are undefined behavior and can crash a caller that explicitly supplies size == 0.

Reproduction

Minimal library-level reproduction; run the program once with each of the three calls enabled individually:

#include "wabt/literal.h"

int main() {
  char output = 'x';
  wabt::WriteFloatHex(&output, 0, 0);
  // wabt::WriteDoubleHex(&output, 0, 0);
  // wabt::WriteUint128(&output, 0, {0, 0, 0, 0});
}

Affected surface

The affected functions are helpers declared in the installed include/wabt/literal.h: WriteFloatHex(), WriteDoubleHex(), and WriteUint128(). WABT does not document every declaration in the installed headers as a supported, stable public API.

Current in-tree production callers of WriteFloatHex() and WriteDoubleHex() pass nonzero fixed-size buffers. WriteUint128() currently has only test callers. I did not find a .wat/.wasm or bundled CLI input path that supplies size == 0. This issue therefore concerns an explicit library-level zero-size call rather than an input-triggerable path in the bundled tools.

Required WebAssembly features

None. No --enable flag is needed.

Candidate fix and validation

I have a small fix that returns before accessing the output for size == 0, plus GoogleTests covering all three functions in the regular unit-test target. The two focused regressions, all 137 WABT unit tests, a three-function ASan/UBSan harness, and the full WERROR=ON CMake build and check target pass on macOS with AppleClang.

I did not run the exhaustive 32-bit hexfloat_test input space.