- What does
nis a big number (e.g.,
- What does
calloc(n, size)return if
sizeare big numbers?
- What does
new T[n]return if
nis a big number (assuming
I believe everyone expects the answer
NULL for all the questions,
NULL is used to indicate out of memory.
Unfortunately, in reality many memory allocators break (or used to
break) these expectations, which could lead to memory corruption and
security vulnerabilities, especially if the allocation size
can be controlled by an adversary.
Here goes a summary of some known and unknown vulnerabilities (notably multiplication, rounding, and pointer overflows) in popular memory allocators.
An infamous multiplication overflow used to exist in the
implementation in GNU libc and Microsoft’s C runtime, as detailed
RUS-CERT Advisory 2002-08:02.
This overflow vulnerability can be easily misused to mount buffer
overflow attacks. It was
in GLIBC in 2002.
I cannot find the revision log of the Hoard allocator, so it’s hard to say what vulnerabilities it used to have. Though it’s easy to exploit the current version (3.8) via a rounding overflow. Below is the buggy code snippet.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
(size_t)-1, a big value, the
align call overflows
sz up to 0. Thus,
malloc(-1) is basically
which could lead to a buffer overflow.
calloc implementation is also vulnerable to multiplication
overflow. With glibc Hoard is injected via
calloc won’t be used. However, the vulnerability can
be triggered on platforms that do not use glibc, such as Mac OS X.
The jemalloc allocator is used by the libc of FreeBSD and NetBSD, as well as Mozilla Firefox.
It used to have the
calloc multiplication overflow vulnerability.
The bug was
fixed in 2006.
A rounding overflow bug was fixed in 2007.
Another interesting signedness bug was fixed in 2008. Below is the simplified code snippet.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
size is unsigned and
incr is signed.
size is big enough,
incr is misinterpreted as negative
and causes the
sbrk system call to shrink the memory.
Bionic, the Android libc, is derived from the BSD libc. When
libc.debug.malloc is set, allocation calls will be rerouted to
its own debugging library.
Again, the debugging library had
calloc multiplication overflows
malloc implementation still has multiple rounding
overflows, at least in in
Hope that no production phone ships the debugging library.
I didn’t spot any serious rounding issues in TCMalloc,
but just a slip in
the error message says “tcmalloc: large alloc 0 bytes == (nil)’’
malloc(-1), because the size to be output overflows.
APR pool is developed for the Apache web server. It is also adopted by other projects such as Subversion. The interface is similar to allocate-and-free allocators, but takes one extra parameter specifying which pool to allocate from.
APR pool had a series of interesting pointer arithmetic fixes, listed as follows.
In Aug 2002, the sanity check
p + size < q was
size < q - p, where
the two pointers
node->endp) point to the start and the end of a memory block,
1 2 3
Over a month later (Sep 2002), the check was
p + size < q.
One week later (Oct 2002), the check was again
size < q - p.
After over five years (Apr 2008), the check was
size <= q - p with an off-by-one fix.
1 2 3 4 5
p + size < q is more dangerous than
size < q - p. One obvious
reason is that
p + size could wrap around to a small pointer given
size, while the latter check doesn’t have the pointer
But how about adding a wrap check
p + size < p? A subtle
problem is that pointer overflow (or more precisely, out-of-bounds
pointer) is undefined in C, which means that the compiler can do
anything to the wrap check
p + size < p, such as optimizing it
I tried to compile the wrap check using GCC 4.6 and Clang 3.0 on x86:
it’s optimized as extracting the sign of
size, without even looking
at the value of
1 2 3
The size rounding vulnerability was also discovered and fixed in APR pool (CVE-2009-2412) in 2009.
Boost Pool has the multiplication overflow
issue in its
1 2 3 4 5 6 7 8 9 10 11
We can see that
total_req_size may wrap around to a small number
given a big
Boehm GC is probably
the most popular open-source garbage collector. It provides C
allocation calls and overloads the C++
calloc implementation, in both
malloc.c and its
calloc(n, size) to
GC_MALLOC((n) * (size)),
thus is vulnerable to multiplication overflow.
It also has the size rounding vulnerability, so
doesn’t return a null pointer either.
Safe by Design
Some allocators are designed to be used in “safe’’
environment. For example, LLVM’s
doesn’t consider out-of-memory or overflow issues.
kLIBC is another example.
1 2 3 4 5 6 7 8 9 10 11 12 13
The C++ array allocation
new T[n] has a potential integer overflow
n is not limited correctly, which could lead to
a buffer overflow. This is similar to
The code generated by GCC simply calculates the
allocation size as
n * sizeof(T) without any overflow checks,
though GCC developers have been
discussing the problem
for a long time.
Clang generates “safe’’ code that defends against integer overflow attacks, via setting the allocation size to -1 if any multiplication overflow happens. This implies an assumption that the underlying allocator must return a null pointer given a large size. As we have seen, this assumption doesn’t hold well in practice.