Repository navigation
USDTs on important functions #1070
Description
Other popular allocators, such as jemalloc have built-in mechanisms for profiling; mimalloc does not. To allow some level of debugging, it would be great if mimalloc added USDTs in important places, such as at the end of malloc and free. USDTs have a very minimal impact when not being traced (inserts a single NOP instruction) and much lower impact than having to use uprobes/uretprobes.
Is this something that would be accepted into the mimalloc project?
Activity
shikhar commentedon Apr 9, 2025
We would love to have more insight into mimalloc behaviour for our workloads and this seems like a great low-cost way to achieve that.
brancz commentedon Jun 13, 2025
Daan (@daanx) with jemalloc being archived, it might start to be more important to make mimalloc more observable. Any chance this is something you'd be interested in accepting?
daanx commentedon Jun 14, 2025
Apologies for the late response. Yikes, I didn't know jemalloc was getting archived -- that's unexpected :-(
Anyways, I am quite interested in USDT's but I would have to study it a bit more -- if the overhead is indeed this low that would be quite interesting. At the moment, there is already quite some support for tracing with tools like valgrind and asan (see
mimalloc-trace.h) and there has been integration with Windows perf counters as well. If we add USDT it would be good if we could reuse the same mechanism? Or at least impact the code as little as possible. I'll study this a bit more soon.brancz commentedon Jun 15, 2025
At first sight I would say that should work. A cherry on top would be if mimalloc sampled allocations every X bytes allocated and only fired the trace probes then (or additionally to trace probes that always fire unconditionally). That way mimalloc doesn’t have to know how to unwind a stack and can offload that part to the profiler, but the profiler doesn’t have to trace every single allocation which would be cost prohibitive in a production environment (which is arguably exactly where you want to profile). That would give us everything we need to build external heap and allocation profilers with very low overhead.
lucab commentedon Jun 16, 2025
I'm also coming from the jemalloc-profiling world and would love to have some way to observe mimalloc allocation patterns under real-world workloads in production environments.
This is a somehow common wishlist item when developing backend/infrastructure services (in my case, Rust binaries running on Linux servers).
For completeness, tracing through valgrind/heaptrack is nice for local development but is not well suited for investigating memory usage-patterns of a remotely deployed binary.
I'm quite excited if this ends up going in the direction suggested by Frederic Branczyk (@brancz), which wouldn't even require active instrumentation in the consuming application and can be directly driven by an external profiler!
danielsn commentedon Apr 17, 2026
I'm really glad to see this issue, I'm also excited about having memory profiling in mimalloc. I came to it from the direction of supporting in-process memory profiling from python (which is using mimalloc for the new NOGIL mode), which it turns out we could do with a fairly small code change.
For allocation profiling, this provides a callback that could be used to trigger the probe for an external profiler, or to directly sample and store the values for in-process profiling.
Supporting live-heap profiling requires a cheap way of knowing whether a given pointer was tracked when
freeis called. Doing so externally turns out to be surprisingly expensive, as even a hash-table incurs cache misses, which are avoided if the metadata lives on the same cache line as an allocator data stucture that will already be read as part of the free operation. In this case, theon_freecallback could either do in-process or out-of-process untracking of the object. In the common case where there is no tracked object on the page, this is a single branch on data that is already in cache.I have a draft PR exploring the approach: #1266
I'm not wedded to this code, and would love to work with you on whatever approach best fits with the mimalloc goals. Does this fit with what you're looking for Frederic Branczyk (@brancz)? If you could give this a try and let me know if there are changes that would work better for your out-of-process goals, that would be very helpful.
danielsn commentedon Apr 21, 2026
I've done some polish on the PR and I think it is ready for review Frederic Branczyk (@brancz) Daan (@daanx).
lucab commentedon Apr 23, 2026
Sorry for not posting this earlier, but last year I did some exploration around mimalloc allocations profiling.
Compared to #1266, I went in a different direction and hooked USDT probes to relevant entrypoints, see the drafted code at lucab#1.
The code in that PR is roughly working as an experiment, but it is not polished nor complete, so I didn't try to push it upstream.
In particular, before proposing this, I still wanted to explore some way to support externally-configured sampling.
I had some plans around trying to apply consistent probabilistic sampling (similar to OpenTelemetry head sampling algorithm) to pointer arithmetic, in order to get consistent sample/no-sample decisions for both allocs and frees based on pointer values (without full state tracking).
An external tracer would use a dedicated USDT semaphore (which IIRC is basically just a volatile
u16at a known location) to set the sampling probability.Unfortunately I didn't found the time/stamina to close the loop on this experiment in the past months. I'm not actively working on these topics, so it's unlikely that I'll manage to conclude this exploration anytime soon. But maybe the code draft and the braindump above could be useful to somebody else.
danielsn commentedon Apr 23, 2026
I did some experiments around doing this in python, and found that at least in python it easy to get into a situation where a set of objects are being allocated/freed repeatedly as python reusused objects, which started to mess with the statistics of which allocations to sample if they happened to all match or not match the determinstic algorithm.
danielsn commentedon May 11, 2026
Luca Bruno (@lucab) I'd be interested in looking into pointer arithmetic based sampling: my experiment was a quick one, and its possible that with proper upscaling it might work better.
danielsn commentedon May 11, 2026
Daan (@daanx) is the approach of adding the sampling decision inside mimalloc, with either direct probes or flexible callbacks when sampling occurs, one you'd be interested in exploring?
brancz commentedon May 12, 2026
In an unrelated context we’ve been playing with usdt semaphores which might be another thing we can use to make sure we’re only paying for the instrumentation when something actually attaches to the USDT(s).
lucab commentedon May 12, 2026
Frederic Branczyk (@brancz) I think stap-based probes can already handle this internally, see https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation#Semaphore_Handling (though I didn't explicitly verify that). But this is still a boolean full-on/full-off behavior when the profiler attaches.
Daniel Schwartz-Narbonne (@danielsn) I didn't consider the skewed bias in pointer values, that's indeed a bummer for an unlucky single-process/single-run case.
This is a bit outside of my area of expertise, but I still think in cases like a distributed profiler observing a N>1 number of executions, the statistical properties of the distribution of values may still make this work. That is, while a single observation can be highly biased, the aggregated set can still expose a useful signal on average.
zackees commentedon Jul 29, 2026
Hi all, I created a fork of mimalloc with built-in pprof heap profiling. Works great. 😄
While I was at it I also found a handful of bugs in upstream (and, by extension, most forks), including TLS cleanup paths that weren’t always being hit correctly.
The fork is based on the community’s v3 mimalloc work.
https://github.com/zackees/mimalloc-pprof
I'll be watching your repo for changes you do and trying to implement. Alex Costa (@deanx) is upstreaming some of my fixes already so pprof may or may not make it's way into microsoft upstream sooner or later.