A lot of times the performance is data dependent, like randomized data can result in poor warp utilization where as sorted data can be much more efficient because you have less warps sitting around idle. We have an upcoming blog post on decompression kernels that show this clearly, stay tuned!
Thanks! I think most performance debugging happens during development, what we're bringing to the table is exposure of system behavior in production which often diverges because of changes in the shape of workloads from dev, which are often simplistic and synthetic. So I'd say its late-stage performance debugging and production observability combined that makes this useful.
Stay tuned for a follow on post where we show how we used this to optimize an FSST decompression kernel for vortex (https://github.com/vortex-data/vortex).
I feel like I've seen Cupti have fairly high overhead depending on the cuda version, but I'm not very confident -- did you happen to benchmark different workloads with cupti on/off?
---
If you're taking feature requests: a way to subscribe to -- and get tracebacks for -- cuda context creation would be very useful; I've definitely been surprised by finding processes on the wrong gpu and being easily able to figure out where they came from would be great.
I did a hack by using LD_PRELOAD to subscribe/publish the event, but never really followed through on getting the python stack trace.
CUPTI is kind of a choose your own adventure thing, as you subscribe to more stuff the overhead goes up, this is kind of minimalist profiler that just subscribes to the kernel launches and nothing else. Still to your point depending on kernel launch frequency/granularity it may be higher overhead than some would want in production, we have plans to address that with some probabilistic sampling instead of profiling everything but wanted to get this into folks hands and get some real world feedback first.
This "low-overhead always on GPU profiler" seems really cool and useful, but we're not using Kubernetes for anything, and the instructions for how to use it seems to only include Kubernetes. Is there a way of running this without Kubernetes?
Does the profiler read any of the GPU's performance counters? Would be super cool to have an open source tool that can capture the same data nsight compute does.
Cache hit rate is probably the most immediately useful. Although given that this is for always-on profiling maybe this project isn't as geared towards optimizing kernels as I originally thought? In theory reading the counters should be low overhead though.
[ All from my experience on home GPUs, and in lah with 2 nodes with 2 80GB H100 each. Not extensively benchmarked ]
Events like kernel launch, which this profiler reads right now, is a very small overhead (1-2%). Kernel level metrics like DRAM utilisation, cache hit rate, SM occupancy, etc usually give you a 5-10% overhead. If you want to plot a flame graph at a instruction level (mostly useful for learning purposes) then you go off the rails - even 25% overhead I have seen. And finally full traces add tons of overhead but that's pretty much expected - they anyways produce GBs of profiling data.
Occupancy and RAM utilization are available from static analysis. A sampling profiler would also obviously not be suitable for this always-on profiler case. But reading the counters [0] from the GSP should be cheap.
I wrote the code and the blog, happy to answer any questions/comments. Very eager to have folks try it out and give feedback! Like is my meme game strong or very strong? J/K
There's some missing bits around FFI and callbacks (i.e. C calling function pointer that is a luajit generated stub back into the interpreter) and curious if anyone actually uses these things in OpenResty workloads. Deploy and enjoy!
I'd love to use this to profile Neovim, where (nowadays) user scripts are often written in Lua 5.1. Neovim by default embeds LuaJIT (close to HEAD). Being able to see Lua functions and C functions in the stack would be amazing.
Is there a single-shot mode that I can use from the command-line? Something like:
It should work to profile neovim, does it use the luajit shared library or is it statically linked? You could run parca-agent and set up a filter to discard all non-neovim samples and leave a parca instance running locally and just restart it periodically.
The answer you are looking for is no. No one can translate this into what humans should eat to avoid heart disease, that will require a ton more research. We still haven't even figured out whether eggs are good or bad for you.
> However, 49% of industry-funded intervention studies reported conclusions that were discordant with study results (ie, net cholesterol increases were described as favorable in the articles’ stated conclusions), compared with 13% of non–industry-funded studies.
I can definitely believe that the average American should probably keep eating eggs since they are better than the average American's egg replacement (more refined grains, sugar). But that doesn't mean eggs are part of a diet that minimizes CVD risk, for example. I think we often confuse the two facts.
I would bet a dollar inhibiting TOR via protein restriction turns out to be one of those things thats good for mice but bad for humans. Whatever aging gains you get are probably over-ridden by earlier onset of sarcopenia. Too many studies show how having muscle and strength is key to aging well, something mice in a cage don't have to worry about. Agree though that 99.9 is probably high.
If you want to do stuff like this, anything you use to do it will have to look a lot like eBPF. eBPF doesn't make it easy, it only makes it possible. But dtrace was not eBPF.
Well, it would have been hard to be eBPF because it pre-dated it. But perhaps you meant to say that eBPF is not DTrace? On that point, certainly agreed.
Dtrace provided huge value for quite little implementation effort. It has taken a positively enormous amount of more grueling, detailed work to make eBPF much more capable. It might not have been done without dtrace demonstrating the value available, but I credit eBPF to the people who did that hard work.
