built-in profiler

[xrme]

I think it may make sense to provide some kind of profiling tool with CCL.

We have always struggled trying to coerce system profiling tools like Shark (on old macOS) and perf (on Linux) info providing useful information.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[eugeneia]

I guess I have two relevant references here:

• [Sketch] OS-level tracing #54 which is a WIP for exposing data required by OS level traving tools. I think ultimately, this is what we want. I think with the Linux tracing landscape in flux and dtrace becoming stable (how does dtrace on OSX look? can we get away with only supporting dtrace (with dtrace4linux)?), there is a new generation of OS-level tracing ecosystems on the rise. Notably, since dynamic language runtimes have become ubiquitous, I see opportunities to influence the tracing ecosystems to expose ABIs that lend themselves well to tracing such runtimes as ours. I think its important to note that these tracing approaches are much more wide-reaching than a traditional profiler, and its impossible(?) to replicate similar functionality from inside CCL.

• Sam could probably just be shipped with CCL. It works well for high-level profiling if all you care about is what happens within CCL.

Actually, I think we want both. An integrated high-level profile is nice because you can program it from within CCL. However, we can’t possibly compete with OS-level tracing tools, and not having a strong integration with those will be a big downside of using CCL on such a platform.

[svspire]

I have a non-sampling metering tool at https://bitbucket.org/svspire/ccl-metering. I'm happy if you want to distribute it with CCL. Not sure it makes much sense to submit it to Quicklisp since it's very CCL-specific.

[xrme]

At some point, perf changed in such a way that it stopped consulting the map file for addresses in the mapped lx86cl64.image file.

Gary Byers sent mail about this to linux-perf-users about this in mid-January 2016. The upshot off it is that perf is no longer fooled by CCL's attempt to make the heap image file look like an ELF object. While perf supports the map file for addresses from an anonymous map, the image file is file-mapped (and the map file isn't/wasn't used in that case). There was some discussion on the list about how to work around this, but I can't find the list archives.

I don't think dtrace supports ustack helpers on macOS, which I think would be necessary for dtrace (and hence Instruments) to interpret a lisp call stack.

It's certainly true that OS-specific tracing tools are going to be more sophisticated than anything we can put into CCL. On the other hand, the operating systems that CCL runs on all have different profiling tools. If there is something basic but useful that could be used the same way on all of the platforms, that would be a good feature to have.

The approach described in https://research.swtch.com/pprof seems like it would not be too hard to adapt for CCL. (See also https://research.swtch.com/macpprof).

[svetlyak40wt]

It would be interesting to have a solution which produces data in a flame graph's format. It is widely adopted by other languages, but I didn't see a library for the Common Lisp. Read more about flame graphs here: https://github.com/brendangregg/FlameGraph

…

On Sat, Dec 9, 2017 at 9:42 PM, R. Matthew Emerson ***@***.*** > wrote: At some point, perf changed in such a way that it stopped consulting the map file for addresses in the mapped lx86cl64.image file. Gary Byers sent mail about this to linux-perf-users about this in mid-January 2016. The upshot off it is that perf is no longer fooled by CCL's attempt to make the heap image file look like an ELF object. While perf supports the map file for addresses from an anonymous map, the image file is file-mapped (and the map file isn't/wasn't used in that case). There was some discussion on the list about how to work around this, but I can't find the list archives. I don't think dtrace supports ustack helpers on macOS, which I think would be necessary for dtrace (and hence Instruments) to interpret a lisp call stack. It's certainly true that OS-specific tracing tools are going to be more sophisticated than anything we can put into CCL. On the other hand, the operating systems that CCL runs on all have different profiling tools. If there is something basic but useful that could be used the same way on all of the platforms, that would be a good feature to have. The approach described in https://research.swtch.com/pprof seems like it would not be too hard to adapt for CCL. (See also https://research.swtch.com/macpprof). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#92 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABg-walsR6fxaszhuE7j5H9NH-S583Oks5s-tSQgaJpZM4Q7xsU> .

-- Alexander Artemenko https://github.com/svetlyak40wt