Add GFX support

[capital-G]

GFX seems really powerful and would allow to write (improved) analysis tools such as a freqscope or write a gui for a parametric eq. But the API used by JSFX is a bit more complex, see https://www.reaper.fm/sdk/js/gfx.php#js_gfx - additionally this would probably need some additional thread synchronization. And we currently don't support strings which we probably should support at this point?

At the same time: This could be maybe also (mis)used for rendering visuals ;)

One thing to keep in mind: We should always also offer a headless version of DynGen!

edit: see https://github.com/JoepVanlier/JSFX for what GFX makes possible!

[Spacechild1]

Well, we would have to:

• create event loops on Windows and Linux resp. hook into the existing macOS event loop
• create our own native windows
• integrate a graphics library for rendering
• implement all the GFX functions

Good luck with that :-D

[Spacechild1]

Generally, I would be a bit careful about feature creep. I feel a tendency for reimplementing JSFX ;-)

If one feels the need to create GUIs or visualizations, maybe they should just use a JSFX plugin (https://github.com/jpcima/ysfx) + VSTPlugin?

[capital-G]

• create our own native windows

• integrate a graphics library for rendering

I think this would be already handled by LICE - Lightweight Image Compositing Engine and WDL Virtual Window system, which are part of WDL - https://www.cockos.com/wdl/

Generally, I would be a bit careful about feature creep. I feel a tendency for reimplementing JSFX ;-)

Yes yes ;) At the same time it would be interesting what people come up with since it seems more performant than the Qt stuff we have and would not require a sclang roundtrip.

I don't plan implementing this feature (yet), but it seems good to collect ideas here. Maybe it isn't too hard after all, since we don't have to be JSFX compliant.

[Spacechild1]

Actually, all the GFX functions are already implemented in eel_lice.h! AFAICT, JSFX just runs the @gfx section periodically on the UI thread, in parallel with the audio thread. The JSFX docs say the following:

The @gfx section gets executed around 30 times a second when the plug-ins GUI is open. You can do whatever processing you like in this (Typically using gfx_*()). Note that this code runs in a separate thread from the audio processing, so you may have both running simultaneously which could leave certain variables/RAM in an unpredictable state.

https://www.reaper.fm/sdk/js/js.php#js_intro

I'm wondering what this means in practice. There must be some thread synchronization.

[capital-G]

Actually, all the GFX functions are already implemented in eel_lice.h!

Yes - this is the reason why I opened that issue b/c most of the building blocks seem actually to be available - otherwise I would have definitely considered it out of scope.

I'm wondering what this means in practice. There must be some thread synchronization.

I was also surprised when reading that line - maybe they can get away b/c EEL2 only has floats at a pre-determined(?) location? Worst case is reading/writing a corrupted float? Or maybe in JSFX all communication between threads happens via block level sliders which are guarded by a LiFo queue/watch(?)

[Spacechild1]

I was also surprised when reading that line - maybe they can get away b/c EEL2 only has floats at a pre-determined(?) location?

That's what I thought as well. All the variables are allocated upfront and do not change locations. RAM blocks are allocated on demand, but the allocation itself is protected by a lock.

Worst case is reading/writing a corrupted float?

At least on 64-bit platforms, reads and writes to/from doubles should be atomic. But on 32-bit systems this can definitely lead to teared writes/reads...

Or maybe in JSFX all communication between threads happens via block level sliders which are guarded by a LiFo queue/watch(?)

EEL2 code just reads and writes to variables. There are no queues involved.

That being said, seems like eel_atomic.h contains the following functions for thread-safe variable access:

NSEEL_addfunc_retval("atomic_setifequal",3, NSEEL_PProc_THIS, &atomic_setifeq);
NSEEL_addfunc_retval("atomic_exch",2, NSEEL_PProc_THIS, &atomic_exch);
NSEEL_addfunc_retval("atomic_add",2, NSEEL_PProc_THIS, &atomic_add);
NSEEL_addfunc_retval("atomic_set",2, NSEEL_PProc_THIS, &atomic_set);
NSEEL_addfunc_retval("atomic_get",1, NSEEL_PProc_THIS, &atomic_get);

Curiously, these are not documented, but they could be used to safely communicate between @gfx and the remaining code sections.

[Spacechild1]

GFX could be very interesting for visualizations, but honestly I would just use https://www.realinks.net/links/ysfx/ + VSTPlugin instead of bloating the DynGen project.

We still need event loops + safe message passing between the UI thread and SC's realtime threads. This is pretty hairy stuff. (I'm speaking from experience.) Typically, all of this is handled by the host program.

Maybe I would consider this once scsynth has native event loops on all platforms that we can hook into with plugin API functions. (I want to do this eventually.)

[Spacechild1]

Curiously, these are not documented, but they could be used to safely communicate between @gfx and the remaining code sections.

Alternatively, one could also use a dedicated code section (e.g. @update) for exchanging data between audio thread and UI thread. This section would share the same lock as the @gfx section. However, it would only use a try-lock, so it never has to wait for the UI thread.

[capital-G]

We still need event loops + safe message passing between the UI thread and SC's realtime threads. This is pretty hairy stuff. (I'm speaking from experience.) Typically, all of this is handled by the host program.

Is it actually that difficult? The main problem seems to me that the GUI thread is accessing RT owned data which could be thrown away at any point and should be freed at this point - but beyond that? Event loops seem actually feasible w/ something like asio.

I can tell that you have lots of experience w/ this so I don't doubt it that its hard and the "devil is in the details" - and I also don't have any plans to implement this currently, but still...

[Spacechild1]

Event loops seem actually feasible w/ something like asio.

We are talking about UI event loops. This is something entirely different from "event loop" for asynchronous I/O. Asio doesn't help us here at all, the code is 100% platform specific. It's totally doable (as proven by VSTPlugin), but I'd rather implement the event loops in SuperCollider so that we only have to call some plugin API functions.