RFC: Backend provider plugins

[p-e-w]

With the basic plugin system now landed (ffaf2b6), and @Piktrip's excellent work on OpenRouter integration (#23) as a starting point, it's time to make it possible for plugins to provide backends that Waidrin can use for inference. The main challenges are being generic enough to serve all possible plugins of this type, and maintaining a clean engine/frontend separation.

Here's my current plan to achieve this:

The engine interface

Waidrin's engine relies on just two functions to communicate with the LLM: getResponse, and getResponseAsObject. The logical first step would be to extract those function's signatures (without the params, as those aren't generic) into a Backend TypeScript interface. The plugin would then provide an implementation of this interface, or, better, any number of them, as a plugin might want to provide multiple backends:

type Plugin = {
  ...
  getBackends(): Promise<Record<string, Backend>>;
};

The response is a map from the backend name to the implementation, and can contain arbitrarily many entries. When the plugin is loaded, the loader checks whether it implements getBackends, and, if it does, calls it and adds the resulting backend entries to a Record<string, Backend> map in the global state. The state also contains an activeBackend: string field that decides which of the backends is used by the engine.

The frontend

The Plugin interface is frontend-agnostic. Any frontend-specific logic is handled by the opaque context parameter passed to init. The context provides environment-specific functionality to the plugin, and for Waidrin's React frontend, will contain various functions that allow the plugin to interact with it.

My idea is to add something like the following function to the context:

addBackendConfigurationPage(backendName: string, displayName: string, pageComponent: () => React.ReactNode): void

When the plugin calls that function, Waidrin adds a new tab to the connection screen, titled displayName, with pageComponent as its contents. The plugin is free to do whatever it wants in this component, and can connect it to the Backend implementation it provides in whatever way it wants. When the user selects this tab, Waidrin sets activeBackend to backendName, and everything should magically work. This way, plugins can provide any number of backend implementations, and separate configuration pages for each of them, while activation is completely controlled by the application. Waidrin's plugin system already has a rudimentary mechanism for plugin-specific settings, which can be used and extended as needed to serve backend plugins.

Backend provider plugins will be a critical feature making Waidrin usable by more people, and it's very important to get them right. If you have any comments on this proposal, now is the time!

[Piktrip]

Do you plan to add a function where the backend can be switched mid playing, or will it always be "only" on the connection page?

[p-e-w]

Eventually yes. There will be a configuration screen that can be brought up at any time, and where this and other parameters can be changed. I also plan to have the option to specify different backends for narration and object generation, which is already hinted at by the current code that supports having different generation parameters for the two.

[Piktrip]

Looks great. Ready to test the plugin ( prepared it already, at least for OpenRouter - more to come ) once you implement the core backend architecture. I think this is a clean way to handle the backend plugins.

[p-e-w]

I have implemented the engine part of my proposal (2d1474e). See state.ts for the method signature that the plugin must implement, and backend.ts for the actual Backend interface.

Backends provided by plugins are loaded automatically on startup. You can use the state debugger to set activeBackend to the name of your plugin's backend, and it should be used instead of the default backend.

Note that I have not tested this at all yet (except for the default backend, of course), so if you encounter bugs, please let me know.

Two important points regarding plugins:

1. The plugin loader uses import() to load the plugin's main module, which expects a vanilla ES6 JavaScript module. If you want to use TypeScript, JSX, or external dependencies, your plugin will need its own build process. In the future, I intend to add a plugin installer, which will run npm install and npm run build in the plugin's directory after cloning.
2. Since the plan is to ship your plugin with Waidrin by default, make sure you add it as an exception to .gitignore. This will also ensure that Biome runs on the plugin's files.

[Piktrip]

I've been testing the plugin system and ran into an issue. When plugins register backends through getBackends(), the backend objects get stored in the Zustand persistent state, but they lose their class methods during serialization/deserialization.

For example, backends.openrouter.isAbortError becomes undefined even though the OpenRouter backend class
implements it correctly. This causes runtime errors when the engine tries to call backend methods.

Could it be that getBackend() should call the plugin's getBackends() method fresh each time instead of using the
stored objects? Just wondering if this is expected behavior or if I'm missing something in my plugin
implementation.

The plugin loads fine and shows up in the console logs, but the methods don't survive the state persistence

[p-e-w]

I had anticipated this problem, and thought I had coded around it: Note that when loading the plugins in page.tsx, I first clear out state.backends, and then re-populate it by calling getBackends() on the freshly loaded plugins.

My understanding (apparently flawed) was that on page load, Zustand would first load the persisted state, then the plugins are loaded, which clears the stale backend objects and replaces them with fresh ones. But now I'm wondering if there is maybe a race condition between loading the persisted state and loading the plugins, which could result in the fresh plugin interfaces being overwritten by the state loaded from persistence.

[p-e-w]

Looking into the Zustand docs appears to confirm my suspicion. The persistence middleware offers an onFinishHydration hook to get notified when the state has fully loaded. I will change the main page logic to wait for this to happen, which should not only fix the issue you're experiencing, but also the annoying FOUC that's sometimes visible when refreshing the page.

[p-e-w]

I can't actually reproduce the problem you're describing. Here is my plugin for testing purposes:

class TestBackend {
  async getNarration(prompt, onToken) {
    return "test";
  }

  async getObject(prompt, schema, onToken) {
    // Required to pass connection test.
    return "waidrin";
  }

  abort() {
    //
  }

  isAbortError(error) {
    return false;
  }
}

export default class BackendTest {
  async getBackends() {
    return { "test-backend": new TestBackend() };
  }
}

This plugin will only get you past the connection test, because further output doesn't validate, but it can do so again and again, even after hydration from state persistence, exactly as intended. Not sure what is happening in your case.

[Piktrip]

The PR will likely happen tomorrow, provided there are no surprises.

[p-e-w]

I have pushed two precautionary changes to guard against state issues:

1. Waidrin now guarantees that plugins are only loaded once state hydration is complete.
2. All properties containing functions are removed from the state debugger, making it impossible to corrupt them by editing those or other properties.

The frontend API for adding backend configuration UIs is coming soon.

[Piktrip]

See #28 (comment)