Channels

[taylordowns2000]

Context

The most common reason for adopting the OpenHIM is for observability. Being able to deploy the OpenHIM between two existing services, and get instant visibility into the requests and responses flowing back and forth between them is extremely helpful in complex systems.

Within this observability there is also a common need for:

• The ability to retry transactions (or batches of transactions). This doesn't always make sense because the response has already been sent by that point though.
• Alerting for failed transactions so that they can be investigated.
• Scheduled reporting (daily, weekly, monthly) to identify trends and spot anomalies.

The next-most important reason for adoption is for access control. Being able to separate the credentials used for authenticating a client, and those used for making requests to the target service provides a lot of flexibility in managing access, especially when combined with channels which can limit requests to a single endpoint.

This is also one of the biggest challenges for adoption though because of the diversity of authentication methods out there and the complexity which comes along with them. Mutual TLS with self-signed certificates are a fairly common approach.

Then a few other features that are commonly used:

• Transformation of request paths from one format to another. This is particularly helpful for (backwards) compatibility with target services.
• Sending a single incoming request to multiple target services for parallel processing or archival.

Proposed Approach

In some cases, the complexity and overhead of a full OpenFn workflow javascript workflow isn't required. An inbound request (GET, POST, PUT, PATCH, DEL) to an OpenFn project channel would not need to spawn a new worklfow run, but would instead use native Elixir/Phoenix to:

1. check if the requester is valid (authenticate using the webhook_auth_method model which should be abstracted to become a "registered client" model which is used by both webhook triggers and channels... if they are kept separate, call it a channel_access_method?)
2. optionally add authentication to the request
3. forward the request to a predefined destination (preserving the path, the body, the query params, etc.)

The goal would be to have a blazing fast reverse-proxy plus some simple authentication.

Requests to these channels could be captured as runs or they could be stored in a different model such as channel_requests. Either way, we'd want them to appear on the history page, searchable and auditable just like runs. See image below, where you can see the traditional history page has been extended to include a "Client" column which tells you who started the run or channel-request.

Open Questions

1. Are they runs or something else (like channel_events)?
2. Do we convert webhook_auth_method/channel_access_method into the broader client concept? This aligns really nicely with the IA change we're making towards "connected systems" for "tech estate management". Your connected systems sometimes have "inbound credentials" (i.e., this/webhook_auth_methods) or "outbound credentials" (i.e., credentials). Either way, we'll want to be able to see a list of all the systems in the tech estate that are connected to OpenFn - both making requests TO and receiving requests FROM OpenFn.
3. Should any data transformation be allowed, or is this only a reverse proxy? (I think no... if you want transformation, use a workflow.)
4. ...more, I presume?

[brandonjackson]

I don't understand the problem this solves. Could someone help me by creating a customer problem statement?

I am... (The Persona): Who is the customer?
I am trying to... (The Goal): What is their desired outcome?
But... (The Barrier): What obstacle is preventing them from succeeding?
Because... (The Root Cause): Why does this obstacle exist?