RFC: High-level architecture

[thewilkybarkid]

Problem

Libero will be a framework for producing publication and presentation platforms. It needs to:

• scale up and down
• allow for a high degree of customisation and extension: allowing users to share features, as well as catering for the bespoke
• be deployable anywhere
• easily integrate with other systems

Suggestion

• Use a (micro)service-oriented architecture
  • Encapsulate features in separate applications
• Overall choreography, local orchestration
  • Communication between services is decoupled and processing is asynchronous
  • Inside services it can be neither
• Agnostic to where content originates and is owned
  • Source of truth can be elsewhere
• Specify communication methods and integration points, allowing for extension
  • Require a small core of structure and data
  • Provide modules for common features
  • “Build your own”
• Allow integration with external systems
  • API
  • Event bus
  • Message schema
• Use existing standards where possible

Concerns

• Distributed systems are more complex to understand, develop and run
  • Barrier to entry
  • Observability: distributed logging and tracing
• Too-high level of abstraction provides little value
• Yet another data format to define and translate data into to satisfy the user experience

[elrossco22]

The Digirati Team (Frank, John R, Stephen and Myself) discussed the above RFC and have the following questions:

• Is there a further detail/expansion that can be given on the problem space

• How does the high level architecture support adoption (forking, modules etc)

• In regards to being deployable anywhere - would additional work to secure the communication between microservices be required where we had previously seen the use of VPC's

• How will the system be easily integrated with other systems - will this be via api or standardisation

• Would it be possible to know more about the use case for Air flow and the other alternatives that were considered

• How will logging and monitoring be implemented in a centralised manner to support the platform being deployable anywhere (will these be additional microservices)

[thewilkybarkid]

How does the high level architecture support adoption (forking, modules etc)

We'll go over this at the sprint, but Libero is really three levels:

1. Specifications
2. Libraries
3. Applications

Code should be avoided in applications, and they will need to be forked if you want to do your own thing. That then avoids having to duplicate the code, unless you're replacing that part. Increases the complexity, of course.

In regards to being deployable anywhere - would additional work to secure the communication between microservices be required where we had previously seen the use of VPC's

Libero will work on the assumption that the API/event bus aren't public, but won't prevent it (and so needs to not be incompatible with it). It would be up to the implementor to allowing/disallowing access as required.

How will the system be easily integrated with other systems - will this be via api or standardisation

Not sure what you mean by standardisation, but there's two main communication methods (API and event bus) which are the primary integration points (and Libero could provide adapter code, eg a Drupal module). Libero will provide services that will interact with other services more directly where relevant (eg sending the article to some repository).

How will logging and monitoring be implemented in a centralised manner to support the platform being deployable anywhere (will these be additional microservices)

This mostly out of scope. We need to make sure that we provide ways to do both easily where relevant (eg by producing sensible logs), but how you do them will be up to the implementor (eg you would need to add in the New Relic PHP agent yourself).

[giorgiosironi]

How will logging and monitoring be implemented in a centralised manner to support the platform being deployable anywhere (will these be additional microservices)

Some other examples of this could be done by an implementor:

• deploy containers on top of a fixed set of virtual machines, stream logs through syslog-ng (eLife does this for some services)
• deploy containers on Kubernetes, set up a DaemonSet to stream logs to a managed service
• extend each container image to add an additional PHP or Python module