Show error messages when running models

[pinkwah]

In addition to errors being logged, we should also tell the user why the model failed. As far as I can tell, these are the types of errors that can occur:

1. Network/protocol errors
2. Input validation errors
3. Output validation errors
4. Model runtime error

1. Network/protocol error

This error type is between the frontend and backend. When the error is between backend and the model, it should be considered a "model runtime error".

This occurs when the API endpoint is incorrect, the wrong parameters are passed, or when the returned documents aren't correctly validated with zod.

In the case of such an error, the frontend should display a red card with the failure information.

2. Input validation error

These occur when the either the concentrations or parameters are not accepted by the model.

We can piggyback off of pydantic's ValidationError and tell the user exactly which inputs were incorrect. When the frontend receives this information, it should either display the message under the input box that contains the error, or as a red card at the top of the model inputs section.

The error's schema could be the following:

{
  "errors": [
    {
      "section": "parameters",
      "field": "temperature",
      "message": "Temperature cannot be above 100K"
    },
    {
      "section": "concentrations",
      "field": "HNO3",
      "message": "Can't have both HNO3 and H2SO4"
    },
    {
      "section": "concentrations",
      "field": "H2SO4",
      "message": "Can't have both HNO3 and H2SO4"
    }
  ],
}

3. Output validation errors

When the model adapter returns something, we should ensure that the data has the correct shape in the backend. If this isn't the case, the frontend should display a red card with the error and tell the user that this is a bug that should be reported to us.

4. Model runtime error

If an unexpected exception occurs, or the job times out or gets cancelled (if we implement this), the message should be displayed to the user. In development mode, it would be nice if the error message is displayed together with a stack trace.

I also believe we should show a stack trace in production (because there isn't anything to hide) but if we are squeamish, we could figure out how to create a link to this error in grafana or something like that.

Like with 3, we should tell the user that this is a bug and that they should report it to us.