The idea for this feature is that an authenticated and authorized user will be able to add time series/forecasts/simulations to the system for inclusion in metrics and dashboards. This feature needs more specific requirements.
We may need to implement #11 first or in conjunction, also #13, maybe.
Process:
- User uploads time series data via the browser. Format is either CSV or Parquet (others?)
- Data lands in either an S3 bucket or Iceberg table
- Data needs to be mapped to the TEEHR time series schema
- A configuration needs to be created in the configurations table before data can be validated. How do we want this to work? Could be automatic but that may create a mess of users do not use the same configuration name for like data. This could be something that an admin needs to do - add any new configurations. This is safer but less automated.
- What are the permissions on the new data? All public? Are there controls on this?
- Data will be validated against the domain and locations tables like is done now for all ingested and loaded data via Jupyter.
- Validation passes - data is loaded. Validation fails...we need to let users know this. How? Some logging that is exposed to users?
- Could Prefect be used for this validation and loading process once we set it up for async analytics API? I envision this having a way to notify users of status asynchronously.
Option 1 - Use S3 as a "drop box"
Option 2 - Use an Iceberg namespace (or table) as a "drop box"
The idea for this feature is that an authenticated and authorized user will be able to add time series/forecasts/simulations to the system for inclusion in metrics and dashboards. This feature needs more specific requirements.
We may need to implement #11 first or in conjunction, also #13, maybe.
Process:
Option 1 - Use S3 as a "drop box"
Option 2 - Use an Iceberg namespace (or table) as a "drop box"