diff --git a/README.md b/README.md index 0b6f864..98905a1 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,6 @@ A powerful federated data processing and analysis system that preserves patient - [Data Management](#data-management) - [High Level Description](#high-level-description) - [Architecture](#architecture) -- [Installation Prerequisites](#installation-prerequisites) # Preamble @@ -59,7 +58,6 @@ This includes documentation on existing algorithm federation approach as well as - [Available federated analysis algorithms](documentation/algorithms.md) - [Exaflow Analytic Engine](https://github.com/madgik/exaflow/tree/1.0.0) - # Data Management For all details relating to the Data Factory, how to manage your data and process it for use with the MIP, please consult the following document @@ -78,14 +76,13 @@ For a high-level description of the MIP please consult: - [The MIP: A powerful federated data processing and analysis system that preserves patient privacy](https://ebrains.eu/data-tools-services/medical-analytics/medical-informatics-platform) on EBRAINS research infrastructure - # Architecture -[High-level view of the architecture](documentation/Architecture.md), the main building blocks and data flows. +- [High-level view of the architecture](documentation/Architecture.md), the main building blocks and data flows. -# Installation Prerequisites +# Onboarding guidance -- See [Deployment Pack](documentation/deployment-pack/README.md) +- [Onboarding to the Medical Informatics Platform MIP](https://wiki.ebrains.eu/bin/view/Collabs/onboarding-to-the-mip/) on EBRAINS Collaboratory # Acknowledgement This project/research received funding from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Framework Partnership Agreement No. 650003 (HBP FPA). diff --git a/deployment/README.md b/deployment/README.md index 78a4e36..b479499 100644 --- a/deployment/README.md +++ b/deployment/README.md @@ -18,4 +18,4 @@ Use the Kubernetes guide for production-like and federated installations. ## Supporting Documentation -- [Data requirements for onboarding new datasets](docs/NewDataRequirements.md): CSV and `CDEsMetadata.json` format rules, including additional constraints for longitudinal data. +- [Data requirements for onboarding new datasets](../documentation/MIP_Data_management_documentation.md): CSV and `CDEsMetadata.json` format rules, including additional constraints for longitudinal data. diff --git a/deployment/docs/NewDataRequirements.md b/deployment/docs/NewDataRequirements.md deleted file mode 100644 index 1fce1d4..0000000 --- a/deployment/docs/NewDataRequirements.md +++ /dev/null @@ -1,50 +0,0 @@ -# Data Requirements - -This document describes the specifications that any new data (CSVs) **must** have in order to be imported properly. - - -## CDEsMetadata Requirements: - -The CDEsMetadata (Common Data Elements Metadata) is a json file that is used to define the type of the variables inside each csv files. - -The metadata file **must** follow these rules: -* It **must** follow a tree structure. The `variables` lists are the leafs and the `groups` lists are the branches where one or more `variables` lists can exist. -* A `variable` inside the `variables` list **must** have these fields: - * **code** (Variable name) - * **isCategorical** (true/false) - * **sql_type** (TEXT, REAL, INT) - * **type** (nominal, real, integer, text) -* It can also contain: - * **min** (Integer) - * **max** (Integer) - * **enumerations** (List of codes) -* The `dataset` CDE is required. -* In the parent dictionary there will be a `version` property, of string format. - -An example can be seen at [CDEs Medatadata](../data/dementia/CDEsMetadata.json). - -After adding the CDEsMetadata file you can add your data the same way as adding **New Data on existing Pathology**. - -### Longitudinal CDEsMetadata Requirements: - -The metadata for the longitudinal analysis pathologies, in addition to the previous, will have the following constraints: -* A pathology will be marked as `longitudinal` with a property in the metadata, similar to the `version`. -* The `subjectID` CDE will be required. -* The `visitID` CDE will be required. -* The `visitID` will be categorical, with enumerations `BL`, `FL1`, `FL2` … - -**Important:** The `subjectID` and `visitID` columns must always exist and be filled, in the csvs. - - -## CSV File Requirements: - -A csv file **must** follow these rules: -* The csv file **must** contain at least one row with the variable names (CDEs), like a header, corresponding to the rest of the rows. -* All the column names that exist in the csvs **must** also exist in the CDEsMetadata.json file. The csv can have less columns than the CDEs in the metadata but **NOT** more. -* The `dataset` CDE is **required** to exist in the CDEsMetadata.json and as a column in the csvs. - -## New Pathology - -If you want to add a new pathology on MIP then you need to create a new folder inside the `data` directory with the name of your pathology. Inside that folder you need to add: -* The CDEsMetadata.json file -* and the CSVs containing the data. \ No newline at end of file diff --git a/documentation/MIP_Data_management_documentation.md b/documentation/MIP_Data_management_documentation.md index 76a16ac..34b839d 100644 --- a/documentation/MIP_Data_management_documentation.md +++ b/documentation/MIP_Data_management_documentation.md @@ -1,46 +1,14 @@ - -# **The Medical Informatics Platform (MIP)** - -# **Data Management Guideline** - -**Author (MIP Support Team)** - -**Version 5.0** - -**2023 ****09 19****** - -# Acronyms - -| HBP | Human Brain Project | -|:--- |:--------------------------------------------------------------------------------------------| -| MIP | Medical Informatics Platform | -| CHUV | Centre Hospitalier Universitaire Vaudois | -| AUEB | Athens University of Economics and Business | -| EHR | Electronic Health Record | -| SQL | Structured Query Language | -| CDEs | Common Data Elements | -| JSON | JavaScript Object Notation; is open-standard file format or data interchange format | -| CSV | Comma-separated values file; is a delimited text file that uses a comma to separate values. | -| MRI | Magnetic Resonance Imaging | -| DICOM | Digital Imaging and Communications in Medicine | -| NIFTI | Neuroimaging Informatics Technology Initiative | -| DQC-Tool | Data Quality Control Tools | -| GUI | Graphical User Interface | -| DC | Data Catalog | +# **Data Management Guideline for the Medical Informatics Platform (MIP)** # **Table of Contents** -* [Acronyms](#acronyms) - * [About this Document](#about-this-document) * [Target Audience](#target-audience) * [Document Overview](#document-overview) -* [Concepts and Definitions](#concepts-and-definitions) - -* [The MIP](#the-mip) + * [Concepts and Definitions](#concepts-and-definitions) * [Data Management and Processing](#data-management-and-processing) @@ -59,7 +27,7 @@ This document is targeting data processors and members of data controllers from The document provides the information needed to understand: - The approach of data processing to achieve data ingestion to the MIP. -- The tools that have been developed to facilitate the data processing (The MIP Data Catalogue, the MRI parallel neuromorphometrics pipeline, and the MIP Data Quality Control Tool (MIP-DQC Tool). +- The tools that have been developed to facilitate the data processing (The MIP Data Catalogue, and the MIP Data Quality Control Tool (MIP-DQC Tool). - Link to the user guide for each of these tools ## Document Overview @@ -68,7 +36,7 @@ This document describes the MIP Data architecture, and for which purpose it was It is a step-by-step guide covering data end to end processing and it provides links to additional documentation and detailed instructions described in the different GitHub repositories and shall not be seen in isolation. -# Concepts and Definitions +## Concepts and Definitions **Common Data Elements (CDEs)** @@ -86,31 +54,10 @@ Data Model (Metadata) describes the structure of database variables found in spe A variable or scalar is a storage address (identified by an index or address) paired with an associated symbolic name, which contains some known or unknown quantity of information referred to as a value. [Knuth, Donald (1997). The Art of Computer Programming. 1 (3rd ed.). Reading, Massachusetts: Addison-Wesley. p. 3-4.] -**EBRAINS** - -EBRAINS is a new digital research infrastructure, created by the EU-funded Human Brain Project, that gathers an extensive range of data and tools for brain-related research. EBRAINS will capitalize on the work performed by the Human Brain Project teams in digital neuroscience, brain medicine, and brain-inspired technology and will take it to the next level. - **Electronic Health Records (EHR)** Health information and clinical records registered per each patient per visit in the hospital's database (Oracle, SQL or any database systems) and usually transferred in db or CSV format. EHR usually contain different levels of data; we might define them in this context as spaces, domain, and sub-domain. For example, General space might include demographic, social status or patient's medical history as different data domains. On the other hand, EHR contain other data spaces related to the specific medical condition such as Dementia or Epilepsy where each space includes specific domain and sub-domain, such as medical assessments and tests, diagnoses, treatment, and operations, etc. -**Medical Conditions** - -Diseases are often known to be medical conditions that are associated with specific symptoms and signs. ["Disease" at Dorland's Medical Dictionary] - -# The MIP - -The [Medical Informatics Platform](https://mip.ebrains.eu/) (MIP) for the [Human Brain Project](https://www.humanbrainproject.eu/) is an innovative platform that provides an interface for various investigators (_i.e._ clinicians, neuroscientists, epidemiologists, researchers, health managers) to access, explore and analyse anonymised medical data that are locked and hosted in their original hospital or research centre. Data exploration and analysis can then be performed without moving the data from the hospital where data reside, and hence without infringing on patient privacy. - -The MIP has been designed adopting a 2-tier architecture, which guarantees hospital data protection and privacy by design. The MIP offers two main solutions (see diagram below): - -- The MIP-Local Node contains data that can only be accessed and analysed by the Local Data Manager and its accredited staff from the hospital. -- The MIP-Federated Node contains anonymized data and can be connected to other MIP-Federated Nodes located in other hospitals when the decision is made to join the MIP Network (medical condition-based federations of hospitals) and to provide access to hospital data to members of the network. - -Upon signed agreements (between the data provider`s institution and the CHUV), and admission to the MIP Network, accredited researchers can query multiple MIP federated Nodes and obtain aggregated results. Queries of the MIP-Federated Nodes does not allow to copy, download, or upload any data, nor to see individual patient's data. - -![data governance](./images/The%20MIP%20Data%20Governance%20flow.jpg) - # Data Management and Processing Data originating from diverse hospitals are highly heterogeneous in nature and hence cannot be uploaded into the MIP _per se_. Multiple steps need to be done that are represented in the following figure and which can be regrouped in two main groups: @@ -120,39 +67,33 @@ Data originating from diverse hospitals are highly heterogeneous in nature and h The standard format of the dataset file is .csv UTF-8. Because csv format doesn't allow to keep the Metadata structure, a second file is required, which corresponds to the CDEs for which the MIP standard format is JSON. -To help data managers to go through these processes, three main tools have been developed until now: +To help data managers to go through these processes, two main tools have been developed until now: -- **The MIP Data Catalogue** : Web-based application that is used to explore and manage MIP Data Models and CDEs pre-defined by the federation's community or the use case group. All the existing Medical Conditions, Federations and Hospitals and their associated Common Data Elements (CDEs) are available within the MIP Data Catalogue, which is the unique source of reference for the MIP and the MIP-DQC Tool. The MIP Data Catalogue can be accessed at the following address ([HERE](https://datacatalogue.hbpmip.link/)). _It is recommended to use any browser except Firefox._ -- **The MRI parallel neuromorphometrics pipeline:** A python wrapper to run the Statistical Parametric Mapping SPM12 pipeline in parallel with Matlab over multiple CPU cores. -- **The MIP Data Quality Control Tool (MIP-DQC Tool)**: A standalone software that provides hospital personnel an easy way to explore, transform and validate their dataset based on the CDEs before uploading them into the MIP. MIP-DQC Tool has both, a Command Line Interface (CLI) and a Graphical User Interface (GUI), with only the latter having the full set of functionalities. The MIP-DQC Tool GUI version has the following functionalities: +- **The MIP Data Catalogue** : Web-based application that is used to explore and manage MIP CDEs pre-defined by the federation's community or the use case group. All the existing Medical Conditions, Federations and their associated Common Data Elements (CDEs) are available within the MIP Data Catalogue, which is the unique source of reference for the MIP and the MIP-DQC Tool. The MIP Data Catalogue can be accessed at the following address ([HERE](https://datacatalogue.mip.ebrains.eu/#federations)). +- **The MIP Data Quality Control Tool (MIP-DQC Tool)**: A standalone software that provides hospital personnel an easy way to explore, transform and validate their dataset based on the CDEs before uploading them into the MIP. MIP-DQC Tool has both, a Command Line Interface (CLI) and a Graphical User Interface (GUI), with only the latter having the full set of functionalities. The MIP-DQC Tool GUI version has the following main functionalities: - Inference of a dataset's schema and producing a schema file in Frictionless json or the MIP Data Catalogue's excel format. - Validating the MIP Data Models or CDEs pre-defined by the federation's community or the use case group. - Translating the MIP Data Models or CDEs from xlsx format to json format. - Validating the hospital tabular (csv) data against the MIP Data Models or CDEs and producing a validation report and some overall statistics about the data. - Data cleaning capability based on the previously performed validation report. - - Designing and performing schema mapping of an incoming hospital dataset to a certain Pathology's Common Data Element (CDE) schema. - - Producing a DICOM MRI validation and statistical report, based on the meta-data headers. - - The tabular (csv) data validation functionality has the option of downloading the pathology`s CDE metadata directly from the MIP Data Catalogue's API. Please note that this option is not available in the CLI version. - - The schema mapping functionality is performed by the MIPMap engine packaged in a Docker container, which runs in the background. Please note that this option is not available in the CLI version. The user guide for each of these three tools can be found in the respective GitHub repositories. - - [The MIP Data Catalogue](https://datacatalogue.hbpmip.link/) + - [The MIP Data Catalogue](https://datacatalogue.mip.ebrains.eu/#federations) - [The MIP Data Quality Control Tool](https://github.com/HBPMedical/DataQualityControlTool/wiki) - - [The MRI parallel neuromorphometrics pipeline](https://github.com/HBPMedical/mri-parallel-nmm-pipeline) Through description of the steps required before being able to upload data on the MIP, the information of which tool can be used is provided in italics. The following table lists the sequential steps as well as the tool that can be used to simplify this process. ***Table 1*: MIP Data extraction and compilation and available Tools** || **1srt step** | **2nd step (optional)** | **3rd step (optional)** | | --- | --- | --- | --- | -| **Tasks** | Extract the clinical data as a csv file | Extract the brain volumes from the nifti to the csv file | Merge the brain volumes with the clinical data | -| **Tools** | | MRI parallel neuromorphometrics pipeline | | +| **Tasks** | Extract the clinical variable as a csv file | Extract relevant variable from medical images | Merge those data to obtain one csv file | +| **Tools** | | | | ***Table 2*: MIP Common Data Element definition and available Tools** -|| **1srt step** | **2nd step** | **3rd step (optional)** | **4th step** | -| --- | --- | --- | --- | --- | -| **Tasks** |Infer the data schema from the csv file to produce the Data Model xlsx file | For the federation, there should be a discussion around the standardization of variables in the Data Model to produce a standard Common Data Element (CDE, or Data Model) as an xlsx file | The MIP metadata dictionary can be used to standardize the variables across all medical conditions | Validate and translate the CDEs xlsx file to JSON format | -| **Tools** | MIP-DQC Tool | | MIP-DQC Tool | MIP-DQC Tool & MIP Data Catalogue | +|| **1srt step** | **2nd step** | **3rd step** | +| --- | --- | --- | --- | +| **Tasks** |Infer the data model from the csv file to produce the Data Model xlsx file | For the federation, there should be a discussion around the standardization of variables in the Data Model to produce a standard Common Data Element (CDE) as an xlsx file | Validate and translate the CDEs xlsx file to JSON format | +| **Tools** | MIP-DQC Tool | | MIP-DQC Tool & MIP Data Catalogue | ***Table 3*: MIP Data cleaning, validation, and upload and available Tools** @@ -161,36 +102,36 @@ Through description of the steps required before being able to upload data on th | **Tasks** | Check the dataset csv file quality against the CDEs | Perform the data cleaning and validation | Upload of the dataset csv file on the local node | | **Tools** | MIP-DQC Tool | MIP-DQC Tool | -![The MIP data flow](./images/The%20MIP%20Data%20flow.jpg) +![The MIP data flow](./images/The%20MIP%20Data%20flow.png) ## Extraction and compilation of clinical data as a csv file ### Extract variables from the Electronic Health Record (EHR) system -A dataset that can be uploaded on the local node is a unique csv file that contains all the variables. The first step is hence to extract the clinical data of interest from the EHR system to a csv file. In most cases, a hospital EHR system contains various examination results that a patient obtains during one or more hospital visits. To keep this information, the csv file must contain: +The dataset that can be uploaded on the local node is a unique csv file that contains all the variables. The first step is hence to extract the clinical data of interest from the EHR system to a csv file. In most cases, a hospital EHR system contains various examination results that a patient obtains during one or more hospital visits. -- a field with a unique ID for each PATIENT -- a field with a unique ID for each VISIT -- a "Number of month since first visit" field for each VISIT +In case of longitudinal dataset, the csv file must contain: + +- a field with a unique ID for each PATIENT: `subjectID` +- a field with a unique ID for each VISIT (BL, FL1, FL2...): `visitID` +- a "Number of month since first visit" field for each VISIT: `timeBL` In terms of csv file structure, the following basic conditions must be met: -- Header (variable) names must: +- Header that contains the variable names and must: - not contain special characters like space, parentheses, hyphen, etc. - use only underscore ("\_") for word separation. - not start with any number character. -- The delimiter must be the comma character (","). +- The csv delimiter must be the comma character (","). - The encoding must be ASCII. -### Extract Imaging Features from MRIs - OPTIONAL - -Not all medical conditions require data extracted from Brain Scans (or MRIs) which explains why this step can be skipped. However, because the MIP recognizes only one csv data format, features from MRI images must be extracted and merged to the csv file that contains EHR variables in case they are valuable for the scientific analysis objectives. +### Extract Medical Images Features - OPTIONAL -Although multiple methodologies can be used to extract those features, we highly recommend following the Statistical Parametric Mapping SPM12 pipeline to harmonize the methodology across the hospitals. +Not all medical conditions require data extracted from medical images which explains why this step can be skipped. However, because the MIP recognizes only one csv data format, features from medical images must be extracted and merged to the csv file that contains EHR variables in case they are valuable for the scientific analysis objectives. -*To simplify the process of feature extraction, you can use the **MRI parallel neuromorphometrics pipeline** that was deployed by the MIP data management team. It is a python wrapper to run the Statistical Parametric Mapping SPM12 pipeline in parallel with Matlab over multiple CPU cores.* +Although multiple methodologies can be used to extract medical images features, we highly recommend following the same extraction pipeline to harmonize the methodology across the hospitals. -As soon as all variables are merged in a unique csv file, you can proceed with the definition of the Common Data Elements for your medical condition and your scientific analysis objectives. +As soon as all variables are merged in a unique csv file, you can proceed with the definition of the Data Model and the Common Data Elements for your medical condition and your scientific analysis objectives. ## Definition of Common Data Elements for a medical condition @@ -202,11 +143,18 @@ The Data Model xlsx file contains information about the hospital variables. It c The descriptions required are listed below; in **bold** the ones that cannot be left empty: -- **name** : The full name of the variable -- **code** : The code version of the variable name -- **type** : Format of the variable. Four formats are recognized in the MIP – nominal, date, real and integer. -- **values** : Only for the nominal format. It lists all the levels of the nominal variable with the correspondence between the code and the value. The format used is - {"Code","Value"},{"Code","Value"}. For example, the sex variable is coded as: {"M","Male"},{"F","Female"}. -- unit: Only for real and integer. Unit of the variable (_e.g.,_ km/h) +- **name** : The full name of the variable that will appear on the MIP interface +- **code** : The code version of the variable name that is used on the csv file of the dataset +- **type** : Format of the variable. Five formats are recognized in the MIP: + - Categorical variable: + - Nominal - multiple level without hierarchy (Handedness for exemple) + - Ordinal - multiple level with a possible hierarchy (graduate stage of the disease for exemple, be carefull that the difference between each steps are similar) + - Binary - two level without hierarchy (Presence/Absence variable for exemple) + - Numerical: + - Integer - Discrete (Age in year for exemple) + - Real - Continuous (Glucose concentration for exemple) +- **values** : Only for the categorical variable. It lists all the levels of the categorical variable with the correspondence between the code and the value. The format used is - {"Code","Value"},{"Code","Value"}. For example, the sex variable is coded as: {"1","Male"},{"2","Female"}. Note that to increase the analitic the code should be integer. +- unit: Only for numerical variables. Unit of the variable (_e.g.,_ km/h) - canBeNull: Mention with yes/no the possibility of this variable to be null. It is mainly useful for MIP federated nodes. - description: A precise description of the variable. - comments: Any comment. For example, a linkage with another variable. @@ -215,7 +163,7 @@ The descriptions required are listed below; in **bold** the ones that cannot be !NOTE : Although the semantics and the conceptPath can be customized to a specific Data Model, we highly recommend following existing Standards such as HL7 FHIR, SNOMED or ICD-10 -*The **Data Quality Control Tool** allows you to infer this Data Model xlsx file based on your data csv file. In the infer option section of the Data Quality Control Tool, indicate the number of rows that the tool will be based on for the schema inference and the maximum number of categories that a nominal variable can have. You can also include the MIP metadata dictionary to infer the global conceptPath directly for your data.* +*The **Data Quality Control Tool** allows you to infer this Data Model xlsx file based on your data csv file. In the infer option section of the Data Quality Control Tool, indicate the number of rows that the tool will be based on for the schema inference and the maximum number of categories that a nominal variable can have.* ### Common Data Elements @@ -223,55 +171,40 @@ Based on the Data Model, a hospital selects the variables that will be made avai In case of a MIP-Federated node, three more steps are required: -- Each variable that could lead to identification of a patient must be deleted (i.e., name, date of birth, address…) and must be replaced by a patient code. Only the data controller has access to the table of correspondence code/patient identity. +- Each variable that could lead to identification of a patient must be deleted (i.e., name, date of birth, address…) and must be replaced by a patient code. - Each hospital that is part of the federation needs to agree on the list of the variables AND on the harmonization of each variable. Indeed, variables from each hospital need to have the exact same codification and meaning for nominal variables and, should have been acquired with a similar methodology to be federated. Here is a list of discussion points: - Agreement on the Variable list - - For each nominal variable, agreement of codification and meaning of this codification - - For each real and integer variable, agreement of the unit and range + - For each categorical variable, agreement of codification and meaning of this codification + - For each numerical variable, agreement of the unit and range - For each variable, agreement of the methodology followed to acquire it -- Add a nominal variable named "Dataset" that have a level for each hospital federated. A code for each hospital needs to be created (e.g. Centre Hospitalier Universitaire Vaudois is coded chuv and is written as {"chuv"," Centre Hospitalier Universitaire Vaudois "} +- Add a nominal variable named `Dataset` that have a level for each hospital federated. A code for each hospital needs to be created (e.g. Centre Hospitalier Universitaire Vaudois is coded `1` and is written as {"1"," Centre Hospitalier Universitaire Vaudois "} -As soon as the CDE is defined, and in the same xlsx format as the Data Model, it can be translated to the json format that is the standard format of the MIP. To do so, you can use the Data Quality Control Tool to have first a validation control to avoid any mistake and second translate it to json format. If you are part of a federation you can do it through the Data Catalogue. +As soon as the CDE is defined, and in the same xlsx format as the Data Model, it can be translated to the json format that is the standard format of the MIP. To do so, you can use the Data Quality Control Tool to have first a validation control to avoid any mistake and second translate it to json format. *The **Data Quality Control Tool** allows you to validate your Data Model or CDEs to avoid missing cells, extra space etc… . In the Validate DC Excel section of the Data Quality Control Tool, upload your Data Model or CDEs in xlsx format and click on Validate. In case there is no mistake, you can translate it automatically to json format. This translation is also made by **the Data Catalogue** web-based application.* Once the CDEs have been defined, the process of preparing hospital data for ingestion to the MIP can be started. -!NOTE: In case a hospital wants to join an existing federation, it should download the pre-defined CDEs from the Data Catalogue web-based application and contact the federation leader to be able to join the federation (see already existing federations and medical conditions available on the MIP in the following sub-sections of this document). - -### Supported Federations and their associated Medical Conditions as of September 2023 - -Each CDE associated with a Medical Condition within a federation of hospitals is under the responsibility of a "Federation Leader". The role of the "Federation Leader" is to ensure that the CDE of a Medical Condition evolves over time in a coordinated and versioned manner. The table below gives an initial view of all existing Medical Conditions with the corresponding versions of CDEs and the leader contact in use as of February 2023. For updated information about the available defined medical conditions, please refer to the Data Catalogue web-based application. - - ***Table 4*: MIP-Federated node** - -| **Medical Conditions** | **Leader contact** | **Hospital** | **CDE version** | -| :--- | :--- | :--- | :--- | -| Dementia | Dr Thibaud Lebouvier (from 1 November 2021) [Thibaud.LEBOUVIER@chu-lille.fr](mailto:Thibaud.LEBOUVIER@chu-lille.fr)| CHRU Lille | V5 | -| Traumatic Brain Injury (TBI) | Dottore Guido Bertolini [guido.bertolini@marionegri.it](mailto:guido.bertolini@marionegri.it) Stefano Finazzi [stefano.finazzi@marionegri.it](mailto:stefano.finazzi@marionegri.it) | Hospedale Mario Negri, Bergamo, Italy | V8 | -| Mental Health | Prof. Pegah Sarkheil [psarkheil@ukaachen.de](mailto:psarkheil@ukaachen.de) | University Hospital RWTH, Aachen | V8 | -| Epilepsy | Prof. Philippe Ryvlin [Philippe.Ryvlin@chuv.ch](mailto:Philippe.Ryvlin@chuv.ch) | Neurosciences Department, CHUV, Lausanne | V9 | -| NeuroNet | Lewis Killin [lkillin@synapse-managers.com](mailto:lkillin@synapse-managers.com) | SYNAPSE Research Management Partners S.L. | V1 | -| Stroke | Prof. Valeria Caso [vcaso@hotmail.com](mailto:vcaso@hotmail.com) & Georgios Tsivgoulis [tsivgoulisgiorg@yahoo.gr](mailto:tsivgoulisgiorg@yahoo.gr) | Azienda Ospedaliera Di Perugia & National and Kapodistrian University of Athens | V1 | +!NOTE: In case a hospital wants to join an existing federation, it should download the pre-defined CDEs from the Data Catalogue web-based application and contact the MIP team support to be able to join the federation. ## Data cleaning, validation, and upload ### Check of the dataset quality against the CDEs -Based on the CDEs defined is the previous steps, the dataset compiled as a csv, needs to be cleaned/modified and validated according to the definition of all variables. The two main violation types that need to be fixed are: +Based on the CDEs defined is the previous steps, the dataset, compiled as a csv, needs to be cleaned/modified and validated according to the definition of all variables. All the column names that exist in the csvs **must** also exist in the CDEsMetadata.json file. The two main violation types that need to be fixed are: - Constraint violations which regroup: - - minimum (for date, integer, numerical) - - maximum (for date, integer, numerical) - - enum (list of enumerations for nominal datatypes) + - minimum (for numerical variable) + - maximum (for numerical variable) + - enum (list of enumerations for categorical variable) - Datatype violations: Case when a value in a column has a different datatype than the one that has been declared for that variable in the CDEs. This also includes the empty cells that should be truly empty and not with "NA", "Nas", "unknowns" etc… ### Perform the data cleaning and validation -All violations highlighted before should be corrected so that each variable present in the csv file, corresponds to the one listed in the CDEs. Be careful to save the csv file in the correct format, which is csv; encoding must be ASCII with comma separation. +All violations highlighted before should be corrected so that each variable present in the csv file, corresponds to the one listed in the CDEs. The csv can have less columns than the CDEs in the metadata but **NOT** more. Be careful to save the csv file in the correct format, which is csv; encoding must be ASCII with comma separation. ### Data set upload To become visible to the MIP end users, the dataset csv must be first renamed as the hospital code defined on the Dataset variable created on the CDE (e.g., "chuv.csv"). The file is then copied to the corresponding "medical condition" folder on the MIP local node and, when the hospital is part of a federation (federated hospital), to the corresponding "medical condition" folder on the MIP federated node. -*The **Data Quality Control Tool** allows you to compare the CDEs json file with your dataset csv file to highlight the violations and suggests possible corrections. If the corrections suggestions are good, the DQC tool can also perform those corrections and save the corrected dataset in the correct format. In case you want to join an existing federation, the DQC tool allows you to perform a data mapping, which transforms the hospital's local variables to a set of variables of a target CDEs medical condition.* +*The **Data Quality Control Tool** allows you to compare the CDEs json file with your dataset csv file to highlight the violations and suggests possible corrections. If the corrections suggestions are good, the DQC tool can also perform those corrections and save the corrected dataset in the correct format.* diff --git a/documentation/deployment-pack/README.md b/documentation/deployment-pack/README.md deleted file mode 100644 index 82e4dfa..0000000 --- a/documentation/deployment-pack/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# MIP Deployment Pack - -The installation prerequisites, hardware requirements and other considerations prior to deploying and running the MIP can be found at: - -- [Deployment Pack (PDF)](../legal-docs/MIP_Executive_Summary_V02.00_7960b8432e.pdf) -- [Installation and License Agreement (PDF)](../legal-docs/MIP_Installation_and_License_Agreement_-_V02.10_210c5d6538.pdf) -- [Installation Prerequisites and Installation Guide](./install-prerequisites.md) -- [Minimal Worker Setup (Tailscale-only)](./minimal-setup.md) -- [Ethics and Legal Requirements](./ethics-legal.md) -- [FAQ](./faq.md) - - -- [Data Sharing and Processing Agreement](../legal-docs/Data_Sharing_and_Processing_Agreement_June2021_v2.pdf) -- [Data Transfer Agreement](../legal-docs/DTA_EBRAINS_June2021_v2.pdf) -- [MIP Service Agreement and Software Licences](../legal-docs/MIP_Service_Agreement_and_Software_Licenses_Dec21.pdf) diff --git a/documentation/deployment-pack/ethics-legal.md b/documentation/deployment-pack/ethics-legal.md deleted file mode 100644 index e30350a..0000000 --- a/documentation/deployment-pack/ethics-legal.md +++ /dev/null @@ -1,266 +0,0 @@ -# Ethics and Legal Requirements - -Purpose -------- - -> The present document outlines the Ethics and Legal requirements and -> responsibilities related to the deployment of the Medical Informatics -> Platform (MIP) into hospitals participating to the MIP network. - -Introduction ------------- - -> MIP relies on citizens and patients allowing researcher to use their -> private personal medical data. -> -> MIP is a platform designed to enable large scale, privacy preserving -> data sharing for research purpose. It is the responsibility of the -> hospitals to make sure that their data subjects and patients have -> given their consent for the collection of the data. It is also the -> responsibility of the hospitals to ensure that this data has been -> properly pseudonymized / anonymized according to the standards and the -> recommendations of the MIP deployment team. - -Data Collection ---------------- - -> Data is not collected in the specific purpose of the MIP. It is -> collected in the course of the patient's health care or for research -> projects and can be further processed and shared using the MIP. - -Consent -------- - -> 'Consent' of the data subject means any freely given, specific, -> informed and unambiguous indication of the data subject's wishes by -> which he or she, by a statement or by a clear affirmative action, -> signifies agreement to the processing of personal data relating to him -> or her (see Article 7 of the GDPR); -> -> In any case, hospitals are the only responsible to obtain informed -> consent from patients whose data are going to be analyzed. Withdrawal -> of consent also takes place at the partner's local hospital. -> -> A record of each consent is kept by the hospital to demonstrate that -> explicit consent has been obtained. -> -> HBP research participation consent is not 'bundled' with medical -> treatment and participating hospitals make it clear to patients that -> the use of their medical data for the secondary research is optional - -Usage of data in the MIP -------------------------- - -> Concerning entering retrospective data in the Federated MIP, there is -> no need for specific consent to reuse data if performing federated -> analysis was already defined as an objective within the frame of a -> clinical study. In all other cases, the signature of an informed -> consent allowing the re-use of patient's data must be obtained (see -> Figure 1A). -> -> In order to collect prospective data to be analysed using the -> Federated MIP, a new research protocol has to be submitted to the -> corresponding EC proposing to elaborate a multicentre cohort study -> justifying the use of the MIP as the tool needed to federate -> distributed cohorts (see Figure 1B). - -### - -### - -### - -*image* - -### - -Pseudoanonymisation - Anonymisation ------------------------------------ - -> **Pseudonymization** -> -> *According to GDPR Art. 4c* -> -> *'pseudonymisation' means the processing of personal data in such a -> manner that the personal data can no longer be attributed to a -> specific data subject without the use of additional information, -> provided that such additional information is kept separately and is -> subject to technical and organisational measures to ensure that the -> personal data are not attributed to an identified or identifiable -> natural person;* -> -> **Anonymization** -> -> *According to the Data Policy Manual of the HBP* -> -> *"Anonymous data: Information which does not relate to an identified -> or identifiable natural person." According to the Swiss Federal Act on -> research involving Human Beings, "Anonymised biological material and -> anonymised health-related data means biological material and -> health-related data which cannot (without disproportionate effort) be -> traced to a specific person;"* -> -> **Application of the GDPR** -> -> The GDPR only applies to personal data or information concerning an -> identified or identifiable natural per- son. If data are anonymised, -> it is no longer considered to be personal and is thus outside the -> scope of GDPR application. In other words, if data accessible in the -> MIP are anonymous, the GDPR does not apply and the data can be -> processed for research purposes without the restrictions of data -> protection law. However, given the difficulty in creating truly -> anonymous data, the bar for anonymisation has been set extremely high -> under EU data protection law. -> -> Regardless of the technique applied (e.g. addition, permutation, -> differential privacy, aggregation, k-anonymity, l-diversity, -> t-closeness, etc.), three main questions should be considered: - -1. Is it still possible to single out an individual? - -2. Is it still possible to link records relating to an individual? - -3. Can information be inferred concerning an individual? - -> On a general basis, data stored on the MIP local will be pseudonymised -> and is attributable to a natural person by the use of additional -> information, which is securely stored using both organizational and -> technical security measures. -> -> Data on the MIP federated node will be anonymized. Taking into account -> all the means reasonably likely to be used for identification, data -> subjects cannot be identified through research data available at this -> level. -> -> Data Providers are responsible for the pseudonymization and the -> anonymization of their data, based on the requirements provided by the -> MIP Deployment team. -> -> All the identifiers are removed or coded and the patient record -> receives a unique encrypted identifier when it is stored on the MIP -> Local server. The look-up table is stored on a different server in the -> hospital level 3 "clinical area" which is not accessible from the -> outside. -> -> A description of the anonymization process is available in the Data -> Processing User Guide. - -Data Privacy Levels -------------------- - -**Level 3 - Data stored in hospital's clinical data storage systems -(EHR, PACS)** - -- Contains Personal Health Identifiers (PHI) - -- Raw data, including full brain images that enable reconstructing the - patient's face, diagnostics and longitudinal information with exact - dates - -- High risk of unauthorized identification - -- General regulatory requirements: Cannot be shared publicly, must be - protected from any un- authorized access. - -- **MIP policy: Such data are not accessible through the MIP** - -**Level 2 - Pseudonymised data stored in MIP local** - -- No Personal Health Identifiers (PHI). - -- Neuroimaging data are being processed in order to deface them in the - case images are shared, or to extract features such as brain - volumes. - -- Medium to Low (from Raw to features) risk of unauthorized - re-identification: identity can be recovered from a lookup table - secured and password protected in a hospital server distinct from - where the pseudonymised data are stored. In hospitals, the look-up - table is stored on the level 3. - -- General regulatory requirements: Can be shared by authorized - investigators provided ethics approval and patient's informed - consent whenever appropriate, but cannot be shared publicly and must - be protected from any unauthorised access. - -- **MIP policy: Such data will be only accessible through the MIP - local by the data provider and his local authorized staff.** - -**Level 1 - Anonymized data stored in MIP federate nodes** - -- Anonymization with no lookup table - -- Only features data obtained after pre-processing of raw data (no - available image that would allow reconstructing the face) - -- Very low risk of unauthorized re-identification: identity cannot be - recovered from a lookup table. Most features do not contain enough - information to find directly or by cross-references the identity of - an individual. - -- General regulatory requirements: Can be shared by authorized - investigators. Must be protected from any unauthorised access. - -- **MIP policy: Such data cannot be explored at the individual - level.** Data are made available for aggregated queries only within - the MIP federate network to investigators authorized by the MIP Data - Governance Steering Committee. Data will not be shared publicly, - must be protected from any unauthorised access. - -**Level 0 - Anonymized data aggregates transmitted to MIP central** - -- Same as above with the following additional features: - -- Only aggregated data (minimum values are set to the algorithms to - ensure there is no singling out) - -- **MIP policy: Data will be made available for aggregated queries - only to any MIP registered users** - -- No data storage happens at level 0, only per-call data queries of - the results - -*image* - -*image* - -> Please refer to the Executive Summary and the MIP Installation -> Prerequisites and Installation Guide for details of the MIP technical -> architecture to support these requirements. - -Data Management Oversight & Governance --------------------------------------- - -*image* - -Basis and Reference -------------------- - -- MIP is complying with the GDPR with special consideration to Privacy - by Design and Privacy by Default. Legislation and Guidance - -- REGULATION (EU) 2016/679 of the European Parliament and of the - Council of 27 April 2016, on the protection of natural persons with - regard to the processing of personal data and on the free movement - of such data, and repealing Directive 95/46/EC (General Data - Protection Regulation)- Applies from May 2018 - -- Directive 95/46/EC of the European Parliament and of the Council of - 24 October 1995 on the protection of individuals with regard to the - processing of personal data and on the free movement of such data, - OJ L 281, 31--50 (henceforth '95/46/EC' or 'the Directive'). - -- Working Party 29 'Opinion 216 05/2014 on Anonymisation - Techniques' (2014) 5 ('WP29 216'). - -- Federal Act on Research involving Human Beings (Human Research Act, - HRA) of 30 September 2011 (Status as of 1 January 2014) - - Contact: --------- - -> Erika BORCEL: - -*** - -*** diff --git a/documentation/deployment-pack/executive-summary.md b/documentation/deployment-pack/executive-summary.md deleted file mode 100644 index f74c18f..0000000 --- a/documentation/deployment-pack/executive-summary.md +++ /dev/null @@ -1,216 +0,0 @@ -Introduction to the MIP -======================= - -The main objective of this document is to provide the reader (hospital -or health organization director and decision maker) with an introduction -to the MIP and its capabilities, a description of the benefits that the -MIP offers in terms of research capabilities once deployed within the -organization, and finally outlines how the organization, working with -the MIP deployment and support team, shall proceed to deploy the MIP on -premise. - -Introduction ------------- - -Brain diseases and related medical conditions, considered as a whole, -affect 165 million European citizens, many whom are being at least -partly managed in hospitals. The clinical data collected from these -patients represent a unique source of information for better -understanding and treating brain diseases and related medical conditions -but are unfortunately not usually available for research. - -The MIP has been developed by the Human Brain Project, an EU Horizon -2020 Flagship project, in order to facilitate access to clinical data -stored in hospitals for research purpose, while preserving data privacy. -The MIP aims at enabling breakthrough medical progress in the field of -brain diseases and related medical conditions through access to an -unprecedented volume of patient's data. - -MIP Description -=============== - -The MIP is an innovative data analysis and data collection system that -provides an interface for various investigators (clinicians, -neuroscientists, epidemiologists, researchers, health managers) enabling -them to access and analyze anonymized medical data currently locked in -hospital, research centers and public databases, without moving the data -from the hospital where they reside, and without infringing on patient -privacy. - -The MIP is designed to help clinicians and researchers aiming to adopt -advanced analytics for diagnosis and research in clinics and to promote -collaborative neuroscience research using hospital data. - -MIP 2-Tier Architecture ------------------------ - -The MIP has been designed adopting a 2-tier architecture which -guarantees total hospital data protection and privacy by design. The MIP -offers two main solutions, the MIP-Local and the MIP-Federated -instances. - -The MIP-Local Node contains anonymized data that can only be -accessed and analyzed by the Local Data Manager and its accredited staff -from within the hospital. - -The MIP-Federated Node contains anonymized data and can be connected to -other MIP-Federated Nodes located in other hospitals when the decision -is made to join the MIP Network (medical condition-based federations of -hospitals) and to provide access to hospital data to members of the -network. - -Upon signed agreements (between the data provider`s institution and the CHUV), and admission to the MIP -Network, accredited researchers can query multiple MIP federated Nodes -and obtain aggregated results. Queries of the MIP-Federated Nodes does -not allow to copy or upload any data, nor to see individual patient's -data. - -The 2-tier MIP architecture (MIP Local, MIP-Federated) has been designed -in order to address the specific challenges of: - -- local deployment adapted to each hospitals' environment, - -- capturing and processing heterogeneous types of data (e.g. - socio-demographic, clinical, biological and neuroimaging data), - -- fulfilling privacy rules, policies and best practices to enable - efficient and secure data sharing, - -- harmonizing data through Common Data Elements for cross-site - comparisons, and - -- integrating readily available statistical and machine learning - tools. - - -MIP functionalities -=================== - -The MIP, after installation at hospital, provides the main components -and end-user functionalities briefly described here after. - -Functionality of the Medical Informatics Platform ---- - -**Components** | **Description** ----------------|---------------- -Platform | The platform acts as the main entry point to the business offerings of the MIP for researchers and clinicians -API Layer | The API layer offers a protected layer of functionality exposed in a uniform and interoperable manner. It acts as a gateway to the MIP offerings, providing horizontal reuse, vertical specialization and separation of concerns and technological restrictions through the employment of a microservices architecture| -Authentication | The authentication mechanism uses proven standards to allow for a wide range of interoperability between the authenticated clients and the platform services -Operational data | A set of operational data assist in the streamlined communication and interaction between the clients and the MIP services -Data Factory | The Data Factory set of services facilitate the ingestion of hospital data within the MIP -Deployment Stack | The deployment stack of the MIP can be split to facilitate disjoint but complementary deployments - MIP Local offers enhanced services and analytical capabilities within the boundaries of each hospital - MIP Federated offers federated analysis over anonymized data, across multiple hospitals - -**End-user functionality** | **Description** -----------------------------------------|---------------- -Import and analyse local data | Pseudo-anonymise selected local data sets and run experiments using AI algorithms with the MIP -Join a federation/pathology | Elect to submit and analyse data for a particular MIP federation -Select and analyse federation variables | Capability to selectively analyse variables for a particular federation -Save & filter variables | Filter and save selected variable analyses -Run experiments using AI algorithms | Select AI algorithms compatible with selected datasets and variables - - -Key Benefits from being part of the MIP Network ------------------------------------------------ - -Since the beginning of the Human Brain Project (HBP) the MIP has been -developed and installed in an increasing number of participating -hospitals. The Lausanne university hospital (CHUV) is the HBP partner -coordination this activity. - -A few key benefits to expect from joining the MIP network are the -followings: - -- Participate in the largest ever funded Europe wide brain research - initiative, - -- Train and use novel state-of-the-art analytical tools, including - machine learning algorithms, - -- Investigate and discover novel findings from its own data using the - MIP-Local Node - -- Participate or lead Federated analyses on big data available in the - network of MIP-Federated Nodes - -- Develop new scientific collaborations - -- Increase the chance of future successful national or European grant - applications - -MIP Deployment Process -====================== - -The MIP deployment process follows several steps to facilitate the -installation flow and progressively allows the hospital staff to gain -experience along the implementation process (see Figure 2). - -The first 5 steps fall within the scope of installing a local version of -the platform -- the MIP-Local -- which allows researchers to compare -their data with the public databases already included in the MIP: - -1. Identify all relevant hospital staff required to proceed to MIP - deployment and present them the platform and the deployment process. - -2. Secure signature of the MIP Installation Agreement by both the - Hospital and CHUV legal representatives. This agreement covers - installation of the MIP software, but not data sharing. - -3. Prepare the IT infrastructure (e.g. server) needed to install the - MIP-Local according to its specifications. This preparation is - typically performed by the Hospital IT staff with assistance from the CHUV IT team, if needed. - -4. Install the MIP-Local software, usually through a combined effort of - the Hospital IT staff and the MIP-CHUV deployment team. Part of the - installation can be performed remotely through VPN connection, if - required by the Hospital. On site, hospital-specific tuning is often - required. - -5. Capture pseudonymized patients' data from the - Hospital or research department aiming to use the MIP, in the MIP-Local , in full - compliance with all local ethics and regulatory procedures. This - step is undertaken under the full responsibility of the Hospital or - research department and its local Data Coordinator. Once data have - been captured in the MIP-Local, the Local Data Coordinator and his - accredited local staff can use the MIP to analyse their data. No - other stakeholder has access to these data. - -Thus, researchers have the opportunity to test the MIP and apply its -algorithms to their own datasets in combination with public pre-existing -cohorts. In case they want to create a federation together with other -centres, they have to follow the next steps: - -1. Secure the signature of the MIP Data Sharing Agreement by both the - Hospital and CHUV legal representatives. This agreement covers the - possibility to perform federated analyses of fully anonymized data - captured in the MIP-Federated node of the hospital. - -2. Secure participation of the Local Data Coordinator to the relevant - MIP DGSC disease-specific board. - -3. Prepare the IT infrastructure (e.g. server) to install the - MIP-Federated Node software (same procedure as previously described - in step 3). - -4. Install the MIP-Federated Node software (same procedure as - previously described in step 4). - -5. Proceed to full anonymization of the data stored in MIP-Local and - then push these data into the MIP-Federated Node database. This step - is undertaken under the full responsibility of the Local Data - Coordinator. - -6. Perform federated analysis. - -Contacts -======== - -For more information you can contact HBP-MIP Leadership team members: - - - - - -*** - -*** diff --git a/documentation/deployment-pack/faq.md b/documentation/deployment-pack/faq.md deleted file mode 100644 index ad7a916..0000000 --- a/documentation/deployment-pack/faq.md +++ /dev/null @@ -1,154 +0,0 @@ -# FAQ - -ACCESS & AUTHORISATION ---- -1. **Do I need an HBP account to connect to the MIP?** -- You do need HBP credentials to access a federated MIP. For MIP Local, access is granted by the local administrator. - -2. **How can I get my credentials for a MIP federation?** -- Please refer to the MIP user manual. - -GENERAL QUESTIONS ---- -1. **How long has the MIP been working?** -- The MIP has been working since 2015. - -2. **Do we need two independent servers to install the MIP?** -- In order to perform federated analysis, the MIP local and the MIP federated must be installed on 2 separated servers. - -3. **What are the requirements to install the MIP?** -- Please refer to the Installation Prerequisites document. - -4. **Can I install the MIP in my laptop?** -- This is an interesting option that is being tested and may be available in the future. - -TRAINING ---- -1. **Will users have a training on the MIP?** -- A set of training videos is already available from the MIP user interface. A more detailed training can be organised on request. - -2. **Are regular webinars planned in order to show the users the last updates of the system and to allow the centres to ask their questions to the HPB team?** -- Yes, webinars will be organised regularly. - -SERVICES ---- -1. **In case of MIP malfunction, is there a technical service support department?** -- Yes we will provide technical support free of charge. In case of technical problems, please write to , we will reach out to you in the briefest delay. - -2. **When the MIP is installed locally, how will the technical team is going to solve the issues without connecting to the local server?** -- Our engineers will provide you support by phone, email and skype. - -3. **If we are not able to solve the problem, is there any possibility that one of your engineers come help us in situ?** -- Your request will be studied on a case-by-case basis. Normally all cases can be resolved by remote access to your servers by our engineer. - -4. **Will the opinions/questions of users be requested by the HBP as part of a quality assessment process?** -- A Quality Assessment program will be implemented in the following months in order to analyse user's MIP evaluation. - -DATA ---- -1. **Which kind of analysis can we perform with the MIP?** -- Run exploratory data analyses, create and share analysis models, execute descriptive statistics, inferential statistics and machine-learning algorithms on user-defined analysis models on the data reunited from an increasing number of datasets. - -As of February 2026, the following algorithms are available in the system: - - ANOVA - - CART - - Calibration Belt - - ID3 - - k-Means Clustering - - Kaplan-Meier Estimator - - Linear Regression - - Logistic Regression - - Naive Bayes Training - - Naive Bayes with Cross Validation - - Pearson Correlation - - Principal Components analysis - - SVM - - T-Test Independent - - T-Test One-Sample - - T-Test Paired - -> All the analysis on MIP is aggregated and can't be made on an individual level. However, sampling and filtering the data within certain groups is possible. - -2. **Will the dataset we introduced to the MIP local be automatically transferred to the MIP federated or we will have to enter the data again?** -- At the time of joining the MIP federation, the hospital will run the "MIP anonymization" module whose function is to anonymize and copy MIP local data to the server (OR NODE). It is this second dataset (anonymized) that will then be used by the federation. - -3. **Who has to anonymize the data?** -- Data anonymization is supported by the "MIP anonymization module" whose main function is to create a second dataset identical to the source be fully anonymized. Running the module is performed by the hospital personnel. - -4. **Can we share any kind of data or there are already pre-selected variables?** -- There definitely is an existing set of variables that the MIP - knows and recognizes. This data set can be extended hospital by - hospital under certain circumstances. If the new variables are - considered "common" they are added to the common set of variables - the MIP uses and recognizes. If considered as local only then - the variable can be added to the MIP but as a local extension. - Note that the objective is to ensure the set of variables (common - or local) the MIP supports must have a justification. There is a - selection / approval process that will take place before the MIP - set of variables is extended or augmented. - - -5. **Can we publish the data obtained with the MIP?** -- No problem with publishing results obtained using the hospital - data on the MIP in local mode. In a federated mode and before - joining the federation of hospitals, the new hospital will be - requested to sign an agreement to use the data made available - (shared) by others wisely. This includes the rights to publish - experiences and papers based on the use of the data accessed. - - -6. **Is there any system to detect/eliminate duplicated patients across the databases?** -- This will have to be done for a large part by the hospital - personnel before starting the data ingestion process. During the - ingestion processes mechanisms are in place to verify and - harmonize data across data from different origins. This includes - detecting duplications. - - -7. **If we analyse some data today, are we going to obtain the same results if we analyse the same datasets in the future? If not, how can we come back to the same dataset to repeat our analysis?** -- Assuming the content of the dataset has not changed, and the - parameters used are the same, the answer is YES. The answer is - NO if new data have been added over time or if parameters - changed. We are aware of this limitation and have a pending - requirement to allow the same "experiment" to be run more than - once over time making the experiment reproducible. It shall be - implemented at a later date. - - -8. **Is it mandatory to share data in the MIP federated if we want to have access to the other datasets?** -- By joining the federation, the new hospital obtains access to - data of other hospital in the federation and by joining - authorizes de facto other hospitals access to its data. - - -9. **Can I choose with which hospitals we want to share our data?** -- Interested groups of researchers with a common use case can - propose to form a federation, define the data format and limit - who can participate. Please contact the MIP management team for - more information. - - -10. **How can I connect clinical data to image data that came from the same patient?** -- You need to add an ID (a number) that connects data from both - sources. The data ingestion process and the data factory set of - tools is designed to identify and manage these cases. Quality and - consistency of data is ensured by such mechanisms. However, this - ID will be anonymized after the anonymization process to assure - the minimum security measurment and prevent people from - reidentifying the record. - - -11. **After pre-processing the images, can they be deleted, or should they be kept stored?** -- For MIP usage the extract from the pre-orocessing step is - sufficient. However, it's within the best practice and the - accounatbility of the hospital to keep a back-up of such - important data as part of their own database. - - -12. **Could it affect other data stored in the same server?** -- MIP could affect the execution of other programs only if the disk/server is saturated and there is not enough free space. - - -*** - -*** diff --git a/documentation/deployment-pack/install-prerequisites.md b/documentation/deployment-pack/install-prerequisites.md deleted file mode 100644 index 2f11064..0000000 --- a/documentation/deployment-pack/install-prerequisites.md +++ /dev/null @@ -1,186 +0,0 @@ -# Installation Prerequisites and Installation Guide - -Preamble --------- - -> This document provides participating hospitals and institutions with a -> description of the pre-requisites necessary to install the MIP -> components, and how to interact with the MIP operations team to build -> a running MIP. It forms part of the Deployment Pack standard document -> set that also includes the following additional documents: - -- [Executive Summary](./executive-summary.md) -- [Installation and License Agreement (PDF)](../legal-docs/MIP_Installation_and_License_Agreement_-_V02.10_210c5d6538.pdf) -- [Ethics and Legal Requirements](./ethics-legal.md) - -Assumptions ------------ - -- The MIP is to be installed for the first time at Hospital site, OR - an earlier version of MIP has been installed at Hospital and will be - replaced or upgraded - -- The MIP Installation and License Agreement document is read and - understood, and the necessary signatures are in place - -- The onsite deployment team has read the MIP Executive Summary, which - includes a high level description of the MIP and explains the key - steps in the MIP deployment process - -- The hospital MIP technical and user community has access to the MIP - User Guide and the Data Factory User Guide - -- All necessary hardware and software for the planned deployment will - be installed and available before the MIP Operations team is - involved. Please note, in order to ensure a smooth installation it - is important that the preparation is executed exactly as described - in the requirements that follow - -- The hospital team understands that the MIP can be progressively - installed, over time. Starting with MIP Local, using demonstration - and/or hospital data for local use only, then adding the MIP - Federated node if required, and making Hospital ANONYMIZED data - available to the "Medical Condition" based federation of choice - -- When the Hospital joins or is part of a Medical Condition based - federation the Hospital adopts the data model defined by the - federation. The federation defines the common data model in - collaboration with other hospitals and subject matter experts - -- The hospital's data shall be prepared and pre-processed by the - hospital team following the Data Factory use guide, so that data are - in the correct format for import / upload to the MIP - -- The MIP operations team will accompany the Hospital team in the - process of preparing for and setting up the MIP until the MIP is - deemed operational - -- The MIP operations team is NOT in charge of the MIP Installation and - set-up, nor responsible for all or part of the MIP application - installation and Hospital data preprocessing, nor the quality of - Hospital data - -Contacts --------- - -Communications, coordination and deployment: - -- Name - -- Email - -- Telephone - - -Technical support (MIP operations team): - -- Name - -- Email - -- Telephone - -Installation Pre-requisites ---------------------------- - -> There are three elements to a full MIP deployment: MIP Local, MIP -> Federated and the Data Factory. In order to deploy the MIP the -> following hardware and software is a prerequisite to installation, and -> must be available before the MIP operations team is involved. - - - -> **MIP Local Node** -> -> This is the most common deployment, and is a "stand-alone" mode. The -> initial deployment will allow you to try the MIP with test data and -> import and analyse your own data. - -| Server| --------- -| Single core | -| 4GB RAM | -| 16GB HDD | - -|OS| ----- -|Ubuntu 20.04| - -> **MIP Federated Node** -> -> In order to participate in a federation and allow analysis of your -> data by other researchers, a second server is required. Before your -> data is imported to the federated node it passes through an -> anonymization process (see the Data Factory user guide). - -| Server| --------- -| Single core | -| 4GB RAM | -| 16GB HDD | - -|OS| ----- -|Ubuntu 20.04| - -> The two servers can be either separate, physical machines or virtual -> machines. The machines need to be "clean" with no software running -> other than that specified. Any exception to this can cause unexpected -> problems and delays with the installation. -> -> In both cases, physical and virtual, there is strict security between -> the machines, and no possibility of researchers from another -> institution having access to your non-anonymised data. Please note -> that during analysis in a federation your data never leaves you -> physical location. -> -> -> -> For a federated deployment there are some connectivity requirements -> that the MIP operations team will supply during collaboration to -> deploy your MIP. In all cases, SSH access (TCP/22) will be required to -> support the deployment. Deployment options for CentOS and RHEL are -> being considered. -> -> - -> **Data Factory** -> -> For a complete MIP installation, and to aid preparation of your data -> for use with the MIP you will need to install the Data Factory. The -> Data Factory includes the data anonymisation module. - -| Server| --------- -| Single core | -| 4GB RAM | -| 16GB HDD | - -|OS| ----- -|Ubuntu 20.04| - -> Once you have fulfilled these pre-requisites according to the MIP you -> plan to deploy, contact the MIP operations team for support with the -> next steps. - -Check List ----------- - -| Check List | No | Yes | -|-----------------------------------------------------------------------------------|----|-----| -| *Initial deployment* | | | -| Are the necessary agreements ready, approved and signed? | | | -| Are the necessary resources assigned and available for the project? | | | -| Is the project objective clear \- MIP Local only, or MIP Local plus MIP Federated? | | | -| Is the necessary hardware and operating system installed and available? | | | -| Is the remote access for the MIP operations team in place and accessible? | | | -| *Prepare for use* | | | -| Is the hospital data prepared for import/upload to the MIP Local? | | | -| Is the hospital data prepared for import/upload to the MIP Federated? | | | -| (See Data Factory user guide) | | | -| Has the MIP user community followed the training video series? | | | - -*** - -*** diff --git a/documentation/deployment-pack/minimal-setup.md b/documentation/deployment-pack/minimal-setup.md deleted file mode 100644 index a547891..0000000 --- a/documentation/deployment-pack/minimal-setup.md +++ /dev/null @@ -1,128 +0,0 @@ -# Minimal Worker VM Installation -**Goal:** Join a hospital-side VM to a MicroK8s cluster as a **worker** using **Tailscale-only** networking (no public MicroK8s ports). -**Security posture:** If host firewall policy is strict, keep inbound closed and allow only the minimum required traffic on `tailscale0`. ---- - -## 0) Federation Variables (The federation administrator should provide these values) -- `TS_AUTHKEY` = short-lived, **one-off** Tailscale auth key (ideally tagged) -- `MASTER_TS` = the master VM's Tailscale IPv4 (e.g., `100.x.y.z`) -- `MICROK8S_JOIN_TOKEN` = the token used to join the microk8s cluster, **one-off**. - -Example of values that should be provided: -```bash -export TS_AUTHKEY="tskey-auth-REDACTED" -export MASTER_TS="1.2.3.4" -export MICROK8S_JOIN_TOKEN="microk8s-join-token-REDACTED" -``` - ---- - -## 1) Install MicroK8s and Tailscale (worker VM) - -```bash -sudo snap install microk8s --classic --channel=1.33/stable -curl -fsSL https://tailscale.com/install.sh | sh -``` - ---- - -## 2) Join the Tailnet (worker VM) - -**Note:** `--accept-dns=false` prevents changing hospital DNS settings. - -```bash -sudo tailscale up --auth-key="$TS_AUTHKEY" --accept-dns=false -tailscale ip -4 -tailscale status -``` - ---- - -## 3) UFW minimum rules (explicit split: public network vs Tailnet) - -Apply this section only if the hospital VM enforces UFW with restrictive defaults. -Assumed baseline policy: `sudo ufw default deny incoming` and a controlled outbound policy per hospital standard. - -Rule scope in this section: -- **Public network rules**: not bound to `tailscale0`; used only so the Tailscale client can establish/maintain connectivity. -- **Tailnet-only rules**: explicitly bound to `tailscale0`; these carry MicroK8s node traffic. - -### 3.1 Public network outbound rules (Tailscale client only) - -```bash -# Public egress needed by the Tailscale daemon for control/relay/connectivity -sudo ufw allow out 443/tcp -sudo ufw allow out 80/tcp -sudo ufw allow out 3478/udp -sudo ufw allow out 41641/udp -``` - -**Important:** These are **public-network egress-only** exceptions. They are not MicroK8s service ports and they are not opened inbound. - -### 3.2 Tailnet-only: worker -> master (MicroK8s join + API) - -```bash -# Restricted to tailscale0 and to the master Tailscale IP -sudo ufw allow out on tailscale0 to "$MASTER_TS" port 25000 proto tcp # cluster-agent -sudo ufw allow out on tailscale0 to "$MASTER_TS" port 16443 proto tcp # Kubernetes API (MicroK8s) -``` - -### 3.3 Tailnet-only: master/cluster -> worker (kubelet secure port) - -```bash -# Restricted to tailscale0 and master Tailscale IP -sudo ufw allow in on tailscale0 from "$MASTER_TS" to any port 10250 proto tcp -``` - -### 3.4 Tailnet-only: Calico VXLAN dataplane between nodes - -```bash -# Calico VXLAN is UDP 4789 between nodes, only on tailscale0 -sudo ufw allow in on tailscale0 to any port 4789 proto udp -sudo ufw allow out on tailscale0 to any port 4789 proto udp -``` - -**Scope note:** VXLAN is node-to-node traffic, so `4789/udp` applies to all cluster node peers reachable on `tailscale0`, not only the master. -Use Tailscale ACLs/tags to ensure only authorized worker/master nodes can join that Tailnet segment. - -**Dependency note:** These VXLAN rules are valid when the cluster backend is Calico VXLAN. If the backend changes (for example IPIP or WireGuard), ports/protocols must be adjusted by the cluster operator. - -### 3.5 Reload and verify UFW rules - -```bash -sudo ufw reload -sudo ufw status verbose -sudo ufw status numbered -``` - ---- - -## 4) Connectivity checks (before `microk8s join`) - -```bash -tailscale ping "$MASTER_TS" - -# Optional reachability checks to master -nc -vz -w2 "$MASTER_TS" 25000 -nc -vz -w2 "$MASTER_TS" 16443 -``` - -Expected: - -* `tailscale ping` replies (no timeout) -* `nc` to 25000 and 16443 succeeds - ---- - -## 5) Join MicroK8s as a worker - -Run the join command provided securely by the master operator (short-lived / one-time): -```bash -sudo microk8s join $MASTER_TS:25000/$MICROK8S_JOIN_TOKEN --worker -``` - -If you get the error `Joining cluster failed. Could not verify the identity of 1.2.3.4. Use '--sk` -Please use the following command and notify the federation administrator: -```bash -sudo microk8s join $MASTER_TS:25000/$MICROK8S_JOIN_TOKEN --worker --skip-verify -``` diff --git a/documentation/images/MIP Architecture Diagrams_v3.pptx b/documentation/images/MIP Architecture Diagrams_v3.pptx deleted file mode 100755 index e143fd8..0000000 Binary files a/documentation/images/MIP Architecture Diagrams_v3.pptx and /dev/null differ diff --git a/documentation/images/MIP Architecture Diagrams_v4.pptx b/documentation/images/MIP Architecture Diagrams_v4.pptx new file mode 100644 index 0000000..fb220f2 Binary files /dev/null and b/documentation/images/MIP Architecture Diagrams_v4.pptx differ diff --git a/documentation/images/The MIP Data flow.jpg b/documentation/images/The MIP Data flow.jpg deleted file mode 100644 index 8ed240f..0000000 Binary files a/documentation/images/The MIP Data flow.jpg and /dev/null differ diff --git a/documentation/images/The MIP Data flow.png b/documentation/images/The MIP Data flow.png new file mode 100644 index 0000000..391fdd5 Binary files /dev/null and b/documentation/images/The MIP Data flow.png differ diff --git a/documentation/legal-docs/DTA EBRAINS_June2021_v2.docx b/documentation/legal-docs/DTA EBRAINS_June2021_v2.docx deleted file mode 100644 index a353441..0000000 Binary files a/documentation/legal-docs/DTA EBRAINS_June2021_v2.docx and /dev/null differ diff --git a/documentation/legal-docs/DTA_EBRAINS_June2021_v2.pdf b/documentation/legal-docs/DTA_EBRAINS_June2021_v2.pdf deleted file mode 100644 index 8984ea1..0000000 Binary files a/documentation/legal-docs/DTA_EBRAINS_June2021_v2.pdf and /dev/null differ diff --git a/documentation/legal-docs/Data Sharing and Processing Agreement_June2021_v2.docx b/documentation/legal-docs/Data Sharing and Processing Agreement_June2021_v2.docx deleted file mode 100644 index 619cedc..0000000 Binary files a/documentation/legal-docs/Data Sharing and Processing Agreement_June2021_v2.docx and /dev/null differ diff --git a/documentation/legal-docs/Data_Sharing_and_Processing_Agreement_June2021_v2.pdf b/documentation/legal-docs/Data_Sharing_and_Processing_Agreement_June2021_v2.pdf deleted file mode 100644 index 8bc39c7..0000000 Binary files a/documentation/legal-docs/Data_Sharing_and_Processing_Agreement_June2021_v2.pdf and /dev/null differ diff --git a/documentation/legal-docs/MIP Architecture README.docx b/documentation/legal-docs/MIP Architecture README.docx deleted file mode 100755 index fb76e9c..0000000 Binary files a/documentation/legal-docs/MIP Architecture README.docx and /dev/null differ diff --git a/documentation/legal-docs/MIP DATA Management Guideline 221221.docx b/documentation/legal-docs/MIP DATA Management Guideline 221221.docx deleted file mode 100644 index 5a6afdc..0000000 Binary files a/documentation/legal-docs/MIP DATA Management Guideline 221221.docx and /dev/null differ diff --git a/documentation/legal-docs/MIP Service Agreement and Software Licenses_Dec21.docx b/documentation/legal-docs/MIP Service Agreement and Software Licenses_Dec21.docx deleted file mode 100644 index 3a29115..0000000 Binary files a/documentation/legal-docs/MIP Service Agreement and Software Licenses_Dec21.docx and /dev/null differ diff --git a/documentation/legal-docs/MIP_DATA_Management_Guideline_221221.pdf b/documentation/legal-docs/MIP_DATA_Management_Guideline_221221.pdf deleted file mode 100644 index 5988521..0000000 Binary files a/documentation/legal-docs/MIP_DATA_Management_Guideline_221221.pdf and /dev/null differ diff --git a/documentation/legal-docs/MIP_Executive_Summary_V02.00_7960b8432e.pdf b/documentation/legal-docs/MIP_Executive_Summary_V02.00_7960b8432e.pdf deleted file mode 100644 index 7493b20..0000000 Binary files a/documentation/legal-docs/MIP_Executive_Summary_V02.00_7960b8432e.pdf and /dev/null differ diff --git a/documentation/legal-docs/MIP_Installation_Prerequisites_V02.00_c9ad266e67.pdf b/documentation/legal-docs/MIP_Installation_Prerequisites_V02.00_c9ad266e67.pdf deleted file mode 100644 index f270287..0000000 Binary files a/documentation/legal-docs/MIP_Installation_Prerequisites_V02.00_c9ad266e67.pdf and /dev/null differ diff --git a/documentation/legal-docs/MIP_Installation_and_License_Agreement_-_V02.10_210c5d6538.pdf b/documentation/legal-docs/MIP_Installation_and_License_Agreement_-_V02.10_210c5d6538.pdf deleted file mode 100644 index 562bfd1..0000000 Binary files a/documentation/legal-docs/MIP_Installation_and_License_Agreement_-_V02.10_210c5d6538.pdf and /dev/null differ diff --git a/documentation/legal-docs/MIP_Service_Agreement_and_Software_Licenses_Dec21.pdf b/documentation/legal-docs/MIP_Service_Agreement_and_Software_Licenses_Dec21.pdf deleted file mode 100644 index ba4f661..0000000 Binary files a/documentation/legal-docs/MIP_Service_Agreement_and_Software_Licenses_Dec21.pdf and /dev/null differ diff --git a/documentation/legal-docs/README.md b/documentation/legal-docs/README.md deleted file mode 100644 index 8758eff..0000000 --- a/documentation/legal-docs/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# MIP Docs - -Collection of Legal and Ethics documents, mostly for internal use. \ No newline at end of file