diff --git a/users/2023/20231129.md b/users/2023/20231129.md new file mode 100644 index 0000000..3b97231 --- /dev/null +++ b/users/2023/20231129.md @@ -0,0 +1,16 @@ +--- +date: 2023-11-29 +type: users +attendees: + - Olivier M.(DGFIP) + - Moncef L.(DGFIP) + - Mehdi B.(DGFIP) + - Idir I. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- **Kickoff** : begin the design and implementation +- first technology to integrate : jupyterhub +- Participation to the KubeCon Paris 2024 +- Update of the github repository with new logo: https://github.com/OKDP +- website domain reserved : okdp.io diff --git a/users/2023/20231206.md b/users/2023/20231206.md new file mode 100644 index 0000000..3a25466 --- /dev/null +++ b/users/2023/20231206.md @@ -0,0 +1,19 @@ +--- +date: 2023-12-06 +type: users +attendees: + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- begin the study of the first technology to integrate : jupyterhub +- separation between OKDP and its requirements in an extra project like kube, minio, longhorn, etc. +- clone used projects into a separate repository in ordre to keep their history +- start with a github jupyterhub-draft project in the okdp repo then design an okdp project to centralize all integrated technologies +- maintaining Issues/PRs + 2 approves like TDP only for official projects and not draft +- creating discussions for decision-making on current topics +- next step: preparation of local environments via kind, vagrant, etc. diff --git a/users/2024/20240117.md b/users/2024/20240117.md new file mode 100644 index 0000000..ebd8262 --- /dev/null +++ b/users/2024/20240117.md @@ -0,0 +1,18 @@ +--- +date: 2024-01-17 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Nicolas V. (DGFIP) + - Rémi PERRIER (ENEDIS) +--- + +- new partner joined okdp meetings : Rémi PERRIER - ENEDIS +- prepare okdp presentation for the TOSIT day (February 9th at BPCE) +- work in progress : 1. OKDP WebSite update with more details about okdp and its roadmap 2. build of jupyterhub/jupyterlab images 3. choice between ArgoCD and FluxCD for CD implementation diff --git a/users/2024/20240228.md b/users/2024/20240228.md new file mode 100644 index 0000000..aa3b6e3 --- /dev/null +++ b/users/2024/20240228.md @@ -0,0 +1,23 @@ +--- +date: 2024-02-28 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- Back to TOSIT Data Day Event +- Front OKDP : Angular or React can be used for the front dev and SpringBoot for the back +- Superset: integrate its integration into the roadmap for dataViz +- Go-lang : can be used for transverse automation layers for OKDP +- OKDP Web site up to date with last version by Moncef L. +- Spark: advanced Metastore, Spark-ui and spark-hs integration by Idir I. +- Spark: FluxCD deployment integration by Serge A. +- FluxCD : OKDP integration done by Serge A. with some features in progress : security, multi-tenancy, etc. +- ArgoCD can be used for applicative deployments +- Trino : first integration in progress by Moncef L. diff --git a/users/2024/20240306.md b/users/2024/20240306.md new file mode 100644 index 0000000..01f4091 --- /dev/null +++ b/users/2024/20240306.md @@ -0,0 +1,22 @@ +--- +date: 2024-03-06 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Edouard ROUSSEAUX (EDF) +--- + +- RoadMap (to be published via the web site) updated with integration of superset : + 1. Spark integration + 2. Metastore integration (hive metastore in progress) + 3. Trino integration (design and requirements) + 4. SuperSets integration (to-do) + 5. Sanbox implementation (to-do) +- Implementation of an authentication module for spark history server (demo to be planned for tosit members) +- implementation of github discussions with decisions and technical details diff --git a/users/2024/20240313.md b/users/2024/20240313.md new file mode 100644 index 0000000..7eb9f58 --- /dev/null +++ b/users/2024/20240313.md @@ -0,0 +1,28 @@ +--- +date: 2024-03-13 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Edouard R. (EDF) + - Lionel H. (Orange) + - Pierre S. (Adaltas) +--- + +- Olivier M.: okdp roadmap to be updated +- Mehdi B.: new project to manage authentication in Spark History and its release management +- Pierre S.: topics to share for new contributions +- Pierre S.: Proposal for other DEV environments with other kubernetes distributions +- Lionel H.: Orange contributions: + - Spark HS to be tested in the Orange context + - dev of an UI to list spark apps via APIs kube + rex to be planned + - share Spark images with the community +- presentation to be planned : + - March 27th: SparkHistory and okdp projects + - April 3: build of spark images and Spark History + - April 4: okdp demo for TOSIT members diff --git a/users/2024/20240320.md b/users/2024/20240320.md new file mode 100644 index 0000000..55cf681 --- /dev/null +++ b/users/2024/20240320.md @@ -0,0 +1,23 @@ +--- +date: 2024-03-20 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Nicolas V. (DGFIP) + - Pierre S. (Adaltas) +--- + +- Olivier M.: confirmation of the following meetings: Thursday April 11 for the demo from 9 a.m. to 10 a.m. +- Olivier M.: DGFIP is looking for alternatives to array (potentially supersets) +- Moncef L.: Update roadmap on the web-site and remove of react +- Moncef L.: implementation of the Helm chart for Spark History Server +- Serge A.: implementation of the Helm chart for Hive Metastore and postgres +- Idir I.: build of spark images +- Mehdi & Chung: finalizing the setup of the test environment with Kind +- Moncef: creation of an OKDP channel on the TOSIT mattermost to discuss current technical subjects diff --git a/users/2024/20240327.md b/users/2024/20240327.md new file mode 100644 index 0000000..dd17498 --- /dev/null +++ b/users/2024/20240327.md @@ -0,0 +1,27 @@ +--- +date: 2024-03-27 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Nicolas V. (DGFIP) + - Pierre S. (Adaltas) + - Lionel H. (Orange) +--- + +- Lionel H.: working on a build container for constant dev env. +- Lionel H.: proposal for a PR on the Spark History authentication project to make the identifier configurable for the JWT token (between sub and email) +- Olivier H.: Participation in the Big Data & AI event with a stand in November 2024. +- Pierre S.: working on multi-tenancy feature in SuperSet +- Serge A.: hive metastore integration and release management +- Serge A.: using FluxCD to deploy requirements (cert manager, metalLb, etc.) in Gitops mode +- Idir I. : build of Spark images +- Moncef L.: build factory +- Mehdi B.: preparation of OKDP slides +- Lionel H.: contribution in the form of PR on the Spark build with: dependency checking + Log4j patch + jre deployment for jdk 11 and 17 + kerberos deployment + tests + observability configuration +- Moncef L.: creation of an OKDP sub-channel for specific discussions diff --git a/users/2024/20240516.md b/users/2024/20240516.md new file mode 100644 index 0000000..396419a --- /dev/null +++ b/users/2024/20240516.md @@ -0,0 +1,18 @@ +--- +date: 2024-05-16 +type: users +attendees: + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Lionel H. (Orange) +--- + +- Moncef L.: integration of Trino in progress via a Helm chart +- Idir I.: integration of Superset with a focus on security +- Serge A.: implementation of the OKDP sandbox via FluxCD +- Moncef L.: preparation of the June demo in progress +- Moncef L.: integration of the Spark History authentication module at the Helm chart level +- Lionel H.: successful testing and integration of the Spark auth module in the Orange context diff --git a/users/2024/20240522.md b/users/2024/20240522.md new file mode 100644 index 0000000..e45df8f --- /dev/null +++ b/users/2024/20240522.md @@ -0,0 +1,20 @@ +--- +date: 2024-05-22 +type: users +attendees: + - Olivier M. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Stephane B. (Orange) + - Lionel H. (Orange) +--- + +- Olivier M.: preparation of the presentation for the Data & AI Leaders +- Confirmation of participation in Tosit Day on Thursday 13/06 for an OKDP presentation (Olivier M., Idir I. and Mehdi B. on site and Moncef L. remotely) +- Olivier M.: contact BPCE and Arkea on a potential collaboration on Trino +- Olivier M.: OKDP internal presentation on 07/06 and slides to be reviewed by Mehdi B. and Moncef L. +- Moncef L.: preparation of the demo and progress on current topics: Trino (Moncef L.), Sandbox via FluxCD (Serge A.) and Superset (Idir I.) +- Stephane B. and Lionel H.: integration of the spark HS authentication module and review of the integration with OIDC in order to avoid declaring a wlidcard for SparkUI redirects +- Lionel H.: internal validation for Orange's participation in the OKDP presentation diff --git a/users/2024/20240529.md b/users/2024/20240529.md new file mode 100644 index 0000000..4ebf515 --- /dev/null +++ b/users/2024/20240529.md @@ -0,0 +1,21 @@ +--- +date: 2024-05-29 +type: users +attendees: + - Olivier M. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Lionel H. (Orange) + - Edouard R. (EDF) + - Nicolas V. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- Olivier M.: preparation of the presentation for the Data & AI Leaders +- Olivier M.: OKDP internal presentation on 07/06 and slides to be reviewed by Mehdi B. and Moncef L. +- Olivier M.: collaboration with an inter-ministerial group around superset +- Edouard R.: proposal for a demo of EDF's containerized Data platform on 06/19/24 +- Olivier M.: ongoing update of the website by Mehdi B. with new events (in progress) +- Lionel H.: creation of a new github issue ( followed by Idir I. ) around Spark History Server and its authentication module in order to make it more generic diff --git a/users/2024/20240605.md b/users/2024/20240605.md new file mode 100644 index 0000000..331ee0e --- /dev/null +++ b/users/2024/20240605.md @@ -0,0 +1,24 @@ +--- +date: 2024-06-05 +type: users +attendees: + - Olivier M. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Lionel H. (Orange) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- Update of the OKDP web site with various events (PR ongoing) +- Presentation of OKDP to Arkéa on Thursday, 30/05 (done) +- Presentation of EDF's proprietary solution on Spark, Wednesday, 19/06/2024. +- Testing of the spark authentication module by new users +- Github issue created by Orange on the Authentication/Ldap (IDP) part on GitHub (ongoing) +- DINUM (BlueHats workshop), Friday, June 7, 2024, from 11:00 to 12:30 (Jupyter, Spark + Spark History authentication module, SandBox, SuperSet, end-to-end build and deployment, etc.) +- TOSiT-Day on the afternoon of 13/06, fill in the framadate (done) +- Roundtable at the Data & AI Leaders on 27 and 28/11 (in progress) +- Packaging SuperSet by Idir, publication and documentation to follow +- Build OKDP: Orange will approach DIODE (DGFiP) to work and provide feedback on the Build part +- Participation of partners (BPCE and Orange) in the Hoche Data & AI, Data Mesh, Data Cloud Platform, Data Fabric & Architectures Meetups diff --git a/users/2024/20240612.md b/users/2024/20240612.md new file mode 100644 index 0000000..689220c --- /dev/null +++ b/users/2024/20240612.md @@ -0,0 +1,18 @@ +--- +date: 2024-06-12 +type: users +attendees: + - Olivier M. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- Update the OKDP website with our various events (done) +- DINUM (BlueHats workshop): Friday, June 7, 2024, from 11:00 AM to 12:30 PM (Jupyter, Spark + Spark History authentication module, SandBox, SuperSet, end-to-end build and deployment, etc.). There were 40 participants (done) +- TOSiT-Day on 06/13 in the afternoon. +- SuperSet: Meeting on Tuesday, 06/18, for a new collaboration on the support and REX part +- Presentation of EDF’s proprietary solution on Spark, Wednesday, 06/19/2024. +- TODO : MLOPS: integration of kubeflow, MLFlow, Onyxia, other UIs, etc. diff --git a/users/2024/20240703.md b/users/2024/20240703.md new file mode 100644 index 0000000..f87b656 --- /dev/null +++ b/users/2024/20240703.md @@ -0,0 +1,14 @@ +--- +date: 2024-07-03 +type: users +attendees: + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Lionel H. (Orange) +--- + +- Serge A.: integration of topolvm with vgs management via helm chart for the sandbox +- Idir I.: create github repo for hive/superset images and their sources +- Lionel H.: propose a solution to secure Docker images (currently being tested on Orange side) diff --git a/users/2024/20240717.md b/users/2024/20240717.md new file mode 100644 index 0000000..1594044 --- /dev/null +++ b/users/2024/20240717.md @@ -0,0 +1,16 @@ +--- +date: 2024-07-17 +type: users +attendees: + - Nicolas V. (DGFIP) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Lionel H. (Orange) +--- + +- Idir: split helm charters in github (done) +- Idir: start the architecture choices on the front part with a preference for go-lang for its watcher support +- Serge A.: integration of topolvm for minio/sandbox (done) +- Serge A.: automation of minio resources creation in declarative mode (in progress) diff --git a/users/2024/20240724.md b/users/2024/20240724.md new file mode 100644 index 0000000..6259c0c --- /dev/null +++ b/users/2024/20240724.md @@ -0,0 +1,18 @@ +--- +date: 2024-07-24 +type: users +attendees: + - Nicolas V. (DGFIP) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Olivier M. (DGFIP) + - Guillaume M. (DGFIP) +--- + +- Updating slides with new UI features +- preparation of the internal presentation with the new roadmap +- Discussion with other UX design teams +- Minio integration and automatic resource management in progress +- split Helm charts on github finalized diff --git a/users/2024/20240731.md b/users/2024/20240731.md new file mode 100644 index 0000000..d01c9eb --- /dev/null +++ b/users/2024/20240731.md @@ -0,0 +1,19 @@ +--- +date: 2024-07-31 +type: users +attendees: + - Olivier M. (DGFIP) + - Guillaume M. (DGFIP) + - Nicolas V. (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Lionel H. (Orange) +--- + +- Idir I.: Split Helm charts (done) +- Idir I.: REST API in Golang + Angular for the UI (in progress) +- Serge A.: MinIO and declarative resource management + RBAC (bucket + policies) (in progress) +- Moncef L.: Take over the Kubernetes clusters topic on private cloud +- Lionel H.: New ticket on the Spark authentication module diff --git a/users/2024/20240911.md b/users/2024/20240911.md new file mode 100644 index 0000000..2a2617d --- /dev/null +++ b/users/2024/20240911.md @@ -0,0 +1,9 @@ +--- +date: 2024-09-11 +type: users +--- + +- Idir I.: Develop some Angular models for the OKDP UI (in progress) +- Serge A.: MinIO and declarative resource management + RBAC (bucket + policies) (in progress) +- Lionel H.: New [ticket ](https://github.com/OKDP/okdp-spark-auth-filter/issues/34)on the Spark authentication module +- Idir I. : Choosing the backend for the UI, potentially in GoLang. diff --git a/users/2024/20240918.md b/users/2024/20240918.md new file mode 100644 index 0000000..003d509 --- /dev/null +++ b/users/2024/20240918.md @@ -0,0 +1,28 @@ +--- +date: 2024-09-18 +type: users +attendees: + - Olivier M. (DGFIP) + - Idir I. (DGFIP) + - Nicolas V (DGFIP) + - Mehdi B. (DGFIP) + - Chung N. (DGFIP) + - Guillaume M. (DGFIP) + - Moncef L. (DGFIP) + - Serge A. (DGFIP) + - Lionel H. (Orange) +--- + +- Backend choice for the UI: + 1. GlassKube: https://glasskube.dev/ + 2. Timoni: https://timoni.sh/ + 3. Open Component Model OCM: https://ocm.software/ + 4. Open Application Model (Microsoft, Alibaba): https://oam.dev/ -> kubevela: https://kubevela.io/ + 5. kusionstack: https://www.kusionstack.io/ +- Criteria: + 1. Integration with fluxcd as the first selection criterion + 2. A user-friendly frontend + 3. Multi-cluster support + 4. CNCF incubation +- Serge A.: Minio integration in progress for resource management + LDAP integration +- Lionel H.: Spark issue in progress diff --git a/users/2024/20240925.md b/users/2024/20240925.md new file mode 100644 index 0000000..9154b2e --- /dev/null +++ b/users/2024/20240925.md @@ -0,0 +1,17 @@ +--- +date: 2024-09-25 +type: users +attendees: + - Idir I. (DGFIP) + - Nicolas V (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Moncef L. (DGFIP) + - Serge A. (DGFIP) +--- + +- Sandbox: Integration of KIND for platforms (not macOS: Linux, Windows) [Kube on containers] +- Sandbox: Integration of Vagrant for internal use [Kube on VMs] +- Backend OKDP UI: Choice in progress and discussions on KubeVela and Crossplane (Open Application Model) +- Backend OKDP UI: Crossplane (Open Application Model) +- MinIO: Resource management (in progress) diff --git a/users/2024/20241002.md b/users/2024/20241002.md new file mode 100644 index 0000000..cadb0f5 --- /dev/null +++ b/users/2024/20241002.md @@ -0,0 +1,29 @@ +--- +date: 2024-10-02 +type: users +attendees: + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Moncef L. (DGFIP) + - Serge A. (DGFIP) + - Moncef L. (DGFIP) + - Lionel H. (Orange) +--- + +- Sandbox: Integration of Minikube for better cross-OS compatibility: Windows, Linux, etc. +- UI: Exclude the KubeVella option for the following reasons: + 1. No versioning + 2. Packaging does not match + 3. No GitOps + 4. Decreasing trend on GitHub +- Instead, proceed with the following architecture : + +![Flux schema](./assets/20241002-schema.png) + +1. A REST API development using Go-lang will be necessary between the frontend and the Git layer. +2. The choice of Go-lang is based on the following arguments : + - [X] Continuity with Kubernetes and existing tools like kubectl + - [x] Simpler exploitation + - [x] Modularity +- Unit and integration tests in progress on the Spark authentication module by Orange diff --git a/users/2024/20241009.md b/users/2024/20241009.md new file mode 100644 index 0000000..41efde5 --- /dev/null +++ b/users/2024/20241009.md @@ -0,0 +1,17 @@ +--- +date: 2024-10-09 +type: users +attendees: + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Chung N. (DGFIP) +--- + +- SANDBOX: minikube tested and validated in containers and VMs mode +- SANDBOX: integration pending the latest updates in KAD +- TODO: documentation of the different GitHub repositories in progress +- KAD: evolution and refactoring in progress +- UI: adding new features: multiple workspaces by namespace and the concept of deployment template +- UI: preparing a Docker image with all dependencies diff --git a/users/2024/20241016.md b/users/2024/20241016.md new file mode 100644 index 0000000..6edc05e --- /dev/null +++ b/users/2024/20241016.md @@ -0,0 +1,18 @@ +--- +date: 2024-10-16 +type: users +attendees: + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Chung N. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Moncef L. (DGFIP) +--- + +- Sandbox: Integration of Minikube into the sandbox validated and integration of the new version of KAD in progress +- OKDP UI: Rest API to be done +- OKDP UI: First version of KAD and the front-end (based on Angular) validated by the team +- Chung N.: Request for a presentation of the work that will take place on Friday, October 18th at 2:00 PM diff --git a/users/2024/20241106.md b/users/2024/20241106.md new file mode 100644 index 0000000..bcc13d9 --- /dev/null +++ b/users/2024/20241106.md @@ -0,0 +1,19 @@ +--- +date: 2024-11-06 +type: users +attendees: + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Chung N. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Olivier M.: update of dgfip OKDP support +- Olivier M.: presentation 01/21/25 BPCE TOSIT Day (preparation of the demo) +- Idir I.: dev of the rest api for the OKDP UI backend +- Serge A.: Refactoring of KAD in progress +- Mehdi B.: integration in progress of the latest version of KAD for the sandbox +- Mehdi B.: preparation of the demo for the Data & AI Leaders Event +- Olivier M.: work internally DGFIP to prepare a dataset for the demos diff --git a/users/2024/20241204.md b/users/2024/20241204.md new file mode 100644 index 0000000..9582c77 --- /dev/null +++ b/users/2024/20241204.md @@ -0,0 +1,34 @@ +--- +date: 2024-12-04 +type: users +attendees: + - Moncef L. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Lionel H.(Orange) + - Olivier M. (DGFIP) +--- + +**Olivier M.** + +- Presentation at the tech show and feedback on the highly anticipated V1 of OKDP. +- TOSIT Day: presentation and demo of OKDP. + +**Lionel H.** + +- Presentation of an AI module (KeyOps) at the TOSIT Day: [Khiops](https://khiops.org/) under BSD license. +- Spark History PR approved on GCP +- compatibility with Java 8/11 in progress (replacement of Caffeine with Guava). + +**Moncef L.** + +- Add the front end to OKDP. +- Prepare the management of Spark jobs. + +**Mehdi B.** + +- Upload the slides to the DGFIP share. +- Refactor the sandbox. +- Documentation for publication on the website. diff --git a/users/2024/20241211.md b/users/2024/20241211.md new file mode 100644 index 0000000..ce4af79 --- /dev/null +++ b/users/2024/20241211.md @@ -0,0 +1,25 @@ +--- +date: 2024-12-11 +type: users +attendees: + - Moncef L. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Lionel H.(Orange) + - Olivier M. (DGFIP) + - Chung N. (DGFIP) + - Nicolas V. (DGFIP) +--- + +- Python Libraries: Request to integrate a Python library for internal teams: Streamlit. +- Slides for TOSIT Data Day: + - Preparation of materials to present strategic progress. + - Preparation of a demonstration using real data, integrating the Spark operator. + - Orange's feedback on the Spark History authentication module. +- Website: Update and definition of the roadmap to enhance visibility. +- Iceberg and Polaris: Analysis of opportunities for integrating the two technologies. +- Apache Incubation: Progress in the process of integrating OKDP into the Apache incubation program. +- Sandbox Refactoring: Integration of the latest features related to KAD and configuration refactoring. +- Superset: Emphasis on the importance of a gouv chart for its integration into OKDP. diff --git a/users/2024/20241218.md b/users/2024/20241218.md new file mode 100644 index 0000000..3c0f3c9 --- /dev/null +++ b/users/2024/20241218.md @@ -0,0 +1,18 @@ +--- +date: 2024-12-18 +type: users +attendees: + - Moncef L. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Streamlit: Validation of the addition of the Streamlit library to Python tools. +- BPCE Day: Preparation underway for the event scheduled on 01/21. +- Roadmap 2025: in progress +- Sandbox: in progress + Decision to resolve DNS via /etc/hosts. +- Superset: Integration of the governance chart into the roadmap to be planned. +- OKDP Server: Implementation in progress. diff --git a/users/2024/assets/20241002-schema.png b/users/2024/assets/20241002-schema.png new file mode 100644 index 0000000..cb18852 Binary files /dev/null and b/users/2024/assets/20241002-schema.png differ diff --git a/users/2025/20250115.md b/users/2025/20250115.md new file mode 100644 index 0000000..e7c1ba1 --- /dev/null +++ b/users/2025/20250115.md @@ -0,0 +1,17 @@ +--- +date: 2025-01-15 +type: users +attendees: + - Moncef L. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Resend the invitation for the weekly meeting with a new web conference link. +- TDP/OKDP Slides for the TOSIT Day: + - TDP: Olivier, Mehdi. + - OKDP: Mehdi, Moncef. +- Update the roadmap (modify in the slides and publish on the website). +- Prepare an end-to-end demonstration with KAD, OKDP Server, and OKDP UI. diff --git a/users/2025/20250122.md b/users/2025/20250122.md new file mode 100644 index 0000000..fd9b5aa --- /dev/null +++ b/users/2025/20250122.md @@ -0,0 +1,20 @@ +--- +date: 2025-01-22 +type: users +attendees: + - Moncef L. (DGFIP) + - Lionel H(Orange) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Confirmation of TOSIT's participation in Paris 2024 at Porte de Versailles +- TODO: Improvements to the KAD UI before publication +- TODO: Finalization of the first publishable version of KAD, with a possible integration of the REST API +- TODO: Development of the first Helm chart for Trino +- TODO: Integration of OKDP UI and REST API into the sandbox +- Collaboration with Orange on the OKDP UI focused on user experience +- PR Orange: PR update with a new version of Java and Guava diff --git a/users/2025/20250129.md b/users/2025/20250129.md new file mode 100644 index 0000000..203ff46 --- /dev/null +++ b/users/2025/20250129.md @@ -0,0 +1,18 @@ +--- +date: 2025-01-29 +type: users +attendees: + - Nicolas V. (DGFIP) + - Lionel H(Orange) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Guillaume M. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Idir: Implementation of continuous integration for OKDP server + UI and configuration via Helm with GitHub Actions. +- Idir: Addressing issues in Spark Authenticator with support of larger refresh tokens. +- Serge: Redesign of the KAD module and optimization of the REST API. +- Lionel: Adding Java 8 compatibility in addition to Java 11 to ensure better backward compatibility of Spark Authenticator. +- Deployment sharing: Sharing OKDP UI, server and KAD with Orange following the latest refactoring. diff --git a/users/2025/20250205.md b/users/2025/20250205.md new file mode 100644 index 0000000..b63946d --- /dev/null +++ b/users/2025/20250205.md @@ -0,0 +1,17 @@ +--- +date: 2025-02-05 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Lionel H. (Orange) + - Mehdi B. (DGFIP) + - Moncef L. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Integration of the Hive Metastore on Orange’s side: ongoing progress, with necessary adjustments to ensure compatibility with the existing setup. +- KAD refactoring: ongoing optimizations to improve modularity and enhance integration with FluxCD. +- Sandbox publication: planned with the latest KAD refactoring. +- Trino: OIDC integration in progress, with considerations on authorization management via OPA. +- Spark Auth Filter patch: published, feedback to be gathered. diff --git a/users/2025/20250212.md b/users/2025/20250212.md new file mode 100644 index 0000000..940ab0b --- /dev/null +++ b/users/2025/20250212.md @@ -0,0 +1,17 @@ +--- +date: 2025-02-12 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Strengthening the team on the DGFIP side +- KAD – Refactoring: Optimizing the architecture for better modularity and maintainability. +- OKDP Server & OKDP UI: Awaiting updates on the KAD side before moving forward with the next steps. +- Spark UI: Development in progress with a modular approach for greater flexibility. +- Presentation to Data & AI Leaders: Follow-up planned to confirm the next presentation. +- Sandbox: Project currently on standby, awaiting new features on the KAD side. diff --git a/users/2025/20250219.md b/users/2025/20250219.md new file mode 100644 index 0000000..8e28282 --- /dev/null +++ b/users/2025/20250219.md @@ -0,0 +1,17 @@ +--- +date: 2025-02-19 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Moncef L. (DGFIP) + - Lionel H. (Orange) +--- + +- Integration of History Server on the Orange side +- Feedback (ReX) on Polaris to be scheduled by Orange (security check related to S3 storage) +- Sandbox with KAD planned in the coming weeks +- Ongoing development of the Spark job management front-end +- Website update scheduled diff --git a/users/2025/20250305.md b/users/2025/20250305.md new file mode 100644 index 0000000..6d94270 --- /dev/null +++ b/users/2025/20250305.md @@ -0,0 +1,15 @@ +--- +date: 2025-03-05 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) +--- + +- Include Graph databases in the strategy +- Refactoring of KAD +- Spark interface under development +- Sandbox finalization and publication in progress diff --git a/users/2025/20250312.md b/users/2025/20250312.md new file mode 100644 index 0000000..e06b1d4 --- /dev/null +++ b/users/2025/20250312.md @@ -0,0 +1,16 @@ +--- +date: 2025-03-12 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) + - Moncef L. (DGFIP) +--- + +- Review and update the OKDP website +- Serge: Refactoring of KAD +- Idir: Finalization of the "spark-history-web-proxy" project +- Moncef: Finalization and publication of the Sandbox (integration of Ingress DNS into minikube is ongoing) diff --git a/users/2025/20250319.md b/users/2025/20250319.md new file mode 100644 index 0000000..bceafb6 --- /dev/null +++ b/users/2025/20250319.md @@ -0,0 +1,17 @@ +--- +date: 2025-03-19 +type: users +attendees: + - Lionel H. (Orange) + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) + - Moncef L. (DGFIP) +--- + +- Serge: Refactoring of KAD (in progress) +- Idir: Finalizing the "spark-history-web-proxy" project (in progress) +- Moncef: Finalizing and publishing the Sandbox, integrating Spark History Server (in progress) +- OKDP V1 planned for the end of 2025 diff --git a/users/2025/20250326.md b/users/2025/20250326.md new file mode 100644 index 0000000..35035a5 --- /dev/null +++ b/users/2025/20250326.md @@ -0,0 +1,20 @@ +--- +date: 2025-03-26 +type: users +attendees: + - Nicolas V. (DGFIP) + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) + - Moncef L. (DGFIP) +--- + +- Serge: Refactoring of KAD (ongoing) +- Idir : Finalizing the spark-history-web-proxy project (ongoing) +- Moncef & Mehdi: Finalization and release of the Sandbox + 1. Integration of Spark History Server (done) + 2. Integration of Minikube (in progress) + 3. Documentation (in progress) +- OKDP V1 planned for the end of 2025 diff --git a/users/2025/20250402.md b/users/2025/20250402.md new file mode 100644 index 0000000..9d71298 --- /dev/null +++ b/users/2025/20250402.md @@ -0,0 +1,35 @@ +--- +date: 2025-04-02 +type: users +attendees: + - Nicolas V (DGFIP) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Lionel H. (DGFIP) +--- + +## Serge + +- Refactoring of KAD (in progress) + +## Idir + +- Finalization of the `spark-history-web-proxy` project (in progress) +- Choosing a new name for the project (in progress) + +## Moncef & Mehdi + +- Finalization and release of the Sandbox: + - Minikube integration (done) + - Make the deployment OS-agnostic (in progress) + - Documentation (in progress) + +## Moncef + +- Trino integration (in progress) + +## Note + +- OKDP V1 delivery planned for end of 2025 diff --git a/users/2025/20250409.md b/users/2025/20250409.md new file mode 100644 index 0000000..46efb36 --- /dev/null +++ b/users/2025/20250409.md @@ -0,0 +1,41 @@ +--- +date: 2025-04-09 +type: users +attendees: + - Guillaume M. (DGFIP) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Lionel H. (DGFIP) + - Lucian (Orange) +--- + +- Ongoing analysis of centralized authorization management using OPA (Open Policy Agent) + +## Serge + +- Refactoring of KAD (in progress) +- Refactoring of the okdp-server module + +## Idir + +- Finalization of the `spark-history-web-proxy` project (in progress) +- Choosing a new name for the project (in progress) +- Refactoring of the okdp-ui interface + +## Moncef & Mehdi + +- Finalization and release of the Sandbox: + - Minikube integration (done) + - Make the deployment OS-agnostic (done) + - Automated DNS resolution using containerized wildcard mode (done) + - Documentation (in progress) + +## Moncef + +- Trino integration (in progress) + +## Note + +- OKDP V1 delivery planned for end of 2025 diff --git a/users/2025/20250416.md b/users/2025/20250416.md new file mode 100644 index 0000000..bd38d88 --- /dev/null +++ b/users/2025/20250416.md @@ -0,0 +1,49 @@ +--- +date: 2025-04-16 +type: users +attendees: + - Guillaume (DGFIP) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Moncef L. (DGFIP) + - Nicolas V. (DGFIP) + - Lionel H. (Orange) + - Lucian (Orange) +--- + +## News + +- Orange team onboarding this week +- Weekly meetings resume after the holidays: **May 7, 2025** + +## Technical Development Updates + +### Authorization Management + +- Study in progress on centralized authorization management using **OPA (Open Policy Agent)** + +### Serge + +- Refactoring of KAD (in progress) +- Refactoring of **okdp-server** + +### Idir + +- Refactoring of **okdp-ui** and **okdp-server** + +### Moncef & Mehdi + +- Finalization and publication of the **Sandbox** (done) +- Review and update of the **OKDP website** (done) + +### Moncef + +- Integration of **Trino** (in progress) + +### Lucian + +- Addition of the **MySQL connector** to the **Hive Metastore** image (done) +- Build of the **Hive Metastore** image version 4.X +- Pull request validation and documentation (in progress) diff --git a/users/2025/20250507.md b/users/2025/20250507.md new file mode 100644 index 0000000..1fc0dd9 --- /dev/null +++ b/users/2025/20250507.md @@ -0,0 +1,50 @@ +--- +date: 2025-05-07 +type: users +attendees: + - Guillaume (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Lionel H. (Orange) + - Lucian (Orange) + - Jean-Marc (CGI) +--- + +## News + +- Abir O. joins the DGFIP team on OKDP topics. + +--- + +## Technical Developments Follow-up + +- **Centralized authorization management**: + - Ongoing study using **OPA (Open Policy Agent)** + +### Serge + +- Refactoring of `okdp-server` _(in progress)_ + +### Idir + +- Refactoring of `okdp-ui` and `okdp-server` _(in progress)_ + +### Moncef & Mehdi + +- GitHub sandbox shared +- Documentation enrichment + DNS resolution add-on _(in progress)_ + +### Moncef + +- Trino integration _(in progress)_ + +### Lucian + +- New image published on [quay.io](https://quay.io) + +### Lionel + +- Study of **kro.run** for managing Kubernetes CRDs + - First use case: managing Spark jobs and associated CRDs diff --git a/users/2025/20250514.md b/users/2025/20250514.md new file mode 100644 index 0000000..3761d80 --- /dev/null +++ b/users/2025/20250514.md @@ -0,0 +1,43 @@ +--- +date: 2025-05-14 +type: users +attendees: + - Abir O. (DGFIP) + - Guillaume (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Lionel H. (Orange) + - Lucian (Orange) + - Moncef L. (DGFIP) +--- + +- Centralized authorization management: ongoing study using Open Policy Agent (OPA) + +## Serge + +- Integration study of Ory Hydra – an OAuth 2.0 and OpenID Connect authorization server (in progress) + +## Idir + +- Refactoring of `okdp-ui` and `okdp-server` (in progress) + +## Mehdi + +- Documentation enrichment +- Add-on for DNS resolution (in progress) + +## Moncef + +- Trino integration (in progress) +- OKDP CLI (Command Line Interface) development (in progress) + +## Lucian + +- New image published on the public quay.io registry (done) +- Security scan of the image (in progress) + +## Lionel & Mehdi + +- Abstract of the presentation to be sent to the Big Data & AI 2025 organizer (in progress) diff --git a/users/2025/20250521.md b/users/2025/20250521.md new file mode 100644 index 0000000..8022b15 --- /dev/null +++ b/users/2025/20250521.md @@ -0,0 +1,48 @@ +--- +date: 2025-05-21 +type: users +attendees: + - Abir O. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Lionel H. (Orange) + - Lucian (Orange) + - Moncef L. (DGFIP) +--- + +**Ongoing study on centralized authorization management (OPA: Open Policy Agent)** + +## Serge + +Integration study of Ory Hydra and Zitadel _(ongoing)_ + +## Idir + +Refactoring of okdp-ui and okdp-server _(ongoing)_ + +## Mehdi + +Documentation enrichment + addon on DNS resolution _(ongoing)_ + +## Moncef + +Integration of Trino _(ongoing)_ + +## OKDP CLI + +Command Line Interface _(ongoing)_ + +## Moncef & Mehdi + +Data catalog and Trino _(on standby)_ + +## Lucian + +Integration of JMX into the Hive metastore image + +## Lionel + +Address the topic of managing multiple metastores through Unity, Polaris, Starburst Catalog, etc. diff --git a/users/2025/20250528.md b/users/2025/20250528.md new file mode 100644 index 0000000..806b2b4 --- /dev/null +++ b/users/2025/20250528.md @@ -0,0 +1,52 @@ +--- +date: 2025-05-28 +type: users +attendees: + - Abir O. (DGFIP) + - Serge A. (DGFIP) + - Idir I. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Lionel H. (Orange) + - Lucian (Orange) + - Moncef L. (DGFIP) + - Guillaume M. (DGFIP) + - Jean Marc J. (CGI) +--- + +- Ongoing study on centralized authorization management (OPA: Open Policy Agent) + +## Serge + +- Integration study of Ory Hydra and Zitadel _(ongoing)_ + +## Idir + +- Refactoring of okdp-ui and okdp-server _(done)_ + +## Mehdi + +- Enrich documentation +- Addon for DNS resolution _(ongoing)_ + +## Moncef + +- Trino integration _(ongoing)_ + +## OKDP CLI + +- Command Line Interface _(nearing completion)_ +- New sandbox version following OKDP Server refactoring + +## Moncef & Mehdi + +- Data catalog and Trino _(on standby)_ + +## Lucian + +- Management of the Hive metastore database through initialization scripts + +## Lionel + +- Build Spark image with a unified base and custom dependencies diff --git a/users/2025/20250604.md b/users/2025/20250604.md new file mode 100644 index 0000000..20a47d6 --- /dev/null +++ b/users/2025/20250604.md @@ -0,0 +1,38 @@ +--- +date: 2025-06-04 +type: users +attendees: + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Abir O. (DGFIP) + - Idir I. (DGFIP) + - Guillaume M. (DGFIP) + - Jean-Marc J. (CGI) + - Lionel H. (Orange) + - Lucian C. (Orange) +--- + +**Olivier** + +- Participation in the Vivatch project postponed. + +**Mehdi** + +- Ongoing study on Metastore solutions: + - Comparison between Unity, Polaris, Nessie, and Starburst Gravity. + +**Lucian** + +- Hive Metastore build in progress (Ubuntu only for now). + +**Lionel** + +- Study on using Distroless as the Docker base image for OKDP. + +**Idir** + +- UI refactoring for OKDP in progress. + +**Serge** + +- Implementation of authentication using OIDC. diff --git a/users/2025/20250611.md b/users/2025/20250611.md new file mode 100644 index 0000000..8996d2d --- /dev/null +++ b/users/2025/20250611.md @@ -0,0 +1,36 @@ +--- +date: 2025-06-11 +type: users +attendees: + - Mehdi B. (DGFIP) + - Lucian C. (Orange) + - G. Mandier (DGFIP) + - Idir I. (DGFIP) + - Moncef L. (DGFIP) + - Serge (DGFIP) +--- + +### Mehdi + +- Metastore study ready for discussion. +- Materialized views on Polaris and Nessie to be checked. + +### Moncef + +- Sandbox in progress, using the latest component versions and no dependency on GitHub. + +### Idir + +- Ongoing improvement of the UI with more user experience features. + +### Serge + +- Discussion on OIDC: + - Limitations identified with Keycloak + Dex. + - Building an OIDC setup with Zitadel or Ory Hydra. +- User and account management. + +### Lucian + +- Metastore image security. +- Scaling: ReplicaSet management. diff --git a/users/2025/20250704.md b/users/2025/20250704.md new file mode 100644 index 0000000..01093db --- /dev/null +++ b/users/2025/20250704.md @@ -0,0 +1,20 @@ +--- +date: 2025-07-04 +type: users +attendees: + - Mehdi B. (DGFIP) + - Abir O. (DGFIP) + - Eliott M. (Adaltas) + - Lionel H. (Orange) + - Marian (Orange) + - Moncef Lahouar (DGFIP) + - Mori H. (Adaltas) + - Idir I. (DGFIP) +--- + +### Version Github + +- **Extension of Spark images** based on Docker/Ubuntu images. +- **Trino initialization** in progress with the latest versions. +- **Paid features to be checked in Trino**: query history integration, Trino UI, and view auto-refresh. +- **Sandbox validation** → sharing in progress following Lionel's validation. diff --git a/users/2025/20250827.md b/users/2025/20250827.md new file mode 100644 index 0000000..d0491b9 --- /dev/null +++ b/users/2025/20250827.md @@ -0,0 +1,25 @@ +--- +date: 2025-08-27 +type: users +attendees: + - Mehdi B. (DGFIP) + - Lionel H. (Orange) + - Jean-Michel (Orange) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Olivier M. (DGFIP) + - Nicolas V. (DGFIP) + - Guillaume M. (DGFIP) + - Redu (Orange) + - Ana (Orange) +--- + +- Mehdi: Preparation of OKDP slides with Lionel for Big Data & AI Paris 2025 +- Mehdi: Integration of Trino with TLS/SSL, OIDC, and UI deployment in the Sandbox +- Moncef: Comparison between Trino Open Source and Starburst version +- Serge: MinIO and OIDC authentication (integration with cert-manager) +- Idir: New version of the UI integrated into the Sandbox +- Lionel: Progress on Spark Docker images +- Ana: Hive Metastore +- Lionel: Adaptation of the Spark Auth Filter in the Orange context +- Mehdi & Lionel: Topics to be addressed – Group Mapping in Trino and OPA for authorization management diff --git a/users/2025/20250903.md b/users/2025/20250903.md new file mode 100644 index 0000000..8ae896a --- /dev/null +++ b/users/2025/20250903.md @@ -0,0 +1,23 @@ +--- +date: 2025-09-03 +type: users +attendees: + - Eliott M. (Adaltas) + - Lionel Herbet (Orange) + - Mehdi B. (DGFIP) + - Moncef Lahouar (DGFIP) + - Mori H. (Adaltas) + - Nicolas V. (DGFIP) + - Olivier M. (DGFIP) + - Serge A. (DGFIP) +--- + +- **Spark image**: Integration of the Orange PR with ongoing work on the build and integration of Spark images. +- **Demo**: Preparation and organization of a demonstration showcasing the new OKDP features. +- **Presentations (Pres)**: Progress on the presentation materials; latest version sent by Lionel and updated by Olivier. +- **Trino and MinIO features**: Discussions on new features to be developed and proposed in a community-driven, unitary, and modular approach (e.g., MinIO OIDC and Group Mapping in Trino). +- **Moncef**: Ongoing topics related to Trino and Starburst. +- **Serge / MinIO**: + - Access management via the console and API. + - OIDC integration with the console and user management. +- **Adaltas**: OKDP point to be scheduled. diff --git a/users/2025/20250910.md b/users/2025/20250910.md new file mode 100644 index 0000000..bdaa367 --- /dev/null +++ b/users/2025/20250910.md @@ -0,0 +1,26 @@ +--- +date: 2025-09-10 +type: users +attendees: + - Ana M. (Orange) + - Jean-Michel B. (Orange) + - Guillaume M. (DGFiP) + - Lionel Herbet (Orange) + - Marian G. (Orange) + - Mehdi B. (DGFiP) + - Moncef L. (DGFiP) + - Mori H. (Adaltas) + - Nicolas V. (modérateur) + - Olivier M. (DGFiP) + - Serge A. (DGFiP) +--- + +- Ana: managing GitHub permissions during Idir’s absence +- OKDP: publication of the sandbox +- Presentation slides: quiz to be added and session to be scheduled +- Mori: Adaltas update to be scheduled +- Mehdi: OKDP interaction with TDP currently being integrated +- Trino: addition of some security features + integration with existing technologies +- Moncef: ongoing study of Trino vs Starburst +- Moncef: integration of Airflow 3 +- Serge: OIDC and JWT as authentication protocols being generalized across all OKDP layers diff --git a/users/2025/20250917.md b/users/2025/20250917.md new file mode 100644 index 0000000..b4782a6 --- /dev/null +++ b/users/2025/20250917.md @@ -0,0 +1,19 @@ +--- +date: 2025-09-17 +type: users +attendees: + - Guillaume M. (DGFiP) + - Idir I. (DGFiP) + - Marian G. (Orange) + - Mehdi B. (DGFiP) + - Mori (Adaltas) + - Olivier M. (DGFiP) + - Pierre S. (Adaltas) + - Serge A. (DGFiP) +--- + +- Idir: latest updates to the Spark auth filter images have been incorporated, and the newest version is being deployed at Orange +- Idir: JupyterLab upgrade in progress +- Mehdi: TDP / OKDP (integration in progress) +- Slides: updates in progress +- Serge: OIDC and token management (Kubernetes integration) diff --git a/users/2025/20250924.md b/users/2025/20250924.md new file mode 100644 index 0000000..bd42279 --- /dev/null +++ b/users/2025/20250924.md @@ -0,0 +1,39 @@ +--- +date: 2025-09-24 +type: users +attendees: + - Abir O. (DGFIP) + - Anass Q. (Kubotal) + - Elliott M. (Adaltas) + - G. Mandier (DGFIP) + - Idir I. (DGFIP) + - Lionel H. (Orange) + - Marian G. (Orange) + - Mohame M. (Kubotal) + - Mehdi B. (DGFIP) + - Moncef L. (DGFIP) + - Mori H. (Adaltas) + - Nicolas V. (DGFIP) + - Olivier M. (DGFIP) + - Remi A. (DGFIP) + - Serge A. (DGFIP) + - Stephane B. (Orange) +--- + +- **GitHub access**: all accesses are now operational for Orange. +- **Mattermost**: to be used as the preferred channel for upcoming discussions. +- **Sandbox**: + - Jupyter: OK. + - Sandbox integration: adding extra parameters to improve. + - Superset: access to a private registry validated. +- **Moncef**: + - Airflow: in progress. + - Vault: to be planned. + - Spark Operator: to be planned. + - Metastore: to be planned. +- **Mehdi**: + - Trino avec TDP/HUE: work in progress. +- **Idir**: + - JupyterLab notebook images in multi-architectures (in progress). +- **Serge**: + - OIDC and sandbox integration (in progress). diff --git a/users/2025/20251015.md b/users/2025/20251015.md new file mode 100644 index 0000000..34f4b6a --- /dev/null +++ b/users/2025/20251015.md @@ -0,0 +1,28 @@ +--- +date: 2025-10-15 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier MAZAIN (DGFiP) + - Moncef Lahouar (DGFiP) + - Octave (Adaltas) + - Serge A. (DGFiP) + - Nicolas V. (DGFiP) + - Abir O. (DGFiP) + - Guillaume M. (DGFiP) + - Marian G. (Orange) + - Stéphane B. (Orange) + - Jean-Michel B. + - Idir I. (DGFiP) +--- + +- **Olivier:** Thanks to the team for the progress made and follow-up on the Apache incubation process. +- **Stéphane:** Testing the sandbox on a Windows/WSL machine (DNS issues to be fixed). +- **Adaltas:** Kubernetes deployments still outside the sandbox, migration planned soon. +- **Mohamed MASLO (Kubotal):** Working on integrating Spark images on OpenShift. +- **Moncef L.:** URSSAF is using the authentication Filter; Spark History Server Helm chart and Spark images are currently being integrated. +- **Anas (Kubotal):** LLM deployment tests within OKDP. +- **Moncef L.:** Ongoing progress on Spark integration with Airflow. +- **Idir I.:** For Stéphane, verifying that packages are automatically submitted. +- **Mehdi B.:** Preparing forms for component declaration and tracking within the sandbox. +- **Mehdi B.:** Ongoing discussions around Hue and JDBC driver integration for Trino. diff --git a/users/2025/20251022.md b/users/2025/20251022.md new file mode 100644 index 0000000..b48a949 --- /dev/null +++ b/users/2025/20251022.md @@ -0,0 +1,33 @@ +--- +date: 2025-10-22 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Idir I. (DGFiP) + - Abir O. (DGFiP) + - Serge A. (DGFiP) + - Alexis M. (Adaltas) + - Octave (Adaltas) + - Eliott M. (Adaltas) + - Jean Philippe S. (DGFiP) + - Stéphane B. (Orange) + - Marian G. (Orange) +--- + +- **No meeting planned for next week.** +- **Mehdi B.:** + - _Trino + Hue:_ Ongoing tests with JDBC and SQLAlchemy for Trino integration. +- **Idir:** + - Publication of JupyterLab images. + - Update of README files for all projects. + - Preparation of a sandbox deployment without Internet access. + - Drafting of the Sandbox OKDP README. +- **Serge A.:** + - _OIDC integration:_ Documentation in progress. + - Preparation of a demonstration for Orange. +- **Stéphane:** + - Added to Mattermost: OK. + - Validation of the Metastore GCS connector integration (merged by Idir) in progress. +- **Adaltas:** + - Sandbox integration in progress. diff --git a/users/2025/20251105.md b/users/2025/20251105.md new file mode 100644 index 0000000..1de3012 --- /dev/null +++ b/users/2025/20251105.md @@ -0,0 +1,25 @@ +--- +date: 2025-11-05 +type: users +attendees: + - Alexandre M. (CGI) + - Anass Q. (Kubotal) + - Mohamed M. (Kubotal) + - Mehdi B. (DGFIP) + - Idir I. (DGFIP) + - Olivier M. (DGFIP) + - Abir O. (DGFIP) + - Octave (Adaltas) + - Alexis M. (Adaltas) + - Moncef L. (DGFIP) + - Paul Adrien (Adaltas) + - Mori (Adaltas) +--- + +- **Olivier M.**: Feedback on the **Data Day at Bercy** +- **Mehdi B.**: Proposal to use **Querybook** as a SQL querying tool for **Trino** +- **Moncef L.**: + - Integration of **Airflow** into the sandbox + - Integration of **Spark Operator** into the sandbox + - **GPU testing** +- **Alexis M.**: Creation of an issue to set **resource** diff --git a/users/2025/20251109.md b/users/2025/20251109.md new file mode 100644 index 0000000..90b0e82 --- /dev/null +++ b/users/2025/20251109.md @@ -0,0 +1,65 @@ +--- +date: 2025-11-09 +type: users +attendees: + - Nicolas V. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Marian G. (DGFIP) + - Alexandre M. (CGI) + - Idir I. (DGFIP) + - Serge A. (DGFIP) + - Moncef L. (DGFIP) + - Rabah (BPCE) + - Lionel H. (Orange) + - Anass Q. (Kubotal) + - Octave (Adaltas) + - Pierre Adrien C. (Adaltas) + - Jerome D. (BPCE) +--- + +## Mehdi B. + +- Study of the integration of QueryBook connected to Trino. + +## Moncef L. + +- Progress on the integration of the DGFiP sandbox (configuration, testing, and documentation). +- Integration and validation of the proxy within the architecture. +- Refactoring of the OKDP UI to improve consistency, modularity, and maintainability. +- Update of the OKDP server and UI (bug fixes, optimization, stabilization). + +## Anass Q. + +- Integration tests of the Secret Manager within OKDP. +- Integration of the Secret Manager at the user interface level. + +## Idir + +- Work on DGFiP sandbox integration, with practical examples ready to be shared. + +## Serge + +- Study of quota management (possible approaches, technologies, and impacts on workloads). + +## Lionel H. + +- Suggestion: Volcano could be a relevant solution for workload management on Kubernetes. + +## Rabah (BPCE) + +- YuniKorn and Capsule were deployed and tested. +- Proposal: further study of Volcano for comparison. + +## Documentation & Communication + +- Updates to the website and technical documentation to be planned / continued. + +## Alexandre M. + +- Update on MinIO, now source-only: need to consider alternatives for deployments (external object storage recommended). +- Documentation: addition of a dedicated note/section explaining how to deploy the stack outside the sandbox. + +## Orange + +- Build and maintenance of Spark images for the OKDP ecosystem. diff --git a/users/2025/20251112.md b/users/2025/20251112.md new file mode 100644 index 0000000..4532420 --- /dev/null +++ b/users/2025/20251112.md @@ -0,0 +1,35 @@ +--- +date: 2025-11-12 +type: users +attendees: + - Idir I. (DGFIP) + - Alexandre M. (CGI) + - Anass Q. (Kubotal) + - Alexis M. (Adaltas) + - Guillaume M. (DGFIP) + - Moncef L. (DGFIP) + - Nicolas V. (DGFIP) + - Mehdi B. (DGFIP) + - Olivier M. (DGFIP) + - Abir O. (DGFIP) + - Octave (Adaltas) + - Lionel (Orange) +--- + +- **Olivier M.:** Review of OKDP **copyright and licenses** +- **Follow-up on:** + - **KubeCon**, **Kube Community Days**, and **CNCFCon** for a potential OKDP presentation + - **Review between Orange and DGFiP** +- **Validation of the selected UIs for Trino:** + - Work on integrating a **Trino JDBC kernel** into **Jupyter** + - Integration with **Superset** + - Deployment and integration with **QueryBook** +- **Monitor Trino-related needs** within **TOSIT** +- **Moncef L.:** Finalization in progress for **Airflow Operator integration** +- **Guillaume M.:** Keep an eye on **StarRocks** and **Apache Doris** +- **Alexis:** Sandbox issue and **resource limits management** created ([#29](https://github.com/OKDP/okdp-sandbox/issues/29)) +- **Keep Apache Hop** in the **OKDP roadmap** +- **Idir I.:** Sandbox refactoring in progress with: + - **Service abstraction** for better integration across contexts + - **Tuning of resource limits** +- **Mohamed M. & Anass Q.:** Integration of an **external secret management system** (e.g., **Vault**, via Secret Engine) diff --git a/users/2025/20251126.md b/users/2025/20251126.md new file mode 100644 index 0000000..40d5533 --- /dev/null +++ b/users/2025/20251126.md @@ -0,0 +1,40 @@ +--- +date: 2025-11-26 +type: users +attendees: + - Nicolas V. (DGFiP) + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Marian G. (DGFiP) + - Alexandre M. (CGI) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Moncef L. (DGFiP) + - Rabah (BPCE) + - Lionel H. (Orange) + - Anass Q. (Kubotal) + - Octave (Adaltas) + - Pierre-Adrien C. (Adaltas) + - Jérôme D. +--- + +## QueryBook + +- Possible alternatives and considerations regarding integration (full, partial, or isolated mode). +- Use of Elastic and Redis, not compatible with the Apache license. + +## Trino + +- Authentication: mechanisms under consideration and their impacts. + +## Contributions and Licensing + +- Copyright: formalization of the “OKDP Authors.” +- Review of dependencies and associated licenses. +- CLA: management of Contributor License Agreements. +- Extraction and consolidation of licenses. +- Code transfer process via PR to be verified. + +## Notebooks + +- Anas: ongoing comparison between Kubeflow Notebook and JupyterHub. diff --git a/users/2025/20251203.md b/users/2025/20251203.md new file mode 100644 index 0000000..ec771bc --- /dev/null +++ b/users/2025/20251203.md @@ -0,0 +1,23 @@ +--- +date: 2025-12-03 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Marian G. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Moncef L. (DGFiP) + - Octave (Adaltas) + - Guillaume M. (DGFiP) + - Alexis (Adaltas) + - Abir O. (DGFiP) + - Stéphane B. (Orange) +--- + +- **Mehdi:** QueryBook license under review and clarification. +- **Mehdi:** Automation of Trino and Hadoop/Hive integration (work in progress). +- **Idir:** Finalizing sandbox integration and preparing examples/samples. +- **Serge:** Evaluation of scheduling options (Unicorn, Volcano, QUEUE — excluding Spark, and Kordinator). +- **Stéphane (Orange):** Integration of Spark Images & Operator — Spark PR in progress (second approval pending). +- **Alexis:** The OKDP topic will be resolved shortly by Idir regarding OKDP sandbox tuning. diff --git a/users/2025/20251210.md b/users/2025/20251210.md new file mode 100644 index 0000000..b921ca6 --- /dev/null +++ b/users/2025/20251210.md @@ -0,0 +1,29 @@ +--- +date: 2025-12-10 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Marian G. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Moncef L. (DGFiP) + - Anas Q. (Kubotal) + - Guillaume M. (DGFiP) + - Alexandre M. (CGI) + - Alexis (Adaltas) + - Mori (Adaltas) +--- + +- **Olivier:** Participation in CIGREF and OSXP + ongoing work on open-source license management. +- **Mehdi:** Automation of Trino integration with Hive and HDFS on OCI. +- **Moncef:** Integration of the CAGIP Spark Proxy (issue being created regarding the minimum log size set to 10 MB in S3). +- **Idir:** Finalization of the sandbox integration. +- **Serge:** Deployment of the infrastructure for the scheduling system POC. +- **Anass:** Kubeflow integration in progress (authorization + authentication), currently using CLI mode. +- **Alexandre M.:** Memory issues in the OKDP sandbox (PR in progress by Idir) + deployment of the Spark Operator (Jupyter and the Operator to come). +- **OKDP V1 Release:** The official release date will be announced soon. +- **Adaltas:** Creation of tickets/issues across components to explore contributions to OKDP. +- **Mori:** Testing Apache Hive in GitOps mode. +- **Alexandre:** Comment regarding the redesign of the OKDP UI. +- **Mattermost:** New members added: Alexis, Anas, and Alexandre. diff --git a/users/2025/20251217.md b/users/2025/20251217.md new file mode 100644 index 0000000..acb72a4 --- /dev/null +++ b/users/2025/20251217.md @@ -0,0 +1,42 @@ +--- +date: 2025-12-17 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Moncef L. (DGFiP) + - Guillaume M. (DGFiP) + - Alexandre M. (CGI) + - Remi A. (DGFiP) + - Mori H. (Adaltas) + - Nicolas V. (DGFiP) + - Stephan B. (DGFiP) + - Stéphane (Orange) + - Paul F. (DGFiP) + - Pierre S. (Adaltas) + - Lionel H. (Orange) + - Octave (Adaltas) + - Pierre Adrien C. (Adaltas) +--- + +- **Lionel**: CLA to be validated internally +- **DigDash**: Study in progress for integration with (and not inside) OKDP +- **URSSAF**: Using DigDash on Kubernetes +- **Mehdi**: New PR for automating Trino integration with HDFS and Hive +- **Mehdi**: V1 features and timeline to be defined +- **Mehdi**: OIDC integration for Trino coordinators and integration with other layers such as Superset and Jupyter +- **Paul**: Dnsmasq PR is awaiting reviews +- **Pierre S.**: + - Why KuboCD? According to Moncef, it only provides shared modules and is not required (Helm charts and images are provided independently of KuboCD) + - Cassandra could be integrated in the future and a proposal was made to do so + - MinIO is used only for testing and must be replaced (Ceph, SeaweedFS S3?) +- **Stéphane**: Log issues with the new Spark version (SLF4J identified; fix in progress) +- **Paul**: Review process to be defined +- **Idir & Abir**: Spark integration with Jupyter (with examples) +- **Serge**: Infrastructure setup in progress (gang scheduling) +- **Mori**: + - Apache Hop and Apache Airflow to be integrated + - Creation of a new project for Hop under the GitHub organization +- **Paul**: Nicolas is working on the CLA and is requesting reviews (link available under `docs`) diff --git a/users/2026/20260107.md b/users/2026/20260107.md new file mode 100644 index 0000000..f348726 --- /dev/null +++ b/users/2026/20260107.md @@ -0,0 +1,25 @@ +--- +date: 2026-01-07 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Moncef L. (DGFiP) + - Abir O. (DGFiP) + - Nicolas V. (DGFiP) + - Stephan B. (DGFiP) + - Mikael D. (DGFiP) + - Lionel H. (Orange) +--- + +- **Olivier**: Initiated the CLA (Contributor License Agreement) on the DGFiP side; website update is upcoming. +- **Mehdi**: Validated the `Client Credentials` flow as the OIDC authentication standard for service accounts. +- **Serge**: Recommended against locking in a specific authentication flow to preserve platform flexibility. +- **Serge**: Continuing work on multi-tenancy, incorporating **Capsule** into the ongoing study. +- **Lionel**: To schedule a technical sync focused on the build process for the various technologies. +- **Idir**: Finalized the sandbox refactoring and test suite; the PR is currently being merged. +- **Moncef**: Upcoming presentation of the new OKDP UI version. +- **Stéphane**: Ongoing integration study regarding the replacement of Minio with **SeaweedFS**. +- **CI/CD**: Choice of **GitHub Actions** confirmed to implement and expose a public build pipeline. diff --git a/users/2026/20260114.md b/users/2026/20260114.md new file mode 100644 index 0000000..12df563 --- /dev/null +++ b/users/2026/20260114.md @@ -0,0 +1,34 @@ +--- +date: 2026-01-14 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Abir O. (DGFiP) + - Stephan B. (DGFiP) + - Mikael D. (DGFiP) + - Guillaume M. (DGFiP) + - Pierre S. (Adaltas) + - Lionel H. (Orange) + - Alexis (Adaltas) + - David W. (Adaltas) + - Paul-Adrien C. (Adaltas) +--- + +- **Mehdi:** Preparation of the presentation for the TOSIT Day on 01/21. +- **Lionel:** Technology build meeting scheduled for 01/15 at 11:00 AM. +- **Mehdi:** Client credentials OK; Trino / Superset OIDC integration in progress. +- **Lionel:** CLA currently under review on the Orange side, following validation on the DGFiP side. +- **Stéphane:** Integration of SeaweedFS as a replacement for MinIO, limited to the sandbox environment (STS to be validated). +- **David:** OIDC client integration via DCR (Lionel may be IdP-specific): + https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#client-registration-approaches +- **David:** Topics to be shared: NiFi Operator and Onyxia kick-off on the OECD side. +- **Olivier:** OKDP website update still in progress. +- **Serge:** Sharing the results of the Capsule study (project abandoned): + https://github.com/OKDP/studies/blob/main/multi-tenancy/capsule/report.md + +## To Be Scheduled + +- Planning session with Alexis to identify first good issues. diff --git a/users/2026/20260128.md b/users/2026/20260128.md new file mode 100644 index 0000000..9184aee --- /dev/null +++ b/users/2026/20260128.md @@ -0,0 +1,35 @@ +--- +date: 2026-01-28 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Abir O. (DGFiP) + - Stephan B. (DGFiP) + - Mikaël D. (DGFiP) + - Paul-Adrien C. (Adaltas) + - David W. (Adaltas) + - Mori H. (Adaltas) + - Alexandre M. (CGI) + - Achraf (Orange) + - Nicolas V. (DGFiP) + - Jean S. (DGFiP) + - Rémi A. (DGFiP) + - Octave (Adaltas) + - Paul (DGFiP) +--- + +- **Olivier:** OKDP presentation scheduled for TOSIT Day and CGI Cloud Native Days. +- **Olivier:** v1 scope to be defined. +- **Olivier:** DigDash integration possible, planning to follow with CGI. +- **Olivier:** CLA review and trademark registration on 02/17 with TOSIT. +- **Olivier:** Friday afternoon OKDP meeting to be scheduled starting in February. +- **Olivier:** Website update prioritized. +- **Mehdi:** Superset & Trino in OIDC (token management). +- **Stephan:** SeaweedFS integration in progress. +- **Paul-Adrien:** STS on SeaweedFS in progress (status OK). +- **Achraf:** Logs not uploading to GitHub, issue to be prioritized. +- **Alexandre M. (CGI):** Sandbox validated. +- **Team:** Technology builds to be prioritized. diff --git a/users/2026/20260204.md b/users/2026/20260204.md new file mode 100644 index 0000000..ba2b86c --- /dev/null +++ b/users/2026/20260204.md @@ -0,0 +1,30 @@ +--- +date: 2026-02-04 +type: users +attendees: + - Mehdi B. (DGFiP) + - Abir O. (DGFiP) + - Jean S. (DGFiP) + - Mikaël D. (DGFiP) + - Stephan B. (DGFiP) + - Nicolas V. (DGFiP) + - Idir I. (DGFiP) + - Pierre S. (Adaltas) + - Octave (Adaltas) + - Alexis (Adaltas) + - Lionel H. (Orange) + - Moncef L. (DGFiP) + - Anass Q. (Kubotal) +--- + +- Mehdi: Spark Trino authentication via code proposed by the Orange team. +- Mehdi: Polaris presentation to be scheduled. +- Mehdi: Technical weekly meeting scheduled for Friday at 2p.m . +- Mehdi: Dalibo starting work on Postgres on Kubernetes. +- Lionel: GitHub discussions created to define the agenda. +- Lionel: Open source IAM to be studied (https://ferriskey.rs/). +- Stephan: Spark Operator on Jupyter Enterprise Gateway OK (except Spark image rebuild + kernel addition). Spark Connect PoC in progress to simplify integration. +- Stephan: Full integration of SeaweedFS OK. +- Moncef: Spark operator integration in progress. +- Moncef: Progress on OKDP backend and UI. +- Abir & Moncef: Airflow integration. diff --git a/users/2026/20260211.md b/users/2026/20260211.md new file mode 100644 index 0000000..cab0b65 --- /dev/null +++ b/users/2026/20260211.md @@ -0,0 +1,29 @@ +--- +date: 2026-02-11 +type: users +attendees: + - Mehdi B. (DGFiP) + - Olivier M. (DGFiP) + - Abir O. (DGFiP) + - Stephan B. (DGFiP) + - Paul (DGFiP) + - Idir I. (DGFiP) + - Serge A. (DGFiP) + - Jean S. (DGFiP) + - Moncef L. (DGFiP) + - Alexis M. (DGFiP) + - Alexandre M. (CGI) + - Lionel H. (Orange) + - Achraf I. (Orange) + - Octave (Adaltas) + - G. Mandier (DGFiP) +--- + +- **Mehdi:** Polaris presentation completed; implementation to be started. +- **Stephan:** Spark Jupyter: continuing discussions on integration implementation methods. +- **Mehdi:** OIDC Superset with Trino: in progress / Orange source code to be updated. +- **OIDC Jupyter:** Pending follow-up. +- **Idir:** Spark ticket for Orange currently in progress. +- **Paul:** Website updated. +- **Olivier:** v1 and roadmap to be defined. +- **Abir:** Airflow operational; DAGs and documentation in progress.