feat(cds): EQC corpus tooling and dataset EQC search#8
Closed
yellowcap wants to merge 11 commits into
Closed
Conversation
* fix: address missing/incorrect steps in readme for initial deploy (and applied them) * fix: unblock workflows for initial deploy
Add an EQC (Evaluation Quality Control) corpus pipeline for the CDS toolset: discovery, fetch, parse, normalize, clean, and a model2vec semantic index, exposed via the get_dataset_eqc and search_eqc tools. Includes a snapshot script with S3 push/pull, a CI workflow to refresh the corpus, and a Dockerfile step to pull the snapshot at build time.
- Dockerfile: always mkdir the cds data dir so the unconditional COPY succeeds for every toolset build (not just cds with a snapshot URI). - lint: broaden the scripts/** ruff S-ignore to **/scripts/** (covers nested toolset scripts), drop a type-narrowing assert (S101), mark the request-jitter random as non-crypto (S311), and ignore boto3 stubs in mypy. Apply ruff format to the new EQC files.
Member
Author
|
@ciaransweet for deployment of the EQC work, we need to build an index and add it to the instances somehow. Lets discuss how to best go about that. Not sure in K8s what the best way is (for instance, does the ECMWF k8s support S3 buckets?) |
Publish the EQC corpus and semantic index (with the bundled embedding model) as the `eqc-data` workflow artifact and download it into the cds docker build context, so the deployed image is self-contained and works offline. Replaces the half-wired S3 snapshot path, which deploy.yml never fed and which left the index absent in deployment. - cds-eqc-data.yml: upload data/ as `eqc-data` artifact; drop S3 steps - deploy.yml: download latest `eqc-data` into the cds build context, with an inline-build cold-start fallback - Dockerfile: drop CDS_EQC_S3_URI/S3 pull; keep the data COPY - remove dead eqc_snapshot.py and unused boto3 dependency - docs: describe the artifact flow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the CDS-side EQC work out of the
evals-frameworkbranch into its own PR.What
Adds an EQC (Evaluation Quality Control) corpus pipeline for the CDS toolset:
cds.eqcpackage: discovery, fetch, parse, normalize, clean, and amodel2vecsemantic index over the corpus.get_dataset_eqcandsearch_eqc.scripts/fetch_eqc_corpus.pyto build the corpus + index.Deployment (corpus baked into the image via build artifact)
The
data/corpus is gitignored, so it ships into the image as a CI artifact rather than being committed — keeping the 32M embedding model out of git history while keeping the runtime fully offline (no HuggingFace dependency):.github/workflows/cds-eqc-data.yml(weekly +workflow_dispatch) builds the corpus + index and uploadstoolsets/cds/data/as theeqc-dataartifact..github/workflows/deploy.ymldownloads the latest successfuleqc-dataartifact into the cds build context before building, with an inline-build cold-start fallback.COPY . .then bakes it in.Dockerfilecopies the data into the image (no S3 pull).Notes
numpy,model2vec,trafilatura;uv.lockregenerated.evals).search_eqc/get_dataset_eqcoffline (--network none) from the baked-in corpus + bundled model.