Repository navigation
Google drive api - #19
Conversation
|
|
||
| per_class = 5 # Fixed set size for CI | ||
|
|
There was a problem hiding this comment.
The value per_class = 5 is currently hardcoded, which limits configurability and might force us to modify the script when CI requirements change or Drive folder grows. Hardcoded operational parameters can be discouraged in production pipelines because they reduce reproducibility and flexibility. GitHub suggests utilizing environment variables as much as possible as opposed to hardcoding, as it allows a little more flexibility when adjusting values across runs: (https://docs.github.com/en/actions/how-tos/deploy/configure-and-manage-deployments/manage-environments)
It appears that the CI_FILES_PER_CLASS environment variable is defined, but never used within the script. We could potentially utilize this environment variable with a safe default (5) to allow us to change this value without directing changing the code.
Example:
per_class = int(os.environ.get("CI_FILES_PER_CLASS", 5))| def download_file_bytes(service, file_id: str) -> bytes: | ||
| request = service.files().get_media(fileId=file_id, supportsAllDrives=_use_all_drives()) | ||
| buf = io.BytesIO() | ||
| downloader = MediaIoBaseDownload(buf, request, chunksize=1024 * 1024) | ||
| done = False | ||
| while not done: | ||
| status, done = downloader.next_chunk() | ||
| # Optionally, could print progress: status.progress() | ||
| return buf.getvalue() |
There was a problem hiding this comment.
Google Drive API calls (especially MediaIoBaseDownload) can return transient 5xx errors, connection resets, or throttling responses. Currently, the download loop performs no retry logic. Without retry logic, the pipeline can intermittently fail, especially if hundreds of PDFs are being downloaded from the useful and not useful folders. Since the function call grants read access to the dataset, it might be worth it to make the workflow should confirm that the credentials only request the minimum OAuth scopes required (read only).
Google recommends retrying transient Drive API errors with exponential backoff:
Google Drive API Best Practices – Retry Handling
https://developers.google.com/drive/api/guides/handle-errors
Google Cloud General Retry Guidance
https://cloud.google.com/storage/docs/retry-strategy
We should consider wrapping the MediaIoBaseDownload chunk loop in a retry loop (maybe 3–5 attempts with exponential backoff?) so that in the case that there are connection issues, it will have a chance to recover without immediately failing.
| if tiny_context: | ||
| # For tiny datasets, attempt a 50/50 split; if that fails, fall back to train-only. | ||
| test_size = 0.5 | ||
| try: | ||
| X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=0.2, random_state=42, stratify=labels) | ||
| X_train, X_test, y_train, y_test = train_test_split(texts, labels, test_size=test_size, random_state=42, stratify=labels) | ||
| except ValueError as e: | ||
| print(f"[ERROR] train_test_split failed: {e}") | ||
| return None | ||
| print(f"[WARN] Stratified split failed on tiny dataset: {e} — training on all data without evaluation.") | ||
| perform_eval = False | ||
| X_train = texts | ||
| y_train = labels |
There was a problem hiding this comment.
The fallback behavior for small CI datasets is to attempt a 50/50 split, then fall back to train only mode if stratification fails. However if the dataset is below a certain threshold, it can potentially produce misleading accuracy metrics and it might be worth it to just avoid the evaluation entirely when the dataset is that small. In other words, we should consider explicitly skipping evaluation when CI_TRAIN = 1 or when the dataset has less than 10 PDFs.
The fallback behavior for small datasets attempts a 50/50 stratified split and then switches to train only mode if the split fails. However, evaluating a model on extremely small datasets (<10 samples or highly imbalanced classes) can produce accuracy metrics that are very misleading. Small sample evaluation introduces misleading variance and does not represent the model's true accuracy. The scikit-learn model evaluation guide warns against test sets that are too small (https://scikit-learn.org/stable/modules/cross_validation.html#limitations). This can affect CI test coverage accuracy because model evaluation acts as a validation check. With extremely small datasets, the metrics become nondeterministic and could cause CI to produce unstable test results.
It might be worth it to just avoid the evaluation entirely when the dataset is that small. In other words, we should consider explicitly skipping evaluation when CI_TRAIN = 1 or when the dataset has less than 10 PDFs instead of allowing it to fail.
Example:
if os.environ.get("CI_TRAIN") or len(labels) < 10:
perform_eval = False
This pull request introduces a robust and flexible pipeline for extracting text from PDFs and training a classification model, with support for both Google Drive streaming and local datasets. The changes refactor the CI and full training workflows, add new utility modules for environment variable loading and Google Drive integration, and update documentation for clarity. The CI pipeline now streams a sample from Drive, while the full pipeline script supports both API and local modes. These improvements enable easier scaling, reproducible runs, and better separation between CI and full training jobs.
Pipeline and Workflow Enhancements:
scripts/ci_pipeline.pyto stream a small, fixed sample of PDFs from Google Drive, extract text, generate labels, and train the model in CI. This script uses environment variables for authentication and folder selection, and does not persist PDFs locally.scripts/full_pipeline.pyto run the complete training pipeline, supporting both API mode (streaming all PDFs from Drive) and local mode (processing locally downloaded PDFs). This script generates labels and trains the model after extraction..github/workflows/working_sw.ymlto integrate the new CI pipeline, enable model artifact caching and uploading, and clarify the separation between CI and full training workflows. Optional steps for local extraction and preprocessing are now commented for clarity.Google Drive Integration:
scripts/google_drive/drive_io.py, a utility module for streaming PDF bytes directly from Google Drive using a service account. It supports shared drives and provides functions for listing, downloading, and sanitizing file names.scripts/google_drive/download_from_drive.pyshowing authentication and credential loading from environment variables.Environment Variable Management:
scripts/env_loader.pyto automatically load environment variables from a.envfile for local development, ensuring sensitive information is never hardcoded.Documentation Updates:
documentation/CONTRIBUTING.mdto explain how to run the application with either a local dataset or via the API, including required folder structure and environment variable setup.Code Cleanup:
scripts/run_pipeline.pyin favor of the new, more flexible pipeline scripts.Minor Fixes:
osinsrc/model/train_model.pyto support new environment variable usage.These changes collectively modernize the pipeline, improve maintainability, and make it easier to run and test the model training process in both CI and local environments.