Feat: devcontainer pytest integration by junkataoka · Pull Request #110 · microsoft/dstoolkit-devcontainers

junkataoka · 2025-12-16T07:58:38Z

This pull request introduces a major refactor and modernization of the CI pipeline infrastructure for both Azure DevOps and GitHub Actions. The main improvements are the switch to modular, template-driven pipeline definitions, enhanced support for devcontainer-based testing, and improved code coverage reporting. The changes streamline CI setup, make it easier to add new projects, and improve compatibility across platforms (Linux and macOS). Documentation and configuration files have also been updated for clarity and consistency.

Azure DevOps Pipeline Modernization

Both .azuredevops/ado-ci-pipeline-ms-hosted.yml and .azuredevops/ado-ci-pipeline-self-hosted.yml are refactored to use a two-stage pipeline (Lint and Test), with jobs and steps defined via new templates for devcontainer testing and coverage publishing. This replaces legacy scripts and direct task definitions, making the pipeline more maintainable and extensible. [1] [2] [3]

Template-Driven CI Architecture

New reusable templates are introduced under .azuredevops/templates/, including test-devcontainer-job.yml for per-project devcontainer testing, run-devcontainer.yml for platform-aware container execution, setup-devcontainer.yml for environment setup, publish-test-results.yml for publishing test results and coverage, and merge-coverage.yml for merging and publishing code coverage reports. [1] [2] [3] [4] [5]

GitHub Actions Pipeline Improvements

.github/workflows/ci.yaml is reworked to use matrix jobs for devcontainer-based testing, artifact upload for results and coverage, and a dedicated lint job. A new job aggregates and publishes combined test results and coverage reports, improving visibility and maintainability. [1] [2]

Documentation and Configuration Updates

References to legacy scripts (e.g., ci-tests.sh) are removed from README.md and .env.example is updated for clarity, reflecting the new devcontainer and pipeline workflow. [1] [2] [3]

Platform Compatibility and Usability Enhancements

The new pipeline templates include logic for handling both Linux and macOS (Apple Silicon) agents, and ensure clean workspace setup and artifact handling for reliable CI runs. [1] [2]

Does this introduce a breaking change?

Yes
No

Author pre-publish checklist

No PII in logs or output
Made corresponding changes to the documentation
All new packages used are included in requirements.txt
Functions use type hints, and there are no type hint errors

Pull Request Type

What kind of change does this Pull Request introduce?

junkataoka · 2025-12-16T08:21:58Z

@junkataoka please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

@microsoft-github-policy-service agree [company="microsoft"]

junkataoka · 2025-12-16T16:39:44Z

@microsoft-github-policy-service agree
@microsoft-github-policy-service agree company="Microsoft"

- Add GitHub Actions workflow using official devcontainers/ci@v0.3 action - Add Azure DevOps pipeline with custom templates for macOS compatibility - Configure pytest with coverage reporting and JUnit XML output - Handle workspace permission issues using /tmp with sudo - Support CPU, GPU, and notebooks project configurations - Add coverage merging and test result publishing for both platforms - Skip pre-commit hook installation in CI when .git not available - Add sync-to-azdo workflow for pipeline synchronization - Update documentation with CI/CD setup instructions GitHub Actions uses the official action which works well on Linux. Azure DevOps uses custom templates to handle macOS-specific Docker issues (path translation and working directory mount namespace errors).

Copilot

Pull request overview

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pyproject.toml

.github/workflows/ci.yaml

fujikosu · 2026-02-06T10:01:08Z

.gitignore

+
+# Speckit / .specify related files
+.specify/
+specs/
+*.speckit
+
+# .github agents and prompts
+.github/agents/
+.github/prompts/


should these be out of .gitignore? we often manage .github/agents/ .github/prompts/ as a team and share prompts

fujikosu · 2026-02-06T10:02:22Z

.env.example

+# For users who need AML, create .env in your project folder (e.g., src/sample_pytorch_gpu_project/.env)
+# Optional: Set these variables if using the AML CLI v2 example under src/sample_pytorch_gpu_project


what's the reason of deletion?

fujikosu · 2026-02-06T10:15:22Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+    "containerEnv": {
+        "PYTHONPATH": "/workspaces/repo"


Is this a wrong path? Also as long as .env is there, it should be picked up automatically by vscode so this one is not needed

fujikosu · 2026-02-06T10:16:58Z

src/sample_cpu_project/.devcontainer/devcontainer.json

-    "postCreateCommand": "pre-commit install --overwrite",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",


USER $USERNAME in Dockerfile side applies the same effect so this would be redundant

fujikosu · 2026-02-06T10:17:06Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+    },
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",


this should be automatically set by "ghcr.io/devcontainers/features/common-utils:2" above?

fujikosu · 2026-02-06T10:18:23Z

src/sample_cpu_project/.devcontainer/devcontainer.json

+                }
+            }
        }
-    },
+    }


is this your vscode's setting? it'd be good to keep diff minimal as this PR is quite large and format change may not be worth including?

fujikosu · 2026-02-06T10:18:29Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

                }
-            },
+            }
        }
-    },
+    }


is this your vscode's setting? it'd be good to keep diff minimal as this PR is quite large and format change may not be worth including?

fujikosu · 2026-02-06T10:21:47Z

src/sample_cpu_project/.devcontainer/devcontainer.json

+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",
+    "postCreateCommand": "cd /workspaces/repo && pre-commit install --overwrite || echo 'pre-commit installation failed; continuing without pre-commit hooks'",


is there any benefit of still going ahead when pre-commit fails? I'd rather it fail there as something would be wrong in that case

fujikosu · 2026-02-06T10:22:01Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",
+    "postCreateCommand": "cd /workspaces/repo && pre-commit install --overwrite || echo 'pre-commit installation failed; continuing without pre-commit hooks'",


same as above

fujikosu · 2026-02-06T10:27:28Z

notebooks/.devcontainer/devcontainer.json

-    "postCreateCommand": "pre-commit install --overwrite",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",


USER $USERNAME in Dockerfile side applies the same effect so this would be redundant

fujikosu · 2026-02-06T10:29:46Z

notebooks/.devcontainer/devcontainer.json

    ],
-    "postCreateCommand": "pre-commit install --overwrite",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",


any strong reason we want to introduce this path and realted settings? Otherwise, I'd prefer to remove it to keep devcontainer.json bare minimum

fujikosu · 2026-02-06T10:30:05Z

notebooks/.devcontainer/devcontainer.json

+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",
+    "remoteUser": "devuser",
+    "postCreateCommand": "cd /workspaces/repo && pre-commit install --overwrite || echo 'pre-commit installation failed; continuing without pre-commit hooks'",


same as others

fujikosu · 2026-02-06T10:30:19Z

src/sample_cpu_project/.devcontainer/devcontainer.json

    ],
-    "postCreateCommand": "pre-commit install --overwrite",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",


any strong reason we want to introduce this path? Otherwise, I'd prefer to remove it to keep devcontainer.json bare minimum

fujikosu · 2026-02-06T10:31:12Z

src/sample_cpu_project/.devcontainer/devcontainer.json

+        "${localWorkspaceFolder}/.env"
    ],
-    "postCreateCommand": "pre-commit install --overwrite",
+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",


if we don't introduce "/workspaces/repo, I think this mount automatically runs without us specifying anything

fujikosu · 2026-02-06T10:53:04Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+            "username": "devuser"
+        }
+    },
+    "initializeCommand": "test -f ${localWorkspaceFolder}/.env || touch ${localWorkspaceFolder}/.env",


how about creating .env from .env.example instead?

fujikosu · 2026-02-06T10:56:12Z

notebooks/.devcontainer/devcontainer.json

        "dockerfile": "Dockerfile"
    },
    "shutdownAction": "none",
+    "updateRemoteUserUID": false,


default setting =true allows devcontainer to match local side UID and avoids file permission issues from happening (ex: files created within container owned by different UID and local can't access it). Is there any benefit with this? I see file access issue being avoided would be better but I'm curious to hear your thoughts

fujikosu · 2026-02-06T10:59:28Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+    "hostRequirements": {
+        "gpu": "optional"
+    },


I didn't know about this option! This is cool as it can still work when gpu is not available while supporting gpu!

fujikosu · 2026-02-06T10:59:41Z

src/sample_pytorch_gpu_project/.devcontainer/devcontainer.json

+    "workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
+    "workspaceFolder": "/workspaces/repo",


same as above

fujikosu · 2026-02-06T15:45:21Z

.azuredevops/ado-ci-pipeline-ms-hosted.yml

+# Azure DevOps pipeline for CI (Microsoft-hosted version)
+# As the Microsoft-hosted agent option has a limit of 10GB of storage for disk outputs from a pipeline,
+# this causes an issue when the Docker images for modules under src require more than 10GB of storage.
+# If you will run into space issues (or other limitations with a Microsoft hosted agent option outlined in
+# https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#capabilities-and-limitations),
+# consider using the .azuredevops/ado-ci-pipeline-self-hosted.yml version or using scale set agents, see
+# this link for more info: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops
+# Note that docker images will only be build for src directories that contain at least one test file, so the
+# total space consumed by Docker builds will be dependent on which modules under src contain tests.
+# For setting up the pipeline in ADO see:
+# https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&tabs=yaml%2Cbrowser


what's the reason for diff showing up here? It looks the same?

fujikosu · 2026-02-06T15:46:26Z

.azuredevops/ado-ci-pipeline-ms-hosted.yml

+        pool:
+          vmImage: 'ubuntu-latest'


I see every stage uses the same pool. so should we stick with the previous design where we define it once for all?

fujikosu · 2026-02-06T15:47:54Z

.azuredevops/ado-ci-pipeline-self-hosted.yml

-# but could be adapated to other OSs.
-
+# Azure DevOps CI Pipeline (Self-Hosted Agent)
+#


this 10GB part was the reason why we needed two different yamls. Could you check if it's ok to delete this part? If we still have this restriction, we likely need some cleanup method we had for self-hosted version specifically

fujikosu · 2026-02-06T15:50:04Z

.github/workflows/ci.yaml

-      - name: Checkout code
-        uses: actions/checkout@v6
+      - name: Checkout repository
+        uses: actions/checkout@v4


why does this degrade version?

fujikosu · 2026-02-06T15:52:09Z

.github/workflows/ci.yaml

+  # Run linter on codebase
+  lint:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Setup Python 3.11
+        uses: actions/setup-python@v5
        with:
-          name: test-and-coverage-results
-          path: |
-            **/test-reuslts-*.xml
-            coverage.xml
+          python-version: 3.11
+
+      - name: Install requirements
+        run: |
+          python -m venv venv
+          source venv/bin/activate
+          python -m pip install --upgrade pip
+          pip install -r requirements-dev.txt
+
+      - name: Run ruff linter
+        run: |
+          source venv/bin/activate
+          ruff check --output-format github


It'd be better to keep this at the previous position as this part would run faster and if we fail at linter, we don't need to proceed further and that'd help us iterate faster

fujikosu · 2026-02-06T15:52:53Z

.github/workflows/ci.yaml

+          filename: 'coverage/**/*.xml'

-      - name: Add code coverage summary markdown to github step summary
+      - name: Add code coverage summary to job summary


I think this is not accurate?

fujikosu · 2026-02-06T15:54:48Z

.github/workflows/ci.yaml

+  # Smoke test for notebooks devcontainer
+  test-notebooks:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Setup .env file
+        run: cp .env.example .env
+
+      - name: Build and smoke test notebooks devcontainer
+        uses: devcontainers/ci@v0.3
+        with:
+          configFile: notebooks/.devcontainer/devcontainer.json
+          push: never
+          runCmd: |
+            cd /workspaces/repo
+            python --version
+            pytest --version
+            ruff --version


could we merge this to the previous section's build check?

fujikosu · 2026-02-06T15:58:55Z

.github/workflows/ci.yaml

@@ -13,44 +13,109 @@ permissions:
  contents: read


I wonder if it's really worth splitting into several jobs instead of having having them as multiple steps based on job complextity we have. One thing I noticed is this does repo clone 4 times as it used to be just one for example and that's one of downsides of going with this structure

fujikosu

Thanks for your PR! I left several comments so please check them

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from 44faaca to ec09cba Compare December 16, 2025 08:43

junkataoka force-pushed the feature/devcontainer-pytest-integration branch 5 times, most recently from 2523a23 to 0ebed45 Compare December 18, 2025 15:06

junkataoka changed the title ~~Feature/devcontainer pytest integration~~ feat: devcontainer pytest integration Dec 18, 2025

junkataoka changed the title ~~feat: devcontainer pytest integration~~ Feat: devcontainer pytest integration Dec 18, 2025

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from d73bb8e to 82fb7b9 Compare December 19, 2025 00:10

junkataoka requested a review from Copilot December 19, 2025 00:20

Copilot started reviewing on behalf of junkataoka December 19, 2025 00:20 View session

This comment was marked as outdated.

Sign in to view

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from fe5a82e to 78f787f Compare December 19, 2025 01:29

junkataoka requested a review from Copilot December 19, 2025 01:48

Copilot started reviewing on behalf of junkataoka December 19, 2025 01:48 View session

This comment was marked as outdated.

Sign in to view

junkataoka force-pushed the feature/devcontainer-pytest-integration branch 3 times, most recently from f22f8ef to d94a2d6 Compare December 19, 2025 02:55

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from 672b9c6 to 164e6b9 Compare December 19, 2025 04:51

patch: remove azdo sync pipeline as it was for testing purpose

5f6316e

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from a279838 to 5f6316e Compare December 19, 2025 05:03

junkataoka requested a review from Copilot December 19, 2025 05:04

Copilot started reviewing on behalf of junkataoka December 19, 2025 05:05 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

.github/workflows/ci.yaml Show resolved Hide resolved

junkataoka marked this pull request as ready for review December 19, 2025 05:21

junkataoka requested a review from bhavikm December 19, 2025 05:21

junkataoka requested a review from fujikosu December 19, 2025 05:21

junkataoka force-pushed the feature/devcontainer-pytest-integration branch from 0e665e8 to 5f6316e Compare December 19, 2025 09:25

fujikosu reviewed Feb 6, 2026

View reviewed changes

fujikosu requested changes Feb 6, 2026

View reviewed changes

		# For users who need AML, create .env in your project folder (e.g., src/sample_pytorch_gpu_project/.env)
		# Optional: Set these variables if using the AML CLI v2 example under src/sample_pytorch_gpu_project

		"workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/repo,type=bind",
		"workspaceFolder": "/workspaces/repo",

Conversation

junkataoka commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Does this introduce a breaking change?

Author pre-publish checklist

Pull Request Type

Uh oh!

junkataoka commented Dec 16, 2025

Uh oh!

junkataoka commented Dec 16, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fujikosu Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fujikosu Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fujikosu left a comment

Choose a reason for hiding this comment

Uh oh!

junkataoka commented Dec 16, 2025 •

edited

Loading

fujikosu Feb 6, 2026 •

edited

Loading

fujikosu Feb 6, 2026 •

edited

Loading