From 844347faf9a771098eae68b56ba00b75379586ff Mon Sep 17 00:00:00 2001 From: Stefan Jansen Date: Mon, 12 Oct 2026 20:33:05 -0400 Subject: [PATCH 1/2] docs: adopt shared contribution policy --- .github/ISSUE_TEMPLATE/bug.yml | 37 ++-- .github/ISSUE_TEMPLATE/config.yml | 6 +- .github/ISSUE_TEMPLATE/documentation.yml | 20 +-- .github/ISSUE_TEMPLATE/feature.yml | 20 +-- .github/PULL_REQUEST_TEMPLATE.md | 38 +++- CONTRIBUTING.md | 219 +++++++---------------- 6 files changed, 127 insertions(+), 213 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml index 83cb6947..b7040d9d 100644 --- a/.github/ISSUE_TEMPLATE/bug.yml +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -1,5 +1,5 @@ name: Bug report -description: Report reproducible incorrect behavior in ecosystem policy or tooling +description: Report reproducible incorrect behavior in ml4t-data title: "[Bug]: " labels: - "type: bug" @@ -10,22 +10,10 @@ body: - type: markdown attributes: value: | - Search existing issues first. Do not disclose suspected vulnerabilities here. - - type: dropdown - id: library - attributes: - label: Affected library - options: - - data - - engineer - - backtest - - specs - - live - - diagnostic - - models - - ecosystem - validations: - required: true + Search existing issues first. Report one problem per issue and use separate issues for + unrelated problems. Read the + [shared contribution policy](https://github.com/ml4t/ecosystem/blob/main/CONTRIBUTING.md) + before submitting. Do not disclose suspected vulnerabilities here. - type: input id: version attributes: @@ -37,7 +25,8 @@ body: id: environment attributes: label: Environment - description: Include Python version, operating system, installation method, and relevant extras. + description: >- + Include Python version, operating system, installation method, and relevant extras. validations: required: true - type: textarea @@ -56,7 +45,17 @@ body: id: reproduction attributes: label: Minimal reproduction - description: Provide the smallest complete example and error output. + description: Provide the smallest complete public API or user workflow and its error output. + validations: + required: true + - type: dropdown + id: numerical + attributes: + label: Numerical impact + options: + - Produces an incorrect numerical result + - Changes a documented definition or convention + - No known numerical impact validations: required: true - type: dropdown diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 80255174..c3df4b47 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -3,6 +3,6 @@ contact_links: - name: Security vulnerability url: https://github.com/ml4t/data/security/advisories/new about: Report suspected vulnerabilities privately. - - name: Usage questions - url: https://github.com/orgs/ml4t/discussions - about: Ask questions that do not describe a defect or requested change. + - name: Documentation and usage guidance + url: https://www.ml4trading.io/docs/data/ + about: Read the supported workflows and provider guidance before opening an issue. diff --git a/.github/ISSUE_TEMPLATE/documentation.yml b/.github/ISSUE_TEMPLATE/documentation.yml index d060f5a8..0ae40819 100644 --- a/.github/ISSUE_TEMPLATE/documentation.yml +++ b/.github/ISSUE_TEMPLATE/documentation.yml @@ -7,21 +7,13 @@ labels: - "status: needs-triage" - "compatibility: none" body: - - type: dropdown - id: library + - type: markdown attributes: - label: Affected library - options: - - data - - engineer - - backtest - - specs - - live - - diagnostic - - models - - ecosystem - validations: - required: true + value: | + Search existing issues first. Report one documentation problem per issue and separate + unrelated changes. Read the + [shared contribution policy](https://github.com/ml4t/ecosystem/blob/main/CONTRIBUTING.md) + before submitting. - type: input id: version attributes: diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml index ca17225c..4d892ef7 100644 --- a/.github/ISSUE_TEMPLATE/feature.yml +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -7,21 +7,13 @@ labels: - "status: needs-triage" - "compatibility: affected" body: - - type: dropdown - id: library + - type: markdown attributes: - label: Owning library - options: - - data - - engineer - - backtest - - specs - - live - - diagnostic - - models - - ecosystem - validations: - required: true + value: | + Search existing issues first. Describe one coherent change per issue and separate unrelated + requests. Discuss the intended behavior here before opening a pull request. Read the + [shared contribution policy](https://github.com/ml4t/ecosystem/blob/main/CONTRIBUTING.md) + before submitting. - type: input id: version attributes: diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 0d924318..76de6fa4 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,26 +1,50 @@ +Read the [ML4T contribution policy](https://github.com/ml4t/ecosystem/blob/main/CONTRIBUTING.md) +before submitting. + ## Owning issue Closes # +The issue and this pull request each describe one problem or coherent change. Separate unrelated +fixes. Automated dependency updates and administrative metadata changes are exempt from the issue +requirement. + ## Outcome -Describe the user-visible or standards outcome. +Describe the problem, expected behavior, and what this change does. -## Compatibility +## Compatibility and numerical behavior - [ ] No public compatibility impact - [ ] Compatibility impact is documented in the owning issue and migration guidance +- [ ] No numerical result or documented definition changes +- [ ] A numerical defect is reproduced below and corrected by this change +- [ ] An intended definition or convention change was agreed in the owning issue and documented ## Verification -- [ ] Tests reproduce the prior failure or verify the requested behavior +For a bug fix, run the same regression through the affected public API or user workflow on both +revisions: + +- Base revision, command, and failing result: +- Pull-request revision, same command, and passing result: + +For other changes, state the relevant commands and results. + +- [ ] Verification exercises the affected public API or user workflow - [ ] Ruff lint and format checks pass - [ ] `ty` passes -- [ ] Test suite passes -- [ ] Package build passes when applicable -- [ ] Strict MkDocs build passes when documentation is affected +- [ ] The applicable test lanes pass +- [ ] The package build passes when applicable +- [ ] Strict MkDocs passes when documentation is affected - [ ] Ecosystem qualification passes +## AI assistance + +State whether AI tools helped produce code or prose and identify the affected parts. + +- [ ] I reviewed and tested this submission and can explain its code, prose, and evidence + ## Documentation and release -State the documentation, release notes, and patch-release work required after merge. +State the documentation, release-note, and patch-release work required after merge. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ad2c0eeb..b328a1bc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,182 +1,89 @@ # Contributing to ml4t-data -Thank you for your interest in contributing to ml4t-data. This guide covers everything you need to get started. +The [ML4T contribution policy](https://github.com/ml4t/ecosystem/blob/main/CONTRIBUTING.md) +defines the issue, pull-request, regression-evidence, and accountability requirements for this +repository. This guide adds the setup and verification steps specific to `ml4t-data`. -## Development Setup +## Before opening an issue -```bash -# Clone the repository -git clone https://github.com/ml4t/data.git ml4t-data -cd ml4t-data - -# Install dependencies (requires uv) -uv sync --dev - -# Install pre-commit hooks -uv run pre-commit install +Search the existing issues first. Each issue must describe one problem or coherent change. Open +separate issues for unrelated work. -# Verify your setup -uv run pytest tests/ -q -``` +A bug report must identify the affected package version and environment, state the observed and +expected behavior, and include the smallest complete reproduction through a public API or user +workflow. Report suspected vulnerabilities through +[private vulnerability reporting](https://github.com/ml4t/data/security/advisories/new), not a +public issue. -**Requirements**: Python 3.11+ and [uv](https://docs.astral.sh/uv/). +Discuss substantial changes in their owning issue before implementing them. A change to an intended +definition, numerical convention, default, or compatibility contract requires agreement on the +expected behavior and migration impact. -## Quality Gates +## Set up the repository -All changes must pass these checks before merging: +`ml4t-data` supports Python 3.12 through 3.14 and uses +[uv](https://docs.astral.sh/uv/) for environments and commands. ```bash -# Lint and format -uv run ruff check src/ tests/ -uv run ruff format --check src/ tests/ - -# Type check -uv run ty check - -# Tests (core suite, excludes integration/slow/paid) -uv run pytest tests/ -q - -# Resource leak check -uv run pytest tests/ -q -W error::ResourceWarning - -# Run everything at once -uv run pre-commit run --all-files -``` - -## Code Style - -- **Formatter/linter**: [ruff](https://docs.astral.sh/ruff/) (100 char line length) -- **Type checker**: [ty](https://docs.astral.sh/ty/) (Astral) -- not mypy -- **Type hints**: required on all public functions -- **Docstrings**: Google style -- **Imports**: sorted by ruff (stdlib, third-party, local) -- **DataFrames**: Polars as primary, Pandas for compatibility - -All style rules are configured in `pyproject.toml` under `[tool.ruff]` and `[tool.ty]`. Ruff auto-fixes most issues via pre-commit. - -## Testing - -### Running Tests - -```bash -# Default: core tests only (fast, no API keys needed) -uv run pytest tests/ -q - -# Verbose with full output -uv run pytest tests/ -v - -# Run a specific file or test -uv run pytest tests/test_core_models.py -q -uv run pytest tests/test_core_models.py::test_function_name - -# Run tests matching a pattern -uv run pytest -k "yahoo" -q - -# Stop on first failure -uv run pytest tests/ -x +git clone https://github.com/ml4t/data.git ml4t-data +cd ml4t-data +uv sync --locked --all-extras --all-groups +uv run pre-commit install ``` -### Test Markers +Create a focused branch from the current `main` branch. Keep provider-specific dependencies +optional, do not require credentials or network access during import, and preserve the PEP 420 +namespace layout under `src/ml4t/`. Use the repository's conventional commit prefixes, such as +`feat:`, `fix:`, `docs:`, `test:`, and `chore:`. -Tests are organized by marker. The default `pytest` invocation skips slow, integration, and paid-tier tests. +Use the test markers already declared in `pyproject.toml`. The default lane excludes live, +credentialed, paid-tier, and slow tests. The +[testing guide](https://www.ml4trading.io/docs/data/contributing/testing/) explains focused tests, +markers, optional dependencies, and live provider contracts. The +[provider guide](https://www.ml4trading.io/docs/data/contributing/creating-a-provider/) defines the +provider interfaces and required contract coverage. -| Marker | Description | Default | -|--------|-------------|---------| -| `slow` | Tests taking >10 seconds | Skipped | -| `integration` | Real API calls to external services | Skipped | -| `requires_api_key` | Needs a specific API key in env | Skipped | -| `paid_tier` | Requires paid API subscription | Skipped | -| `expensive` | High API cost | Skipped | +## Verify the change -To run a specific marker lane: +Run focused tests while editing, then run the repository checks before requesting review: ```bash -# Integration tests (requires API keys in environment) -uv run pytest -m integration -q - -# All tests, ignoring marker filters -uv run pytest -m "" -q -``` - -### Writing Tests - -- Place tests in `tests/` mirroring the source layout -- Use `pytest` fixtures in `conftest.py` for shared setup -- Mark integration tests with `@pytest.mark.integration` -- Ensure OHLCV invariants hold: `high >= low`, `high >= open`, `high >= close` - -## Pull Request Workflow - -1. **Create a feature branch** from `main`: - ```bash - git checkout -b feat/my-feature - ``` - -2. **Make your changes** with tests. Ensure all quality gates pass: - ```bash - uv run pre-commit run --all-files - uv run pytest tests/ -q - ``` - -3. **Push and open a PR**: - ```bash - git push -u origin feat/my-feature - gh pr create --title "feat: description" --body "..." - ``` - -4. **CI runs automatically** on the PR. All checks must pass before merge: - - Lint (ruff check + format) - - Type Check (ty) - - Test Core (Python 3.11, 3.12, 3.13) - - Package Build - -5. **Address review feedback**, then the PR is squash-merged into `main`. - -## Commit Message Format - -Use conventional commit prefixes: - -| Prefix | Use for | -|--------|---------| -| `feat:` | New feature or capability | -| `fix:` | Bug fix | -| `docs:` | Documentation changes | -| `refactor:` | Code restructuring (no behavior change) | -| `test:` | Adding or updating tests | -| `chore:` | Maintenance, dependencies, CI | - -Examples: -``` -feat: add Stooq provider for international equities -fix: handle empty response in Yahoo provider -test: add integration tests for Binance websocket -refactor: extract common HTTP retry logic to base class +uv run ruff check src tests scripts +uv run ruff format --check src tests scripts +uv run ty check +uv run pytest tests -q -ra +uv run pytest tests -q -ra -W error::ResourceWarning +uv run mkdocs build --strict +uv build +actionlint .github/workflows/*.yml +pre-commit run --all-files ``` -## Adding a New Provider +Bug fixes require executed regression evidence. Run the same command against the relevant base +revision and the proposed fix. Record the revisions, command, failing base result, and passing fix +result in the pull request. The regression must exercise the affected public API or user workflow. +Tests of copied logic or source text do not establish that behavior is corrected. -Providers inherit from `BaseProvider` and implement three methods: +Correct verified numerical defects. Do not preserve an incorrect result solely for compatibility. +When a change intentionally alters a definition or convention, document the agreed behavior and +migration impact in the owning issue and user documentation. -1. `name()` -- lowercase provider identifier -2. `_fetch_raw_data()` -- API call, returns raw response -3. `_transform_data()` -- converts raw data to standard Polars DataFrame +## Open a pull request -See `docs/contributing/creating-a-provider.md` for a full walkthrough and `src/ml4t/data/providers/` for existing implementations. +Each pull request must address one issue and include a closing reference such as `Closes #123`. +Automated dependency updates and administrative metadata changes do not require a separate issue. -## Project Structure +The pull request must describe the outcome and compatibility impact, report the commands and +results used for verification, and state any documentation, release-note, or patch-release work. +Disclose whether AI tools helped produce code or prose, identify the affected parts, and confirm +that you reviewed and tested the submission and can explain its evidence. -``` -src/ml4t/data/ -├── providers/ # 20+ data source implementations -├── storage/ # Hive-partitioned Parquet backends -├── futures/ # CME/CBOE/ICE futures pipelines -├── validation/ # OHLCV schema and quality checks -├── config/ # Pydantic configuration models -├── cli/ # Click CLI interface -└── utils/ # Rate limiting, retry, formatting -``` +CI runs the repository and ecosystem qualification checks. Address review findings and failed +checks before merge. Maintainers squash-merge accepted pull requests. -## Getting Help +## Get help -- [GitHub Issues](https://github.com/ml4t/data/issues) -- bug reports and feature requests -- [GitHub Discussions](https://github.com/ml4t/data/discussions) -- questions and ideas +- Use [GitHub Issues](https://github.com/ml4t/data/issues) for reproducible defects and accepted + changes. +- Read the [Data documentation](https://www.ml4trading.io/docs/data/) for supported workflows and + provider guidance. File a documentation issue when that guidance is missing or incorrect. From 941d1897a90dec1cf59659fb5291095b11dc190e Mon Sep 17 00:00:00 2001 From: Stefan Jansen Date: Tue, 13 Oct 2026 12:28:17 -0400 Subject: [PATCH 2/2] docs: align contribution checks with coverage gate --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b328a1bc..4cf74cd8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -51,7 +51,7 @@ Run focused tests while editing, then run the repository checks before requestin uv run ruff check src tests scripts uv run ruff format --check src tests scripts uv run ty check -uv run pytest tests -q -ra +uv run pytest tests -q -ra --cov=ml4t.data --cov-report=term-missing uv run pytest tests -q -ra -W error::ResourceWarning uv run mkdocs build --strict uv build