Skip to content

ci: backport remaining rhoai-3.3 compatibility fixes - #697

Closed
NeriCarcasci wants to merge 12 commits into
red-hat-data-services:rhoai-3.3from
NeriCarcasci:chore/ci-unblock-rhoai-3.3
Closed

NeriCarcasci wants to merge 12 commits into
red-hat-data-services:rhoai-3.3from
NeriCarcasci:chore/ci-unblock-rhoai-3.3

Conversation

@NeriCarcasci

@NeriCarcasci NeriCarcasci commented Aug 24, 2026 •

Copy link
Copy Markdown

Purpose

Unblock the rhoai-3.3 CVE updates in #656 and #677 by fixing the remaining base-branch CI and dependency-compatibility failures that are unrelated to either CVE diff.

Although this PR's commit history contains earlier investigation and merge-resolution work, the final review scope is seven files. The table below is the authoritative description of every surviving change.

Baseline evidence

The failures are reproducible without either CVE change:

Final changes and reasons

Path Change Why it is needed Provenance
mlflow/R/mlflow/Dockerfile.dev Upgrade rocker/r-ver from 4.2.1 to 4.4.2 and install libcurl4-openssl-dev and libuv1. Current R dependencies require a newer libcurl than the Ubuntu 20.04-based image provides, and the fs package requires the libuv.so.1 runtime. This affects the development/docs image only, not the shipped MLflow runtime image. Backports upstream mlflow/mlflow#22839 and mlflow/mlflow#22685.
.github/workflows/examples.yml Restore the workspace PIP_CONSTRAINT. Examples jobs must honor this maintenance branch's constraints instead of resolving newly released, incompatible dependencies. Backports upstream mlflow/mlflow#20396.
.github/workflows/examples.yml Restrict the Docker selector to the repository-root Dockerfile and .dockerignore, and print the actual matched files. The previous unanchored expression also matched nested example Dockerfiles, incorrectly selecting the repository Docker job for unrelated example changes. Branch-specific CI correction.
requirements/extra-ml-requirements.txt Add tensorboard without a duplicate version bound. TensorFlow 2.21+ no longer supplies TensorBoard as an implicit dependency, causing the Keras example to raise TBNotInstalledError. TensorFlow's metadata selects the compatible TensorBoard version. Backports upstream mlflow/mlflow#21879.
mlflow/sentence_transformers/__init__.py Convert pandas Series/DataFrame inputs to lists before encode, including an empty-input guard. Sentence Transformers 5.4 rejects pandas containers that earlier releases accepted. Backports upstream mlflow/mlflow#23108.
tests/tracking/test_mlflow_artifacts.py Initialize and dispose the SQLite store before launching the test server. On Windows, database setup could consume the server's readiness window and make the artifact test time out. Backports upstream mlflow/mlflow#19266.
tests/server/auth/test_auth.py Allow up to 10 seconds for startup, poll every 0.5 seconds, and return directly when ready. The previous fixed five-second window could expire while SQLite/Alembic initialization was still running in python (3). Backports upstream mlflow/mlflow#19818.
examples/mlflow_artifacts/docker-compose.yml Replace the unpinned Docker Hub references with pinned MinIO server and client images from Quay. The old community Docker Hub repositories now return pull access denied. Pinned Quay tags keep the existing MinIO-based example and make the test reproducible. The zero-diff #736 result demonstrates that this failure exists on the base branch. Uses the image locations and tags published in MinIO's official Helm values; independently corroborated by #735.

Scope and provenance

  • Each direct MLflow backport above links to its upstream PR so the original authorship and rationale are visible. The PIP_CONSTRAINT restoration is explicitly attributed to mlflow/mlflow#20396 here.
  • The MinIO image change is intentionally limited to the two Compose image references. It does not change the example's API, service layout, credentials, or ports.
  • No production dependency is pinned by this PR. The Sentence Transformers compatibility fix is the only user-facing runtime behavior change; the remaining changes affect CI, tests, examples, or the R development/docs image.

Testing

At current head 284ad5a8b, the complete reported GitHub Actions matrix passes, including:

  • Examples 1-2 and R
  • Python 1-3, Python Skinny, database, flavors, models 1-2, evaluation 1-2, GenAI, and pyfunc 1-4
  • Windows 1-4
  • build variants, Docker, lint, and macOS lint

Additional targeted validation:

  • repository-pinned Ruff 0.12.10 passes on the changed auth test
  • the changed Python tests compile successfully
  • both pinned Quay image manifests are anonymously accessible
  • git diff --check is clean

For comparison, the zero-diff baseline #736 failed python (2) at the Docker Hub pull, while #697 passes that job with the two pinned Quay references.

Documentation

  • No user-documentation update is required. The maintenance rationale and upstream provenance are recorded in this PR.

Release notes

  • User-facing bug fix: restore Sentence Transformers 5.4+ prediction compatibility.
  • The remaining changes are CI and maintenance-branch compatibility backports.

CI on rhoai-3.3 started failing after unpinned tools drifted: openai 3
dropped httpx, opentelemetry-sdk 1.43 added processor _metrics, and
Alembic 1.19 compares named CHECK constraints. Pin the drifting deps and
match the 3.5 schema/processor one-liners so unrelated CVE PRs can merge.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copilot AI lite review requested due to automatic review settings August 24, 2026 09:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The pyfunc lock tests resolved unconstrained openai from PyPI (3.x uses
httpx2, not httpx). The otel error-logging test asserted a logger call
shape that no longer fires. Accept httpx or httpx2, and assert stored
traces instead of exporter internals.

Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comment thread tests/pyfunc/test_model_export_with_class_and_artifacts.py Outdated
Comment thread tests/tracing/test_otel_logging.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several material dependency, runtime, and model-behavior changes are absent from the stated scope and validation details.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 12/12 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread mlflow/R/mlflow/Dockerfile.dev
Comment thread requirements/extra-ml-requirements.txt
Comment thread mlflow/sentence_transformers/__init__.py
@NeriCarcasci NeriCarcasci changed the title chore: pin openai/otel and align schema metadata to unblock rhoai-3.3 CI ci: stabilize rhoai-3.3 dependency and test compatibility Sep 10, 2026
…hoai-3.3

# Conflicts:
#	mlflow/store/tracking/dbmodels/models.py
#	tests/pyfunc/test_model_export_with_class_and_artifacts.py
#	tests/tracing/test_otel_logging.py
@NeriCarcasci NeriCarcasci changed the title ci: stabilize rhoai-3.3 dependency and test compatibility ci: backport remaining rhoai-3.3 compatibility fixes Sep 11, 2026
@kramaranya
kramaranya requested a balanced review from Copilot September 11, 2026 14:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes are consistent with the documented upstream backports and no unresolved correctness issues were identified.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Use pinned MinIO images from Quay for the artifact example and backport the upstream auth-server readiness polling fix.

Signed-off-by: Neri Carcasci <nericarcasci@gmail.com>
@NeriCarcasci
NeriCarcasci force-pushed the chore/ci-unblock-rhoai-3.3 branch from e342e2d to 284ad5a Compare September 15, 2026 13:57
@NeriCarcasci

Copy link
Copy Markdown
Author

Closing in favor of #737 after re-auditing the current rhoai-3.3 head and all open overlapping PRs.

#737 contains only the three remaining example-dependency compatibility backports, with upstream contributors recorded as co-authors and DCO sign-offs. The artifact-image replacement is intentionally excluded because #735 already owns that work. Both PRs are intended to unblock #656 and #677 without carrying the exploratory history or unrelated changes from this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants