Skip to content

Add supported ops sync skill workflow - #2096

Open
parthosa wants to merge 5 commits into
devfrom
feature/supported-ops-sync-skill
Open

parthosa wants to merge 5 commits into
devfrom
feature/supported-ops-sync-skill

Conversation

@parthosa

@parthosa parthosa commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the repo-local supported-ops-sync skill workflow and sync-helper script updates split out from #2095.

This PR documents the boundary between deterministic automation and support validation: the CSV sync mechanics are automated, but support promotion remains evidence-gated for parser behavior, plugin code paths, Spark versions, datasource formats, Spark configs, and event-log coverage.

The new TNEW override config entries stay in #2095 because they are part of that concrete supported-ops data sync.

Changes

  • Adds a workflow for syncing supported operator CSVs from RAPIDS plugin generated files.
  • Adds OpenAI agent metadata and a PR checklist reference for future supported-ops sync PRs.
  • Moves the sync-helper script update that preserves tools-side SQL function aliases when plugin generated CSVs are blank or contain a strict subset of tools aliases.
  • Hardens SQL function alias parsing for NaN-like values and SupportLevel sentinels such as NS, S, and CO.
  • Documents that generated CSV drift is not enough to promote Exec/DataSource support; promotions need plugin code-path review plus event-log/parser evidence.

Automation Boundary

  • Deterministic sync work is handled by the workflow: stage plugin CSVs, run the sync helper, preserve tools-side aliases, gate new rows as TNEW, update score files, and generate a report.
  • Expressions can often be validated from parser output, but still need parser/event-log evidence before promotion.
  • Exec and datasource changes require extra review when support is config-gated, format-specific, datatype-specific, Spark-version-specific, or implemented in phases.
  • High-risk cases such as Iceberg, Delta, merge/write execs, datasource V2 paths, and physical-plan parser gaps should remain gated until evidence exists.

Testing

  • conda run -n aether_env_3_11 python /Users/psarthi/.codex/skills/.system/skill-creator/scripts/quick_validate.py .claude/skills/supported-ops-sync
  • conda run -n aether_env_3_11 python -m py_compile scripts/sync_plugin_files/process_supported_files.py
  • conda run -n aether_env_3_11 python -c targeted alias parser assertions for NaN, SupportLevel sentinels, subset preservation, and alias rename behavior
  • git diff --check origin/dev...HEAD
  • Commit hooks ran during git commit -s.

parthosa added 2 commits May 29, 2026 12:37
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
@greptile-apps

greptile-apps Bot commented May 29, 2026

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

The PR appears safe to merge, with a non-blocking request to add regression coverage for the new alias-preservation behavior.

Findings

  1. P2 Alias Logic Lacks Tests

Summary

  • Documents staging, preview, application, score synchronization, testing, and draft-PR steps.
  • Adds agent metadata and a supported-ops PR checklist.
  • Normalizes blank, NaN-like, and support-level sentinel values during alias comparison.
  • The helper behavior would benefit from committed regression tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Plugin generated supported CSVs] --> B[Stage under temporary plugin-root/000]
  B --> C[Run process_supported_files.py preview]
  D[Tools supported CSVs] --> C
  E[Override configuration] --> C
  C --> F[Preserve tools-only SQL aliases]
  C --> G[Gate new operators as TNEW]
  C --> H[Generate sync report and new operator list]
  H --> I[Review support and parser evidence]
  I --> J[Apply synchronized CSVs]
  J --> K[Update operator score files]
  K --> L[Run targeted test suites]
  L --> M[Create draft pull request]
Loading

Comment thread scripts/sync_plugin_files/process_supported_files.py
Comment thread scripts/sync_plugin_files/process_supported_files.py
parthosa added 3 commits May 29, 2026 12:49
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
Signed-off-by: Partho Sarthi <psarthi@nvidia.com>
@parthosa parthosa self-assigned this May 29, 2026
Comment on lines +86 to +116
def get_sql_func_aliases(sql_func_value):
"""
Return SQL function aliases from a semicolon-separated SQL Func cell.
"""
if pd.isna(sql_func_value):
return set()
sql_func_str = str(sql_func_value).strip()
if sql_func_str == "" or sql_func_str.lower() in ("none", "nan"):
return set()
return {
alias.strip() for alias in sql_func_str.split(";")
if alias.strip() and alias.strip() not in SupportLevel.__members__
}


def should_preserve_tools_string(column_name, tools_value, plugin_value):
"""
Preserve tools-side expression aliases when plugin generated metadata is less informative.
SQL function aliases are parser lookup keys in tools, and the generated plugin CSV can omit
aliases for newer Spark versions even though older-version aliases should still be recognized.
Preserve only when the plugin aliases are blank or a strict subset of the tools aliases, so an
intentional plugin alias rename is accepted instead of being hidden by a string-length heuristic.
"""
if column_name != "SQL Func":
return False

tools_aliases = get_sql_func_aliases(tools_value)
plugin_aliases = get_sql_func_aliases(plugin_value)
if not tools_aliases:
return False
return not plugin_aliases or plugin_aliases < tools_aliases

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Alias Logic Lacks Tests

The new alias normalization and preservation logic has no committed automated coverage. Focused tests should cover blank and NaN-like values, support-level sentinels, strict subsets, disjoint renames, and the final emitted CSV value; otherwise, a later sync change could silently remove parser lookup aliases.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants