Skip to content

Fix ETL tool issues - #24

Open
stdavis wants to merge 10 commits into
mainfrom
fix/etl-error-message
Open

Fix ETL tool issues#24
stdavis wants to merge 10 commits into
mainfrom
fix/etl-error-message

Conversation

@stdavis

@stdavis stdavis commented Aug 24, 2026

Copy link
Copy Markdown
Member

No description provided.

@stdavis
stdavis enabled auto-merge (rebase) August 24, 2026 16:23
@stdavis
stdavis requested a lite review from Copilot August 24, 2026 16:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens and clarifies the county-to-UTRANS ETL behavior by improving failure reporting, preserving invalid domain values in audit notes, and enforcing target field-length constraints to prevent invalid appends.

Changes:

  • Add clearer error context when adding missing template fields fails, and validate target string lengths during normalization.
  • Preserve invalid domain values by appending them to UTRANS_NOTES, and only copy them into destination fields when they fit.
  • Standardize county boundary name matching to uppercase (with special cases) and log total CLI elapsed time.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
cli/src/utrans/etl_common.py Adds detailed AddField failure context and optional target-length validation in normalization.
cli/src/utrans/etl_mappers.py Preserves invalid domain values in notes and conditionally assigns them based on destination field length.
cli/src/utrans/etl.py Updates normalization call signature, uppercases county boundary lookup values, and logs elapsed time.
cli/tests/test_etl_common.py Adds tests for AddField error reporting, field-length enforcement, and _fits_field_length.
cli/tests/test_cli.py Adds coverage for uppercase county boundary name normalization.
cli/PROFILES.md Documents the updated invalid-domain preservation + field-length behavior.
Suppressed comments (1)

cli/src/utrans/etl.py:196

  • If switching to perf_counter() for the start time, the elapsed calculation should also use perf_counter() to keep units and monotonic behavior consistent.
    elapsed = time.time() - start_time
    log(f"Time elapsed: {elapsed:.2f}s")

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cli/tests/test_etl_common.py
Comment thread cli/src/utrans/etl.py
@stdavis
stdavis force-pushed the fix/etl-error-message branch from 23c0a8e to cf94198 Compare August 24, 2026 17:04
@stdavis
stdavis requested a review from acneville August 24, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants