Fix ETL tool issues - #24
Open
stdavis wants to merge 10 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR tightens and clarifies the county-to-UTRANS ETL behavior by improving failure reporting, preserving invalid domain values in audit notes, and enforcing target field-length constraints to prevent invalid appends.
Changes:
- Add clearer error context when adding missing template fields fails, and validate target string lengths during normalization.
- Preserve invalid domain values by appending them to
UTRANS_NOTES, and only copy them into destination fields when they fit. - Standardize county boundary name matching to uppercase (with special cases) and log total CLI elapsed time.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/src/utrans/etl_common.py | Adds detailed AddField failure context and optional target-length validation in normalization. |
| cli/src/utrans/etl_mappers.py | Preserves invalid domain values in notes and conditionally assigns them based on destination field length. |
| cli/src/utrans/etl.py | Updates normalization call signature, uppercases county boundary lookup values, and logs elapsed time. |
| cli/tests/test_etl_common.py | Adds tests for AddField error reporting, field-length enforcement, and _fits_field_length. |
| cli/tests/test_cli.py | Adds coverage for uppercase county boundary name normalization. |
| cli/PROFILES.md | Documents the updated invalid-domain preservation + field-length behavior. |
Suppressed comments (1)
cli/src/utrans/etl.py:196
- If switching to perf_counter() for the start time, the elapsed calculation should also use perf_counter() to keep units and monotonic behavior consistent.
elapsed = time.time() - start_time
log(f"Time elapsed: {elapsed:.2f}s")
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
stdavis
force-pushed
the
fix/etl-error-message
branch
from
August 24, 2026 17:04
23c0a8e to
cf94198
Compare
…e configs For implementing more county-specific etl logic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.