Skip to content

cicd: unicode updater#4

Merged
sevenzing merged 4 commits into
masterfrom
ll/cicd
Oct 20, 2025
Merged

cicd: unicode updater#4
sevenzing merged 4 commits into
masterfrom
ll/cicd

Conversation

@sevenzing

@sevenzing sevenzing commented Oct 20, 2025

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features

    • Introduced EnsNameNormalizer API with tokenize, process, normalize, and beautify methods.
    • Added specific error types (CurrableError, DisallowedSequence) for improved error reporting.
    • Expanded token support for mapped, ignored, disallowed, stop, NFC, and emoji variants.
  • Documentation

    • Added Crates.io version badge to README.
  • Chores

    • Added automated Unicode update workflow for continuous data synchronization.

@coderabbitai

coderabbitai Bot commented Oct 20, 2025

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

Introduces automated Unicode update tooling via a new GitHub Actions workflow and Python script, reorganizes the public API with a new EnsNameNormalizer struct offering tokenization and normalization methods, adds granular error types (CurrableError and DisallowedSequence), expands the token model with additional variants, and applies various precision improvements including lifetime clarifications and warning suppression.

Changes

Cohort / File(s) Summary
Workflow & Tooling
.github/workflows/unicode-update.yml, tools/unicode-update.py
Added daily automated Unicode update workflow with Python script for downloading, hashing, and detecting changes to Unicode normalization files; generates pull requests when updates are available.
API Reorganization
src/normalizer.rs
Introduced EnsNameNormalizer struct with new() constructor and instance methods (tokenize, process, normalize, beautify); added ProcessedName struct; provided no-cache module-level functions for backward compatibility.
Error Handling
src/error.rs
Added public CurrableError enum (7 variants) and DisallowedSequence enum (5 variants) for granular error reporting.
Token System
src/tokens/types.rs
Extended EnsNameToken enum with variants Mapped, Ignored, Disallowed, Stop, Nfc, Emoji; introduced new token types (TokenMapped, TokenIgnored, TokenDisallowed, TokenStop, TokenNfc); expanded TokenEmoji with additional fields; added CollapsedEnsNameToken enum.
Code Points & Specs
src/code_points/specs.rs, src/code_points/types.rs
Clarified lifetime in finditer_emoji return type signature; added #[allow(dead_code)] attributes to suppress warnings on unused variants.
Library Exports
src/lib.rs
Reduced visibility of code_points module from public to crate-private.
Documentation & Examples
README.md, src/tokens/tokenize.rs, examples/benchmark.rs
Added Crates.io version badge to README; removed embedded example annotations from tokenize module; optimized benchmark string construction using iter::repeat_n.
Tests
tests/ens_tests.rs
Updated only_cases return type to explicitly include lifetime parameter in Vec<IndexedTestCase<'_>>.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant API as Module API
    participant Normalizer as EnsNameNormalizer
    participant Specs as CodePointsSpecs
    
    Note over User,Specs: Direct Function (no-cache)
    User->>API: normalize(input)
    API->>Normalizer: default().normalize(input)
    
    Note over User,Specs: Instance Method (with cache)
    User->>Normalizer: new(specs)
    Normalizer->>Specs: store specs
    User->>Normalizer: tokenize(input)
    Normalizer->>Specs: use cached specs
    Normalizer-->>User: TokenizedName
    User->>Normalizer: process(input)
    Normalizer->>Normalizer: tokenize → validate
    Normalizer-->>User: ProcessedName {labels, tokenized}
    User->>Normalizer: normalize(input)
    Normalizer->>Normalizer: process → extract normalized
    Normalizer-->>User: String
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

The PR spans multiple files with heterogeneous changes: new error types, a significant API reorganization introducing EnsNameNormalizer and expanded token variants, a Python automation tool, and various precision improvements. While some cohorts (documentation, warning suppression) are straightforward, the API surface expansion and token model changes require careful review for backward compatibility and correctness.

Poem

🐰 Hops with joy through tokenized trails,
New normalizers and error tales,
Specs get cached, updates run free,
Unicode harmonies for all to see!
A rabbit's code, refined with care,
Shared with the world, beyond compare!

Pre-merge checks and finishing touches

❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
Title Check ❓ Inconclusive The title "cicd: unicode updater" accurately describes a significant and real part of the changeset—the new GitHub Actions workflow and Python script for automated Unicode normalization updates. However, the pull request contains multiple substantial changes beyond the CI/CD aspect, including major refactoring of the normalizer API (new EnsNameNormalizer struct with multiple public methods), introduction of new error types (CurrableError and DisallowedSequence), and expansion of the token type system. These code changes appear equally significant to the CI/CD additions, yet they are not reflected in the title. This creates ambiguity about whether the title captures the primary intent of the pull request or only addresses one component of a multi-faceted changeset. To clarify the PR's scope, consider whether this is primarily a CI/CD change with incidental code refactoring, or a broader refactoring effort that includes CI/CD infrastructure. If the normalizer API refactoring is the main objective, the title should reflect that (e.g., "refactor: add EnsNameNormalizer abstraction and unicode update automation"). If CI/CD is truly the focus, clarify why the substantial code changes are included or separate them into a follow-up PR.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sevenzing

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 20, 2025

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (2)
.github/workflows/unicode-update.yml (2)

27-30: Update to latest action version.

The actions/setup-python@v4 action version is outdated. Consider updating to @v5 for the latest features and security fixes.

Apply this diff:

     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: '3.11'

72-111: Update to latest action version.

The peter-evans/create-pull-request@v5 action may have a newer version available. Consider updating to the latest version for improved functionality and security.

#!/bin/bash
# Check the latest version of peter-evans/create-pull-request
gh api repos/peter-evans/create-pull-request/releases/latest --jq '.tag_name'
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8d1aad0 and ef1e5f8.

📒 Files selected for processing (12)
  • .github/workflows/unicode-update.yml (1 hunks)
  • README.md (1 hunks)
  • examples/benchmark.rs (1 hunks)
  • src/code_points/specs.rs (2 hunks)
  • src/code_points/types.rs (2 hunks)
  • src/error.rs (3 hunks)
  • src/lib.rs (1 hunks)
  • src/normalizer.rs (3 hunks)
  • src/tokens/tokenize.rs (0 hunks)
  • src/tokens/types.rs (3 hunks)
  • tests/ens_tests.rs (1 hunks)
  • tools/unicode-update.py (1 hunks)
💤 Files with no reviewable changes (1)
  • src/tokens/tokenize.rs
🧰 Additional context used
🪛 actionlint (1.7.8)
.github/workflows/unicode-update.yml

28-28: the runner of "actions/setup-python@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue

(action)


74-74: the runner of "peter-evans/create-pull-request@v5" action is too old to run on GitHub Actions. update the action's version to fix this issue

(action)

🪛 Ruff (0.14.1)
tools/unicode-update.py

29-29: Probable use of requests call without timeout

(S113)


195-195: Do not catch blind exception: Exception

(BLE001)

🔇 Additional comments (16)
examples/benchmark.rs (1)

7-8: No issues found.

The project does not specify an MSRV, so using std::iter::repeat_n (stabilized in Rust 1.82.0) is appropriate. The refactoring improves readability by making the repetition count explicit and is a sound change.

src/lib.rs (1)

12-12: LGTM! API surface intentionally reduced.

Restricting code_points to crate-private visibility is a reasonable choice to limit the public API surface and prevent external dependencies on internal implementation details.

README.md (1)

4-5: LGTM! Nice addition to project visibility.

The Crates.io version badge improves discoverability and helps users quickly identify the latest published version.

tools/unicode-update.py (1)

192-194: Reconsider exit code logic.

Exiting with code 1 when no updates are detected (sys.exit(0 if should_update else 1)) will cause the GitHub Actions step to fail. This may be intentional to trigger conditional logic, but it could be confusing since "no updates" is not an error condition.

Verify that the workflow at .github/workflows/unicode-update.yml correctly handles the exit code 1 case without marking the workflow as failed.

tests/ens_tests.rs (1)

97-97: LGTM! Good lifetime clarification.

Explicitly specifying the lifetime '_ in the return type improves code clarity and helps the compiler verify borrow relationships.

src/tokens/types.rs (3)

6-14: LGTM! Well-structured token type expansion.

The expanded EnsNameToken variants provide a comprehensive representation of different token types in ENS name processing. The additions align with the ENSIP-15 tokenization specification.


75-124: LGTM! Clear token type definitions with good documentation.

The new token types are well-documented and provide appropriate granularity for representing different ENS name components. The field naming is clear and purposeful.


125-139: LGTM! Useful abstraction for collapsed tokens.

The CollapsedEnsNameToken enum provides a clean abstraction for the two main token categories after processing, simplifying downstream logic.

src/code_points/specs.rs (1)

99-101: LGTM! Explicit lifetime improves clarity.

Specifying the explicit lifetime 'a for the iterator's Item type clarifies the borrow relationship between the returned iterator and the input string, making the code more maintainable.

src/error.rs (3)

3-19: LGTM! Well-structured error type with clear documentation.

The ProcessError enum is well-designed with descriptive error messages and proper use of thiserror. The documentation clearly explains its purpose.


21-38: LGTM! Clean error enum for curable errors.

The CurrableError enum provides granular error cases with clear, descriptive messages. The documentation correctly describes these as errors that can be handled by the normalizer.


40-53: LGTM! Well-defined disallowed sequence errors.

The DisallowedSequence enum covers the key validation cases with appropriate error messages. The use of #[from] in ProcessError enables convenient error conversions.

src/normalizer.rs (4)

6-25: LGTM! Clean API design for ENS name normalization.

The EnsNameNormalizer struct provides a well-organized interface for ENS name processing. The use of CodePointsSpecs as a private field allows for potential future caching optimizations.


13-20: LGTM! Well-structured result type.

The ProcessedName struct appropriately encapsulates both the intermediate tokenization result and validated labels, providing a clean separation of processing stages.


27-49: LGTM! Methods follow a clear processing pipeline.

The methods (tokenize, process, normalize, beautify) form a logical progression from tokenization through validation to final output. The error handling is consistent throughout.


62-80: LGTM! Convenient no-cache wrapper functions.

The free functions provide a convenient API for one-off operations where caching the CodePointsSpecs isn't needed. The documentation clearly indicates these are no-cache versions.

Comment thread .github/workflows/unicode-update.yml
Comment thread src/code_points/types.rs
Comment thread src/code_points/types.rs
Comment thread src/normalizer.rs
Comment thread tools/unicode-update.py
Comment thread tools/unicode-update.py
@sevenzing
sevenzing merged commit 18a771b into master Oct 20, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant