Skip to content

Convert GitHub Actions to Alpine Linux and refactor year extraction - #5

Merged
mjc merged 11 commits into
mainfrom
fix/elixir-1.19-compatibility
Sep 17, 2025
Merged

mjc merged 11 commits into
mainfrom
fix/elixir-1.19-compatibility

Conversation

@mjc

@mjc mjc commented Sep 17, 2025

Copy link
Copy Markdown
Owner

Overview

This PR converts GitHub Actions from Ubuntu with external dependencies to Alpine Linux container with comprehensive caching, and refactors year extraction logic for better maintainability.

GitHub Actions Migration to Alpine Linux

Infrastructure Changes

  • Container: Switched to Alpine Linux 3.21 container for lighter, more predictable builds
  • Dependencies: Install Erlang 27 and Elixir 1.19 directly from Alpine community repository
  • Build Tools: Added cmake and make for native dependency compilation (lazy_html)
  • Video Stack: FFmpeg with x264-libs, x265-libs, svt-av1 for complete AV1 encoding support

Rust Toolchain Modernization

  • Toolchain: Use rustup instead of outdated Alpine Rust packages
  • ab-av1: Install latest version via cargo with proper Rust edition support
  • PATH Management: Ensure cargo binaries available across workflow steps

Comprehensive Caching Strategy

  • Rust Cache: Cache rustup installation and cargo registry data
  • Binary Cache: Cache compiled ab-av1 binary with conditional installation
  • Mix Cache: Cache Elixir dependencies and compiled build artifacts
  • Native Cache: Cache cmake-built native dependencies to avoid rebuilds

Year Extraction Refactoring

Centralized Implementation

  • New Module Function: Added Parsers.extract_year_from_text/1 for centralized year parsing
  • String-Based Logic: Replaced multiple regex patterns with simple string manipulation
  • Priority Matching: Parentheses > brackets > dots > spaces > standalone numbers

Consumer Updates

  • Rules Module: Updated to use centralized year extraction function
  • MediaInfo Converter: Simplified year extraction logic using Parsers module
  • Sync Module: Removed duplicate year parsing code, uses centralized function

Testing Coverage

  • Comprehensive Tests: Added test/reencodarr/core/parsers_year_test.exs
  • Real-World Examples: Test cases with actual video filename patterns
  • Edge Cases: Boundary years, invalid ranges, missing data handling

Elixir 1.19 Compatibility

Regex Pattern Updates

  • ab_av1 Parsers: Moved regex patterns from module attributes to functions
  • Process Caching: Use process dictionary for pattern caching to maintain performance
  • Progress Parser: Updated pattern compilation for Elixir 1.19 compatibility

Struct Syntax Updates

  • Dashboard State: Updated struct update syntax per Elixir 1.19 patterns
  • Type Safety: Maintained proper struct types while using new syntax

Development Environment

Flake Updates

  • Elixir Version: Updated to support Elixir 1.19.0-rc.0
  • OTP Version: Use erlang_28 for latest features
  • Tools: Added ab-av1 and mediainfo to development shell

Benefits

  • Faster CI: Comprehensive caching reduces build times after initial run
  • Reliable Builds: Alpine container provides consistent, reproducible environment
  • Maintainable Code: Centralized year extraction eliminates code duplication
  • Modern Toolchain: Latest Rust and Elixir versions for better compatibility
  • Better Testing: Comprehensive test coverage for year extraction edge cases

Copilot AI review requested due to automatic review settings September 17, 2025 18:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR converts the GitHub Actions CI workflow from Ubuntu-based runner to NixOS with aggressive caching strategies and fixes Elixir 1.19.0-rc.0 compatibility issues.

Key changes include:

  • CI Infrastructure: Migrated from ubuntu-22.04 to nixos-latest runner with comprehensive caching for Nix store, flake inputs, and Mix dependencies
  • Elixir Compatibility: Fixed regex pattern injection issues by moving patterns from module attributes to function-local definitions
  • Development Tooling: Added local CI testing script and updated flake dependencies to include required video processing tools

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
.github/workflows/elixir.yml Complete workflow conversion to NixOS with multi-layer caching strategy
flake.nix Updated to Elixir 1.19.0-rc.0, OTP 28.1, and added video processing dependencies
lib/reencodarr/ab_av1/progress_parser.ex Moved regex patterns from module attributes to function scope for Elixir 1.19 compatibility
lib/reencodarr/ab_av1/output_parser.ex Converted pattern definitions to function-based approach
lib/reencodarr/dashboard_state.ex Fixed struct syntax deprecation warnings
test-ci-locally.sh New script for local CI workflow testing

Comment thread lib/reencodarr/dashboard_state.ex
Comment thread lib/reencodarr/dashboard_state.ex
Comment thread lib/reencodarr/dashboard_state.ex
Comment thread lib/reencodarr/ab_av1/progress_parser.ex Outdated
- Move regex patterns from module attributes to functions to avoid
  Elixir 1.19 compilation errors with pattern injection
- Fix struct update syntax deprecation warnings by using map syntax
- Update flake.nix to use OTP 28.1 (latest) with Elixir 1.19.0-rc.0
- Add ab-av1 and mediainfo packages to development environment
- Convert GitHub Actions to use NixOS runner with aggressive caching
- Add local CI testing script for development workflow

Fixes compilation errors:
- ArgumentError: cannot inject attribute @patterns into function/macro
- Deprecation warnings for struct update syntax in Elixir 1.19
@mjc
mjc force-pushed the fix/elixir-1.19-compatibility branch from 839c1c2 to 4bada15 Compare September 17, 2025 19:00
mjc added 10 commits September 17, 2025 13:02
- nixos-latest runner is not available in GitHub Actions
- Use cachix/install-nix-action to install Nix on Ubuntu runner
- Use cachix/cachix-action for improved caching
- Keep all Elixir 1.19 compatibility fixes
- Maintain aggressive caching strategy for faster builds
Address Copilot PR feedback: Moving regex patterns from function-local
compilation to process dictionary caching improves performance for
frequently called parsing functions while maintaining Elixir 1.19 compatibility.

- progress_parser.ex: Cache patterns in process dictionary to avoid recompilation on every parse_line/2 call
- output_parser.ex: Use cached_patterns() instead of get_patterns() for frequently called functions
- Maintains Elixir 1.19 compatibility by avoiding module attributes with regex references
- Significant performance improvement for video processing telemetry parsing
- Replaced complex optimization experiments with simple, maintainable string parsing
- Uses pure string manipulation (String.split, String.graphemes) - no regex required
- Added custom all_digits?/1 helper for character-by-character validation
- Maintains priority order: parentheses > brackets > dots > spaces > standalone
- Comprehensive test coverage for all supported patterns
- Removed performance optimization complexity in favor of code readability
- Updated all consumer modules to use centralized Parsers.extract_year_from_text/1

Refactored to reduce function nesting for better code quality:
- Extracted check_year_candidate/1 and parse_and_validate_year/1 helpers
- Added extract_year_from_part/2 to flatten nested logic
- Maintains same functionality with improved readability

This approach prioritizes maintainability and readability over micro-optimizations,
making it easy for any developer to understand and modify the year extraction logic.
- Replaced complex Nix setup with lightweight Alpine 3.20 container
- Install Erlang 27 and Elixir 1.19 from Alpine edge repository
- Build ab-av1 from source using Rust/Cargo (more reliable than binary downloads)
- Install system dependencies (ffmpeg, mediainfo) via apk package manager
- Simplified workflow removes Nix complexity and cachix dependencies
- Maintains same functionality with faster, more reliable CI builds
- Updated cache keys to reflect Alpine environment

This approach should be much more stable and faster to set up than the
previous NixOS-based approach, while still providing the same level of
testing and validation.
- Resolved merge conflict in GitHub Actions workflow
- Added svt-av1 and libSvtAv1Enc packages for AV1 encoding support
- Complete codec stack now includes FFmpeg, x264/x265, VP9, and SVT-AV1
- Essential for ab-av1 functionality in Alpine Linux container
- Upgrade to Alpine 3.21 for better package availability
- Simplified FFmpeg codec installation to avoid libplacebo/glslang conflicts
- Remove problematic packages (ffmpeg-libs, libvpx, opus, lame, libSvtAv1Enc)
- Keep essential codecs: x264-libs, x265-libs, svt-av1 for core ab-av1 functionality
- Install Erlang/Elixir from community repo instead of edge/testing
- Reduces package conflicts while maintaining AV1 encoding capability

This should resolve the glslang dependency errors while keeping the
essential video encoding functionality for ab-av1.
- Switch from bleeding-edge git version to stable ab-av1 from crates.io
- Avoids Rust edition 2024 compatibility issues with Cargo 1.83.0
- Uses ab-av1 v0.10.1 which is stable and compatible with current Rust toolchain
- More reliable than building from latest git commits

This resolves the 'edition2024 feature required' error by using a
stable release instead of development code that requires nightly Rust.
- Replace Alpine's outdated Rust 1.83.0 with rustup-managed latest stable
- Install rustup via official installer for up-to-date Rust toolchain
- Add cargo to PATH for subsequent workflow steps
- Ensures ab-av1 can build with required Rust features and editions

Alpine's packaged Rust version is too old for modern ab-av1 requirements.
Using rustup gives us the latest stable Rust with all needed features.
- lazy_html requires cmake to build lexbor native library
- Added cmake and make to Alpine package installation
- Cache rustup and cargo installation to avoid repeated downloads
- Cache ab-av1 binary compilation with conditional installation
- Cache Mix dependencies and compiled build artifacts
- Cache native dependencies (lazy_html lexbor, etc.)
- Use proper cache keys with file hashing for invalidation
- Significantly faster CI builds after first run
@mjc mjc changed the title Convert GitHub Actions to NixOS with Elixir 1.19 compatibility fixes Convert GitHub Actions to Alpine Linux and refactor year extraction Sep 17, 2025
@mjc
mjc merged commit 79c1602 into main Sep 17, 2025
1 check passed
@mjc
mjc deleted the fix/elixir-1.19-compatibility branch September 17, 2025 23:14
mjc added a commit that referenced this pull request Mar 5, 2026
- Mark Stage 1 item #5 as completed
- Document successful test suite validation (414 tests pass)
- Note successful server boot with Nix PostgreSQL on macOS
- Foundation is ready for distributed architecture implementation
mjc added a commit that referenced this pull request Mar 5, 2026
Convert GitHub Actions to Alpine Linux and refactor year extraction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants