Implement daily file processing with automatic fallback to weekly files by Copilot · Pull Request #33 · pcunning/hamcall

Copilot · 2025-06-30T14:32:32Z

This PR implements daily file processing for FCC ULS data as requested in the issue. The system can now process daily delta files instead of always downloading the complete weekly files, which significantly reduces download time and bandwidth usage for incremental updates.

Key Features

Environment Variable Control: Set ULS_USE_DAILY=true to enable daily file processing
Automatic Fallback: If daily files fail to download, unzip, or lack essential data, the system automatically falls back to weekly files
Day-Aware Downloads: Downloads appropriate daily files based on current day of week (e.g., l_am_mon.zip, a_am_tue.zip)
Data Validation: Ensures essential files (AM.dat, EN.dat, HD.dat) exist and contain data before proceeding
Backward Compatibility: Weekly processing remains the default behavior when the environment variable is not set

Usage

# Enable daily processing
ULS_USE_DAILY=true ./hamcall -dl

# Default weekly processing (unchanged)
./hamcall -dl

Implementation Details

The implementation adds three new functions:

DownloadDailyLicenses(): Downloads daily license files with fallback to weekly
DownloadDailyApplications(): Downloads daily application files with fallback to weekly
dailyFilesContainEssentialData(): Validates essential files exist and have content

Daily files are downloaded from:

License data: https://data.fcc.gov/download/pub/uls/daily/l_am_{day}.zip
Application data: https://data.fcc.gov/download/pub/uls/daily/a_am_{day}.zip

Testing

Added comprehensive tests covering:

Daily file validation logic
Environment variable behavior
Fallback mechanisms
Integration with existing processing pipeline

All existing tests continue to pass, ensuring no breaking changes to current functionality.

Fixes #10.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

api.backblazeb2.com
- Triggering command: /tmp/go-build3412660826/b001/exe/hamcall -m stats (dns block)
invalid-url-that-does-not-exist.com

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: pcunning <171210+pcunning@users.noreply.github.com>

pcunning · 2025-07-24T06:05:05Z

@copilot daily files are added the day after and not on Monday. So on Tuesday after noon eastern mondays file should be available. On saturday fetch Friday. The daily files only contain a delta of the changes since the last complete file.

There should be two run modes with daily files.

Partial and full.

On full download the last weekly complete file and all daily files since the weekly should be downloaded. The weekly file is published Sunday. So if running full on Wednesday we would download the complete file and the daily file for Monday, and Tuesday.

On partial mode we only need to download the daily delta. Because this mode is only getting a partial dump we need to change some of the functionality of tue other data points like geo LOTW and radioid. These currently initialize a blank callsign when it does not exist in the ULS data, on partial it needs to not do that so we don't overwrite good data from the last full download.

Instead of using http to download the daily files use the ftp server the weekly files are downloaded from.

…ownloads Co-authored-by: pcunning <171210+pcunning@users.noreply.github.com>

Copilot · 2025-07-24T06:18:05Z

@copilot daily files are added the day after and not on Monday. So on Tuesday after noon eastern mondays file should be available. On saturday fetch Friday. The daily files only contain a delta of the...

I've implemented all the requested changes in commit 5cd0bb5:

Two Run Modes:

ULS_MODE=partial: Downloads only previous business day's delta files
ULS_MODE=full: Downloads weekly file + all daily files since Sunday publication

Correct Day Logic:

Uses previous business day (Monday file available Tuesday after noon)
Handles weekends properly (Saturday gets Friday's file)

FTP Downloads: Switched from HTTP to FTP server as requested:

ftp://wirelessftp.fcc.gov:21/pub/uls/daily/l_am_{day}.zip
ftp://wirelessftp.fcc.gov:21/pub/uls/daily/a_am_{day}.zip

Partial Mode Data Protection:

GEO, LOTW, and RadioID sources only update existing callsigns in partial mode
Prevents overwriting good data from last full download

Full Mode: Downloads complete weekly file + all daily deltas since Sunday, merges before processing

Usage:

ULS_MODE=partial ./hamcall -dl   # Delta only
ULS_MODE=full ./hamcall -dl      # Weekly + all dailies  
./hamcall -dl                    # Weekly (default)

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

invalid-url-that-does-not-exist.com

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Copilot

Pull Request Overview

This PR implements a comprehensive daily file processing system for FCC ULS data with intelligent fallback to weekly files. The system introduces three operational modes: partial (daily only), full (weekly + daily merge), and default (weekly only).

Adds environment variable ULS_MODE to control processing behavior with automatic fallback mechanisms
Implements day-aware daily file downloads with business day logic and data validation
Updates all supplementary data processors (radioid, lotw, geo) to support partial mode operation

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
source/uls/uls.go	Core implementation of daily processing modes, download functions, and fallback logic
source/uls/uls_test.go	Comprehensive test coverage for daily processing functions and validation
main.go	Integration of partial mode flag into the main processing pipeline
source/radioid/radioid.go	Added partial mode support to skip creating new callsigns
source/lotw/lotw.go	Added partial mode support to skip creating new callsigns
source/geo/geo.go	Added partial mode support to skip creating new callsigns
.github/workflows/run-process.yml	Updated weekly workflow for full mode processing
.github/workflows/daily-partial-update.yml	New workflow for automated daily partial updates

Comments suppressed due to low confidence (3)

source/uls/uls_test.go:281

[nitpick] The test function name TestDownloadSelection doesn't clearly describe what it's testing. Consider renaming to TestULSModeEnvironmentVariable to better reflect that it's testing environment variable behavior.

	os.Unsetenv("ULS_MODE")

source/uls/uls.go:160

The variable name dailyUrl should use camelCase naming convention. It should be dailyURL following Go naming conventions for acronyms.

		dailyUrl := fmt.Sprintf("ftp://wirelessftp.fcc.gov:21/pub/uls/daily/l_am_%s.zip", day)

source/uls/uls.go:225

The variable name dailyUrl should use camelCase naming convention. It should be dailyURL following Go naming conventions for acronyms.

		dailyUrl := fmt.Sprintf("ftp://wirelessftp.fcc.gov:21/pub/uls/daily/a_am_%s.zip", day)

Copilot · 2025-07-25T21:10:29Z

+	}
+
+	// For partial mode, check if daily files contain essential data and fallback if needed
+	if ulsMode == "partial" {


The string literal "partial" is repeated multiple times throughout the code. Consider defining a constant for ULS mode values to improve maintainability and reduce the risk of typos.

Suggested change

if ulsMode == "partial" {

if ulsMode == ULSModePartial {

Copilot · 2025-07-25T21:10:29Z

+
+	// For partial mode, check if daily files contain essential data and fallback if needed
+	if ulsMode == "partial" {
+		if !dailyFilesContainEssentialData("l_amat") {


The directory path "l_amat" is hardcoded. Consider extracting this as a constant or variable to improve maintainability and consistency with other directory references.

Suggested change

if !dailyFilesContainEssentialData("l_amat") {

if !dailyFilesContainEssentialData(DefaultExtractDir) {

Copilot · 2025-07-25T21:10:29Z

+	if ulsMode == "partial" {
+		if !dailyFilesContainEssentialData("l_amat") {
+			fmt.Println("Daily files missing essential data, downloading weekly files...")
+			wg.Add(1)


Adding to the WaitGroup inside a goroutine that's already being waited on can cause synchronization issues. The WaitGroup increment should happen before starting the goroutine, not inside the fallback function.

Suggested change

wg.Add(1)

Initial plan

287cf0d

Copilot AI assigned Copilot and pcunning Jun 30, 2025

Copilot started work on behalf of pcunning June 30, 2025 14:32 View session

Implement daily file processing with automatic fallback to weekly files

983df57

Co-authored-by: pcunning <171210+pcunning@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Process Daily files~~ Implement daily file processing with automatic fallback to weekly files Jun 30, 2025

Copilot AI requested a review from pcunning June 30, 2025 14:42

Copilot finished work on behalf of pcunning June 30, 2025 14:42

Copilot started work on behalf of pcunning July 24, 2025 06:05 View session

Implement enhanced daily processing with partial/full modes and FTP d…

5cd0bb5

…ownloads Co-authored-by: pcunning <171210+pcunning@users.noreply.github.com>

Copilot finished work on behalf of pcunning July 24, 2025 06:18

pcunning added 2 commits July 25, 2025 14:01

fix merge order

5892852

update workflow

8b30814

pcunning marked this pull request as ready for review July 25, 2025 21:09

Copilot AI review requested due to automatic review settings July 25, 2025 21:09

Copilot AI reviewed Jul 25, 2025

View reviewed changes

lol

44f2a9e

pcunning merged commit 7696846 into main Jul 25, 2025
1 check passed

pcunning deleted the copilot/fix-10 branch July 25, 2025 21:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement daily file processing with automatic fallback to weekly files - #33

Implement daily file processing with automatic fallback to weekly files#33
pcunning merged 6 commits into
mainfrom
copilot/fix-10

Copilot AI commented Jun 30, 2025 •

edited

Loading

Uh oh!

pcunning commented Jul 24, 2025

Uh oh!

Copilot AI commented Jul 24, 2025 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

Copilot AI Jul 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	if !dailyFilesContainEssentialData("l_amat") {
	if !dailyFilesContainEssentialData(DefaultExtractDir) {

Conversation

Copilot AI commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Features

Usage

Implementation Details

Testing

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pcunning commented Jul 24, 2025

Uh oh!

Copilot AI commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 30, 2025 •

edited

Loading

Copilot AI commented Jul 24, 2025 •

edited

Loading