Skip to content

Add logging to data pipeline#254

Open
vipinsamreddy wants to merge 4 commits into
mainfrom
feature/add-logging-data-pipeline
Open

Add logging to data pipeline#254
vipinsamreddy wants to merge 4 commits into
mainfrom
feature/add-logging-data-pipeline

Conversation

@vipinsamreddy

Copy link
Copy Markdown
Collaborator

Description

  • created a dedicated pipeline report directory under RUNNER_TEMP

  • kept per-stage JSON summaries for source versioning, input download, SQL generation, maintenance enable/restore, ETL, review-seed
    backup, environment backup, and review restore

  • preserved ETL and pg_dump logs into the report artifact, instead of only leaving them in working files

  • recorded retry metadata explicitly for ETL: attempts, retry count, retry classification

  • made skipped paths explicit for backup and restore stages

  • added a final markdown run summary to GITHUB_STEP_SUMMARY showing each stage’s status, retries, and concise summary

  • uploaded the whole structured report as one artifact: data-pipeline-stage-report-
    -also fixed the maintenance-restore summary so it no longer depends on the brittle inline || expression.

Related Trello Ticket

Provide the link to the trello card here.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • Tested on [Environment: Dev/Staging/Local]

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have verified accessibility requirements are met

Screenshots (if applicable)

Add UI changes, logs, console outputs anything to help reviewers.

@github-actions

Copy link
Copy Markdown

Code Coverage Report: Only Changed Files listed

Package Coverage
Overall Coverage 🟢 77.52%

Minimum allowed coverage is 0%, this run produced 77.52%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant