feat: resilient region publishing, staging validation, ECR decoupling, and Slack failure reporting by aavinash-nr · Pull Request #470 · newrelic/newrelic-lambda-layers

aavinash-nr · 2026-04-28T10:52:54Z

Summary

Hardens the Lambda layer publishing pipeline across all runtimes (Node.js, Python, Ruby, Java, Dotnet, Extension). Previously, a single AWS region error would abort the entire publish via set -e, silently skipping all remaining regions and ECR pushes. This PR fixes that, adds a staging validation gate before production publish, and surfaces all failures clearly in Slack and the GitHub Actions Summary tab.

Changes

Staging Validation Gate

Layers are published to a staging environment first before production
A Lambda orchestrator is invoked to run end-to-end validation against the staged layers
poll-validation.sh polls the orchestrator results bucket on S3 every 5 min (up to 30 min timeout)
Production publish only proceeds if validation passes; staging layers are always cleaned up afterwards

Region-Resilient Publishing

Added run_region_loop in libBuild.sh — iterates all configured regions, captures per-region pass/fail, and continues even when one region throws an AWS error (fixes the set -e abort bug)
After each publish, a region summary table is written to the GitHub Actions Summary tab
Failed region names are included in the Slack notification (e.g. 1/18 regions failed: sa-east-1)

ECR Publishing Decoupled from Layer Publishing

For Python, Ruby, Dotnet, and Extension: ECR push was previously bundled inline in the same script block as layer publishing — a region failure would kill the script before ECR ran
Replaced with publish_ecr_safe / finalize_ecr_results helpers: ECR always runs regardless of layer region failures, and ECR failures are reported separately without affecting the layer publish exit code

Configurable Infrastructure via Actions Vars

LAYER_REGIONS, S3_BUCKET_PREFIX, and ECR_REPOSITORY are now read from GitHub Actions repository variables instead of being hardcoded in scripts

Automatic Region Retry on Workflow Re-run

On re-run, only previously failed regions are retried (via region-retry composite action) instead of republishing to all regions

Structured Slack Notifications

New notify-slack-layer composite action sends one consolidated Slack message per workflow run
Message includes per-runtime publish status, failed region list, ECR failure summary, and a direct link to the Actions run
fail-fast: false added to all matrix jobs so a node20 failure no longer auto-cancels node22/node24

Test Plan

Trigger a publish workflow with a valid tag and verify staging validation runs before production publish
Verify Slack message includes runtime status, failed regions (if any), and run URL
Simulate a region failure — confirm remaining regions still publish and the summary table appears in the Actions Summary tab
Simulate an ECR failure — confirm layer publish result is unaffected and ECR failure appears in Slack
Re-run a failed workflow — confirm only failed regions are retried
Verify staging layers are deleted after validation regardless of outcome

…ne staging, validation, cleanup, and release processes

…30 min Polling every 15s was noisy and unnecessary — validation takes several minutes on the orchestrator side. Switch to 300s sleep so the job only checks when results are realistically ready. Default VALIDATION_TIMEOUT_S raised from 600s to 1800s so at least 5–6 polls occur before giving up. Added timeout-minutes: 35 on the validate job to match.

…mes to improve error handling and reporting

… regions - Add publish_layer_safe + run_region_loop to libBuild.sh so a single region failure no longer aborts the entire publish loop; all regions are attempted and a pass/fail summary is printed at the end. - Fix empty layer_version propagation: add || return 1 after publish-layer-version command substitution so failures inside publish_public_layer are correctly surfaced when set -e is inactive (i.e. when called from an if-context). - Make GITHUB_STEP_SUMMARY and GITHUB_OUTPUT writes non-fatal (|| true) to avoid permission-denied failures inside Docker containers. - Replace all bare for-region loops in nodejs, python, java, ruby, dotnet, extension publish scripts with run_region_loop. - Add strategy.fail-fast: false to all matrix publish jobs so e.g. node22/node24 are not cancelled when node20 fails. - Mount GITHUB_OUTPUT + GITHUB_STEP_SUMMARY into Docker steps (node, java21) so failure_summary can propagate out of the container. - Add outputs: failure_summary to every publish job; include "N/total regions failed: <list>" in all Slack payloads when set.

…s on layer releases

…otification

… via env vars

…s vars to publish steps

…echanism Docker containers run as root but cannot write to GITHUB_OUTPUT/GITHUB_STEP_SUMMARY (owned by the runner user, mode 600). Failures were silently dropped, so no artifact was ever saved and re-runs always published to all regions. Fix: - run_region_loop writes to FAILED_REGIONS_FILE (new env var) when set - publish-node.yml and publish-java.yml pre-create a world-writable temp file, mount it into Docker, then relay its contents to GITHUB_OUTPUT after docker exits - chmod a+rw GITHUB_STEP_SUMMARY before Docker so the summary table renders

…mpt inputs for better error tracking

ECR publish now runs independently (if: always()) so a region failure in layer publishing no longer prevents Docker images from being pushed. Slack notification now downloads failed-regions-* artifacts and shows which specific regions failed per version instead of generic message. Added failure_key to all versions_json entries and run_attempt input to all 6 notify-slack jobs. ECR failures still surfaced separately.

…es and improve error handling

thisistrinadh · 2026-05-07T09:12:51Z

+    required: true
+  versions_json:
+    description: |
+      JSON array: [{"key":"20","label":"Node.js 20","job":"publish-node (20)","fallback":"success","failure_key":"nodejs-20"}]


I hope these are example data. Can you update them to looks like example data.

Hi @thisistrinadh , Thank you for highlighting. I have updated the description.

thisistrinadh

Minor Comment. Can be taken later as well.

Sashwatdas123

LGTM, thank you @aavinash-nr for raising this pr 👍🏻

aavinash-nr added 15 commits April 24, 2026 13:40

Enhance CI workflows for staging and validation of Lambda layers

d109ba2

Refactor CI workflows for Java, Node.js, Python, and Ruby to streamli…

e6eb859

…ne staging, validation, cleanup, and release processes

feat: implement run_region_loop for layer publishing across all runti…

8e78d79

…mes to improve error handling and reporting

feat: implement Notify Slack Layer action for structured notification…

db916d2

…s on layer releases

feat: add automatic region retry on re-run for all publish workflows

aeac43a

fix: remove expression syntax from composite action description fields

3c6998f

feat: add ECR error handling, per-image failure tracking, and Slack n…

60509e7

…otification

feat: make regions, S3 bucket prefix, and ECR repository configurable…

6a9f62b

… via env vars

fix: pass LAYER_REGIONS, S3_BUCKET_PREFIX, ECR_REPOSITORY from Action…

d8bc3b4

…s vars to publish steps

feat: enhance Slack notification action with failure_key and run_atte…

6065ca9

…mpt inputs for better error tracking

feat: enhance layer publishing scripts to include ECR failure summari…

11bcf42

…es and improve error handling

aavinash-nr requested review from Sashwatdas123 and ashishsinghnr May 4, 2026 09:43

ashishsinghnr previously approved these changes May 6, 2026

View reviewed changes

thisistrinadh reviewed May 7, 2026

View reviewed changes

thisistrinadh previously approved these changes May 7, 2026

View reviewed changes

chore: updated description to be generalized comments.

d556865

aavinash-nr dismissed stale reviews from thisistrinadh and ashishsinghnr via d556865 May 7, 2026 10:14

Sashwatdas123 approved these changes May 7, 2026

View reviewed changes

aavinash-nr merged commit a939fbf into newrelic:master May 7, 2026
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: resilient region publishing, staging validation, ECR decoupling, and Slack failure reporting#470

feat: resilient region publishing, staging validation, ECR decoupling, and Slack failure reporting#470
aavinash-nr merged 16 commits into
newrelic:masterfrom
aavinash-nr:notify-slack

aavinash-nr commented Apr 28, 2026

Uh oh!

thisistrinadh May 7, 2026

Uh oh!

aavinash-nr May 7, 2026

Uh oh!

thisistrinadh left a comment

Uh oh!

Sashwatdas123 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

aavinash-nr commented Apr 28, 2026

Summary

Changes

Staging Validation Gate

Region-Resilient Publishing

ECR Publishing Decoupled from Layer Publishing

Configurable Infrastructure via Actions Vars

Automatic Region Retry on Workflow Re-run

Structured Slack Notifications

Test Plan

Uh oh!

thisistrinadh May 7, 2026

Choose a reason for hiding this comment

Uh oh!

aavinash-nr May 7, 2026

Choose a reason for hiding this comment

Uh oh!

thisistrinadh left a comment

Choose a reason for hiding this comment

Uh oh!

Sashwatdas123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants