Unable to use New Relic layer in lambda that is already using the aws-lambda-web-adapter layer

[lucaswehmuth]

Description

The lambda runtime is unable to locate the newrelic_lambda_wrapper.handler in a lambda that uses the aws-lambda-web-adapter layer (https://github.com/awslabs/aws-lambda-web-adapter) which apparently is another wrapper.

So the issue is trying to set a wrapper inside another wrapper.

Steps to Reproduce

Define a lambda resource using the Serverless Application Model (SAM) like the one below:

  LambdaRestApiWebhooksAuth:
    Type: AWS::Serverless::Function
    DependsOn: RestApiWebhooksAuth
    Properties:
      CodeUri: ./src/handler
      Handler: newrelic_lambda_wrapper.handler
      FunctionName: !FindInMap [StageMappings, !Ref StageName, lambdaNameHandler]
      Role: !Sub "arn:aws:iam::${AWS::AccountId}:role/role-lambda-api"
      Architectures:
        - arm64
      Runtime: python3.10
      MemorySize: 128
      Environment:
        Variables:
          AWS_LAMBDA_EXEC_WRAPPER: /opt/bootstrap
          PORT: "8000"
          ENV: !Ref StageName
          NEW_RELIC_LAMBDA_HANDLER: run.sh
          NEW_RELIC_ACCOUNT_ID: "<ACCOUNT_ID>"
          NEW_RELIC_TRUSTED_ACCOUNT_KEY: "<ACCOUNT_ID>"
          NEW_RELIC_LICENSE_KEY: "<API_KEY>"
      Layers:
        - !Sub arn:aws:lambda:${AWS::Region}:753240598075:layer:LambdaAdapterLayerArm64:20
        - "arn:aws:lambda:us-east-1:451483290750:layer:NewRelicPython310ARM64:27"
      Timeout: 15

or use the sample code provided by AWS itself and then apply the following environment variables:

AWS_LAMBDA_EXEC_WRAPPER: /opt/bootstrap
PORT: "8000"
ENV: !Ref StageName
NEW_RELIC_LAMBDA_HANDLER: run.sh
NEW_RELIC_ACCOUNT_ID: "<ACCOUNT_ID>"
NEW_RELIC_TRUSTED_ACCOUNT_KEY: "<ACCOUNT_ID>"
NEW_RELIC_LICENSE_KEY: "<API_KEY>"

Expected Behaviour

The value inside the environment variable NEW_RELIC_LAMBDA_HANDLER should be able to be located by the New Relic wrapper

Relevant Logs / Console output

Lambda log error:
/opt/bootstrap: line 3: /var/task/newrelic_lambda_wrapper.handler: No such file or directory

Your Environment

• AWS Lambda
• Python 3.10 ARM runtime
• LambdaAdapterLayerArm64 layer
• NewRelicPython310ARM64 layer

Additional context

Is there any suggested workaround for this lambda scenario? I could not find anything in the docs and neither in any past issues from github

[workato-integration]

https://new-relic.atlassian.net/browse/NR-279309

[chaudharysaket]

Working on it

[chaudharysaket]

We can use the layerless approach which installs only the New Relic Extension that can transmit telemetry to New Relic. In the layerless approach, we don't need to set custom New Relic handler.

For lambda managed runtimes, need to investigate how to use the Agent+Extension layers as the AWS_LAMBDA_EXEC_WRAPPER is set to /opt/bootstrap while language agents also wrap the lambda handler.

[anentropic]

In general I find there's a lack of detail in the docs about what things actually do, so when the "happy path" instructions don't work I don't know where to go next...

the layerless docs show this:

import newrelic.agent
from newrelic_lambda.lambda_handler import lambda_handler

newrelic.agent.initialize()

@lambda_handler()
def handler(event, context):
  ...

it's unclear what the lambda_handler does and I am unsure how to integrate it in the Lambda Web Adapter case where I have a Django app

I am guessing I should ignore all the newrelic_lambda stuff and just treat it as a Django app that happens to run on Lambda

similarly I would love to know what NEW_RELIC_SERVERLESS_MODE_ENABLED does, to know whether I should really enable it or not in this set up

[yuzujoe]

It seems that the lambda extension for newrelic does not work well with the Lambda Web Adapter, so I switched to using OpenTelemetry with Next.js to get traces, and it worked.

[chaudharysaket]

We are working on releasing support for aws-lambda-web-adapter

The aws-lambda-web-adapter is a Lambda Extension that transforms the Lambda events into HTTP requests to support web frameworks such as FastAPI, Flask, etc.

The New Relic Layer uses wrapper file to automatically wrap lambda handler, but the Lambda using web-adapter does not have the conventional Lambda handler.
In our tests, we created a sample app, which uses the FastAPI zip awslabs example shared above. The sample app performs following:

• Modifies run.sh, uses newrelic-admin to start and instrument the app
• Uses middleware to read request-id, lambda-arn and lambda-version from header x-amzn-lambda-context
• Uses NEW_RELIC_SERVERLESS_MODE_ENABLED environment variable instead of NEW_RELIC_LAMBDA_HANDLER as no wrapper file is required.

Please use the sample FastAPI zip application example. We are considering packaging the middleware into the Lambda Layer to streamline the instrumentation process for web-adapter apps.
Thank you!