Combine with datadog

[KnisterPeter]

Hi there,

we would like to integrate the lambda-adapter with a datadog agent.
According to their docs (https://docs.datadoghq.com/serverless/aws_lambda/installation/nodejs/?tab=containerimage) we have to set the DD_LAMBDA_HANDLER env-var to the original handler.
When using the lambda-adapter, there is no real handler, can you give guidance on what to do in this case?

[bnusunny]

I'm converting this into a discussion.

[bnusunny]

With Lambda Web Adapter, the app runs as a standalone web server in Lambda. So for monitoring, it should be monitored as a web server. Please consult Datadog docs for how to monitor a web server.

One thing to note for tracing: if you want to capture x-ray traces, LWA forwards the x-ray trace id in the http header 'X-Amzn-Trace-Id', which can be extracted and used for tracing.

[KnisterPeter]

Thanks for the quick answer

[astuyve]

Hey @KnisterPeter - thanks for opening this discussion. (Thanks for the ping @bnusunny!)

We have some work to do on our end to support this project. Our instrumentation mostly expects the full Lambda payload as a struct in basically each language, so it's not super straightforward.

We're prioritizing this work soon and I can update folks when it's ready.

Thanks!
AJ

[b3nk3]

I too would be interested to know if there has been any progress made on this.

[KnisterPeter]

Hi @astuyve, can you give an ETA when this will be ready? I was told that it might be late last year. Any plans on continue with this work or was it droped from schedule?

[astuyve]

Hey folks! We're still working on this! I'll keep you updated, thanks!

[b3nk3]

hello any update to share @astuyve?

[alexgallotta]

We are back working at it!
For now there is a small change in the lwa side on my fork
alexgallotta@186ddd2
Happy to get feed back on that for now. The os set env is not ideal, but the client settings are so deep in the libraries and cannot be changed since it fetched the env variable

The bulk of the change will be on the datadog extension side

[astuyve]

@bnusunny We'll probably need to proxy the Lambda Runtime API environment variable, but since wrapper scripts only apply to the function process (not other extensions), we couldn't do it automatically.

Would you merge a PR which honors an ALTERNATE_LAMBDA_RUNTIME_API env variable?

[bnusunny]

Yes, happy to.

[alexgallotta]

This PR has the change to support Datadog
#588

[alexgallotta]

The support for Datadog is going to be released soon.
If anyone wants to try it before the release, let me know

[KnisterPeter]

These are the linked libraries:

root@87075a7fe526:/opt/extensions# ldd datadog-agent 
        linux-vdso.so.1 (0x00007ffc0c70b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f56dc018000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f56dc013000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f56dc00e000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f56db320000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f56dc009000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56db13f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f56dc03c000)

[KnisterPeter]

When trying to run the setup with a static node binary I get this error:

node: symbol lookup error: /app/node_modules/.pnpm/@datadog+native-metrics@3.1.1/node_modules/@datadog/native-metrics/prebuilds/linuxglibc-x64/node-napi.node: undefined symbol: napi_module_register

[alexgallotta]

Currently Datadog doesn't provide the fully statically linked extension.
Is it strictly related to the Lambda Web Adapter use case? If not, it is better to open a specific ticket/request somewhere else more Datadog-specific

[alexgallotta]

I am preparing the Datadog customer facing documentation and I was thinking about linking the examples in #602

if I can get a review or looks ready to merge, it would be great so I can link them to the docs!

[bnusunny]

@alexgallotta I will review it tomorrow.

[KnisterPeter]

@alexgallotta It seems that it's not working with the lambda function-url response_stream mode.
I've tried it without datadog and that does seem to run fine. As soon as I've enable the respnse_stream mode with datadog enabled, it does fail to send correct responses.

[KnisterPeter]

@lym953 any update on this?

[lym953]

@KnisterPeter Hey I'm still working on this.

[lym953]

@KnisterPeter Could you try extension v84? @duncanista added this support.
Here's an example app: #608

[KnisterPeter]

Thank you @lym953. I'm going to test that next week at the company.

[KnisterPeter]

@lym953 I can report back that it does work now. Thank you for supporting that. ❤️