Skip to content

feat(terraform): add code-based eval executors (LFE-9943)#61

Draft
wochinge wants to merge 1 commit into
mainfrom
port-code-based-evals
Draft

feat(terraform): add code-based eval executors (LFE-9943)#61
wochinge wants to merge 1 commit into
mainfrom
port-code-based-evals

Conversation

@wochinge
Copy link
Copy Markdown

Summary

  • Adds tenant-isolated Python and Node.js Lambda executors for code-based evals, packaged from bundled runner handlers.
  • Wires the Langfuse Helm release to use the AWS Lambda code eval dispatcher and enables the code eval worker queue consumer behind a single module flag.
  • Creates a dedicated no-internet executor VPC, Lambda IAM role, deny policy for function-code ENI management, and Langfuse IRSA invoke permissions.
  • Documents the community-facing enablement path and exposes only the executor Lambda function names for debugging.

Linear

Major Decisions

  • Keeps enable_code_based_eval_executors as the only code-based eval enable flag so community users do not have to coordinate separate infra and worker settings.
  • Keeps the isolated eval VPC managed by the module, with only the CIDR block configurable to avoid customer network collisions.
  • Exposes Lambda function names rather than ARNs or isolated VPC internals to avoid turning implementation details into public API.

Review Focus

  • Lambda tenant isolation and no-egress VPC/security group setup.
  • Helm environment wiring for dispatcher, function names, queue shard count, and worker concurrency.
  • Public module interface changes in variables.tf, outputs.tf, and README docs.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to review

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to review

}
}

module "code_based_eval_executor_vpc" {
Copy link
Copy Markdown
Author

@wochinge wochinge May 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also kick out the VPC to make the setup easier - what do you think @Steffen911 ?

@wochinge wochinge marked this pull request as ready for review May 26, 2026 08:56
@wochinge wochinge requested a review from Steffen911 May 26, 2026 08:56
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbb12ba98a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +51 to +55
data "archive_file" "code_based_eval_executor" {
for_each = var.enable_code_based_eval_executors ? local.code_based_eval_executor_lambda_configs : {}

type = "zip"
output_path = "${path.module}/.terraform/code_based_eval_executor_${each.key}.zip"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid plan-time archive generation for Lambda packages

Using data "archive_file" here makes packaging happen during terraform plan, but aws_lambda_function.filename is consumed during apply. In multi-phase workflows (separate plan/apply jobs or runners), that zip file often is not present at apply time, so Lambda creation fails with a missing artifact. The archive provider docs explicitly call out this behavior and recommend the archive_file resource when artifacts cannot be persisted between phases.

Useful? React with 👍 / 👎.

@Steffen911 Steffen911 marked this pull request as draft May 26, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant