Skip to content

Raise default ingest consumer concurrency to 8 loops per process #SUPERLOG #530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
superlog-app wants to merge 1 commit into from
Closed

Raise default ingest consumer concurrency to 8 loops per process #SUPERLOG #530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/proxy/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ INGEST_QUEUE_URL=
INGEST_OVERSIZE_BUCKET=
INGEST_OVERSIZE_PREFIX=otlp-oversize
INGEST_QUEUE_CONSUMER_ENABLED=true
INGEST_QUEUE_CONSUMER_CONCURRENCY=4
INGEST_QUEUE_CONSUMER_CONCURRENCY=8
INGEST_QUEUE_MAX_MESSAGE_BYTES=240000
# Region for the SQS/S3 ingest buffer (only needed when INGEST_QUEUE_URL is set).
AWS_REGION=
Expand Down
8 changes: 8 additions & 0 deletions apps/proxy/src/ingest-queue.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,14 @@ test("getIngestQueueConfig reads bounded consumer concurrency", () => {
assert.equal(config?.consumerConcurrency, 32);
});

test("getIngestQueueConfig uses 8 concurrent loops by default", () => {
const config = getIngestQueueConfig({
INGEST_QUEUE_URL: "https://sqs.us-west-2.amazonaws.com/123/superlog-test-ingest",
});

assert.equal(config?.consumerConcurrency, 8);
});

test("encodeIngestMessage stores small payloads inline", () => {
const encoded = encodeIngestMessage(
baseInput,
Expand Down
7 changes: 6 additions & 1 deletion apps/proxy/src/ingest-queue.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,12 @@ const DEFAULT_SQS_MAX_SOCKETS = 64;
const DEFAULT_WAIT_TIME_SECONDS = 20;
const DEFAULT_VISIBILITY_TIMEOUT_SECONDS = 120;
const DEFAULT_BATCH_SIZE = 10;
const DEFAULT_CONSUMER_CONCURRENCY = 4;
// Eight concurrent receive loops per process gives the ClickHouseIngestWriter
// enough concurrent callers to keep its per-table insert pipeline saturated
// (all eight loops' messages coalesce into one insert per batchLingerMs window)
// without overwhelming a single ClickHouse node. Deployments that set
// INGEST_QUEUE_CONSUMER_CONCURRENCY explicitly override this default.
const DEFAULT_CONSUMER_CONCURRENCY = 8;
const MAX_COLLECTOR_ERROR_BODY_CHARS = 1_000;
// SQS hard limits on a single SendMessageBatch/DeleteMessageBatch call: at most
// 10 entries, and (for sends) at most 256 KiB of total payload across entries.
Expand Down
Loading