Skip to content

perf(api): drop the default COUNT from list queries, add hasMore and opt-in includeTotal #218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Miracle656 merged 1 commit into from
Sep 30, 2026
Merged

perf(api): drop the default COUNT from list queries, add hasMore and opt-in includeTotal #218

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,8 @@ console.log(data);

// Expected response
// {
// "total": 42,
// "hasMore": true,
// "nextCursor": "MTIzNDU=",
// "limit": 10,
// "offset": 0,
// "transfers": [
Expand All @@ -239,6 +240,13 @@ console.log(data);
// }
```

> **Pagination and totals.** List endpoints (`/transfers/*`, `/nfts/transfers`, GraphQL `transfers`)
> do not run a `COUNT` by default. They fetch one row past the page and return
> `hasMore` (plus `nextCursor` to continue), which is all pagination needs. Pass
> `?includeTotal=true` (GraphQL: `includeTotal: true`) to also get an exact
> `total`. This runs a `COUNT` over every matching row, so it is markedly slower
> on large result sets. Without it, `total` is absent from the response.

***

### `GET /transfers/address/:address` — All transfers (sent & received, merged)
Expand All @@ -263,7 +271,7 @@ console.log(data);

// Expected response (same shape as /transfers/incoming — adds "direction" per row)
// {
// "total": 85,
// "hasMore": true,
// "limit": 20,
// "offset": 0,
// "transfers": [
Expand Down Expand Up @@ -367,8 +375,8 @@ both chains in a single round-trip:

```graphql
{
testnet: transfers(address: "GABC…", network: TESTNET) { total }
mainnet: transfers(address: "GABC…", network: MAINNET) { total }
testnet: transfers(address: "GABC…", network: TESTNET, includeTotal: true) { total hasMore }
mainnet: transfers(address: "GABC…", network: MAINNET, includeTotal: true) { total hasMore }
}
```

Expand Down
111 changes: 101 additions & 10 deletions openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -697,6 +697,17 @@
"description": "Comma-separated field list",
"name": "$select",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand All @@ -708,7 +719,12 @@
"type": "object",
"properties": {
"total": {
"type": "integer"
"type": "integer",
"description": "Exact match count. Only present when the request set includeTotal=true"
},
"hasMore": {
"type": "boolean",
"description": "True when another page exists"
},
"limit": {
"type": "integer"
Expand Down Expand Up @@ -798,7 +814,7 @@
}
},
"required": [
"total",
"hasMore",
"limit",
"offset",
"transfers"
Expand Down Expand Up @@ -1031,6 +1047,17 @@
"description": "Comma-separated field list",
"name": "$select",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand All @@ -1042,7 +1069,12 @@
"type": "object",
"properties": {
"total": {
"type": "integer"
"type": "integer",
"description": "Exact match count. Only present when the request set includeTotal=true"
},
"hasMore": {
"type": "boolean",
"description": "True when another page exists"
},
"limit": {
"type": "integer"
Expand Down Expand Up @@ -1132,7 +1164,7 @@
}
},
"required": [
"total",
"hasMore",
"limit",
"offset",
"transfers"
Expand Down Expand Up @@ -1365,6 +1397,17 @@
"description": "Comma-separated field list",
"name": "$select",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand All @@ -1376,7 +1419,12 @@
"type": "object",
"properties": {
"total": {
"type": "integer"
"type": "integer",
"description": "Exact match count. Only present when the request set includeTotal=true"
},
"hasMore": {
"type": "boolean",
"description": "True when another page exists"
},
"limit": {
"type": "integer"
Expand Down Expand Up @@ -1466,7 +1514,7 @@
}
},
"required": [
"total",
"hasMore",
"limit",
"offset",
"transfers"
Expand Down Expand Up @@ -1639,6 +1687,17 @@
"description": "Comma-separated list of event types",
"name": "eventType",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand Down Expand Up @@ -2465,6 +2524,17 @@
"description": "Comma-separated field list",
"name": "$select",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand All @@ -2476,7 +2546,12 @@
"type": "object",
"properties": {
"total": {
"type": "integer"
"type": "integer",
"description": "Exact match count. Only present when the request set includeTotal=true"
},
"hasMore": {
"type": "boolean",
"description": "True when another page exists"
},
"limit": {
"type": "integer"
Expand Down Expand Up @@ -2566,7 +2641,7 @@
}
},
"required": [
"total",
"hasMore",
"limit",
"offset",
"transfers"
Expand Down Expand Up @@ -3572,6 +3647,17 @@
"description": "Comma-separated field list",
"name": "$select",
"in": "query"
},
{
"schema": {
"type": "boolean",
"nullable": true,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate"
},
"required": false,
"description": "Also return an exact `total`. Runs a COUNT over every matching row, so it is markedly slower on large result sets; omit it and use `hasMore` / `nextCursor` to paginate",
"name": "includeTotal",
"in": "query"
}
],
"responses": {
Expand All @@ -3583,7 +3669,12 @@
"type": "object",
"properties": {
"total": {
"type": "integer"
"type": "integer",
"description": "Exact match count. Only present when the request set includeTotal=true"
},
"hasMore": {
"type": "boolean",
"description": "True when another page exists"
},
"limit": {
"type": "integer"
Expand Down Expand Up @@ -3657,7 +3748,7 @@
}
},
"required": [
"total",
"hasMore",
"limit",
"offset",
"transfers"
Expand Down
118 changes: 118 additions & 0 deletions src/__tests__/listPagination.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
/**
* List queries must not pay for a COUNT by default (#206).
*
* db.ts resolves its client from `globalThis.prisma`, so a recording stub keeps
* this a unit test with no database. The assertions are about which calls were
* issued (no `count` on the default path) and about the n+1 boundary that
* replaces it, not about row contents.
*/
import { StrKey } from "@stellar/stellar-sdk";

type AnyRecord = Record<string, any>;

const ALICE = "GDWCO35QUYQLGO6P7OLW4BZWNMMGGUWNPLRVPLCBVG7YNVDZKUDIW4KN";
// Built from raw bytes so it is a valid strkey by construction.
const CONTRACT = StrKey.encodeContract(Buffer.alloc(32, 7));

const calls: Array<{ model: string; op: string; args: AnyRecord }> = [];
let rows: AnyRecord[] = [];
let exactCount = 0;
let transactionCalls = 0;

const model = (name: string) =>
new Proxy(
{},
{
get: (_t, op: string) => (args: AnyRecord) => {
calls.push({ model: name, op, args: args ?? {} });
if (op === "count") return Promise.resolve(exactCount);
if (op === "findMany") return Promise.resolve(rows.slice(0, args?.take ?? rows.length));
return Promise.resolve(null);
},
}
) as AnyRecord;

const stub: AnyRecord = {
tokenTransfer: model("tokenTransfer"),
nftTransfer: model("nftTransfer"),
accountSummary: model("accountSummary"),
$transaction: (ops: unknown) => {
transactionCalls++;
return Array.isArray(ops) ? Promise.all(ops) : (ops as (tx: AnyRecord) => unknown)(stub);
},
};
(globalThis as AnyRecord).prisma = stub;

// Imported after the stub is installed — db.ts binds its client at module load.
import * as db from "../db";

const makeRows = (n: number) => Array.from({ length: n }, (_, i) => ({ id: n - i, ledger: 1000 + n - i, amount: "10000000" }));
const countCalls = () => calls.filter((c) => c.op === "count");

beforeEach(() => {
calls.length = 0;
rows = [];
exactCount = 0;
transactionCalls = 0;
});

it("fixtures are valid Stellar strkeys", () => {
expect(StrKey.isValidEd25519PublicKey(ALICE)).toBe(true);
expect(StrKey.isValidContract(CONTRACT)).toBe(true);
});

const listQueries: Array<[string, string, (extra?: AnyRecord) => Promise<AnyRecord>]> = [
["queryTransfers", "tokenTransfer", (e) => db.queryTransfers({ address: ALICE, direction: "incoming", ...e })],
["queryAllTransfers", "tokenTransfer", (e) => db.queryAllTransfers({ address: ALICE, ...e })],
["queryNftTransfers", "nftTransfer", (e) => db.queryNftTransfers({ contractId: CONTRACT, ...e })],
["queryAccountSummaries", "accountSummary", (e) => db.queryAccountSummaries({ address: ALICE, ...e })],
];

describe.each(listQueries)("%s", (_name, modelName, run) => {
it("issues no COUNT by default and omits total", async () => {
rows = makeRows(3);
const result = await run({ limit: 10 });

expect(countCalls()).toHaveLength(0);
expect(transactionCalls).toBe(0);
expect(calls.filter((c) => c.model === modelName && c.op === "findMany")).toHaveLength(1);
expect(result).not.toHaveProperty("total");
});

it("reports hasMore=false and no cursor when the page is exactly full (boundary)", async () => {
rows = makeRows(5); // limit 5 -> fetches 6, gets 5
const result = await run({ limit: 5 });

expect(result.hasMore).toBe(false);
expect(result.nextCursor).toBeNull();
expect(result.transfers).toHaveLength(5);
});

it("reports hasMore=true with a cursor when one row exists past the page (boundary)", async () => {
rows = makeRows(6); // limit 5 -> fetches 6, gets 6
const result = await run({ limit: 5 });

expect(calls.find((c) => c.op === "findMany")!.args.take).toBe(6); // n+1
expect(result.hasMore).toBe(true);
expect(result.nextCursor).toEqual(expect.any(String));
expect(result.transfers).toHaveLength(5);
});

it("returns an exact total only when includeTotal is requested", async () => {
rows = makeRows(2);
exactCount = 1234;
const result = await run({ limit: 10, includeTotal: true });

expect(countCalls()).toHaveLength(1);
expect(countCalls()[0].model).toBe(modelName);
expect(result.total).toBe(1234);
expect(result.hasMore).toBe(false);
});

it("includeTotal=false behaves like the default", async () => {
rows = makeRows(2);
const result = await run({ limit: 10, includeTotal: false });
expect(countCalls()).toHaveLength(0);
expect(result).not.toHaveProperty("total");
});
});
3 changes: 2 additions & 1 deletion src/__tests__/network.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,8 @@ describe("reads are network-scoped", () => {

it("counts and rows agree on the network, so total matches the page", async () => {
process.env.STELLAR_NETWORK = "mainnet";
await db.queryTransfers({ address: ADDR, direction: "incoming" });
// The count is opt-in (#206); ask for it so both queries are recorded.
await db.queryTransfers({ address: ADDR, direction: "incoming", includeTotal: true });
expect(lastWhere("tokenTransfer", "count").network).toBe("mainnet");
expect(lastWhere("tokenTransfer", "findMany").network).toBe("mainnet");
});
Expand Down
Loading
Loading