v0.7.x: Data inserted via tabledata.insertAll is not visible to subsequent queries

[jan-pluta-getmoss]

What happened?

After upgrading from v0.6.6 to v0.7.0 (or v0.7.1), data inserted via tabledata.insertAll (streaming inserts using the Java BigQuery client) is no longer visible to subsequent queries. The queries return 0 rows.

The exact same code works correctly with v0.6.6.

What did you expect to happen?

The query should return the inserted row. This works correctly in v0.6.6.

How can we reproduce it (as minimally and precisely as possible)?

1. Start the emulator with --data-from-yaml pointing to a YAML schema file defining tables (empty data:)
2. Insert rows using the Java BigQuery.insertAll() API (equivalent to tabledata.insertAll REST endpoint)
3. Query the table — returns 0 rows

Schema (YAML):

projects:
  - id: dummy-data-project
    datasets:
      - id: dummy_services_sensitive
        tables:
          - id: test_table
            columns:
              - name: id
                type: STRING
                mode: REQUIRED
              - name: value
                type: STRING
                mode: REQUIRED
            data:

Insert (Java):

bigQuery.insertAll(InsertAllRequest.newBuilder(
        TableId.of("dummy-data-project", "dummy_services_sensitive", "test_table"))
    .addRow(Map.of("id", "1", "value", "hello"))
    .build());
// Response has no errors

Query (Java):

TableResult result = bigQuery.query(QueryJobConfiguration.newBuilder(
    "SELECT * FROM `dummy-data-project.dummy_services_sensitive.test_table`")
    .build());
// Returns 0 rows

Anything else we need to know?

• The emulator's own e2e conformance suite does not test tabledata.insertAll followed by queries — it only tests queries against pre-loaded data (via --data-from-yaml). This may explain why this regression was not caught.
• The InsertAllResponse from the emulator reports no errors — the insert appears to succeed.
• This was discovered in a CI pipeline where Testcontainers starts the emulator fresh (no container reuse). On local dev machines with withReuse(true), the old v0.6.6 container may persist, masking the issue.

[goccy]

Thanks for the very detailed report. I tried hard to reproduce this and could not — streamed rows are visible to the subsequent query in every setup I tried, including against the exact published image with your exact repro inputs.

What I tried

1. Go client + in-process emulator (built from main) — insertAll then query: row visible.

2. All six official clients (Python, Ruby, PHP, Node.js, Java, bq CLI) against a freshly started emulator, with the table preloaded via --data-from-yaml (empty data:) and rows streamed in through tabledata.insertAll, then queried — every client sees the row.

3. The published ghcr.io/goccy/bigquery-emulator:0.7.1 image, started with your exact YAML (dummy-data-project / dummy_services_sensitive / test_table, empty data:), then a raw REST tabledata.insertAll followed by a query:

   insertAll → {}                              (HTTP 200, no errors)
   query     → rows: [{"v":"1"},{"v":"hello"}]
   

So with the information in the issue I cannot reproduce the regression.

What changed

You correctly pointed out that the e2e conformance suite only covered read-only queries and never exercised insertAll → query. That gap is now closed: #471 adds an insert_all_then_select case that preloads a table (modelling --data-from-yaml), streams rows in through tabledata.insertAll, and asserts they are visible to the following query — run across all six clients. It passes everywhere, and it stays in as a permanent regression guard.

Could you help narrow it down

If this still reproduces for you, it would help to have:

• Emulator output with --apilog=stderr around the failing insertAll + query
• The actual test code (the insertAll and the query as your CI runs them)
• Whether it still reproduces against a build from main (or the next release)

If there is a specific trigger, we can add it as an e2e case and pin the fix. For now I will leave this open pending more detail — happy to reopen the investigation as soon as we can reproduce it.

[nabetama]

Jumping in here — sorry to chime in from the side.

I ran into what looks like the same problem under slightly different conditions, so I dug into it and think I found a reproduction.

I can reproduce this deterministically, and I think I found why it didn't reproduce on your side.
It only happens when the same dataset.table is defined in more than one project in the --data-from-yaml file. With a single project it works fine, which matches the minimal YAML you tried.

The streamed data is not lost. It is correctly written to the target project's table. The actual problem is name resolution on read maybe. A query that references the table without the project (dataset.table) resolves to the wrong project's (empty) table.

Minimal reproduction

I call raw REST API via curl, no client libraries.

min.yaml — the same ds.t defined in two projects, empty data:

projects:
  - id: project_a
    datasets:
      - id: ds
        tables:
          - id: t
            columns: [{name: id, type: STRING, mode: REQUIRED}]
            data:
  - id: project_b
    datasets:
      - id: ds
        tables:
          - id: t
            columns: [{name: id, type: STRING, mode: REQUIRED}]
            data:

bigquery-emulator --project=project_b --data-from-yaml=min.yaml

# streaming insert -> {}  (HTTP 200, no errors)
curl -s -XPOST localhost:9050/bigquery/v2/projects/project_b/datasets/ds/tables/t/insertAll \
  -d '{"rows":[{"json":{"id":"1"}}]}'

# project-less reference -> 0 rows  (the bug?)
curl -s -XPOST localhost:9050/bigquery/v2/projects/project_b/queries \
  -d '{"query":"SELECT COUNT(*) FROM `ds.t`","useLegacySql":false}'           # -> 0

# fully-qualified reference -> 1 row  (the data is actually there)
curl -s -XPOST localhost:9050/bigquery/v2/projects/project_b/queries \
  -d '{"query":"SELECT COUNT(*) FROM `project_b.ds.t`","useLegacySql":false}' # -> 1

Notes

• With a single project and one table, a project-less query returns 1 row (works).
• With two projects defining the same dataset.table, a project-less query returns 0 rows, while the fully-qualified project.dataset.table returns 1 (bug?).
• With two projects defining different table names, a project-less query returns 1 row (works).

So the trigger is a duplicate dataset.table across projects. The symptom is that an unqualified / dataset.table reference no longer resolves to the connection's project once a same-named table exists under another project. v0.6.6 resolves it correctly; a build from main still reproduces.

[tc-lraigoso]

Hello!

I've been using this emulator and I really appreciate the work you've put into it. It's a fantastic tool.

As a token of appreciation, I've taken the liberty of fixing this regression. The issue was that tables created or populated via streaming inserts (tabledata.insertAll) were being stored/resolved with qualified names that didn't match the search path when queries were unqualified. I've forced full qualification (project.dataset.table) for all table metadata to ensure consistent resolution.

I've also fixed #482 (nested TIMESTAMP formatting) which was affecting my workflows.

You can find the fixes in my fork: https://github.com/lraigosov/bigquery-emulator/tree/feat/maintenance-and-fixes

I tried to open a PR but it seems this repository might have restrictions on PRs from non-collaborators via API. Please let me know if there's a preferred way to contribute these fixes!

Best regards!

[lraigosov]

Correction: I've updated my git configuration. The previous comments and the code in the fork/PR are correctly attributed to lraigosov. I've fixed the regressions for #470 and #482.

PR: #492