Read storage API error selecting array field

[vladDotH]

What happened?

When i try to read field with type via grpc, emulator produces error

What did you expect to happen?

Successful quiering including array fields

How can we reproduce it (as minimally and precisely as possible)?

code, where i create table with array and read it via grpc service: https://gist.github.com/vladDotH/1571031a34a384248870ff8bbb82f2ae

docker-compose.yml:

services:
  bq-emulator:
    container_name: bq-emulator
    image: ghcr.io/goccy/bigquery-emulator:0.6.6
    command: --project=testproject --database=testdatabase --dataset=testdataset --log-level=debug 
    ports:
      - 9050:9050
      - 9060:9060

Anything else we need to know?

I suppose it works correct, because i see my data in logs, but something goes wrong at arrow serializing stage

Stacktrace:

bq-emulator | [bigquery-emulator] REST server listening at 0.0.0.0:9050
bq-emulator | [bigquery-emulator] gRPC server listening at 0.0.0.0:9060
bq-emulator | 2025-04-09T20:42:53.008Z INFO server/middleware.go:63 POST /projects/testproject/jobs {"query": "alt=json&prettyPrint=false"}
bq-emulator | 2025-04-09T20:42:53.013Z INFO contentdata/repository.go:166 {"query": "\n\t\tcreate table if not exists testdataset.testtable(\n\t\t\tx int64,\n\t\t\ty string,\n\t\t\ta Array\n\t\t);\n\n\t\tinsert into testdataset.testtable(x, y, a)\n\t\tvalues \n\t\t(1, '1 str', [1,2,3]), \n\t\t(2, '2nd str', []), \n\t\t(33, '3rd string', [0,0,0]);\n\t", "values": []}
bq-emulator | 2025-04-09T20:42:53.018Z DEBUG contentdata/repository.go:245 query result {"rows": []}
bq-emulator | 2025-04-09T20:42:53.041Z INFO server/middleware.go:63 GET /projects/testproject/queries/zvmrIEDeyc5yn7s1ez2Y5C6neIQ {"query": "alt=json&formatOptions.useInt64Timestamp=true&location=&maxResults=0&prettyPrint=false"}
bq-emulator | 2025-04-09T20:42:53.145Z INFO server/middleware.go:63 GET /projects/testproject/jobs/zvmrIEDeyc5yn7s1ez2Y5C6neIQ {"query": "alt=json&fields=status%2Cstatistics&prettyPrint=false"}
bq-emulator | 2025-04-09T20:42:53.148Z INFO server/middleware.go:63 POST /projects/testproject/queries {"query": "alt=json&prettyPrint=false"}
bq-emulator | 2025-04-09T20:42:53.151Z INFO contentdata/repository.go:166 {"query": "select * from testdataset.testtable;", "values": []}
bq-emulator | 2025-04-09T20:42:53.152Z DEBUG contentdata/repository.go:245 query result {"rows": [[1,"1 str",[1,2,3]],[2,"2nd str",[]],[33,"3rd string",[0,0,0]]]}
bq-emulator | 2025-04-09T20:42:53.159Z INFO contentdata/repository.go:166 {"query": "SELECT x,y,a FROM testtable ", "values": []}
bq-emulator | 2025-04-09T20:42:53.160Z DEBUG contentdata/repository.go:245 query result {"rows": [[1,"1 str",[1,2,3]],[2,"2nd str",[]],[33,"3rd string",[0,0,0]]]}
bq-emulator | panic: arrow/array: field 2 has 6 rows. want=3
bq-emulator |
bq-emulator | goroutine 335 [running]:
bq-emulator | github.com/apache/arrow/go/v10/arrow/array.(*RecordBuilder).NewRecord(0xc000b9b8c0)
bq-emulator | /go/pkg/mod/github.com/apache/arrow/go/v10@v10.0.1/arrow/array/record.go:321 +0x28e
bq-emulator | github.com/goccy/bigquery-emulator/server.(*storageReadServer).sendARROWRows(0xc000c839b0?, 0xc000f13b80, 0xc00093be50, {0x30fb580, 0xc000d812a0})
bq-emulator | /work/server/storage_handler.go:339 +0x1a5
bq-emulator | github.com/goccy/bigquery-emulator/server.(*storageReadServer).ReadRows(0xc000c839b0, 0xc000b9af40, {0x30fb580, 0xc000d812a0})
bq-emulator | /work/server/storage_handler.go:166 +0x11c
bq-emulator | cloud.google.com/go/bigquery/storage/apiv1/storagepb._BigQueryRead_ReadRows_Handler({0x2ce8c20?, 0xc000c839b0}, {0x30f9698, 0xc000ac03c0})
bq-emulator | /go/pkg/mod/cloud.google.com/go/bigquery@v1.60.0/storage/apiv1/storagepb/storage.pb.go:3048 +0xd0
bq-emulator | google.golang.org/grpc.(*Server).processStreamingRPC(0xc000153600, {0x30f44e8, 0xc000d85fb0}, {0x31034e0, 0xc000b73860}, 0xc000af6fc0, 0xc000c83a70, 0x4b68380, 0x0)
bq-emulator | /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1680 +0x1288
bq-emulator | google.golang.org/grpc.(*Server).handleStream(0xc000153600, {0x31034e0, 0xc000b73860}, 0xc000af6fc0)
bq-emulator | /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0xfbb
bq-emulator | google.golang.org/grpc.(*Server).serveStreams.func2.1()
bq-emulator | /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x8b
bq-emulator | created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 338
bq-emulator | /go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x135
bq-emulator exited with code 2

[ohaibbq]

I've implemented and released a fix for the Arrow array serialization panic in the Recidiviz fork of the emulator
https://github.com/Recidiviz/bigquery-emulator/releases/tag/v0.6.6-recidiviz.0

[lraigosov]

Fixed in PR #496. The root cause was in AppendValueToARROWBuilder (internal/types/types.go): listBuilder.Append(true) was called inside the per-element loop, creating one Arrow list slot per element instead of one per row. A row with [1,2,3] produced 3 list slots instead of 1, making that field's row count diverge from the others and triggering the arrow/array: field N has M rows. want=K panic in NewRecord.