Skip to content

fix(server): Support google.protobuf.Timestamp values for the storage write API#416

Open
coxley wants to merge 6 commits into
goccy:mainfrom
coxley:storage-timestamppb
Open

fix(server): Support google.protobuf.Timestamp values for the storage write API#416
coxley wants to merge 6 commits into
goccy:mainfrom
coxley:storage-timestamppb

Conversation

@coxley

@coxley coxley commented Jul 16, 2025

Copy link
Copy Markdown

Summary

While BigQuery recommends to send TIMESTAMP columns encoded as int64, they also support google.protobuf.Timestamp: https://cloud.google.com/bigquery/docs/supported-data-types. This is critical for workloads that ingest JSON data, where RFC3339 strings can naturally decode into a timestamppb.Timestamp.

The client SDK making this easier with the following issue and PR:

Testing these changes work against Google, but panic the emulator:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x60 pc=0x10387d398]

goroutine 8764 [running]:
github.com/goccy/bigquery-emulator/types.normalizeData({0x10541b980?, 0x14002391318?}, 0x0)
        /Users/coxley/projects/fork-bigquery-emulator/types/types.go:547 +0x88
github.com/goccy/bigquery-emulator/types.normalizeData({0x105529660?, 0x1400264d080?}, 0x140026da1e0)
        /Users/coxley/projects/fork-bigquery-emulator/types/types.go:575 +0x394
github.com/goccy/bigquery-emulator/types.NewTableWithSchema(0x14002330248, {0x14001471238, 0x1, 0x103a9316c?})
        /Users/coxley/projects/fork-bigquery-emulator/types/types.go:479 +0x398
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).insertTableData(0x140025dcba0, {0x105869c28, 0x1400264d0b0}, 0x1400220c3d8, 0x1400083c150, {0x14001471238?, 0x4?, 0x10544dca0?})
        /Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:691 +0x44
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).appendRows(0x140025dcba0, 0x1400083c2a0, {0x10588ca80, 0x14001612000}, {0x1058750d0, 0x14002430980})
        /Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:515 +0x428
github.com/goccy/bigquery-emulator/server.(*storageWriteServer).AppendRows(0x140025dcba0, {0x1058750d0, 0x14002430980})
        /Users/coxley/projects/fork-bigquery-emulator/server/storage_handler.go:437 +0xb0
cloud.google.com/go/bigquery/storage/apiv1/storagepb._BigQueryWrite_AppendRows_Handler({0x105720fa0?, 0x140025dcba0}, {0x105870a20, 0x140026da3c0})
        /Users/coxley/.go/pkg/mod/cloud.google.com/go/bigquery@v1.60.0/storage/apiv1/storagepb/storage.pb.go:3365 +0xdc
google.golang.org/grpc.(*Server).processStreamingRPC(0x140015d5000, {0x105869c28, 0x1400264cbd0}, {0x10587b040, 0x14000652d00}, 0x1400263e6c0, 0x140025dcc60, 0x106c72ac0, 0x0)
        /Users/coxley/.go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1680 +0xe54
google.golang.org/grpc.(*Server).handleStream(0x140015d5000, {0x10587b040, 0x14000652d00}, 0x1400263e6c0)
        /Users/coxley/.go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0xac4
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /Users/coxley/.go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x84
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 8622
        /Users/coxley/.go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x138

This PR ensures that google.protobuf.Timestamp messages are converted to time.Time early in decodeRowData.

It also fixes a bug for RECORD columns containing TIMESTAMP cells — but it depends on goccy/go-zetasqlite#232 to make the test pass.

Test Plan

Added a new test case that reproduces the problem, and is fixed with these changes:

> go test ./...
?       github.com/goccy/bigquery-emulator/cmd/bigquery-emulator        [no test files]
?       github.com/goccy/bigquery-emulator/internal/connection  [no test files]
?       github.com/goccy/bigquery-emulator/internal/contentdata [no test files]
?       github.com/goccy/bigquery-emulator/internal/logger      [no test files]
?       github.com/goccy/bigquery-emulator/internal/metadata    [no test files]
?       github.com/goccy/bigquery-emulator/internal/types       [no test files]
ok      github.com/goccy/bigquery-emulator/server       2.749s
?       github.com/goccy/bigquery-emulator/types        [no test files]

@coxley

coxley commented Jul 16, 2025

Copy link
Copy Markdown
Author

I assume @goccy is the best maintainer to review? Thanks in advanced!

adw-okamu added a commit to adw-okamu/bigquery-emulator that referenced this pull request Feb 18, 2026
Remove RECORD column handling for nested timestamps as it depends on
go-zetasqlite#232 which is not yet merged. The core
google.protobuf.Timestamp support for the storage write API is retained.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant