Skip to content

fix(storage): revert optimize gRPC writer with zero-copy #14630

Open
cpriti-os wants to merge 1 commit into
mainfrom
revert-zero-copy-opt
Open

fix(storage): revert optimize gRPC writer with zero-copy #14630
cpriti-os wants to merge 1 commit into
mainfrom
revert-zero-copy-opt

Conversation

@cpriti-os
Copy link
Copy Markdown
Contributor

This reverts commit df64147.

@cpriti-os cpriti-os requested review from a team as code owners May 22, 2026 11:43
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label May 22, 2026
@cpriti-os cpriti-os changed the title Revert "fix(storage): optimize gRPC writer with zero-copy fix(storage): revert optimize gRPC writer with zero-copy May 22, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request simplifies the gRPCWriter by removing the zero-copy write path and transitioning from lazy to immediate buffer allocation. Feedback from the reviewer highlights that immediate allocation increases memory overhead and that relying on cap(w.buf) instead of a dedicated chunkSize field could result in a division-by-zero panic if the chunk size is zero. It is recommended to re-introduce lazy allocation and the chunkSize field to ensure robustness and efficiency.

Comment thread storage/grpc_writer.go
Comment on lines +388 to 402
if len(w.buf)+len(v.p) <= cap(w.buf) {
// We have not started sending yet, and we can stage all data without
// starting a send. Compare against cap(w.buf) instead of
// w.writeQuantum: that way we can perform a oneshot upload for objects
// which fit in one chunk, even though we will cut the request into
// w.writeQuantum units when we do start sending.
origLen := len(w.buf)
w.buf = w.buf[:origLen+len(v.p)]
copy(w.buf[origLen:], v.p)
close(v.done)
} else {
// Too large. Handle it in writeLoop.
w.currentCommand = cmd
return nil
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This logic assumes cap(w.buf) is non-zero. If chunkSize is 0 (which is a valid setting in the storage package), this could lead to a division-by-zero panic in the handle method. Re-introducing lazy allocation with a chunkSize field and ensuring a minimum value would be safer.

Suggested change
if len(w.buf)+len(v.p) <= cap(w.buf) {
// We have not started sending yet, and we can stage all data without
// starting a send. Compare against cap(w.buf) instead of
// w.writeQuantum: that way we can perform a oneshot upload for objects
// which fit in one chunk, even though we will cut the request into
// w.writeQuantum units when we do start sending.
origLen := len(w.buf)
w.buf = w.buf[:origLen+len(v.p)]
copy(w.buf[origLen:], v.p)
close(v.done)
} else {
// Too large. Handle it in writeLoop.
w.currentCommand = cmd
return nil
}
if w.buf == nil {
w.buf = make([]byte, 0, w.chunkSize)
}
if len(w.buf)+len(v.p) <= w.chunkSize {
// We have not started sending yet, and we can stage all data without
// starting a send. Compare against w.chunkSize instead of
// w.writeQuantum: that way we can perform a oneshot upload for objects
// which fit in one chunk, even though we will cut the request into
// w.writeQuantum units when we do start sending.
origLen := len(w.buf)
w.buf = w.buf[:origLen+len(v.p)]
copy(w.buf[origLen:], v.p)
close(v.done)
} else {
// Too large. Handle it in writeLoop.
w.currentCommand = cmd
return nil
}

Comment thread storage/grpc_writer.go

buf: nil, // Allocated lazily on first buffered write.
chunkSize: chunkSize,
buf: make([]byte, 0, chunkSize),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pre-allocating the buffer here increases memory overhead for every writer instance. Consider keeping the lazy allocation pattern (initializing buf to nil and storing chunkSize in the struct) to improve memory efficiency, especially when many writers are opened or for small uploads.

Suggested change
buf: make([]byte, 0, chunkSize),
buf: nil,
chunkSize: chunkSize,

Comment thread storage/grpc_writer.go

buf []byte
chunkSize int
buf []byte
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Removing the chunkSize field forces the implementation to rely on cap(w.buf) for chunking logic. Keeping this field is necessary to support lazy buffer allocation and provides a more robust way to track the intended chunk size.

Suggested change
buf []byte
buf []byte
chunkSize int

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant