Skip to content

CBG-5900: Add support to compact channel history with Channel Names #8857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RIT3shSapata wants to merge 2 commits into
base: main
Choose a base branch
from
Open

CBG-5900: Add support to compact channel history with Channel Names #8857

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5999cc2
Add support to compact channel history with Channel Names
RIT3shSapata Oct 6, 2026
12b0976
fix api spec
RIT3shSapata Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion db/attachment_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1987,7 +1987,7 @@ func TestWritePathsPreserveUnmigratedAttachmentMetadata(t *testing.T) {
return rev2ID
},
run: func(t *testing.T, ctx context.Context, _ *Database, collection *DatabaseCollectionWithUser, docID, _ string) {
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")
},
Expand Down
28 changes: 24 additions & 4 deletions db/crud.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,8 +288,9 @@ func (c *DatabaseCollection) GetDocChannelHistory(ctx context.Context, docid str
}

// CompactDocChannelHistory removes channel history entries that ended at or before the given sequence number.
// If seq is zero, it removes the history entries of the named channels instead.
// This is used to prune stale channel assignment history to reduce storage overhead.
func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid string, seq uint64) ([]string, error) {
func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid string, seq uint64, channels []string) ([]string, error) {

@adamcfraser adamcfraser Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming the new parameter 'channelsToDelete' might make the multiple references to channel and channels below more readable.

key := realDocID(docid)
if key == "" {
return nil, base.HTTPErrorf(400, "Invalid doc ID")
Expand Down Expand Up @@ -326,23 +327,42 @@ func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid
compactedChannels := make(base.Set)

doc.SyncData.ChannelSetHistory = slices.DeleteFunc(doc.SyncData.ChannelSetHistory, func(channel ChannelSetEntry) bool {
del := channel.End <= seq
var del bool
if seq != 0 {
del = channel.End <= seq
} else if len(channels) > 0 {
del = slices.Contains(channels, channel.Name)
Comment on lines +330 to +334

@adamcfraser adamcfraser Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comment is valid, particularly given some of the known use cases related to large numbers of channels per doc - the iteration over channels to create the map wouldn't be significantly more expensive than a single slices.Contains, and you'd only need to do it once.

}
if del {
compactedChannels.Add(channel.Name)
}
return del
})

doc.SyncData.ChannelSet = slices.DeleteFunc(doc.SyncData.ChannelSet, func(channel ChannelSetEntry) bool {
del := channel.End != 0 && channel.End <= seq
var del bool
if seq != 0 {
del = channel.End != 0 && channel.End <= seq
} else if len(channels) > 0 {
del = channel.End != 0 && slices.Contains(channels, channel.Name)
}
if del {
compactedChannels.Add(channel.Name)
}
return del
})

for chanName, chanEntry := range doc.SyncData.Channels {

@factory-droid factory-droid Bot Oct 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Avoid compacting Channels entries with Seq==0

SyncData.getCurrentChannels treats both nil removals and non-nil removals with Seq==0 as current channel memberships. CompactDocChannelHistory currently deletes non-nil entries whenever chanEntry.Seq <= seq (seq mode) or when the channel name is listed (channels mode), which can drop a channel that is still considered current and change the effective channel set.

Consider skipping deletions for chanEntry.Seq==0 (consistent with GetDocChannelHistory, which only reports removals where Seq != 0), for example:

Suggested change
for chanName, chanEntry := range doc.SyncData.Channels {
var del bool
if seq != 0 {
del = chanEntry.Seq != 0 && chanEntry.Seq <= seq
} else if len(channels) > 0 {
del = chanEntry.Seq != 0 && slices.Contains(channels, chanName)
}

if chanEntry != nil && chanEntry.Seq <= seq {
if chanEntry == nil {
continue
}
var del bool
if seq != 0 {
del = chanEntry.Seq <= seq
} else if len(channels) > 0 {
del = slices.Contains(channels, chanName)
}
if del {
compactedChannels.Add(chanName)
delete(doc.SyncData.Channels, chanName)
}
Expand Down
2 changes: 1 addition & 1 deletion db/hybrid_logical_vector_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -671,7 +671,7 @@ func TestRestampVersionCASMou(t *testing.T) {
// A second metadata-only write, from a different path - channel history compaction, which unlike
// resync applies to a tombstone as well as a live document. Both previous values have to be
// carried forward, or they stop naming the last write to the body.
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")

Expand Down
2 changes: 1 addition & 1 deletion db/import_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1948,7 +1948,7 @@ func TestMetadataOnlyUpdateWritePaths(t *testing.T) {
return rev2ID
},
run: func(t *testing.T, ctx context.Context, collection *DatabaseCollectionWithUser, docID, _ string) {
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100)
compacted, err := collection.CompactDocChannelHistory(ctx, docID, 100, []string{""})
require.NoError(t, err)
require.NotEmpty(t, compacted, "compaction should have pruned channel history, or no write happens")
},
Expand Down
40 changes: 31 additions & 9 deletions docs/api/paths/admin/keyspace-_channel_history-compact.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@ parameters:
post:
summary: Compact Channel History of Document
description: |-
Compacts channel history for a specified document. Channel history older than the specified sequence will be removed.
Compacts channel history for a specified document. The request must contain either `seq` or `channels`, but not both.

This endpoint removes all channel entries (for sequences before the specified sequence number where the document left the channel),
effectively cleaning up historical channel membership information while preserving active channels and recent changes.
With `seq`, this endpoint removes all channel entries where the document left the channel at or before the specified sequence number.
With `channels`, this endpoint removes the channel history entries for the named channels, whatever sequence the document left them at.
Channels that the document is still in are not removed.

Compaction cleans up historical channel membership information while preserving active channels and recent changes.
This can be useful for reducing metadata size for documents that frequently gain and lose access to channels.

Required Sync Gateway RBAC roles:
Expand All @@ -25,23 +28,42 @@ post:
application/json:
schema:
type: object
required:
- seq
properties:
seq:
description: |-
Channel history having end sequences earlier than this sequence will be removed from the specified document's metadata.
Channel history having end sequences at or before this sequence will be removed from the specified document's metadata.

Cannot be used together with `channels`.
type: integer
format: int64
minimum: 1
example: 12345
channels:
description: |-
Channel history for these channels will be removed from the specified document's metadata.

A non-empty array cannot be used together with `seq`.
type: array
items:
type: string
example:
- channel1
- channel2
oneOf:
- required:
- seq
- required:
- channels
properties:
channels:
minItems: 1
responses:
'200':
description: |-
Successfully compacted channel history from the specified document.
Returns a list of channels that were compacted.

If the response has an empty array, it means either no channels were compacted.
If the response has an empty array, it means no channels were compacted.

content:
application/json:
Expand All @@ -55,13 +77,13 @@ post:
description: |-
Array of channel names that were compacted.
description: |-
A array of all the compacted channels
An array of all the compacted channels
example:
compacted_channels:
- channel1
- channel2
'400':
description: 'Bad request. This could be due to invalid request parameters such as invalid seq value.'
description: 'Bad request. This could be due to invalid request parameters such as an invalid seq value, an empty channels array with no seq, or both seq and a non-empty channels array in the same request.'
content:
application/json:
schema:
Expand Down
Loading
Loading