Clarity Around Aux Offset Encoding

[XAMPPRocky]

Reading through the spec for encoding, I'm unsure how you would correctly calculate an offset for aux without specifying it manually or jumping through a number of hoops if you're assuming meta and main have no fixed size.

If you want to encode the aux section directly at the end of main and want to find what offset to put it in meta, you first need the encoded version of meta.

However since we have variable length integer encoding, the offset at which you encode meta is dependent on the length of the meta + main section + aux_region_offset, which would also change the length of the encoding, which would affect aux_region_offset. Adding the offset could be between 2–4 bytes of extra encoding depending on the amount of data and figuring out when it would cross that threshold is challenging.

I feel like there could be something in the spec that makes this process easier.

Example:

section	length (bytes)
Meta (without aux_region_offset)	1
Main	3
Total	4

aux_region_offset 6 (2 bytes)

section	length (bytes)
Meta (without aux_region_offset)	32
Main	220
Total	252

aux_region_offset 255 (3 bytes)

section	length (bytes)
Meta (without aux_region_offset)	508
Main	512
Total	1024

aux_region_offset 1028 (4 bytes)

[CZDanol]

Hello :)

I'm unsure how you would correctly calculate an offset for aux without specifying it manually

That is the intended method. You just dedicate a fixed amount of space for the aux region.

I understand your problem though. I'd suggest reserving some amount of space for the meta region (say 8 B) and computing your aux region position based on that. Yes, it wouldn't be always directly after the main region, but the few bytes shouldn't be that hard of a sacrifice.

I feel like there could be something in the spec that makes this process easier.

Do you have any proposals? The limitation is that we cannot make the readers read the whole main section to determine the position of the aux region.

[XAMPPRocky]

Do you have any proposals? The limitation is that we cannot make the readers read the whole main section to determine the position of the aux region.

I've been thinking about it. I think if aux section presence could be determined differently, then you could get the offset from the other properties.

I'm not proposing this as a solution but just as an example. If aux_region_offset was bool | int, and then it being true meant it was directly following main, then as a implementer you would only need main_region_size to be able to read just the aux section as you can determine the aux's position as either:

• If main_region_offset is set: main_region_offset + main_region_size
• Else current_position + main_region_size

[XAMPPRocky]

@CZDanol What do you think? :)

[CZDanol]

The idea sounds realizable, but variants don't fit well into the system. I'd rather opt for a separate field a la aux_region_after_main.

[XAMPPRocky]

I think that would work, I mentioned it as a variant as it would have minimal impact on size but a separate field would also work.

[CZDanol]

Why would a separate field have a bigger impact on the size?

[XAMPPRocky]

Oh well right now to be compatible, the aux_region_offset still needs to be present for indicating presence, but the standard could be updated to say either aux_region_offset or aux_after_main indicate presence.

[CZDanol]

Yep, that's what I meant :)