Privacy concerns, alternative design

[supercooltest2024]

As mentioned in the readme, there is already skepticism of a privacy-preserving way for shared client storage.

"We acknowledge a downside with this approach related to lack of shared client storage for model weights — it would be a better experience if the browser only had to download large weights one time. We don’t know of a privacy-preserving way to do this, short of high level APIs like these which abstract away the details of inference."

• not a user need example, but rather skepticism of shared storage and suggesting high level inference apis instead. (the high level APIs they were referring to Writing Assistance APIs)

This api appears to try to cover 2 different use cases (for both data bandwidth and storage costs):

• Extremely common libraries (wasm/js/css/etc) that are usually relatively small (<10MB) e.g. jquery
• Extremely uncommon blobs greater than a GB. e.g. AI models

It appears this api was designed for common libraries which have less tracking/fingerprinting risk; the first case. But the motivating goals are for the second case (AI models). The Public Hash List implementation page suggests that AI models should be accepted even though they are uncommon. The exact set of large uncommon AI models cached carries a lot of entropy for fingerprinting. Even worse, it probably identifies the obscure sites the user that actually use each individual model, especially for models trained for specific purposes (like models trained to identify medical conditions).

Even the common libraries use case is a privacy problem. If a common library is only on specific kinds of sites (google, social media, etc), it gives away what kind of sites the user visits. And of course with enough common libraries, it will be uncommon to have them all making the set of cached libraries a fingerprinting vector. If a library is used on practically every site then if it is absent it is a strong signal the user is in incognito mode.

It is well known that bucketing users is a more effective privacy measure than randomizing api results (greasing). In this case, the 'greasing' will only be able to lie in one direction. It can't pretend to have something cached that it doesn't, so anything that is reported cached is known to not be a lie. Detecting fingerprinting attempts to stop api access will be a balancing act that will likely fail (ML is not the answer to all problems).

Solutions

Storage space on device fix

if both example.com and example.org each require the same 8 GB AI model, this would result in 16 GB of downloaded data and a total allocation of 16 GB on the user's device This can be solved today without any new api. If two sites store the exact same file the browser can dedup and just store one file with a note that both sites have it cached. Clearing storage for one site would clear the note that that site had it cached. This wouldn't allow a site to use cached files from another site before caching the files themselves, removing privacy risks. Thus even though 16GB of data is downloaded, only 8GB needs to be stored on the user's device. (in fact, this is solved already on some file systems where files are deduped automatically).

Common libraries over fetching fix

Common libraries have been over fetched forever. If it actually is a problem though, they should be bucketed instead of greased. That is, the set of common-files entries should be one of a few possible sets. Probably by storage capacity: small (1GB), medium (10GB) and large (100GB). Where each class contains all most common files under a certain storage size. Phones might use small, laptops might use medium, and desktops might use large. What is in each set shouldn't be updated very often, as that could reveal the last time the user updated.

The api for this could look largely the same as the current proposed cross-origin-storage, but with buckets instead of greasing and without the need to specify an origins list (the api would only be for things that are likely in one of the global buckets). To prevent developers from depending on files being in the current buckets, the buckets should not be used if the cache is turned off in devtools. To maximize effectiveness, the cross-origin-storage(shared-files) html attribute should only be needed if the site has a service worker, otherwise if the html integrity attribute matches that should be enough to benefit for existing sites.

Large files over fetching fix

Large uncommon files like game engines and AI models need a different solution. They need permission prompts. Thus the files need user friendly names. The hash and the name should match to be considered the same (so that a malicious actor can't confuse the user with a misleading name). There are a few cases here.

• The user has the large file cached by the same origin or has been allowed before
  • Allow without prompt
• The user has the large file cached by a different origin
  • Do you want to allow example.com to reuse file 'UnityWebEngineV1.zip' from the following sites to save 10GB of data? <bulleted list of origins>.
• The user doesn't have the file cached
  • we need a permission prompt here to prevent timing attacks, so:
  • example.com wants file 'UnityWebEngineV1.zip', do you want to select it locally to save 10GB of data? followed by a file picker if yes. If the user actually has the file locally and selects it, it should first be copied to browser storage so that it will be immediately available on subsequent visits.

The api for this could also look largely the same as the current proposed cross-origin-storage, but with a name field and again no need for an origins list. This shouldn't be the same api as the one for common libraries, as those shouldn't require permission prompts.

[tomayac]

Thanks for the thoughts. Comments inline…

As mentioned in the readme, there is already skepticism of a privacy-preserving way for shared client storage.

"We acknowledge a downside with this approach related to lack of shared client storage for model weights — it would be a better experience if the browser only had to download large weights one time. We don’t know of a privacy-preserving way to do this, short of high level APIs like these which abstract away the details of inference." - not a user need example, but rather skepticism of shared storage and suggesting high level inference apis instead.

The "We acknowledge a downside […]" part is a quote from a Mozilla engineer.

This api appears to try to cover 2 different use cases (for both data bandwidth and storage costs):

• Extremely common libraries (wasm/js/css/etc) that are usually relatively small (<10MB) e.g. jquery
• Extremely uncommon blobs greater than a GB. e.g. AI models

That's correct.

It appears this api was designed for common libraries which have less tracking/fingerprinting risk; the first case. But the motivating goals are for the second case (AI models).

The API was explicitly designed for and motivated by both use cases.

The Public Hash List implementation page suggests that AI models should be accepted even though they are uncommon.

The prototype implementation explicitly caps the list of models based on popularity. See more detail about this in the Public Hash List (PHL) implementation README and the PHL explainer.

The exact set of large uncommon AI models cached carries a lot of entropy for fingerprinting. Even worse, it probably identifies the obscure sites the user that actually use each individual model, especially for models trained for specific purposes (like models trained to identify medical conditions).

By sorting the models by number of downloads, the prototype implementation makes sure that only highly popular (as per the AI model hub's ranking) models make it onto the PHL.

Even the common libraries use case is a privacy problem. If a common library is only on specific kinds of sites (google, social media, etc), it gives away what kind of sites the user visits. And of course with enough common libraries, it will be uncommon to have them all making the set of cached libraries a fingerprinting vector. If a library is used on practically every site then if it is absent it is a strong signal the user is in incognito mode.

By limiting the number of probes, abusing this indeed theoretically existing tracking vector would be made infeasible in practice. See the implementation notes for the current thinking.

It is well known that bucketing users is a more effective privacy measure than randomizing api results (greasing). In this case, the 'greasing' will only be able to lie in one direction. It can't pretend to have something cached that it doesn't, so anything that is reported cached is known to not be a lie. Detecting fingerprinting attempts to stop api access will be a balancing act that will likely fail (ML is not the answer to all problems).

I think combined with Safe Browsing and other signals browsers have about origins, for example, for ad tagging, plus dynamic attempts at bursty probing (which the implementations should render impossible), we should be in a pretty good spot to detect abuse.

Solutions

Storage space on device fix

if both example.com and example.org each require the same 8 GB AI model, this would result in 16 GB of downloaded data and a total allocation of 16 GB on the user's device This can be solved today without any new api. If two sites store the exact same file the browser can dedup and just store one file with a note that both sites have it cached. Clearing storage for one site would clear the note that that site had it cached. This wouldn't allow a site to use cached files from another site before caching the files themselves, removing privacy risks. (in fact, this already is solved on some file systems where files are deduped automatically).

The storage part would indeed be easy to fix, and some browsers (or, as you say, file systems) may already do this.

Common libraries over fetching fix

Common libraries have been over fetched forever. If it actually is a problem though, they should be bucketed instead of greased. That is, the set of observable cross-origin-storage entries should be one of a few possible sets. Probably by storage capacity: small (1GB), medium (10GB) and large (100GB). Where each class contains all most common files under a certain storage size. Phones might use small, laptops might use medium, and desktops might use large. What is in each set shouldn't be updated very often, as that could reveal the last time the user updated.

Note that the core defense line against abuse is the PHL, which is strictly popularity-based per source (See the PHL explainer), and only applies to resources stored with origins: '*'. If you will, the per-source popularities that then get merged into the global list are your buckets. GREASE'ing is meant to be more like a grain of sand in the gears.

The api for this could look largely the same as the current proposed cross-origin-storage, but with buckets instead of greasing and without the need to specify an origins list (the api would only be for things that are likely in one of the global buckets).

If you specify an origins list, the PHL doesn't apply, and your list of origins can't become arbitrarily large (see the explainer). The listing of origins is meant for developers who want to share a resource that is not popular, but should be shared across the developer's app's origins (like an office suite with a shared corporate web font).

To prevent developers from depending on files being in the current buckets, the buckets should not be used if the cache is turned off in devtools.

This sounds more like a UA-specific behavior, not something that the spec should dictate.

Large files over fetching fix

Large uncommon files like game engines and AI models need a different solution. They need permission prompts. Thus the files need user friendly names. The hash and the name should match to be considered the same (so that a malicious actor can't confuse the user with a misleading name).

There are a few cases here.

• The user has the large file cached by the same origin or has been allowed before

  • Allow without prompt

• The user has the large file cached by a different origin

  • Do you want to allow example.com to use 'UnityWebEngineV1.zip' from the following sites to save 10GB of data? <bulleted list of origins>.

• The user doesn't have the file cached

  • we need a permission prompt here to prevent timing attacks, so:
  • example-a.com wants file 'UnityWebEngineV1.zip', do you want to select it locally to save 10GB of data? followed by a file picker if yes. If the user actually has the file locally and selects it, it should first be copied to browser storage so that it will be immediately available on subsequent visits.

We initially thought about permission prompts, but partner feedback was negative throughout. Also see #47 for some additional background. Instead, we landed on availability gating (the PHL) as a strictly better alternative.

The api for this could also look largely the same as the current proposed cross-origin-storage, but with a name field and again no need for an origins list. This shouldn't be the same api as the one for common libraries, as those shouldn't require permission prompts.

Letting apps specify content to be shown in browser UI like a permission prompt is generally deemed a no-go on the Web. It's different with apps in stores that have a review process, where, for example, letting an app explain why it needs a given permission is a good thing, as it adds context, and as abuse (in theory at least) will be caught during the review.

[tomayac]

Closing the Issue as completed for now, but please re-open or comment if you have additional thoughts.

[supercooltest2024]

I guess, I still don't think my concerns are addressed.

• Incognito mode will be identifiable by common files from other sites not being cached. Unless the cache is shared with incognito mode, which presents it's own problems
• The PHL is not equivalent to pre-downloaded buckets of files, because not every file on the PHL will be cached by all users. That is, every user will have their own set of cached files from the PHL. As each user will have a different set, that makes them identifiable.
• The referenced Constants reference Burst capacity: 2000 (sized around large sharded-AI-model loading — some architectures ship weights as ~25 MiB shards, so 2000 covers roughly a 50 GiB model's worth of shards in one burst — comfortably past any realistic model size, while still bounding worst-case memory for the token-bucket map). implies that 2000 requests can be queried in a burst. 2000 queries is plenty to identify a user based on what they have cached.
• The implementation-notes and readme read like an AI wrote them This is the kind of thing that's easy to miss on a first pass, because it's not part of the spec at all — it's purely an implementation detail that can still become a real, unbounded memory leak. This doesn't bode confidence that points are well considered.
• I don't think We initially thought about permission prompts, but partner feedback was negative throughout is enough to introduce a privacy risk to users. Certainly some partners could have alterior motives.
• I would want to know if some random website I visited was invisibly checking to see if I have used 'popular' heavy AI models / game engines locally. Because in a browser they simply are not common. If I played a lot of unity games in browser, I would consider it reasonable to have a permission prompt to avoid a multi-gigabyte download on new sites. Then I would know if non-gaming sites were randomly trying to see if I played unity games in browser (to identify me).

[tomayac]

• Incognito mode will be identifiable by common files from other sites not being cached. Or if the cache is shared with incognito mode, that presents it's own problems

Incognito mode starts empty as if it were a fresh browser profile as per the current thinking, but it could likely also be seeded by the truly most common resources. All files accumulated during an incognito session will be deleted when incognito mode ends.

• The PHL is not equivalent to pre-downloaded buckets of files, because not every file on the PHL will be cached by all users. That is, every user will have their own set of cached files from the PHL. As each user will have a different set, that makes them identifiable.

Oh, I misunderstood your bucket idea. Downloading these buckets sounds rather wasteful and feels like a no-go, as it would exclude low-storage devices that can't fit the whole bucket.

• The referenced Constants reference Burst capacity: 2000 (sized around large sharded-AI-model loading — some architectures ship weights as ~25 MiB shards, so 2000 covers roughly a 50 GiB model's worth of shards in one burst — comfortably past any realistic model size, while still bounding worst-case memory for the token-bucket map). implies that 2000 requests can be queried in a burst. 2000 queries is plenty to identify a user based on what they have cached.

It's a careful balance to avoid breaking actual use cases like sharded AI models and catching abuse. Previous experiment results from where the HTTP cache was about to be isolated showed cache hit rates were rather unaffected from cache isolation, suggesting the chosen number could work, but of course we can only tell once we roll this out in a real at-scale test implementation (like an origin trial in Chrome).

• The implementation-notes and readme read like an AI wrote them This is the kind of thing that's easy to miss on a first pass, because it's not part of the spec at all — it's purely an implementation detail that can still become a real, unbounded memory leak. This doesn't bode confidence that points are well considered.

The notes are AI-generated, but the numbers chosen in the likewise AI-assisted prototype implementations were not.

[supercooltest2024]

Yeah, my proposed bucket sizes are probably too big. They would only not be wasteful if they eliminated enough common file downloads across sites. Definitely a balancing act between not having enough files and using more space than is saved. There probably isn't 100GB of common files anyway. Would need experimenting. But if a device is so low-storage that it cannot have a 1GB bucket of common web files, it probably can't handle much caching anyway. Bucket-wise, I think such a device should store no common files. If such a device were using a small subset of entries from PHL instead, I think that small subset would greater reflect sites that user commonly uses (more identifiable). So either way, I don't think low-storage devices should cache files cross origin.

For Large files over fetching fix, I agree Letting apps specify content to be shown in browser UI like a permission prompt is generally deemed a no-go on the Web. So my solution should not use named files, so instead the prompt should only mention the size and what sites have it cached in shared cache.

Previous experiment results from where the HTTP cache was about to be isolated showed cache hit rates were rather unaffected from cache isolation I am not familiar with the experiment. But I suspect the experiment was to determine the average performance impact, not the privacy impact of insulating the cache (didn't reflect outlier tracking scripts that probed caches from many pages). I am not sure how the experiment relates to the chosen number of burst api requests. Certainly sharded models could be in 2000 pieces, but I don't think that means 2000 requests to arbitrary PHL shared resources should be allowed in burst. The requests should be required to be correlated somehow, ideally in a well defined way. Perhaps requiring the resources to have the same set of origins. So something like, a request to PHL entry with a set of origins this page requested before would take 100 tokens, otherwise only 1 token. Then only 20 arbitrary requests to PHL can be bursted but 2000 could be bursted from correlated PHL entries. Individual AI models / game engine assets would likely have the same set of origins, so their use case would be fulfilled.

If every entry on the PHL was on 99.9% of sites without correlated origins requirement then using shannon entropy a user can be identified among 2**(-(.001log_2(.001)+.999log_2(.999))*2000)= 7381668 other users given 2000 requests. But with the correlation requirement, this drops significantly. 20 uncorrelated queries where each file is in only 90% of sites only identifies 1 person among 666. $$H(X_i) = -\left( 0.9 \log_2(0.9) + 0.1 \log_2(0.1) \right) \approx 0.469 \text{ bits}$$ $$H(X) = \sum_{i=1}^{20} H(X_i) = 20 \times 0.469 \approx \mathbf{9.38 \text{ bits}}$$. 2**9.38=666.

Even with the above correlation constraint to PHL, I don't think other browser vendors will adopt this api. Native File System Access API wasn't even adopted, and it requires clear permission prompts before allowing cross origin access to files (highly regrettable, as it is very useful. could even replace use cases of this api if it didn't require a file picker). I think investigating other approaches could enable wider adoption enabling use cases across browsers such as are mentioned in the readme (flutter applications, game engines, large assets, AI models, etc).

[tomayac]

Yeah, my proposed bucket sizes are probably too big. They would only not be wasteful if they eliminated enough common file downloads across sites. Definitely a balancing act between not having enough files and using more space than is saved. There probably isn't 100GB of common files anyway. Would need experimenting. But if a device is so low-storage that it cannot have a 1GB bucket of common web files, it probably can't handle much caching anyway. Bucket-wise, I think such a device should store no common files. If such a device were using a small subset of entries from PHL instead, I think that small subset would greater reflect sites that user commonly uses (more identifiable). So either way, I don't think low-storage devices should cache files cross origin.

Anything that makes a device stand out, in the extreme case as a "poor person's phone" for example, is to be avoided.

For Large files over fetching fix, I agree Letting apps specify content to be shown in browser UI like a permission prompt is generally deemed a no-go on the Web. So my solution should not use named files, so instead the prompt should only mention the size and what sites have it cached in shared cache.

Again, all partners we circled the idea with were strictly against prompts, so the problem doesn't immediately raise. However, it could be useful for debugging purposes to show the size and the origins using a resource. This was added to the spec earlier.

Previous experiment results from where the HTTP cache was about to be isolated showed cache hit rates were rather unaffected from cache isolation I am not familiar with the experiment. But I suspect the experiment was to determine the average performance impact, not the privacy impact of insulating the cache (didn't reflect outlier tracking scripts that probed caches from many pages). I am not sure how the experiment relates to the chosen number of burst api requests. Certainly sharded models could be in 2000 pieces, but I don't think that means 2000 requests to arbitrary PHL shared resources should be allowed in burst. The requests should be required to be correlated somehow, ideally in a well defined way. Perhaps requiring the resources to have the same set of origins. So something like, a request to PHL entry with a set of origins this page requested before would take 100 tokens, otherwise only 1 token. Then only 20 arbitrary requests to PHL can be bursted but 2000 could be bursted from correlated PHL entries. Individual AI models / game engine assets would likely have the same set of origins, so their use case would be fulfilled.

See the HTTP cache partitioning explainer.

If every entry on the PHL was on 99.9% of sites without correlated origins requirement then using shannon entropy a user can be identified among 2**(-(.001log_2(.001)+.999log_2(.999))*2000)= 7381668 other users given 2000 requests. But with the correlation requirement, this drops significantly. 20 uncorrelated queries where each file is in only 90% of sites only identifies 1 person among 666. $$H(X_i) = -\left( 0.9 \log_2(0.9) + 0.1 \log_2(0.1) \right) \approx 0.469 \text{ bits}$$ $$H(X) = \sum_{i=1}^{20} H(X_i) = 20 \times 0.469 \approx \mathbf{9.38 \text{ bits}}$$. 2**9.38=666.

There is no way for sites to enumerate the resources a site has stored in COS. You'd very quickly hit the probing limit, plus the results could always be GREASE'ed.

Even with the above correlation constraint to PHL, I don't think other browser vendors will adopt this api. Native File System Access API wasn't even adopted, and it requires clear permission prompts before allowing cross origin access to files (highly regrettable, as it is very useful. could even replace use cases of this api if it didn't require a file picker). I think investigating other approaches could enable wider adoption enabling use cases across browsers such as are mentioned in the readme (flutter applications, game engines, large assets, AI models, etc).

Standards position requests for the other vendors and also the W3C TAG were filed, so we can let the vendors speak for themselves.

[supercooltest2024]

Anything that makes a device stand out, in the extreme case as a "poor person's phone" for example, is to be avoided.

What I mean is that there should be a buckets for multiple size clases. If a device wants to save space, then an available bucket could be of zero size. The bucket sizes could be 0, 500MB, 2GB, and 10GB or whatever makes sense. My point is that, if a user regularly uses most files in a bucket, then it is not a "waste" to download them. There are reasons to save space even if you aren't on a "poor person's phone" (any device can run low on space). 4 buckets is bounded at revealing 2 bits of information, which is highly correlated with existing observable information to attackers (approximate storage space). This is strictly better than practically unbounded possibilities of a user's current PHL subset.

There is no way for sites to enumerate the resources a site has stored in COS

This is not necessary for the attack in question. As I understand the 'probing limit' is 2000 tokens in a burst. As the PHL is public, an attacker would be able to construct a list of 2000 entries from the list which have the smallest amount of correlation. Then an attacker just needs to query each of those 2000 hashes using their 2000 tokens to see which ones are cached. The math might not be perfect, as it is assuming the entries to be uncorrelated, but it indicates that even if only practically ubiquitous files (99.9% of sites) are on the PHL it still is highly identifiable if an attacker can make 2000 queries.

You'd very quickly hit the probing limit

I would like to know how the probing limit would quickly be reached, if it allows 2000 binary queries.

plus the results could always be GREASE'ed

This just makes it a bit harder to get a stable signal, but by selecting the nearest user the randomness of the GREASE can be mitigated in an attack. (and, some entries are never GREASE'ed by design, so they would always be reliable signals)

See the HTTP cache partitioning explainer.

I am aware that caches were changed to be partitioned for privacy reasons. But it does look like the given link is about the effect that partitioning has on performance (as I suspected). So I am still not sure how it relates to the choice of 2000 tokens.

Again, all partners we circled the idea with were strictly against prompts

I was merely explaining how my proposal could be adjusted to no longer have parts of a prompt be attacker controllable. I am aware that my proposal uses prompts, and partners don't like that.

we can let the vendors speak for themselves.

Of course. I just think they have finite resources, and would thus have an easier time reviewing new apis if the apis were already secure, private, and well defined. At least, that is the impression I got from reading the thread on Mozilla's position on Native File System Access, and the quote from a Mozilla engineer (which was specifically about privacy concerns with only downloading large AI models once).

You didn't comment on my suggested privacy improvement of the COS feature as it exists. Currently it appears each COS query takes 1 token. A simple enhancement is to make queries take 100 tokens if the query identifies a PHL entry whose current used origins set is new to the querying site. That is, if there are 30GB AI models on example-abc.com, example-abcd.com, and example-cd.com: if example.com requests model 'a' then model 'b', then 'b' shouldn't have a penalty; but then requesting model 'c' or 'd' would have the penalty. The goal being that then at most 20 different sets of origins would be queryable by an attacker.

[tomayac]

Thanks for continuing to dig into this, a few responses below.

What I mean is that there should be a buckets for multiple size clases. If a device wants to save space, then an available bucket could be of zero size. [...] 4 buckets is bounded at revealing 2 bits of information, which is highly correlated with existing observable information to attackers (approximate storage space). This is strictly better than practically unbounded possibilities of a user's current PHL subset.

That's a cleaner framing of the bucket idea than the earlier one, and the "2 bits vs. a user's practically unbounded PHL subset" comparison is a fair point about entropy. I'd note the comparison isn't quite apples to apples, though: a fixed, publicly known bucket set reveals a device class, which is stable and shared by millions of devices, whereas a user's actual PHL subset is shaped by their own browsing history, which is a different (and arguably more sensitive) kind of signal even before you get to the bit-count.

This is not necessary for the attack in question. As I understand the 'probing limit' is 2000 tokens in a burst. As the PHL is public, an attacker would be able to construct a list of 2000 entries from the list which have the smallest amount of correlation. [...] it still is highly identifiable if an attacker can make 2000 queries.

With the size of the PHL, choosing 2,000 (random or ordered by popularity) examples and hoping for COS hits still seems very unlikely to reveal meaningful information in practice. The real limit is one thing we'll figure out once the API is usable in the wild. On top you always have GREASE'ing, so you never quite know how much you can trust your probes.

I would like to know how the probing limit would quickly be reached, if it allows 2000 binary queries.

2000 is a burst ceiling sized for legitimate sharded-model loading, not a small number chosen to make probing impractical, so I shouldn't have implied it was a meaningful deterrent on its own against exactly the kind of scripted, upfront enumeration you're describing.

This just makes it a bit harder to get a stable signal, but by selecting the nearest user the randomness of the GREASE can be mitigated in an attack. (and, some entries are never GREASE'ed by design, so they would always be reliable signals)

Agreed that GREASE'ing raises the cost of a clean signal rather than eliminating it, and that an attacker with repeat access (multiple sessions, or entries exempted from GREASE'ing [though those would have a significantly lower probability of cache hits]) can average it out. That's consistent with how GREASE'ing is framed elsewhere: as one layer among several, not a standalone guarantee.

I am aware that caches were changed to be partitioned for privacy reasons. But it does look like the given link is about the effect that partitioning has on performance (as I suspected). So I am still not sure how it relates to the choice of 2000 tokens.

You're right that the link supports the performance side (cache hit rates held up under partitioning) but doesn't itself justify the specific token number from a privacy-budget perspective. For probing to be meaningful, you need COS hits. The point of mentioning these prior results was to show that after isolating the caches, the negative impact was surprisingly low. The 2,000 figure is presently sized for legitimate large-model bursts, not derived from a privacy analysis like the one you're doing here.

I was merely explaining how my proposal could be adjusted to no longer have parts of a prompt be attacker controllable. I am aware that my proposal uses prompts, and partners don't like that.

Understood, thanks for clarifying.

Of course. I just think they have finite resources, and would thus have an easier time reviewing new apis if the apis were already secure, private, and well defined.

That's a reasonable expectation, and it's part of why this back-and-forth is useful before this goes further through review.

You didn't comment on my suggested privacy improvement of the COS feature as it exists. Currently it appears each COS query takes 1 token. A simple enhancement is to make queries take 100 tokens if the query identifies a PHL entry whose current used origins set is new to the querying site. [...] The goal being that then at most 20 different sets of origins would be queryable by an attacker.

Correlating token cost to whether a queried entry's origin-set is "new" to the requesting site is a more targeted mitigation than a flat per-query cost, and it's a reasonable direction for bounding exactly the enumeration attack you describe above. It would, however, penalize the legitimate sharded-model case, as upon a first visit to a new AI app, you'd have to download the model (that you might already have), which is exactly a reason for this API to exist.

[supercooltest2024]

With the size of the PHL, choosing 2,000 (random or ordered by popularity) examples and hoping for COS hits still seems very unlikely to reveal meaningful information in practice. The real limit is one thing we'll figure out once the API is usable in the wild. On top you always have GREASE'ing, so you never quite know how much you can trust your probes.

The amount of information to gain is dependant on the quality of the PHL subset the the attacker chooses to test. The quality of entries is how well each subsequent entry divides remaining users (that is, the closer the probability of an entry being cached is to 50%, the better). That is, a 10% cached chance is better than 95% cached chance for an attacker. GREASE'ed entries can only lie in one direction (saying an entry isn't cached when it is) and large uncommon files aren't ever GREASE'ed. A decision tree could be used for entries that are never greased to determine which PHL entries an attacker should query next to maximize how well their queries divide remaining users, but such a tree would break when a user's PHL subset grows (would be a temporary fingerprint). For choosing the nearest user, all entries that were marked cached can be trusted, only 'uncached' entries need to be considered for the nearest user check (this conveniently also tracks users as their PHL subset grows over time). Even if all of the above somehow doesn't reveal meaningful information in practice, detecting incognito mode seems as simple as checking a ubiquitous set of files (where at least one is usually cached) which aren't preseeded. If all ubiquitous files on the PHL were preseeded, it would be approaching the buckets strategy.

Correlating token cost to whether a queried entry's origin-set is "new" to the requesting site is a more targeted mitigation than a flat per-query cost, and it's a reasonable direction for bounding exactly the enumeration attack you describe above. It would, however, penalize the legitimate sharded-model case, as upon a first visit to a new AI app, you'd have to download the model (that you might already have), which is exactly a reason for this API to exist.

The intention was to enable a first visit to a new AI app to not need to redownload a model. The first origin-set requested isn't intended to be penalized. The idea is that all shards of a sharded model would be from the same origin-set. Though, this expectation would be foiled by other sites using only portions of the model. In the case that no site has cached all parts of a sharded model, which parts of the model are cached could potentially identify which sites the user use (that use different portions of the ai model). I guess ideally, if an entire model is cached then there would be no penalties for requesting any subset of it.. Maybe a better approach would be for the PHL to indicate entries that can be grouped, and having every entry in a group cached would remove penalties for entries in that group.

To summarize, I don't think this API can be privacy preserving as it is. Unfortunately, the privacy risks are the worst for users who would like to use this api for large files (the main use case), as these files are uncommon, thus having them provides a lot of information. Retrofitting penalties doesn't remove all concerns either (an attacker could avoid penalties by requesting many entries from the same origin-set but which are only requested on certain kinds of pages on those origins). I still think buckets for common files and a different api for large uncommon files is needed.

[tomayac]

The amount of information to gain is dependant on the quality of the PHL subset the the attacker chooses to test. The quality of entries is how well each subsequent entry divides remaining users (that is, the closer the probability of an entry being cached is to 50%, the better). That is, a 10% cached chance is better than 95% cached chance for an attacker. GREASE'ed entries can only lie in one direction (saying an entry isn't cached when it is) and large uncommon files aren't ever GREASE'ed. A decision tree could be used for entries that are never greased to determine which PHL entries an attacker should query next to maximize how well their queries divide remaining users, but such a tree would break when a user's PHL subset grows (would be a temporary fingerprint). For choosing the nearest user, all entries that were marked cached can be trusted, only 'uncached' entries need to be considered for the nearest user check (this conveniently also tracks users as their PHL subset grows over time). Even if all of the above somehow doesn't reveal meaningful information in practice, detecting incognito mode seems as simple as checking a ubiquitous set of files (where at least one is usually cached) which aren't preseeded. If all ubiquitous files on the PHL were preseeded, it would be approaching the buckets strategy.

I agree that your attack approach looks plausible and practical in its attempt to improve the likelihood of success, but the argument still holds that a burst probing limit will stop it, paired with other signals like the origin's reputation the attack came from, etc.

Regarding incognito mode, each vendor could pre-seed its COS cache in incognito mode with a slightly randomized variant of the, based on telemetry, most popular resources globally across the fleet. But it's a valid concern, and COS certainly shouldn't become an incognito oracle.

The intention was to enable a first visit to a new AI app to not need to redownload a model. The first origin-set requested isn't intended to be penalized. The idea is that all shards of a sharded model would be from the same origin-set. Though, this expectation would be foiled by other sites using only portions of the model. In the case that no site has cached all parts of a sharded model, which parts of the model are cached could potentially identify which sites the user use (that use different portions of the ai model).

In practice, WebLLM shards their models. An app can only work once all shards are downloaded. There's no way an app can work with just parts of the model.

I guess ideally, if an entire model is cached then there would be no penalties for requesting any subset of it.. Maybe a better approach would be for the PHL to indicate entries that can be grouped, and having every entry in a group cached would remove penalties for entries in that group.

It gets complex really quickly, and I'd rather avoid introducing "can be grouped" as a concept. For example, React and React-DOM could be grouped, too. User agents can still exploit their knowledge about hashes, for example, in its COS settings UI or in DevTools, a browser could say that a resource with the hash abc…123 is part of an LLM with the name "Some LLM".

To summarize, I don't think this API can be privacy preserving as it is. Unfortunately, the privacy risks are the worst for users who would like to use this api for large files (the main use case), as these files are uncommon, thus having them provides a lot of information. Retrofitting penalties doesn't remove all concerns either (an attacker could avoid penalties by requesting many entries from the same origin-set but which are only requested on certain kinds of pages on those origins). I still think buckets for common files and a different api for large uncommon files is needed.

We worked with both the AI frameworks and the meta framework & bundler communities, as well as with major Wasm users like Emscripten and Flutter, and also font directories like Google Fonts (see the section Web / Framework developer views notes on ChromeStatus). Their use cases were all met by the existing API. The AI models in the PHL were strictly chosen by popularity as reported by Hugging Face, with the objective of including general purpose LLMs or very specific task-based models like background removal models, but certainly not medical expert models. It is still early days, and we will certainly make tweaks for the PHL and some limits, like the probing limit.

[tomayac]

Coming back to this with a concrete result rather than another round of argument.

Your central objection was right, and I want to say so plainly. You pointed out that the 2,000-token burst limit was not derived from any privacy analysis, that 2,000 binary queries is more than enough to identify a user, and that a limit sized for legitimate sharded-model loading is not a limit that constrains an attacker. I conceded the first part here but did not have an answer to put in its place. That gap is what #73 exists to close.

The change replaces rate limiting as the privacy mechanism with a budget derived from the quantity that actually matters. Each distinct hash a site can resolve cross-origin is worth at most one bit, so the width of any identifier built this way is exactly the number of probes granted. That makes the limit an entropy bound rather than a throughput bound, which is the framing your Shannon-entropy calculation was pressing for. It is counted in distinct hashes rather than calls, charged whether or not the file is found, and keyed by site rather than origin.

Your framing also changed how the documents describe the problem. They previously ran together two threats: an attacker inferring something from resources it did not choose, and an attacker writing the resources itself and using their presence purely as storage for a value it minted. The Public Hash List and GREASE'ing address the first and do nothing about the second, because a k-anonymity bar constrains an attacker that can only observe state, not one that can set it. That is close to the point you were making about GREASE'ing only being able to lie in one direction, and it is now stated explicitly instead of being implied.

I am also removing the concrete token-bucket numbers from the implementation notes, since they predate all of this and were being read, reasonably, as the design.

Two things I do not want to overstate. The budget's actual value is unsettled and there is no cross-vendor agreement on how much entropy is tolerable here, so this is a mechanism rather than a finished answer. And several of your other points are untouched by that PR: the bucket alternative, prompts for large uncommon files, and incognito detection. #73 says Refs #72 rather than Closes #72 for exactly that reason, so this issue stays open on its own terms.

Thanks for the persistence here. The entropy argument in particular did more to move the design than anything else in the thread.