ByteSize — Evidence-Aware Conversational Search with Value-of-Information Stopping

TikTok TechJam 2026 result: this entry did not place. The competition has concluded and this submission was not among those selected. Everything below is what was built and measured — each number traces to a committed artifact, and none of it was ever a placement claim.

TechJam Track 4: Conversational E-Commerce Search Challenge.

A multi-turn shopping agent that finds a customer's hidden target product inside a frozen 50,000-item Amazon catalog. The agent decides, every turn, whether it has enough evidence to recommend or should spend one more turn asking. It runs fully offline at 0 tokens and $0 per session.

Headline result: on 800 unseen, ID-disjoint sessions the stopping controller produces +60 rank-1 recommendations with zero Hit-rate loss, improving on 8 of 8 shards.

Set	n	Hit@10	MRR	MTTC	TechnicalScore	Rank-1
Public (labelled)	200	1.000	0.954167	2.75	0.95125	184
Our ID-disjoint holdout	200	0.980	0.864845	2.885	0.911753	162
Random 800, 8×100 shards	800	0.97375	0.888018	2.8975	0.91533	672
Weak BM25 starter (reference)	200	0.125	0.068034	9.81	—	—

TechnicalScore = 0.50 × Hit@10 + 0.30 × MRR + 0.20 × Efficiency, where Efficiency = clip((11 - MTTC) / 10, 0, 1).

Each row comes from exactly one committed file:

Row	Source of every number in it
Public (labelled)	results_contest_public.json
Our ID-disjoint holdout	holdout/ours_holdout.json
Random 800, 8×100 shards	holdout/shards/a_vs_e123.json → pooled_mean_of_shards.e123
Weak BM25 starter (reference)	docs/baseline_results.json

tests/test_reported_claims.py reparses this README together with that source table and checks every cell against the declared file: the Public and holdout aggregates are recomputed from their 200 per-session records through evaluator.local_evaluator.metric_summary, the Random-800 row from its eight per-shard records. Rank-1 exists only in the per-session records and is always recounted. Changing a digit in the score table, pointing a row at a different file, or letting a result file drift turns the CI badge red.

Two older artifacts in holdout/ measure the same 800 sessions under an earlier configuration and are superseded: holdout/shards/out_0..7.json and holdout/random800_compare.json pool to Hit 0.9725 / MRR 0.824885 / MTTC 2.69125 / TechnicalScore 0.89989, recorded before the E123 freeze commit 3a31ace. They are kept as the research trail, not as the reported result; each now carries a snapshot field saying so.

Every number above was produced with usage = 0 prompt tokens, 0 completion tokens. The public-set score is not a prediction for the organizer's private 800 sessions; that is why we built our own ID-disjoint holdout and an 8-shard robustness study.

Quick Start

Python 3.10 or later. Developed and measured on Python 3.11.5, Windows 10.

Start here without the catalog, API keys, models or third-party packages:

# Replay an Agent dialogue using two original synthetic products.
# This demonstrates the runtime; it does not reproduce the 50,000-item score.
python -S demo/fixture.py

# Replay a real labelled session end to end, on the committed catalog slice.
python -S demo/run_demo.py --session public_0002   # Intent Override
python -S demo/run_demo.py --scenario browsing     # public_0007

# Full test suite: 182 tests; the staged-ZIP check skips until a ZIP is built.
python -S -m unittest discover -s tests -v

demo/run_demo.py uses data/catalog.jsonl when that file is present and otherwise falls back to data/catalog.mini.jsonl (1,631 rows, 2.2 MB, committed), naming on its first line the file it actually read. The slice keeps the complete shelf of the four sessions the script defaults to — public_0001, public_0002, public_0007, public_0035 — and each of those shelves is larger than min_candidates, so the candidate pool is never padded from the rest of the catalog and the four replay turn for turn as they do on the full 50,000 items. Any other session also runs and the script says it is not pool-equivalent; a score computed on the slice means nothing. scripts/make_mini_catalog.py rebuilds the slice from the frozen catalog.

Full public-set scoring additionally requires the organizer's frozen 50,000-item data/catalog.jsonl. Obtain the original competition file from the organizer or your existing participant kit; this repository does not redistribute that file. The upstream data source is Amazon Reviews 2023, but downloading a different source export will not recreate the frozen catalog. The import command checks its exact SHA256, 50,000 unique product IDs and all 200 public-session targets before copying it. See data/README.md and DATA_ATTRIBUTION.md for the checksum and source boundaries.

# Accepts the original .jsonl or .jsonl.gz; use --check-only to avoid copying.
python -S scripts/import_catalog.py "/path/to/original/catalog.jsonl.gz"

# Run the official harness using the standard-library fallback.
# The scored entry point is starter.agent.Agent.
python -S -m evaluator.local_evaluator --catalog data/catalog.jsonl \
    --dataset data/public_set.jsonl --output results.json

# With the frozen catalog in place, the same replay runs on all 50,000 items.
python -S demo/run_demo.py --session public_0002   # Intent Override

The fixture tests need no environment variables and run without model packages. For full-catalog runs, the scored path can use a local MiniLM tie-breaker; without its dependencies/weights the standard-library fallback runs, but is not score-equivalent to the frozen results above. See models/README.md. Set TECHJAM_DENSE_OFFLINE=1 to prevent optional model downloads when using an environment with model packages.

Verified again on 2026-09-12 with Python's -S standard-library path and the original catalog: Public 200 Hit@10 1.000, MRR 0.961131, MTTC 2.75, TechnicalScore 0.953339, reported tokens 0. These are the fallback results, separately from the historical MiniLM results in the opening table.

Additional evaluation entry points: eval_contest.py --only public scores the public 200 and writes results_contest_public.json; eval_holdout.py scores our own holdout 200; eval_shard.py scores one shard of the random 800.

How It Works

The scored path is starter.agent.Agent → ContestAgent + the PUBLIC config. The central idea is not "conjunction filter plus popularity" — it is keep asking while evidence is insufficient, then rank with popularity-first late fusion.

User turn
 ↓
[1] Dialogue state            slots, scoped intent override, scenario
 ↓
[2] Exact-evidence AND        category lock + verbatim conjunction (empty filter is skipped)
 ↓
[3] Value-of-Information controller
      ├ evidence insufficient → ask one more `other`
      └ evidence sufficient   → recommend
 ↓
[4] Popularity-first late fusion
      popularity 1.0 + exact field line 0.35 + distinctive title phrase 0.15 + MiniLM 0.1
 ↓
[5] Optional listwise LLM rerank (shortlist ≤ 10, disabled by default)

The key insight is that a small candidate pool does not mean the agent has enough information. Slot-disclosure trajectories differ by scenario — Buying tends to go 1 → +2 → +1 → exhausted, Browsing 0 → +2 → +2 → exhausted. The controller decides using scenario, number of disclosed slots, whether "no additional preference" was already received, and pool size. It never inspects the remaining turn budget and never changes rank().

MiniLM is applied as late fusion only: the conjunction produces a hard pool first, then score += 0.1 × min-max(cosine). It never replaces popularity and never participates in recall.

Detailed design notes live in report/: architecture.md (data flow), attribution.md (per-component contribution), robustness.md (8-shard study), methods.md (accepted and rejected ablations), freeze.md (frozen configuration).

Model Choice, Cost, Tokens, Latency

Component	Default	Cost	Behaviour without network
Category lock + verbatim AND + popularity	Standard-library in-memory index	$0	Works
MiniLM sentence encoder	Bundled sidecar, then HF cache, then pinned Hub revision	$0	Works from the bundled sidecar
Listwise LLM rerank	Disabled	Per token if enabled	Falls back to the 0-token ranking

Reported token usage across the 200-session public evaluation: 0 prompt tokens, 0 completion tokens, $0. The scored configuration sets llm_listwise=False and calls no LLM.

Latency, measured locally on Windows with the 50,000-item catalog: index build about 8 s, then about 35 s for 200 sessions, roughly 0.2 s per session including MiniLM.

Network: the submission requires no network access. Official final scoring may disable it; our scored path is unaffected because the encoder ships with the bundle.

MiniLM is a scoring dependency, not an optional extra

The frozen configuration's numbers use sentence-transformers/all-MiniLM-L6-v2 at revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf (Apache-2.0), with model.safetensors sha256 53aa51172d142c89d9012cce15ae4d6cc0ca6895895114379cacb4fab128d9db.

Load order: TECHJAM_DENSE_HOME → models/all-MiniLM-L6-v2 (the bundled sidecar) → Hugging Face cache → the same pinned revision from the Hub when network is allowed. Rebuild the sidecar with python scripts/vendor_minilm.py.

If the weights are absent the agent still runs, but it is not score-equivalent:

Set	With MiniLM	Without MiniLM
Public 200	Hit 1.000 / 0.95125	Hit 1.000 / 0.953339
Holdout 200	Hit 0.980 / 0.911753	Hit 0.975 / 0.909529

The Value-of-Information stopping behaviour does not depend on MiniLM; public MTTC stays at 2.75 either way. Force the offline path with TECHJAM_DENSE_OFFLINE=1.

The loader and submission packer now share a code-bound, official-revision manifest for the six MiniLM load files. Invalid local assets are rejected before model import and reported as nodense diagnostics; that fallback is not score equivalence. Packing validates copied assets and staged imports before atomically replacing a ZIP, preserving the old ZIP on failure. These offline integrity checks do not remeasure the historical scores. See manifest and boundaries.

Optional tiered model backend (off by default)

An optional, explicitly tiered model path exists for experimentation and stays usable offline: DeepSeek API → local OpenAI-compatible endpoint → deterministic fallback. It is enabled only by environment variables, never by default. If a configured backend times out or returns an HTTP, JSON, schema or validation error, the next tier is attempted, and the semantic ranker repairs invalid, duplicate, unknown or omitted IDs using the original deterministic candidate order.

SHOPPING_AGENT_DEEPSEEK_API_KEY       # enables the DeepSeek tier
SHOPPING_AGENT_DEEPSEEK_BASE_URL      # default: https://api.deepseek.com
SHOPPING_AGENT_DEEPSEEK_MODEL         # default: deepseek-v4-flash
SHOPPING_AGENT_LOCAL_BASE_URL         # enables the local tier together with LOCAL_MODEL
SHOPPING_AGENT_LOCAL_MODEL            # model name sent to the local endpoint
SHOPPING_AGENT_LOCAL_API_KEY          # optional Authorization token for local servers
SHOPPING_AGENT_MODEL_TIMEOUT_SECONDS  # per-request hard timeout; default 8
SHOPPING_AGENT_MODEL_CANDIDATE_LIMIT  # semantic-ranker candidate cap; default 30
SHOPPING_AGENT_RETRIEVAL_LIMIT        # retrieval budget; default 100
SHOPPING_AGENT_MODEL_MAX_TOKENS       # completion cap; default 512
SHOPPING_AGENT_MODEL_TEMPERATURE      # default 0

Teams manage their own credentials; no API key is committed to this repository.

Agent Interface

class Agent:
    def reset(self, session_id: str, user_profile: dict) -> None:
        ...

    def respond(self, session_id: str, user_message: str, turn: int, top_k: int) -> dict:
        return {
            "message": "Do you have a material preference?",
            "ask_attribute": "material",
            "recommendations": [{"parent_asin": "B000..."}, {"parent_asin": "B001..."}],
            "usage": {"prompt_tokens": 120, "completion_tokens": 30},
        }

ask_attribute is one of category, material, color, size, style, brand, budget, feature, use_case, other, or null. See docs/agent_api_contract.json. Only exact parent_asin equality counts as a hit, and only the first 10 valid unique IDs are scored.

Repository Layout

starter/agent.py                     Agent = ContestAgent PUBLIC (scored); LegacyAgent = earlier pipeline
starter/shopping_agent/contest_*.py  the scored implementation
starter/shopping_agent/*.py          supporting modules (dialogue state, catalog, ranking, response guard)
evaluator/local_evaluator.py         official public-set simulator and scorer, unmodified
eval_contest.py                      scores the public 200 → results_contest_public.json
eval_holdout.py                      scores our own ID-disjoint holdout 200
eval_shard.py                        scores one shard of the random 800
demo/run_demo.py                     replays one multi-turn session
demo/fixture.py                      original synthetic dialogue, no data or dependencies
models/                              pinned MiniLM sidecar and its README
report/                              architecture, ablations, robustness, freeze notes
holdout/                             our own test sets and comparison JSON (not the organizer's private 800)
scripts/pack_submission.py           builds the clean Devpost ZIP
scripts/import_catalog.py            verifies and imports an existing frozen catalog
data/public_set.jsonl                200 labelled development sessions
docs/                                competition specification, API contract, scoring config

Branches

Apart from pull-request branches, this repository has two branches, and main is the one that was scored.

Branch	starter.agent.Agent	Purpose
main	ContestAgent PUBLIC	The scored implementation. Every number in this README was produced by this code
v2/shelf-recall	ContestAgent PUBLIC + a shelf-recall experiment	Later experiment, kept for the record; no reported number comes from it

Four documents were written at submission time and still name the branch contest/public, a teammate branch legacy/qwen, and an earlier repository name: report/freeze.md, report/submit.md, SUBMISSION_CHECKLIST.md and report/devpost_description.md. Neither of those branches is published here. Those files are left as they were written, each with a note at the top, because they are the record of what was submitted rather than a description of the repository today.

The scored configuration is frozen. report/freeze.md records the algorithm-freeze commit and the gates that would be required to reopen it.

The Challenge

Build an AI shopping agent that asks useful follow-up questions and recommends the customer's hidden target product within at most 10 turns.

For each session the agent receives an anonymized preference profile and a short customer message. Raw user IDs, review text, timestamps and purchase history are never disclosed. On every turn the agent may ask a natural clarification question in message while naming one requested field in ask_attribute, return a ranked list of up to 10 catalog parent_asin values, or do both. A session ends when the target appears in the scored Top 10, or after turn 10. Sessions cover Buying, Browsing, Intent Override and Boundary behaviour.

Metrics: Hit Rate@10 is the fraction of sessions that find the target within 10 turns; MRR is the mean reciprocal rank, with a miss contributing zero; MTTC is the mean first-hit turn, with a miss assigned turn 11. Reported token usage is a feasibility metric, not part of the core technical score.

Participant rules are in docs/submission_rules.md and docs/competition_specification.md.

Data Source

The catalog and sessions are derived from Amazon Reviews 2023 by McAuley Lab, UCSD. Sessions are sampled deterministically from the official Clothing 5-core leave-last-out split and joined to the frozen catalog. See DATA_ATTRIBUTION.md before using or redistributing the data.