Audio description for content that has none —
generated, spoken on your Fire TV, and answerable mid-scene.
Build, Ship, Shape: Amazon Developer Hackathon 2026 · Fire TV (Vega OS) track
▶ Demo video (3 min) · ▶ Open Source QA video (1:20) · Live demo · Watch it work · Try it yourself · vs a phone app · Architecture · Evidence · AWS · Tests · Where AI is used · Why it works this way · Limitations
Watch the three-minute demo · Watch the Open Source contribution video (audio-description-qa)
The quickest way to check this is real rather than described:
| to see | open |
|---|---|
| the whole thing in three minutes | the demo video — the app on the device, audio description + sign-language voicing, and a question answered live |
| that it works | the live demo — press play and listen |
| that it runs on the device | app/ — a real Vega OS app; probes/ shows what the platform would and would not do |
| that the numbers are real | Evidence — two of them reproduce with the commands given |
| that the design is not guesswork | Why it works this way — two blind reviewers, quoted directly, including the seven faults one of them found |
| that it works on a real story | three minutes of the film — the trailer withholds its story by design; this does not |
| that it is not one hand-tuned clip | four different videos described by the same pipeline — a film, a 1951 instructional film, an ASL sign-language video, and the trailer |
| where it does NOT work | docs/where-it-works.md — measured. An advert produces zero descriptions |
| what the platform cost us | FRICTION-LOG.md — 15 findings from building on Vega, written while building |
| everything we would ask Amazon to fix | FEEDBACK.md — 21 requests with priorities, plus what worked and what did not, per tool |
| what we learned about Vega | FRICTION-LOG.md (15 entries) and VEGA-FIELD-NOTES.md |
| what is not finished | STATUS.md — kept honest, including what no blind user has tested yet |
| the Open Source entry | audio-description-qa (▶ 80s explainer video) — the blind-tester QA harness, released separately under MIT so anyone writing description can use it |
Fastest check of all: tushartechs.github.io/sightline — press play and listen. Nothing to install.
| criterion | where to look |
|---|---|
| Technological implementation | Architecture and AWS integration. Speech detection rather than silence detection, 2.5 usable seconds becoming 39.8. Every line measured against its gap before it is kept. A second audio stream on the device so description ducks the film rather than fighting it |
| Design | Try it without a Fire TV. Every control confirms itself aloud, because the user cannot see the HUD. VoiceView could not be enabled on the virtual device (FL-011), so the app speaks for itself |
| Potential impact | How we knew what to build. Two blind reviewers, quoted directly. The co-viewing mode exists for one room: a blind viewer and a sighted viewer who want different things from the same screen |
| Quality of the idea | What this does that a phone app pointed at a screen does not, and Limitations. It describes what was never going to be described, and it says where it stops |
| Friction log bonus | FRICTION-LOG.md, 15 entries, each with task, steps, expected against actual, severity, workaround and suggestion. Summarised with priorities in FEEDBACK.md |
| AWS Builder mini challenge | AWS integration. Transcribe, Polly, S3, Lambda and API Gateway, with measurements. Bedrock is included as a documented failure rather than omitted |
| Open Source mini challenge | audio-description-qa (▶ video), MIT, created during the hackathon window |
Four of the six, and not by stretching. The two it does not touch are named rather than skipped.
| category | how |
|---|---|
| AI-enhanced viewing | This is the whole project. Description that does not exist is written and spoken while you watch, for content no human describer was ever paid to cover. |
| Multi-modal UX | Voice out (a synthesised description track and spoken answers), voice in (hold to ask a question), D-pad for every control, on-screen captions for the room, and a second screen — the description can go to one person's phone while the television plays untouched. |
| Computer vision | Description is generated by diffing frame pairs across each gap and reasoning about what changed, not from metadata or subtitles. The same frame analysis answers questions about the moment you are on. |
| Family entertainment | The co-viewing mode exists for a specific room: a blind viewer and a sighted viewer watching the same screen, who want different things from it. Nothing else we found lets both have what they want at once. |
| Not targeted. Live content has no gaps to write into and no time to generate. | |
| Not targeted. |
Apps exist that will describe an image or a video on a phone. This is a different thing in four ways, and the combination is the point:
- It runs on the television, not on a second device held up to one. The description comes out of the same speakers as the film, on a Fire Stick.
- It fits the gaps. Description is placed where nobody is speaking and measured to fit before it is spoken, so it never talks over the dialogue. That requires knowing where the dialogue is, which is why Transcribe is in the pipeline rather than a silence detector.
- It survives a speed change. Ranking is fixed once, host-side. At 2x you hear less of the same story in the same order, never a different story — a rule a blind reviewer gave us and the single hardest constraint in here.
- It can be interrupted. A question about the frame on screen right now is answered from that frame. No pre-rendered description track can do this, because the question does not exist when the track is made.
And one thing it does that nothing else seems to: the description can go to one person's phone while the room hears the film untouched. A blind viewer and a sighted viewer watching the same screen want different things, and every other approach makes one of them compromise.
Most of what people watch has never been described. Not because description is hard to write, but because somebody has to be paid to write it, title by title, and nobody is paying for the long tail. Every existing AI tool in this space is a service you send a catalogue to.
Sightline describes what is on your television now, for content nobody will ever describe by hand — and because it runs at playback time rather than in a batch job months earlier, you can interrupt it and ask a question.
| Watches | frame pairs — it describes what changed, not what is on screen |
| Speaks | in the gaps where nobody is talking, over music without hesitation |
| Ranks | once — so playing faster gives you less of the same story, never a different one |
| Shares a room | description goes to your phone while everyone else hears the film untouched |
| Answers | "who else is in this scene?" — from the frame you are actually on |
▶ The three-minute demo video — the app running on the Vega virtual device, a real generation run against a clip nothing had ever described, sign-language and on-screen title voicing, and a question typed and answered live on a second screen.
▶ The Open Source contribution video (audio-description-qa, 1:20) —
how the four automated quality checks (preflight, audio_events, stability,
shape_test) catch broken bundles before a blind viewer ever hears them.
Running on the Fire TV app. Every word was written by the system from what changed on screen, placed where nobody is speaking, and measured to fit before it was spoken.
Nothing in that frame is in the dialogue or the soundtrack. The figure is a few pixels across, and it is the shot.
 |
 |
| Quiet during playback — the screen belongs to the film | Titles are described, but rank last: at speed they are the first thing dropped |
Open it on anything and press Play with description. No device, no SDK, no install — the same timeline and the same generated audio that the Fire TV app uses, driving a plain web page. About 6 MB, so it starts quickly on mobile data.
Built for a screen reader: large targets, one primary action, semantic landmarks, and live regions that announce state changes without reading the clock aloud. Several of those details are there because a blind reviewer reported the earlier versions doing the wrong thing.
Four more things worth opening, all linked from that page:
▶ /film/ — three minutes of the actual film
The clip above is a trailer, and a trailer withholds its story deliberately. A blind reviewer pointed out that made it a poor test: "is it possible that the information isn't enough to draw a line through it?" This is the film itself, where the story is present and the description can be judged on whether it conveys one.
▶ /signs/ — an American Sign Language video
Twenty-five seconds of Parents Supporting Children (US public domain), signed
in American Sign Language by Certified Deaf Interpreter Amy E. Peterson with a
completely silent audio track (-91.0 dB). Amazon Transcribe hears zero words;
Sightline's --signs pass samples a 6-frame chronological strip across each
window, translates 4 signed passages into spoken English via Amazon Polly, and
writes [Sign: "..."] entries into the transcript.
▶ /duck/ — a 1951 civil defence film
Nine minutes, narrated almost end to end. A much harder case than a film,
because continuous narration leaves almost nowhere to speak — 77% of it is
dialogue against Sintel's 4.5%. It produces 11 descriptions in the gaps that
exist, and docs/where-it-works.md measures where
that floor finally gives out.
▶ /listen/ — can you hear what the machine hears?
Nineteen clips, scored in your browser, no data collected. Deciding whether a noise is an event worth describing or just the score swelling was the hardest problem in here; this is the test of whether the distinction is audible to a listener at all. Three clips are played twice to measure your own consistency, because a machine that changes its mind is only unreliable if it does so more often than a person.
Asking questions works directly on the hosted pages — type a question (or
use your phone keyboard's dictation key) or tap one of the presets. The page
sends the current playhead to a live AWS Lambda endpoint
(infra/lambda_function.py), which reads a
chronological strip of frames leading up to that second (so hand signs, silent
gestures, sign language, and transient on-screen signs are caught even if the
viewer pauses a beat later) along with the spoiler-free transcript and
description history up to that second, asks Claude Opus 5, and speaks the
answer via Amazon Polly.
One command after cloning adds live Fire TV playhead following and on-demand generation for new videos:
SIGHTLINE_BUNDLE=/path/to/bundle pipeline/.venv/bin/python companion/server.pyThen open http://localhost:8190/ and press Play with description (or
Follow a Fire TV instead when the Vega app is running). You get the film
with generated description ducked over it, and Ask about this moment pulls
the exact frame from the local video.
On the same network, other devices can reach it at your machine's LAN address
(http://192.168.x.x:8190/) — useful for testing on a phone, but it is a
private address and will not work from anywhere else.
It is the product minus the television. The page is built for a screen reader:
large targets, one primary action, semantic landmarks, and an aria-live region
that announces state.
Pre-rendered clips, if you would rather just play a file:
| clip | what to listen for |
|---|---|
sintel-described-1.0x.mp4 |
eight generated descriptions placed between the lines of a real trailer |
sintel-described-2.0x.mp4 |
same story, fewer words — "Now checked" rather than "The checkbox is now ticked" |
walkthrough-described.mp4 |
pause-and-explain mode on a software walkthrough |
flowchart LR
subgraph host["Pipeline — host or cloud"]
V["video"]
V --> T["Amazon Transcribe<br/>word timings"]
V --> F["frame pairs"]
T --> G["gaps = where<br/>nobody is speaking"]
A["audio onset analysis<br/>what did they already hear?"]
V --> A
F --> C["Claude Opus 5<br/>what changed · does it matter"]
G --> C
A --> C
C --> R["rank once"]
R --> P["Amazon Polly<br/>16-bit 48kHz PCM"]
P --> B[("bundle<br/>fragmented mp4 · PCM · timeline")]
end
subgraph tv["Fire TV — Vega OS"]
B --> M["MSE player"]
B --> S["Scheduler<br/>rank fixed, depth varies"]
S --> AS["AudioPlaybackStream<br/>USAGE_ACCESSIBILITY"]
S --> TN["drop tone"]
end
subgraph ph["Companion phone"]
PL["follows the playhead"]
QA["asks a question"]
end
M -. "playhead" .-> PL
B --> PL
QA -. "frame + question" .-> C
Why each of those choices, with the evidence
MSE, not URL mode. player.src = url fails on this platform for every media
type, before it issues a network request. The same player plays the same footage
when the bytes are handed to it through SourceBuffer. Both probes are in
probes/; the diagnosis is in
BUG-REPORT-media-playback.md.
keplerscript-audio-lib, not w3cmedia, for description. Raw PCM gives
frame-accurate timing, which every gap budget depends on, and
USAGE_ACCESSIBILITY makes the platform duck the film for us. It also needs no
media server, so the description channel was never blocked by the bug above.
Speech detection, not silence detection. Looking for silence finds 2.5 usable seconds in a 52-second trailer, because the score never stops. Detecting speech and treating everything else as available finds 39.8 seconds. A 16x difference, and the reason is that describers talk over music all the time — what they avoid is dialogue.
Ranking fixed, depth variable. Assigning descriptions to whichever gap fell nearby meant that at 1x you learned one thing and at 2x a different thing. Rank is now decided once and speed only changes how far down the list you get.
Almost every design decision here came from correspondence with blind reviewers on the ACB Audio Description Project mailing list — not from guesses about what blind viewers want. Two of them, and they did different jobs.
A blind accessibility professional set the description rules, and kept correcting them long after they looked settled:
"A walkthrough needs you to describe what changed, not what's there. The information lives in the difference between two moments." → the pipeline diffs frame pairs; it never describes a single frame.
"A change matters when it changes what you can do next." → the salience filter, and the same rule decides when to pause.
"Take the ranking off the clock. Same order every time. More of it at 1x, less at 2x. What I can't live with is a different story at a different speed." → ranking is fixed host-side; the device never reorders.
"Don't spend words telling me something got skipped. Words are the thing you haven't got. Use a sound." → a 140ms two-tone fall, not a sentence.
"The change I can't hear beats the change I can. The soundtrack is already doing half your job. Spend the gap on what's silent." → onset analysis tells the model which moments were already audible.
"It isn't that it's camera language. It's that I can't use it." → the ban is a usability test, not a vocabulary list.
"A swell and a hit are different shapes, not different sizes. Measure how fast the level got there rather than how far it got." → onset detection measures attack and decay, not magnitude. Following his next correction — measure the fall against the level before the onset, not against the peak — separated hits from swells cleanly, and the perturbation test he then insisted on found a bug in the audio loader: it was reading 16-bit and clipping every file, turning impacts into swells before the rule ever saw them.
Dave Matters used the player and found what no amount of local testing had:
"It goes well until just after the 2 minute mark. I get no further description after that." → a limit meant to cap how many descriptions get written was truncating the film instead. On a feature it would have described the first minute and gone silent for the remaining thirteen. Invisible on a 52-second trailer, which is all that had ever been tested.
"The description was drown out by the video volume." →
volumeis read-only on iOS, so ducking silently did nothing on exactly the devices most likely to be running a screen reader. The film now routes through a gain node.
"VO repeating things over and over… even when returning to the homescreen." → a live region rewritten four times a second, and a transcript that prepended each new line so the reader's cursor was shoved out from under it.
"Is that a pan up to that perspective kind of shot? Just trying to construct it in my head." → there was nobody there. A high shot through rafters had been described as "someone watches her sleep". It had turned a camera angle into a person, and a listener cannot check an invented character against the picture.
There was no way to stop except closing the tab. →
runningwas set true once and never cleared, and pausing the video left an in-progress description speaking through its own audio pool. The primary button now toggles Pause and Resume — halting the voice pool as well as the film and releasing the duck — and Escape stops playback from anywhere on the page.
pipeline/src/preflight.py now fails a clip
before anyone hears it, for each class of fault above that can be detected
mechanically.
They disagreed about one thing, which is why it is a setting rather than a
rule: whether to describe the filmmaking. One said camera vocabulary is
unusable; the other enjoys directorial style and wanted it. The research finds
most blind and partially sighted viewers prefer the cinematic style, so
--cinematic exists and is off by default.
| claim | measured |
|---|---|
| Finds the moments something changed | 9/9 ground-truth changes, 0 false positives |
| Applies the salience rule correctly | precision 1.00, recall 1.00 |
| Fits descriptions to the gap at speed | verified by synthesis at 1x, 1.5x, 2x — nothing time-compressed |
| Finds describable time in real film | 39.8s of a 52s trailer, vs 2.5s by silence detection |
| Never speaks over the dialogue | 0.000s of overlap in sintel-described-1.0x.mp4; the closest description starts 0.70s after the line before it |
| Costs little to run | $0.042 of AWS per minute of film, $0.0046 per description |
| Plays video on Fire TV | MSE reaches canplay → playing, audio audible |
| Plays description concurrently | PCM on an accessibility stream, ducking confirmed |
| Translates sign language & silent gestures | 4 signed passages translated into speech and [Sign: "..."] transcript entries on Parents Supporting Children (ASL, 0% spoken dialogue) |
Reproduce the first two:
The frames are not in the repository, because they are generated rather than
authored: scene.html is a pure function of ?t=, so re-rendering gives
byte-identical frames, which is verified: a fresh clone re-rendered all 88 and
every one matched the original byte for byte. Render them first. This needs
Chrome or Chromium and no network, model or AWS account, and takes about three
minutes because it launches one headless Chrome per frame:
cd pipeline
fixtures/walkthrough/render.sh
.venv/bin/python src/detect_changes.py fixtures/walkthrough/frames --fps 8 --out out/changes.json
.venv/bin/python src/evaluate.py fixtures/walkthrough/ground-truth.json --changes out/changes.jsonDETECTION 9/9 ground-truth changes found (0 extra not in ground truth)
The salience figures need one model call per change, so they need a key:
.venv/bin/python src/salience.py fixtures/walkthrough/frames out/changes.json --fps 8 --out out/judged.json
.venv/bin/python src/evaluate.py fixtures/walkthrough/ground-truth.json --judged out/judged.jsonSALIENCE correct on 9/9 matched events
precision 1.00 (spoke when it shouldn't: 0)
recall 1.00 (stayed silent when it mattered: 0)
Honest caveat, stated because it matters: that fixture is synthetic and we wrote it, so a perfect score shows the pipeline implements the rule faithfully — not that the rule generalises. The film results are on real footage; the walkthrough results are not.
The one thing audio description must never do is talk over the dialogue.
Every description in sintel-described-1.0x.mp4 was measured against
Transcribe's word timings for the same film:
| descriptions | 8 |
| overlapping a spoken word | 0, totalling 0.000s |
| closest start after a line ends | 0.70s |
| closest finish before the next line | 0.42s |
| speech placed / gap available | 26.3s / 39.8s |
The 0.70s is not slack left over, it is the guard band in
pipeline/src/detect_speech.py, and it exists because a blind reviewer
missed a line of the film when description followed it after 0.15s. The
pause after someone speaks is when the listener understands them.
Measured on the same trailer, at us-east-1 list prices:
| per minute of film | |
|---|---|
| Amazon Transcribe | $0.0240 |
| Amazon Polly, generative | $0.0178 |
| AWS total | $0.0418 |
| model layer | 28,920 input tokens (16 frames of 1280x720), 713 output |
A 90 minute feature therefore costs $3.76 in AWS services. The model layer is quoted in tokens rather than dollars because the price depends on which model you point it at, and the pipeline runs against Bedrock or a direct API without changing the tokens it sends.
Everything here runs on macOS, Linux and Windows. The pipeline and the companion service are plain Python with no platform specific calls, and the app is built with the Vega CLI, which Amazon ships for all three.
The macOS specific things in this repository are the scripts in tools/ used
to record the demo video. They are not needed to run, build or evaluate
anything. The fixture renderers under pipeline/fixtures/ look for Chrome or
Chromium in the usual place for each platform, and take a CHROME= override
if yours lives somewhere else.
| Python | 3.10 or newer |
| ffmpeg | on your PATH. brew install ffmpeg, sudo apt install ffmpeg, or winget install ffmpeg |
| AWS credentials | for Transcribe and Polly, if you want to generate a description from scratch |
| Vega SDK and CLI | only for the device app. Not needed for the pipeline or the web player |
Pipeline only, no Fire TV needed
macOS and Linux
cd pipeline
python3 -m venv .venv
.venv/bin/pip install anthropic boto3 pillow numpy
export ANTHROPIC_API_KEY=... # or write it to ~/.config/sightline-anthropic-key
.venv/bin/python src/detect_speech.py video.mp4 --cache out/t.json --out out/gaps.json
.venv/bin/python src/film_cues.py video.mp4 out/gaps.json --transcript out/t.json --out out/cues.json
.venv/bin/python src/export_device_bundle.py out/cues.json video.mp4 --gaps out/gaps.json --out /tmp/bundle
.venv/bin/python src/render_from_bundle.py /tmp/bundle --rate 1.0 --out out/described.mp4Windows, PowerShell
cd pipeline
py -3 -m venv .venv
.venv\Scripts\pip install anthropic boto3 pillow numpy
$env:ANTHROPIC_API_KEY = "..."
.venv\Scripts\python src\detect_speech.py video.mp4 --cache out\t.json --out out\gaps.json
.venv\Scripts\python src\film_cues.py video.mp4 out\gaps.json --transcript out\t.json --out out\cues.json
.venv\Scripts\python src\export_device_bundle.py out\cues.json video.mp4 --gaps out\gaps.json --out $env:TEMP\bundle
.venv\Scripts\python src\render_from_bundle.py $env:TEMP\bundle --rate 1.0 --out out\described.mp4Requires AWS credentials for Transcribe and Polly. If your network inspects
TLS, set SIGHTLINE_CA_BUNDLE to your corporate bundle; boto3 does not read
SSL_CERT_FILE, which is its own small trap.
On the Fire TV virtual device
Paths below are POSIX. On Windows use .venv\Scripts\python and $env:TEMP
in place of /tmp, as above.
# 1. build a bundle (above), plus the app's own voice
cd pipeline && .venv/bin/python src/build_ui_voice.py --out /tmp/bundle
# 2. serve it, and the companion phone page
SIGHTLINE_BUNDLE=/tmp/bundle pipeline/.venv/bin/python companion/server.py
# 3. build, install, run
cd app && npm install && npm run build:debug
vega virtual-device start
vega device install-app -p build/aarch64-debug/sightline_aarch64.vpkg
vega device launch-app -a com.sightline.tv.mainThen open http://<your-lan-ip>:8190/ on a phone on the same network and press
Follow a Fire TV instead.
Remote: Select plays and pauses · Right changes speed · Up switches mode · Down moves description to the phone · Menu speaks the controls.
Nothing installed at all
tushartechs.github.io/sightline runs the same timeline and the same generated audio in any browser, on any operating system. Press Play with description.
110 tests. Each one locks a behaviour this project argues for, or a fault a blind reviewer found. None of them need AWS, a Fire TV or a network.
python -m pytest pipeline/tests -q # 95
cd app && npm test # 15| what it protects | why it exists |
|---|---|
| Gaps come from speech, not silence | Looking for quiet found 2.5 usable seconds in a 52 second trailer because the score never stops. Inverting detected speech found 39.8. The tests assert gaps never land inside dialogue, never overlap, and do not depend on how loud the film is |
| The word budget uses the measured rate | We ask Polly for 170 wpm and it delivers 203. Budgeting at the requested rate overflowed every line. A test now fails if the two are ever reconciled the wrong way |
| A continuity repair cannot outgrow its gap | The repair pass rewrites references across independently written lines. A better sentence that no longer fits is a line spoken over dialogue, so it is refused and the original kept |
| Preflight still catches the real faults | Camera language, dialogue collisions, missing audio, a truncated film. And the inverse: an instructional film that narrates for nine minutes is not blamed for silence it never had room to fill |
| Attenuating the file does not move a classification | The test a blind reviewer proposed. Every quantity is a difference between two levels, so a uniform gain must cancel. It did not, which is how the 16-bit clipping was found. Now asserted to 0.01 dB |
| Speed changes the depth, never the order | "What I can't live with is a different story at a different speed." What survives at 2x must be a rank-ordered prefix of what survives at 1x. Asserted at 1.25x, 1.5x, 2x and 3x |
| Questions never leak spoilers, and work at the end of a clip | The hosted answerer excludes dialogue, signs and descriptions after t, fetches a multi-frame strip leading up to t to catch gestures, omits an empty image block on ?probe=1, and steps back to the final extracted frame when a clip stops at a fractional second |
| Silent communication is captured across frame strips and merged into the transcript | Amazon Transcribe hears nothing when someone makes a hand sign, signs in ASL/BSL, or holds up a written note, and a two-frame diff misses gestures that happen mid-gap. signs.py samples a chronological strip, translates signs into spoken audio and tagged transcript entries ([Sign], [Gesture], [On-screen sign]), and preserves word-timing items |
CI runs the pipeline suite on Linux, macOS and Windows across two Python versions, because this README claims all three and a claim in a README is not a test.
Check the deployed endpoint (with or without a live model probe):
curl https://s4i9xx7htg.execute-api.us-east-1.amazonaws.com/health
curl "https://s4i9xx7htg.execute-api.us-east-1.amazonaws.com/health?probe=1"Judges should be able to tell these apart, so they are separated here.
AI is the product, in three places. These are not development aids, they are what the thing does:
| Writing the descriptions | A vision language model reads frame pairs from each gap and writes the line. This is the core of the project |
| Answering questions | The same model, given the frame you are on plus the dialogue and descriptions so far |
| Speech and timing | Amazon Polly speaks every line, Amazon Transcribe supplies the word timings that define the gaps |
AI assisted the development, as a coding assistant, the way most software is now written. It wrote code under direction and was corrected constantly.
What is not AI, and is the actual work:
- Choosing the problem, and checking it was real. Audio description does not exist for most video because someone has to be paid to write it.
- The rules the descriptions follow. These came from blind reviewers on the ACB Audio Description Project mailing list, not from a model and not from me. Nothing during dialogue, nothing that repeats the soundtrack, never "the camera pans".
- The architecture. Detecting speech rather than silence, which turned 2.5 usable seconds into 39.8. Ranking descriptions so the right ones survive at speed. Measuring every line against its gap before keeping it.
- Finding out where it fails. The measurements in
docs/where-it-works.md, including the advert that correctly produces nothing. - Getting it tested by blind users, and acting on what came back. Two reviewers, seven faults, all fixed, and a QA harness released so the next person does not repeat them.
- The fifteen findings in
FRICTION-LOG.md, written while building.
A model can write a line of description. It cannot tell you that the line is wrong because a blind viewer does not care where the camera is.
Five AWS services do real work across the pipeline and the live question endpoint — not decoration, and each earned its place by solving a specific problem.
pipeline/src/detect_speech.py
The first version of this looked for silence. On a real trailer that finds 2.5 usable seconds in 52, because the score never stops, and it would make film description essentially impossible. But describers talk over music constantly — what they avoid is dialogue.
Transcribe's word-level timestamps give the speech intervals directly; everything else is available. 2.5 seconds became 39.8. A 16x difference, and the single most consequential correction in the project.
It also returns the dialogue text, which is fed to the model so a description never repeats a line the viewer just heard.
tr.start_transcription_job(TranscriptionJobName=job,
Media={"MediaFileUri": f"s3://{bucket}/{key}"},
MediaFormat="mp3", LanguageCode="en-US")pipeline/src/speech.py
Generative engine, falling back to neural where a voice or region lacks it.
Chosen over the platform's own speech for three reasons: it is portable (the
alternative was macOS-only, and this project has to build elsewhere), the
generative voices are markedly better, and it returns PCM natively — which
is exactly what AudioPlaybackStream.writeAsync() takes on Vega.
Two things we learned the hard way and corrected in code:
- Polly pads both ends with silence, and in fit-the-gaps mode that padding is charged against the gap budget. It is trimmed before measurement.
- The speaking rate has to be measured, not assumed. We guessed 170 wpm;
once the padding is trimmed the voice delivers 203. Every line overflowed
its gap until we measured.
python3 src/speech.py calibratere-derives it.
Transcribe reads its input from S3, so audio is uploaded, transcribed and the object deleted in the same call. The bucket is private with public access blocked.
infra/lambda_function.py · infra/template.yaml · infra/deploy_ask.py
The public demo is static files on GitHub Pages, and a static page cannot hold a
model key or run ffmpeg. Putting the question endpoint behind an HTTP API
(s4i9xx7htg.execute-api.us-east-1.amazonaws.com) and an arm64 Python 3.12
Lambda function made mid-scene questions work on the public link without a local
server.
Instead of downloading the video on every question, the function fetches the
pre-extracted 11 KB JPEG for that second (frames/00015.jpg) plus the cached
Transcribe timings and timeline up to the viewer's playhead (t <= now, so
nothing later in the film can leak as a spoiler), calls the model, synthesises
the reply with Amazon Polly (Ruth, generative), and returns the MP3 inline as
base64 to save a second round trip on a mobile connection. If the model is
unavailable, it falls back to a clearly labelled recap of the dialogue and
descriptions heard so far rather than returning an error.
Bedrock. It is refused at the account level on this account —
Error 002: Access to Bedrock models is not allowed for this account, in every
region, for every model, while S3, Polly and Transcribe all work on the same
credentials. The Bedrock client is written and shipped
(pipeline/src/salience.py, --backend bedrock, using AnthropicBedrockMantle)
and the first-party API is used instead. The diagnosis is in
FEEDBACK.md and STATUS.md.
app/ |
the Vega OS Fire TV app |
pipeline/ |
description generation, evaluation, rendering |
companion/ |
phone channel and the question endpoint |
infra/ |
AWS Lambda + API Gateway question endpoint (template.yaml, deploy_ask.py) |
probes/ |
minimal apps that isolate what works on this platform |
FRICTION-LOG.md |
15 entries on developing for Vega |
VEGA-FIELD-NOTES.md |
practical notes for other developers |
STATUS.md |
what is done and what is not, honestly |
FRICTION-LOG.md has 15 entries from real work, and
VEGA-FIELD-NOTES.md is the developer-facing companion —
the things that cost us days and are in neither the documentation nor the forum.
The sharpest: vega device run-cmd is a sandboxed app context, so platform
probes run from it return confident false negatives. Three of them told us the
device had no audio while it was playing an audible boot chime.
We also filed the media playback bug upstream with a controlled reproduction (topic 29117).
Kept current in STATUS.md. The ones worth knowing before you judge:
- Description is generated ahead of playback, not live as you watch. Questions are answered live. Generating once for content nobody will ever describe by hand is the design; we would rather say so than imply otherwise.
- Blind reviewers have used the web player, not the Fire TV app. They found seven faults in it, all fixed. The pipeline faults among them — the truncated timeline, the invented character — affect the television app equally, because it is the same generated description. The iOS-specific ones do not. Nobody blind has operated the app on a television.
- It describes what it is left room to describe. Measured across three
content types in
docs/where-it-works.md: an advert produces zero descriptions, because it fills every second it paid for. Gap-filling has a floor and this is where it is. - It does not understand the film. It describes moments accurately and does not connect them. A reviewer asked what the dagger meant; no amount of better description answers that.
- Answers take about 7 seconds.
- Never run on physical Fire TV hardware — the virtual device only.
MIT — see LICENSE.
Required attribution. Demo footage is Sintel, © Blender Foundation, durian.blender.org, licensed CC-BY 3.0. Any video using this footage must carry that credit. Full details in attribution/CREDITS.md.
Built with — not an attribution requirement, listed because it is the first thing anyone evaluating this asks: Amazon Transcribe for speech timings, Amazon Polly for the description voice, and Claude Opus 5 for the descriptions and the answers.