Improve gesture recognition robustness and latency

[zayan-sheikh]

What changed

• normalize each pose by shoulder width so distance from the camera does not distort the KNN feature space
• weight arm and hand landmarks most heavily, because Baboon commands are upper-body gestures, while reducing noisy lower-body influence
• use inverse-distance neighbor voting so a very close match counts more than a marginal neighbor
• shorten the temporal window from 60 frames at 80% consensus to 20 frames at 65% consensus for faster, more jitter-tolerant recognition
• add a deterministic benchmark command: npm run benchmark:recognition

Measured result

The benchmark uses seed 1847 and 9,000 leave-one-sample-out cases (180 source samples × 50 variants). Every tested pose is excluded from its own training set, randomly scaled from 0.65× to 1.45×, and given Gaussian coordinate jitter with σ=0.01.

Metric	Before	After	Change
Overall accuracy	90.64%	98.59%	+7.95 pp
Correct cases	8,158 / 9,000	8,873 / 9,000	+715
Error count	842	127	-84.9%
Full-window response at 30 FPS	2.00s	0.67s	66.7% faster

Accuracy by gesture

Gesture	Before	After
dup2	89.00%	100.00%
endFunc	97.80%	100.00%
neut	98.80%	99.10%
one	90.50%	89.90%
plus	83.40%	99.90%
runFunc	82.20%	99.50%
startFunc	88.10%	98.90%
undo	97.50%	100.00%
zero	88.50%	100.00%

one is the only class with a small regression in this synthetic stress set (-0.6 pp); the overall model still removes 715 of 842 baseline errors. This benchmark measures robustness against camera-distance and landmark jitter using the existing dataset. It is reproducible evidence, but it is not a substitute for a larger real-world multi-person validation set.

Verification

• npm test (10/10 passing)
• npm run benchmark:recognition
• npm run check (0 errors, 0 warnings)
• npm run build

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
baboon	Ready	Preview	Sep 5, 2026 12:08am UTC

[amazon-q-developer]

Summary

This PR introduces significant improvements to gesture recognition accuracy (+7.95 pp) and latency (66.7% faster) through pose normalization, weighted features, and inverse-distance voting. The comprehensive benchmark demonstrates robustness against scale and noise variations.

Critical Issues

Two numerical stability defects must be fixed before merge:

1. Numerical overflow in inverse-distance voting (line 39): Perfect matches produce vote weights of ~1M, causing instability
2. Division by zero in shoulder scaling (line 50-56): Missing or corrupted shoulder landmarks cause classification failure

Both issues will cause runtime crashes or incorrect predictions in production when processing real-world landmark data.

Verification Required

After addressing the critical issues, please re-run:

• npm test to ensure all tests pass
• npm run benchmark:recognition to verify accuracy improvements are maintained

---

You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

[amazon-q-developer]

🛑 Numerical Instability Risk: When neighbor.distance equals 0 (perfect match), the vote weight becomes 1/1e-6 ≈ 1,000,000, which can dominate all other votes and cause numerical overflow in vote tallies. Replace with safer inverse-square weighting or cap the maximum vote weight.

Suggested change

	const voteWeight = 1 / (Math.sqrt(neighbor.distance) + 1e-6);
	const voteWeight = 1 / (neighbor.distance + 1e-6);

[amazon-q-developer]

🛑 Division by Zero Risk: If both shoulders have identical coordinates (corrupted/missing landmarks), Math.hypot(dx, dy, dz) returns 0, making queryScale and sampleScale equal to 1e-6. This causes division operations in line 29 to produce extremely large values (~1e5 scale) that distort the distance calculation and break classification.

Suggested change

	function getShoulderScale(features) {
	const leftShoulder = 11 * 3;
	const rightShoulder = 12 * 3;
	const dx = features[leftShoulder] - features[rightShoulder];
	const dy = features[leftShoulder + 1] - features[rightShoulder + 1];
	const dz = features[leftShoulder + 2] - features[rightShoulder + 2];
	return Math.max(Math.hypot(dx, dy, dz), 1e-6);
	function getShoulderScale(features) {
	const leftShoulder = 11 * 3;
	const rightShoulder = 12 * 3;
	const dx = features[leftShoulder] - features[rightShoulder];
	const dy = features[leftShoulder + 1] - features[rightShoulder + 1];
	const dz = features[leftShoulder + 2] - features[rightShoulder + 2];
	const distance = Math.hypot(dx, dy, dz);
	if (distance < 1e-3) {
	throw new Error('Invalid pose: shoulder landmarks are too close or identical');
	}
	return distance;
	}

[zayan-sheikh]

Addressed the numerical-stability review feedback in ad418a0:

• perfect KNN matches now return immediately instead of creating an extreme inverse-distance vote
• invalid or zero shoulder spans now use a finite RMS body-scale fallback
• added regression coverage for exact matches and zero-width shoulders
• reran all checks: 12/12 tests pass, Svelte check has 0 diagnostics, the production build succeeds, and the benchmark remains 90.64% → 98.59%