Sub-band alignment ignores sign: anti-phase orientations are misaligned or cancel (recovery drops to r≈0.85–0.91 for vertical motion)

[joeljose]

Severity: High. It degrades the core output, depending on which way the surface vibrates.

Problem

postprocess_phase_signals aligns each (level, orientation) phase signal to the reference band (level 0, orientation 0) using the time shift that maximises the cross-correlation, then adds the bands together with np.roll:

shift_matrix[i, j] = find_best_shift(ref_vector, phase_signals[:, i, j])   # argmax(correlate)
sound_raw += np.roll(phase_signals[:, i, j], int(shift_matrix[i, j]))

A displacement d seen through an oriented band gives a phase change ∝ k_θ · d. The DTCWT's 6 orientations cover 180°, so for most motion directions some bands see the motion with the opposite sign. With only "maximise correlation" available, an inverted band either:

• gets shifted by about half a period, which is roughly right for a pure tone and wrong for anything broadband, or
• gets added in anti-phase and cancels part of the signal.

Two related problems:

• The lag is unbounded. find_best_shift searches over all ±N lags. For a band that is mostly noise, the "best" lag is arbitrary; two unrelated 22,000-sample noise bands gave −1235 samples.
• np.roll wraps around, so the end of the clip is moved to the start.

Evidence (synthetic, known ground truth)

The test used a 128×128 texture moved with exact sub-pixel Fourier shifts of 0.05 px peak, with frames sampled at 2200 fps plus sensor noise, then run through extract_audio with -fl 100. The table shows correlation with the true signal (sign-agnostic, best lag within ±50):

motion	signal	current	sign-aware, bounded-lag alignment
horizontal	tones 250/440/660 Hz	0.994	—
horizontal	broadband 150–800 Hz	0.999	0.999
vertical	tones	0.854	0.998
vertical	broadband	0.906	0.999

Correlation signs of each band against the truth (level 1, 6 orientations) for vertical motion: [+, +, +, −, −, −] at level 2 and a mixed pattern at level 1. So half the bands are inverted.

The GPU path gives identical numbers: CPU vs GPU-path correlation 1.000, with the GPU code run on the CPU via PyTorch.

Suggested fix

def align(ref, x, max_lag):
    c = signal.correlate(ref, x, mode="full")
    mid = len(x) - 1
    win = c[mid - max_lag: mid + max_lag + 1]
    k = np.argmax(np.abs(win))                  # allow anti-correlation
    lag, sgn = k - max_lag, np.sign(win[k])
    y = np.zeros_like(x)                        # zero-pad instead of wrapping
    if lag >= 0: y[lag:] = x[:len(x) - lag]
    else:        y[:lag] = x[-lag:]
    return sgn * y

• Bound max_lag to a few samples (e.g. --max-lag, default ≈ 0.005 * fps). The sub-bands come from the same frames, so real delays are tiny.
• Remove the mean from, or band-pass, each band before correlating (see the drift issue). Otherwise DC and drift dominate the correlation.
• Optionally weight each band by its correlation or SNR instead of adding them equally. Davis et al. effectively rely on high-amplitude bands dominating.
• Pick the reference band by energy rather than always (0, 0), so a weak or noisy reference band doesn't hurt everything.

Acceptance criteria

• Synthetic vertical-motion recovery gives correlation ≥ 0.99 (a test using the generator attached to the testing issue).
• The lag is bounded, and there's no circular wrap.