Implement virtual scroll for large dataframes

[severo]

It replaces #347 (the remaining parts). The current commit will be recreated later, but I wanted to save my work.

remaining:

• fix the update loop (lift some state up?)
• when navigating down, the header row is sometimes focused instead of the next cell
• slow: too many renders
• check that everything was copied from [wip] fix scroll for big datasets #347 (CSS? mock row label location in the DOM?)
• fix bottom border of the last row (see Implement virtual scroll for large dataframes #369 (comment))
• add tests (with playwright / vitest browser?)

Later?

• handle edge cases (eg. row / header height > viewport height)

Screencast.From.2026-01-03.01-35-13.mp4

[Copilot]

Pull request overview

This WIP pull request implements virtual scrolling for the HighTable component to handle large datasets efficiently. The implementation introduces coordinate transformation between physical canvas height and virtual canvas height, along with custom scroll handling logic. The PR is currently broken due to an update loop issue as noted in the title.

Key changes:

• Implements virtual scroll provider with coordinate transformation between canvas and virtual canvas spaces
• Adds IntersectionObserver-based focus management for cells in virtual scroll mode
• Updates scroll behavior to support options parameter for smooth/instant scrolling

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
src/providers/ScrollModeVirtualProvider.tsx	Main implementation of virtual scrolling logic with state management, coordinate transformations, and scroll-to-row functionality
src/providers/ScrollModeProvider.tsx	Passes required props (numRows, headerHeight) to ScrollModeVirtualProvider
src/hooks/useCellFocus.ts	Adds IntersectionObserver-based focus logic for virtual scroll mode with horizontal scroll handling
src/contexts/ScrollModeContext.ts	Extends context interface with shouldScrollHorizontally flag and behavior options for scrollToTop
src/components/HighTable/Scroller.tsx	Updates scrollTo implementation to accept behavior options (instant/smooth)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 10 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[severo]

Finally, I have a valid PR for virtual scrolling of huge dataframes. Would you like to have a look @platypii?

[severo]

The bottom border is not visible when scrolling past the end of the table (it only happens in "virtual scroll mode")

Screencast.From.2026-01-07.14-44-18.mp4

[platypii]

Still getting weird behavior: if I scroll down even one "notch" on my scroll wheel, I jump to row 4000+.

Worse: when I try to scroll back up, there's no way to get back to row 1

simplescreenrecorder-2026-01-07_12.06.49.mp4

[severo]

Hmmm, I don't know how to change this while letting the browser natively handle the scroll.

The current logic is:

• the scrollbar is only for global navigation. Ie: 1% scroll = 1% of the number of rows, which might mean a gap of 4000 rows, indeed. It is controlled by any scroll event received from the browser (mouse wheel, drag/drop of the scroll handle, click on the scroll bar), plus some keyboard events when the whole component is focused
• local navigation is only available through the keyboard (arrows, pgup/pgdn).

I understand the UX is maybe not what we expected, but I don't think we would be able to do more with the current concepts/logic.

Alternatives:

1. apply some kind of non-linear reaction to changes to the native 'scrollTop' value: if the change is small, act "locally", if it's big, act "globally". I can try this, but 1. I'm not sure the UX will be predictable for the user (it might feel weird), 2. I'm afraid of having corner cases where the behavior is hard to define,
2. replace the scrollbar with a custom component: to be honest, I'm not sure it would fix anything, as I think one scrollbar cannot provide global navigation and local precision at the same time.
3. provide additional controls/buttons, to make the navigation explicit.

For 3., an option might be to use the scrollbar for local scrolling, eg from row 0 to row 8.000.000, then:

• at the bottom of the scroll area: show a button 'Load more', that would change the local range to 4.000.000 -> 12.000.000 and put the local scroll to 50%, keeping (or trying to keep) the current visible rows. It might be autoloading instead of an explicit button.
• same at the top of the scroll area, when scrolling backwards
• additionally, show a minimap on the right, with the range highlighted. It might be another native scrollbar, or a custom component (more clarity if the scrollbar and the minimap are visually different).

[platypii]

NICE!! This is feeling much better! 👏

I noticed one weird behavior: if you scroll all the way to the bottom of the "Large Data" storybook you get "stuck" and need to scroll up a bunch to make it "unstick" from the bottom.

[severo]

Indeed, I'll fix it tomorrow, and add some tests (I isolated most of the code that contains logic to independent functions)

[severo]

I noticed one weird behavior: if you scroll all the way to the bottom of the "Large Data" storybook you get "stuck" and need to scroll up a bunch to make it "unstick" from the bottom.

hmmm, finally, I cannot reproduce. Can you share more details, or a video? I tried on Firefox and Chrome

[severo]

why not 10¹⁵ rows?

[platypii]

hmmm, finally, I cannot reproduce. Can you share more details, or a video? I tried on Firefox and Chrome

It only happens on my macbook when scrolling up from the bottom. It's still happening but felt worse yesterday for some reason. I don't think its a blocker necessarily. Video (all i'm doing is scrolling up slowly):

bottom.of.scroll.mov

[severo]

Ok thanks, I'll try to reproduce and find why it happens

[bleakley]

The code looks great, and I build hyperparam with this and tried looking at some datasets with 1m rows, generating new columns, etc, and everything looks good 👍

[severo]

OK. I'll merge, and open an issue for the scroll issue for the last rows, as I cannot reproduce.

[severo]

I published v0.25.3

[severo]

I opened #389 for the stuck scroll

[platypii]

I published v0.25.3

It may not be semver, but this is such a major upgrade i would have called it 0.26.0 :-)

[severo]

Yes, I would have changed the second number if the first one was not zero. Until 1.0.0, I was thinking that changes that don't break anything must go to the third number. Anyway, if you prefer the second number for communication matters, I can release 0.26.0 too.

[severo]

Done:

https://www.npmjs.com/package/hightable/v/0.25.3

https://www.npmjs.com/package/hightable/v/0.26.0

[severo]

I updated the demos: try it at https://hyparam.github.io/demos/hightable/#/large or https://hyparam.github.io/demos/hyparquet/?key=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fcalabi-yau-data%2Fws-5d%2Fresolve%2Fmain%2Fnon-reflexive%2Fws-5d-non-reflexive-0000.parquet