Optimize "Load Visualization" Performance for Large Datasets

[Weiyu-Kong]

Description
Currently, when a user initiates a "Load Visualization" request (either from the web or extension version), the backend loads data for every epoch, calculates neighbors for each point, and then sequentially transmits the data to the frontend for storage.
When the dataset is large, the time overhead for neighbor calculation and sequential transmission becomes significant, causing the loading process to be extremely slow.

Expected Behavior
The "Load Visualization" request should respond much faster, providing a smoother user experience even with large datasets.

Your Tasks

1. Optimize Neighbor Calculation:
   • Implement dynamic neighbor calculation (e.g., calculate only when the frontend switches epochs or when a user selects a specific point).
   • Alternatively, utilize more efficient libraries for calculating high-dimensional and low-dimensional neighbors.
2. Parallel Data Transmission:
   • Instead of calculating and transmitting data for each epoch sequentially, adopt a parallel approach to handle multiple epochs or data chunks simultaneously to accelerate the transfer.

Relevant Codebase

• Backend Server: tool/server/server.py (Handling load_visualization request)
• Command Entry: /extension/src/control.ts (Triggering loadVisualization command)
• Frontend Data Handling: web/src/state/state.unified.ts(Receiving and storing visualization data)
• Frontend Entry: web/src/views/plotView.tsx(Handling Load Visualization command)

Your changes are not limited to the files listed above.