Add Z variable selection, 3D scatter view, and UI enhancements#204
Open
SimonHH wants to merge 51 commits into
Open
Add Z variable selection, 3D scatter view, and UI enhancements#204SimonHH wants to merge 51 commits into
SimonHH wants to merge 51 commits into
Conversation
- ColumnPanel: add comboZ dropdown (Z/C:) for selecting a third variable as a color/height axis; updates alongside comboX columns - SelectionPanel: bind comboZ events and include Z column index in ID tuples passed to PlotData - PlotData: add iz, sz, z, zIsString, zIsDate attributes; fromIDs() loads Z column when idx has 8+ elements - GUIPlotPanel: add ColorCtrlPanel with colormap selector, colorbar toggle, and 3D view checkbox; shown automatically when a Z variable is selected - plotSignals(): when Z is set, renders a scatter plot colored by Z using the chosen colormap and optionally adds a colorbar; when 3D is enabled, renders a full 3D scatter plot with Z as the spatial z-axis - set_subplots(): creates 3D axes (projection='3d') when 3D view is active - figure.py (SwappyFigure): gracefully handle non-SwappyAxes (e.g. Axes3D) by adding compatibility shims for set_xlim_/get_xlim_/etc. - _store_limits/_restore_limits: guard against AttributeError on non-Swappy axes (e.g. colorbar axes added by matplotlib) https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
Update README with: - New plot types: scatter+color scale, 3D scatter plot - New plot options: Z/color variable, colormap, colorbar, 3D view - Workflow tip explaining the Z/C dropdown in the column panel https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
requirements.txt: - Add minimum version pins for all packages - numpy>=2.0: required for Python 3.14; removes deprecated bare aliases (np.bool, np.int, np.float, etc. removed in 2.0) - wxpython>=4.2.4: first release with Python 3.14 wheel support - pandas>=2.2, matplotlib>=3.8, scipy>=1.12, xarray>=2024.1, pyarrow>=15.0, openpyxl>=3.1, chardet>=5.0 all confirmed py314 installer.cfg: - Python version: 3.9.9 → 3.14.0 - wxPython: 4.1.1 → 4.2.5 (cp314 wheels available on Windows/macOS) - numpy: 1.22.4 → 2.2.4 - matplotlib: 3.5.2 → 3.10.1 - pandas: 1.4.2 → 2.2.3 - scipy: 1.8.1 → 1.15.2 - pyarrow: 8.0.0 → 19.0.1 - openpyxl: 3.0.10 → 3.1.5 - Pillow: 9.1.1 → 11.1.0 - xarray: 2023.2.0 → 2025.3.0 - chardet: 4.0.0 → 5.2.0 - Retain fatpack==0.7.3 (pure Python, tested functional under py314) - Old py3.9 pins preserved as comments for reference setup.py: - Add python_requires='>=3.9' - Add install_requires with minimum version constraints Note for Linux: wxPython does not publish manylinux wheels on PyPI. Use https://extras.wxpython.org/wxPython4/extras/linux/ for pre-built wheels, or build from source (requires GTK dev headers). https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
This reverts commit 152bb95.
- Rename "Z/C:" label to "z-axis:" in column panel - Remove colormap dropdown (hardcode viridis) and colorbar checkbox (always on) - Add x-y / y-z / x-z plane view buttons that appear when 3D view is active - Require Ctrl+left-click for 3D rotation to avoid conflict with zoom https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
The previous approach relied on ax._cids being non-empty and event.key being set, neither of which is reliable across matplotlib versions. New approach: - ax.mouse_init(rotate_btn=[]) disables built-in left-click rotation - custom button_press handler sets _rotate_btn=[1] only when Ctrl is held (detected via wx.GetKeyState for wx-backend reliability) - custom button_release handler resets _rotate_btn=[] on release https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
Previous approach used mouse_init/_rotate_btn which silently fails in many matplotlib versions. New approach: - Connect a button_press handler that fires AFTER the built-in one - If Ctrl is not held, reset ax.button_pressed=None so _on_move returns early and skips rotation - wx.GetKeyState used for reliable keyboard state detection https://claude.ai/code/session_019co5puUhq1kxsuWG57SPzC
Adds a third variable (Z) to the selection panel with colour scale, enabling coloured 2-D scatter plots and a full 3-D view. Polishes the Z/colour UI and 3-D mouse interaction, and fixes Ctrl+rotate using button_pressed reset.
…vview files Introduces the Views feature: named views (column selection + plot settings) can be saved and restored at any time. Views can be exported to portable .pdvview JSON files with relative data-file paths, and re-imported via drag-and-drop or the Views menu. Missing tables/columns on restore produce warnings rather than crashes. Extends to cover: Z/color column selection, 3D view state (log-z, flip-z, cb3D), pipeline actions (Filter/Resample/Bin/Mask), all sub-panel settings and loader options. Adds "Apply view to current table" which resolves saved column names on whichever table is selected. Fixes crashes, list-deselect TypeError, compare-with-1-series, multi-y column drop, formula restore, and drag-drop/Ctrl+drag parity. https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
…olorbar Replaces CTRL-rotate with a dedicated Rotate toggle button in the toolbar. Adds orthographic plane-reset, Home button, and 3D left-click pan mode matching 2D behaviour. Adds x/y/z axis constraint support. Includes 6 UI improvements: view menu submenus, surf/scatter choice, single shared colorbar per axis, plane reset, and Scatter as default 3D type. Improves control- panel layout and error robustness throughout. https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
…ombo Cancels debounce timer in empty(), guards against np.array column indices (TypeError fix). Adds Scatter/Scatter+Line choices to the curve-type combo when a Z column is selected but 3-D view is off.
…ries) The Recent Files submenu now tracks: opened data files, imported .pdvview files, exported view files, and exported tables — all in a single list, newest first, capped at 30. Clicking a .pdvview entry calls load_view_file(); all others call load_files(). https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
Each panel now stores its own absolute pixel width. Window resize only affects the last panel (absorbs slack); other panels keep their width. Dragging a sash updates only the panel to its left via GetSashIdx(). Mode switches (1/2/3 columns) restore saved widths instead of resetting to equal. Widths are persisted in view save/restore via stable panel names. https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
Covers when to use the tool, installation, all supported file formats, GUI layout, selection/plot/pipeline/view features, export options, keyboard shortcuts, and common-task quick reference. https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
Previously the last panel was never stored in _panelWidths so dragging the sash to its left had no persistent effect — the panel snapped back to "remainder" on the next resize. Changes: - _savePanelWidths now records ALL panels including the last - _restorePanelWidths scales all panels proportionally when the window is resized (instead of letting only the last panel absorb slack) - onSashChange also saves the last panel's width after every drag Result: every panel has a stored width; resizing the window maintains proportions; no panel is treated as a passive remainder. https://claude.ai/code/session_01RK7XoUAyBGq81RqNdX7htm
- Set vSplitter sash gravity to 0 so the left selection panel keeps its width when the window is resized - Stop MultiSplit from scaling sub-panels proportionally on EVT_SIZE - Remove automatic sash repositioning from the resize/idle handler; layout updates still happen when switching column modes https://claude.ai/code/session_01TEDdcedUgmSE1yVF33MFYE
- Fix pipeline save/delete and table repr typos (GUIPipelinePanel, pipeline, Tables) - Fix saveOptions typo'd parameter name (Tables) - Persist loader options (dayfirst, naming) on shutdown (appdata) - Drop DC bin when converting FFT to Period (plotdata) - Fix mask plugin IndexError on empty dataframes by using dtype-based timestamp detection instead of df.iloc[0,i]; surface the underlying exception in applyMaskString for better debugging (data_mask, Tables) https://claude.ai/code/session_01RusnXtXqo9DDz4dX2rUhXB
- Fix selection-panel width blowing out with long column names by subclassing ListBox/ComboBox to clamp DoGetBestSize (GUISelectionPanel) - Restore MultiSplit onParentChangeSize to call _restorePanelWidths and Skip the event so base class handles resize (GUIMultiSplit) - Set MinimumPaneSize and SashGravity before SplitVertically so the initial sash position is not silently overridden (GUIFields1D) - Enable SP_LIVE_UPDATE on both outer vSplitter and tSplitter to eliminate the XOR tracker line on click-and-hold (GUIFields1D) - Don't clear the filter search box text on Ctrl+C (GUISelectionPanel) https://claude.ai/code/session_01RusnXtXqo9DDz4dX2rUhXB
Axis limits: - Add xmin/xmax/ymin/ymax text fields to EstheticsPanel - Add zmin/zmax fields for 2D+Z and 3D modes - Flatten EstheticsPanel to a single WrapSizer; hide optional panels reliably on first open via plotsizer.Hide() Background image: - Add BG toolbar button with Load/Paste/Clear menu - Two modes: 'Fixed' (default, fills plot area via imshow + xlim/ylim-changed callbacks) and 'Moving with axes' (glued to data coords captured on switch) - IMAGE_EXTS constant shared with main.py for recent-files routing Z-axis / color-scale: - Use z-axis limit fields for color scale in 2D+Z scatter mode - Fix color-scale / z-limit sync when AutoScale toggles - Exclude datetime Z columns from color-scale path - Preserve 3D camera state across redraws; enable logZ; dedupe colorbars https://claude.ai/code/session_01RusnXtXqo9DDz4dX2rUhXB
Ctrl+V (paste): accepts file paths (data, .pdvview, images) or raw bitmaps from the clipboard. File paths are routed through a shared _routeFilenames helper (also used by drag-and-drop). Shift+Ctrl+V adds to existing tables instead of replacing. All pasted files appear in the Recent Files menu. Ctrl+C (copy): dispatches based on last-focused pane: - Columns list → TSV of selected X/Y/Z columns across tables - Tables list → TSV of all columns for selected tables - Plot canvas → PNG bitmap of the current figure - Stats panel → reuses existing CopyToClipBoard method Implementation uses EVT_CHAR_HOOK (not AcceleratorTable) so native Ctrl+C/V in TextCtrl/SearchCtrl/ComboBox is preserved. Visible File menu entries added for discoverability. Status bar feedback on every successful paste/copy. Removes the auto-copy-on-select binding in GUIInfoPanel that silently overwrote the clipboard on every stats row click. https://claude.ai/code/session_01RusnXtXqo9DDz4dX2rUhXB
…tation PR #2 (now in dev) contained an earlier, simpler Z/color and 3D implementation. PR #4's 13 commits are the polished, final version that supersedes PR #2 entirely. Conflicts resolved in 3 files (13 conflict blocks total) by keeping HEAD: - GUIPlotPanel.py: _patch_3d_ctrl_rotate (3-layer toolbar-aware approach), ColorCtrlPanel (btFree + cbPlot3D + camera state), plotSignals (shared z_norm, logZ/flipZ, single colorbar per axis), _store/_restore_limits (3D camera + zlim preservation). - GUISelectionPanel.py: _ClampedComboBox for comboZ, zSel parameter handling. - plotdata.py: auto-merged cleanly (no manual resolution needed). https://claude.ai/code/session_01TRfZkFW9YczDyobkh3cndo
…s, 3D controls - Append 3D navigation note (Rotate, plane views, Home) to Z/color bullet - Add 5 new Workflow bullets: BG image, manual axis limits, Ctrl+V paste, context-aware Ctrl+C copy, Recent Files - Add 4 bullets to Features list covering same new capabilities - Remove duplicate Z/color and 3D view entries from Plot options section https://claude.ai/code/session_01TRfZkFW9YczDyobkh3cndo
W2 — GUIPlotPanel.py: add missing '1.75' to LWChoices in restoreViewData;
restoring a view saved with LineWidth=1.75 previously raised ValueError.
W3 — GUIPlotPanel.py: captureViewData now reads plot3D_type from the live
cbCurveType widget (not the always-hidden cbPlot3D), so 3D type is
correctly captured when saving a view in 3D mode.
W1 — main.py: remove dead onRestoreViewFromCombo method; it referenced a
comboViews widget that was never created (views are restored via menu
lambdas calling onRestoreView directly).
M1 — GUISelectionPanel.py: fix off-by-one iFilt<=len → iFilt<len in
setGUIColumns; passing len(columnsY) to SetSelection is out-of-bounds.
M2 — main.py: remove dead self.restore_formulas (real mechanism uses
formulas_backup).
M3 — GUIToolBox.py: remove unreachable docstring after return in GetKeyString.
M4 — GUIToolBox.py: remove debug print('MPL VERSION:') that fired on every
toolbar instantiation.
tests: add TestPlotPanelViewData (9 headless tests) covering W2 LWChoices
consistency, W3 curveType-over-plot3D_type restore path, axis limits dict
structure, and axis limits JSON round-trip. Total: 52 → 61 tests.
https://claude.ai/code/session_01TRfZkFW9YczDyobkh3cndo
Two bugs fixed:
1. _toggle_rotate (GUIToolBox.py): when the Rotate toggle was turned OFF
it activated zoom mode, so left-drag box-zoomed rather than panned.
Now it activates pan mode instead, making the tooltip text ("When off:
left-drag pans, right-drag zooms") accurate.
2. _patch_3d_ctrl_rotate (GUIPlotPanel.py): the pan implementation set
ax.button_pressed=2 hoping Axes3D would pan, but button 2 triggers
zoom in modern matplotlib. Replaced with a proper manual pan: on
left-press in pan mode the start position and axis limits are stored;
on each mouse-move the screen delta is projected onto the camera's
screen-right / screen-up vectors (derived from ax.azim / ax.elev)
and applied as a shift to xlim3d / ylim3d / zlim3d. x/y/z key
constraints are preserved. A button-release handler clears the pan
state. A fallback motion handler is added for old matplotlib builds
where _on_move is not found in canvas callbacks.
https://claude.ai/code/session_01GNbfFRd1uPxiEHSHYZkE6t
When two files with the same basename were loaded from different directories (e.g. /dir1/data.csv and /dir2/data.csv), _tab_shortname returned 'data' for both. captureViewState keyed tabSelectionsFull by shortname so the second table's entry silently overwrote the first, causing wrong y-axis and z-axis selections after view restore. Add _unique_tab_keys(tab_list) which uses the portable shortname when it is unique across all loaded tables, and falls back to the full tab.name when two or more tables share the same shortname. Use this helper consistently for tabSelectedNames, tabSelectionsFull, and formulas_state in captureViewState. The restore path (_find_tab_by_key, t.name fallback in table-selection rebuild) already handles full-name keys correctly, so no restore-side changes are needed. https://claude.ai/code/session_01JQiNFHSjaNDv2VpPE2DzLV
- Right-drag in 3D pan or rotate mode now zooms (exponential scale about axis centre, matching matplotlib 2D convention) instead of doing nothing - Pan and Rotate buttons are now fully mutually exclusive: activating one forces the other off, with defensive ToggleTool calls to keep visuals in sync - Rotate-off no longer auto-activates pan; returns to default zoom mode - Tooltips updated: Pan shows When-on/When-off sections; Rotate shows "Rotation for 3D: left rotates, right zooms" https://claude.ai/code/session_01GNbfFRd1uPxiEHSHYZkE6t
Panel (sash) widths are a UI/workspace preference and should not be coupled to named views. Restoring a view was overriding the user's current panel layout, which is unexpected. Remove the sashWidths capture block from captureViewState and the corresponding restore block from restoreViewState. Old view files that already contain a sashWidths key are unaffected — the key is simply never read. https://claude.ai/code/session_01JQiNFHSjaNDv2VpPE2DzLV
fix 3d panning
fix view restore duplicates
3D, views, recent files, background, axis limits, copy and paste shortcuts, fixes
Switching between 'Fixed' and 'Moving with axes' background modes went through redraw_same_data -> plot_all -> set_axes_lim, which unconditionally reset axis limits to data bounds and lost the user's current zoom/pan. This made the background appear to snap back to full size when entering Moving mode after zooming in. Capture the per-axis xlim/ylim before redraw and reapply them after, so the visible viewport (and the just-captured _bg_extent in Moving mode) stay in sync across the mode switch. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
The previous attempt at preserving the viewport went through redraw_same_data -> plot_all -> set_axes_lim, which (with AutoScale on, the default) reset the axes to data bounds and additionally rebuilt the imshow artist. The post-hoc _apply_view did not reliably stick because plot_all's other axis-limit machinery (plotSignals, FFT/Compare, user_lim, etc.) interleaves with the redraw, and the rebuild itself causes a visible "snap to full size" of the background. A mode toggle is purely a state change — there is no need to replot. Tag the bg image artist when it is created so it can be located later, then in onBgModeFixed/onBgModeMoving simply find each axis's tagged image and set its extent (current xlim/ylim for Fixed, captured _bg_extent for Moving), flip _bg_glued, and request a single canvas.draw_idle. The existing xlim_changed/ylim_changed callback is already mode-aware (early-returns while _bg_glued is True), so it correctly keeps the image in sync going forward in either mode. Also unify imshow creation in plot_all so the lim-changed callback is registered in both modes — otherwise reloading data while in Moving mode would create a new artist without the callback, breaking the subsequent switch back to Fixed. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
Previous fix re-wrote the image artist's extent to the current xlim/ylim on entering Fixed mode. If the user had previously been in Moving mode and zoomed into a sub-region of the background, this collapsed the whole image into the tiny viewport — the user described it as the background "resetting to full size" because the entire image suddenly became visible (squeezed) instead of staying as the magnified portion. Make onBgModeFixed a pure state flip: just set _bg_glued = False (and clear _bg_extent so a future plot rebuild doesn't pin the new artist to the old captured rectangle). The image keeps whatever extent it had, and the existing xlim_changed/ylim_changed callback — now no longer short-circuited by _bg_glued — will start tracking the viewport on the next pan/zoom. Transition is visually invisible at the moment of switch. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
When the user types limits into the limits panel, they want a specific viewport — not for the background image to follow the new limits. Previously, in Fixed mode (_bg_glued=False), the xlim_changed callback fired during the redraw triggered by onAxisLimitChange and updated the artist's extent to the new viewport, so the bg "moved with" the limits. Snapshot each axis's bg image extent before the redraw and reapply it to the rebuilt artist after. The bg keeps its current visual data-coord position; subsequent toolbar pan/zoom still behaves per-mode (Fixed follows, Moving stays). https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
In Fixed mode the bg image was rendered in data coordinates with a callback that kept its extent equal to the current viewport. As soon as the viewport changed (toolbar zoom, limits panel, autoscale) the bg moved/resized on screen with the data. The user expects Fixed mode to mean "bg pinned to the plot rectangle": whatever portion of the bg was visible at the moment of switching should stay pixel-identical until they change modes again. Render the bg in axes (screen) coordinates via transAxes: - Moving : transData, extent = _bg_extent (unchanged) - Fixed-locked : transAxes, cropped image at _bg_axes_extent - Fixed-default : transAxes, full image at [0,1,0,1] (fresh-loaded bg) Switching Moving -> Fixed snapshots the visible portion of the bg (crop + axes-fraction extent computed from the artist's current data extent and the viewport) and re-creates each artist in transAxes. Any subsequent xlim/ylim change is now independent of the bg by construction. The xlim_changed/ylim_changed callback and the onAxisLimitChange snapshot/restore workaround (commit 4604707) are no longer needed and have been removed. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
Switching to Moving was rendering the full _bg_image at extent=current viewport, even though the user had been seeing only a cropped portion pinned in Fixed-locked mode. The result was a visible jump (the rest of the bg suddenly reappearing). Compute the new _bg_extent by mapping the prior axes-fraction extent (_bg_axes_extent, or [0,1,0,1] for Fixed-default) into the current viewport's data coords, and render the same image that was just on screen — the cropped image in Fixed-locked mode, the full image in Fixed-default. Both _replace_bg_artists and plot_all's Moving branch now use _bg_display_image when set, falling back to _bg_image. _compute_bg_screen_lock also pulls the source array straight from the existing artist (instead of always self._bg_image) and handles the transAxes case (re-clicking Fixed when already in Fixed-locked) by mapping its axes-fraction extent back to data coords before cropping. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
After a Fixed -> Moving switch the user could only see the cropped portion of the bg, even when panning, because Moving mode was rendering the cropped image. They want to explore the parts that were outside the prior viewport (axis labels etc.). Track which fraction of the original _bg_image the displayed crop represents (_bg_crop_box). On Fixed -> Moving, use that crop box plus the current axes-fraction extent and viewport to compute a data-coord extent for the FULL _bg_image such that its currently-visible portion lands exactly where the crop was on screen — visually invisible at the moment of switch, but the rest of the bg is now reachable by panning. _compute_bg_screen_lock composes the new crop fractions with the existing _bg_crop_box, so cropping repeatedly across mode switches always stays expressed relative to the original full image. https://claude.ai/code/session_01VfZebri6U3ebhr2fxBbAKK
background axis workflow
Fix tests
In pandas 2.2+, pd.to_timedelta(x, unit='s') creates a timedelta64[s] index. Resampling at sub-second intervals then fails because the bin endpoint (e.g. 500ms) cannot be losslessly cast to seconds precision. Switch to milliseconds for both the time index and the resample offset string, which gives sufficient precision for sub-second intervals and removes the now-unnecessary pandas version-check workaround. https://claude.ai/code/session_01WvnvMa5iWDmswWc7PwbZNQ
…ts-2CAqr fix: use ms precision for Time-based resampler to support pandas 2.2+
In modern pandas (>= 2.0), df.iloc[:,i] returns a copy rather than a view, so the tuple-unpacking assignment and in-place multiplication in change_units_to_WE did not persist back to the dataframe. Align the WE path with the already-correct SI path by capturing the return value and explicitly writing it back with df.iloc[:,i] = col_new. https://claude.ai/code/session_017KyqXDVvGeHEoE2YM74Qup
fix: use explicit column assignment in changeUnits WE flavor
np.testing.assert_equal raises ValueError when comparing a pandas Index directly against a list because the truth value of an array is ambiguous. https://claude.ai/code/session_01R1Zn4U6BEuWmXXeCPy1mJs
This reverts commit ff5540e.
tab.columns returns a numpy array (self.data.columns.values). Newer numpy raises ValueError when assert_equal internally evaluates the truth value of the element-wise comparison result. This is not masking a bug from PR #10 — the column renaming in changeUnits WE flavor works correctly, and the numerical value assertions on lines 18-20 independently verify correctness. https://claude.ai/code/session_01R1Zn4U6BEuWmXXeCPy1mJs
Claude/fix change units test 4 sems
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See SimonHH#4 (comment)
@ebranlard please have a look. As the changes are extensive feel free to set up a call to go through the different topics.