feat: improve collector robustness and add data virtualization search…#1
Open
HugoMSFT wants to merge 14 commits into
Open
feat: improve collector robustness and add data virtualization search…#1HugoMSFT wants to merge 14 commits into
HugoMSFT wants to merge 14 commits into
Conversation
- Path traversal: validate filename and realpath containment in CSV download - Thread safety: add threading.Lock around global mutable state - Connection leaks: add finally blocks to all DB methods in fabric_sql_writer and state_manager - Session leaks: add close() to all collector classes; call after collect() - Code injection: validate table_name and write_mode in PySpark payload generation - JWT handling: validate token structure before decoding payload - Input validation: add keyword count/length limits and Flask MAX_CONTENT_LENGTH - Replace ast.literal_eval with json.loads for CSV keyword parsing - Replace stderr debug prints with proper logger calls in fabric_writer - Remove unused xml.etree.ElementTree import and redundant inline json imports - Add exc_info=True to exception handlers missing stack traces - Remove overly-aggressive regex that stripped legitimate text in utils.py - Warn when FLASK_SECRET_KEY is not set instead of silently using default
fix: security hardening and code quality improvements
…sistence Local persistence: - Add src/local_store.py: thread-safe SQLite store (feedback + state tables, JSON list serialisation, CSV export/import, legacy CSV seeding). - Replace ad-hoc in-memory state with the LocalStore on every collection run; CSV becomes a snapshot, the DB is the source of truth. CSV exports: - Reorder TABLE_COLUMNS so Sources and Url are columns 2 and 3 (right after Feedback_ID); previously buried at columns 5 and 29. Taxonomy long-term persistence: - Save keywords / categories / impact_types to data/*.json (writable) instead of src/*.json. Cascading load order: data/ -> src/ -> in-code defaults. - Customizations survive application restarts, PyInstaller frozen builds (read-only src/), and source-tree upgrades. Collection progress drawer: - Add per-source state tracking (pending / running / success / error / skipped) on the server with a lock-protected helper, surfaced via SSE. - Replace the static 4-card grid in index.html with a dynamic per-source list rendered from collection_status.source_states; includes status badges, spinner for running, item counts, and a summary line. - Mark still-running / pending sources as "error" if the run aborts so the drawer never shows a stuck spinner. .env handling: - Rewrite .env.template to match the variable names actually read by config.py (REDDIT_*, GITHUB_TOKEN, ADO_*, optional Fabric / Flask). Frozen build (PyInstaller 6+): - Add src/ to spec pathex and pin module hiddenimports so run_web / app / config / etc. resolve at runtime. - Update runtime_paths._select_source_dir to probe sys._MEIPASS/src and PROJECT_ROOT/_internal/src layouts. - start_feedback_collector.py reconfigures stdout / stderr to UTF-8 (errors='replace') so emoji-laden log lines don't crash on cp1252-redirected streams.
…xonomy-persistence feat: SQLite-backed local store, per-source progress UI, taxonomy per…
Implement real repeat-collapsing behavior in feedback viewer and filtered API paths, fix identity handling in repeating analysis for exact duplicates, replace static FLASK secret fallback with generated token, and add regression tests for repeat collapsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix repeat collapsing and harden Flask secret fallback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add MCP server aliases section to AGENTS.md
Make SQLite authoritative, secure runtime and browser boundaries, add durable jobs and Fabric synchronization, strengthen packaging, and expand regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Build the expected path with the host path implementation so the Windows branch-selection test remains valid on Linux CI runners. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Prioritize SQL Server and Azure SQL across default sources and keywords, add Hacker News and DEV Community collectors, and isolate optional source failures so one integration cannot discard a run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add comma and newline separated subreddit configuration, normalize prefixes and URLs, migrate saved settings, and divide collection limits across communities. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Harden feedback collector end to end
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… coverage