Repair/phase 0 audit - #4
Conversation
Add a large set of experiment output artifacts (plots, per-seed tables, aggregated statistics) under experiments/results and experiments/results/plots; add experiments/results/statistics CSVs. Introduce a new script (scripts/generate_ablation_table.py) and update multiple analysis scripts (aggregate_multi_seed_results.py, generate_figures.py, generate_lambda_tradeoff_plot.py, generate_tables.py, run_evaluation.py) plus visualization code (src/visualization/node_performance_plot.py) to support multi-seed aggregation, CI/error bars, and updated plotting/aggregation flows. Replace FedAgent_Chain_Remediation_Plan.md with a condensed, phase-based publication hardening plan and update README, Makefile and .gitignore (ignore the remediation plan). These changes bundle results, plotting, and documentation updates to support reproducible multi-seed analysis and the repo hardening workflow.
Instrument clients and server to record runtime and communication metrics (time_local_training, time_blockchain, model_size_bytes/kb, time_aggregation, bytes_transmitted, mean_time_local_training) and compute total_communication_volume_mb. Add scripts/generate_system_overhead_plots.py to generate runtime_breakdown.pdf, communication_costs.pdf and CSV outputs. Commit includes the generated experiment artifacts (system_overhead.csv, communication_costs.csv, plots) and updates run_federated_simulation.py to print comm. volume on completion. README updated with a Systems Overhead table, scalability discussion, and instructions to regenerate plots.
Add reproducibility and scientific-hardening documentation, an experiments manifest, generated results, and evaluation tooling. Key changes: - Add docs/reproducibility.md and docs/scientific_hardening.md to describe verification steps, threats to validity, and ethical considerations. - Add experiments/manifest.yaml and multiple experiments/results artifacts (class_distribution.csv, demos, error_analysis.md, seed_42 tables and plots) and paper_figures PDFs to capture generated outputs. - Add scripts/generate_agent_demonstrations.py and scripts/generate_dataset_diagnostics.py to create qualitative demos and dataset diagnostics. - Update scripts/run_evaluation.py to import matplotlib/sklearn, generate confusion matrix PDFs, and invoke the new plotting function. - Update README.md with expanded project description, badges, dataset transparency, example scenarios, and links to the new docs. - Update CITATION.cff metadata and authorship/url/license information. - Minor .gitignore cleanup. These changes provide reproducibility guidance, dataset diagnostics, demo generation, and enhanced evaluation outputs (confusion matrices and regenerated seed_42 results) for Phase 4–5 of the project.
Add experimental artifacts and helper scripts for Phase 4/5. Includes new result markdowns (scenario_3_governance_risk_detection.md, error_analysis.md), confusion matrix PDFs under experiments/results/seeds, multiple paper figures (communication_costs, fairness_disparity, fl_convergence, lambda_tradeoff_ci, node_f1_scores, runtime_breakdown and confusion matrices), and two scripts: generate_agent_demonstrations.py (simulates three demo scenarios and writes markdown demonstrations) and generate_dataset_diagnostics.py (computes per-node class distribution and saves experiments/results/class_distribution.csv). These additions support qualitative demos and dataset transparency analyses.
Add docs/paper_results_inventory.md as a master catalog of all figures, tables, statistics, and qualitative demos for assembling the manuscript and appendix. Update README.md to include a new "Paper Writing Resources" section that links to the inventory, publication figures, and case study reports to help authors locate and prioritize assets for the paper.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a78d53248
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| fedavg_agg = load_aggregated_history("ablation_no_fairness") | ||
| fairness_agg = load_aggregated_history("fedagent_chain_full") |
There was a problem hiding this comment.
When generate_figures.py is run against a fresh results directory that only has the new results/seeds/seed_* outputs produced by the updated make run-evaluation, results/table_3_fairness_results.csv is absent and the fallback path is used. This block still references fedavg_history/fairness_history, but this change now only creates fedavg_agg/fairness_agg, so Figure generation raises NameError instead of completing the reproduction pipeline.
Useful? React with 👍 / 👎.
| if seed is not None: | ||
| pattern = f"{experiment_name}_seed{seed}_*" |
There was a problem hiding this comment.
The seed-specific glob now requires an underscore after the seed, but scripts/run_baselines.py writes baseline outputs as experiments/runs/baseline_local_seed42 and baseline_centralized_seed42 with no trailing timestamp/underscore. In the normal make run-baselines + make generate-figures flow, find_run_dir(..., "baseline_local", 42) returns None, so Figure 4 silently drops the Local Baseline and Centralized bars.
Useful? React with 👍 / 👎.
1 participant
Pull Request
Summary
Brief description of what this PR changes and why.
Type of Change
Related Issues
Closes #
Checklist
make lintpasses)make testpasses)CHANGELOG.mdconfigs/default.yamlTesting
Describe how the changes were tested:
Notes for Reviewers
Any additional context that reviewers should know.