Add Student-t distribution and comparison mode to Validation.py - #2
Add Student-t distribution and comparison mode to Validation.py#2XCODESSS wants to merge 1 commit into
Conversation
Co-authored-by: XCODESSS <164984183+XCODESSS@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
📝 WalkthroughWalkthroughThe PR introduces a Distribution Model UI to the validation dashboard enabling selection between Normal, Student-t, or Compare Both distributions. The backtesting flow is expanded to execute distribution-specific Monte Carlo simulations, compute appropriate bounds, and store results accordingly. Result processing and reporting are enhanced to support per-model metrics and comparison analysis when comparing both distributions. Changes
Sequence Diagram(s)sequenceDiagram
participant UI as Validation Dashboard
participant Engine as Backtesting Engine
participant MC as Monte Carlo Simulator
participant Processor as Result Processor
participant Reporter as Report Generator
participant Plotter as Plot Engine
UI->>Engine: Select "Compare Both" + Historical Data
activate Engine
Engine->>MC: Run Normal Distribution MC
activate MC
MC-->>Engine: Normal Bounds & Prices vs Bounds
deactivate MC
Engine->>MC: Run Student-t Distribution MC (df param)
activate MC
MC-->>Engine: Student-t Bounds & Prices vs Bounds
deactivate MC
Engine->>Processor: Store Separate Normal/Student-t Results
deactivate Engine
activate Processor
Processor->>Processor: Compute Per-Model Hit Rates
Processor->>Processor: Analyze Interval Widths
Processor-->>Reporter: Processed Metrics for Both Models
deactivate Processor
activate Reporter
Reporter->>Reporter: Generate Model Comparison Narrative
Reporter->>Reporter: Determine Better Model Indicator
Reporter-->>UI: Display Comparative Results & Recommendations
deactivate Reporter
UI->>Plotter: Request Visualization
activate Plotter
Plotter->>Plotter: Plot Bounds & Prices for Both Models
Plotter-->>UI: Render Comparison Plots
deactivate Plotter
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@pages/Validation.py`:
- Around line 399-411: The interval-width comparison can raise ZeroDivisionError
when avg_width_normal is 0; in the block that computes avg_width_normal,
avg_width_student_t and width_difference (variables and DataFrame columns:
df['Interval Width Normal'], df['Interval Width Student-t'], avg_width_normal,
avg_width_student_t, width_difference), add a guard: if avg_width_normal == 0
(or nearly zero via a small epsilon) set width_difference to None or a sentinel
and render "N/A" instead of computing the percentage; update the Streamlit
writes that output width_difference so they print "N/A" when width_difference is
None (or use conditional formatting) to avoid division by zero.
- Around line 617-633: The plotting block assumes df_valid contains 'Lower
Bound', 'Upper Bound', and 'Within Bounds', which crashes when distribution ==
"Compare Both"; update the code that builds the plot (the block using dates =
pd.to_datetime(df_valid['Test Date']) and subsequent ax.fill_between/ax.plot and
hits/misses) to first detect distribution == "Compare Both" (or test for missing
columns in df_valid) and either select one model's result (e.g., choose a column
prefix or a selected_model variable) to create/alias 'Lower Bound'/'Upper
Bound'/'Within Bounds' for plotting, or skip the confidence-interval and bounds
rendering when those columns are absent; ensure you reference df_valid and the
plotting variables so later ax.fill_between, ax.plot and the hits/misses filters
always operate on existing columns.
- Line 420: The st.info call uses an unnecessary f-string prefix with no
interpolations; update the call in Validation.py (the st.info("ℹ️
**Recommendation:** Normal distribution is adequate for this stock/timeframe.")
invocation currently written as st.info(f"...")) by removing the leading f so it
becomes a plain string, and then run the linter to confirm Ruff F541 is
resolved.
| # Calculate interval widths | ||
| df['Interval Width Normal'] = (df['Upper Bound Normal'] - df['Lower Bound Normal']) / df['Starting Price'] * 100 | ||
| df['Interval Width Student-t'] = (df['Upper Bound Student-t'] - df['Lower Bound Student-t']) / df['Starting Price'] * 100 | ||
|
|
||
| avg_width_normal = df['Interval Width Normal'].mean() | ||
| avg_width_student_t = df['Interval Width Student-t'].mean() | ||
| width_difference = ((avg_width_student_t - avg_width_normal) / avg_width_normal) * 100 | ||
|
|
||
| st.write("---") | ||
| st.subheader("📏 Interval Width Analysis") | ||
| st.write(f"**Normal Distribution:** Average interval width: {avg_width_normal:.1f}% of stock price") | ||
| st.write(f"**Student-t Distribution:** Average interval width: {avg_width_student_t:.1f}% of stock price") | ||
| st.write(f"**Difference:** Student-t intervals are {width_difference:.1f}% wider") |
There was a problem hiding this comment.
Guard interval-width ratio against zero-width normals.
If avg_width_normal is 0 (flat-volatility or deterministic runs), width_difference raises a ZeroDivisionError. Add a guard and render an explicit “N/A” when the normal width is zero.
🔧 Proposed fix
- width_difference = ((avg_width_student_t - avg_width_normal) / avg_width_normal) * 100
+ if avg_width_normal > 0:
+ width_difference = ((avg_width_student_t - avg_width_normal) / avg_width_normal) * 100
+ width_diff_text = f"{width_difference:.1f}% wider"
+ else:
+ width_difference = 0.0
+ width_diff_text = "N/A (normal width is 0)"
@@
- st.write(f"**Difference:** Student-t intervals are {width_difference:.1f}% wider")
+ st.write(f"**Difference:** Student-t intervals are {width_diff_text}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Calculate interval widths | |
| df['Interval Width Normal'] = (df['Upper Bound Normal'] - df['Lower Bound Normal']) / df['Starting Price'] * 100 | |
| df['Interval Width Student-t'] = (df['Upper Bound Student-t'] - df['Lower Bound Student-t']) / df['Starting Price'] * 100 | |
| avg_width_normal = df['Interval Width Normal'].mean() | |
| avg_width_student_t = df['Interval Width Student-t'].mean() | |
| width_difference = ((avg_width_student_t - avg_width_normal) / avg_width_normal) * 100 | |
| st.write("---") | |
| st.subheader("📏 Interval Width Analysis") | |
| st.write(f"**Normal Distribution:** Average interval width: {avg_width_normal:.1f}% of stock price") | |
| st.write(f"**Student-t Distribution:** Average interval width: {avg_width_student_t:.1f}% of stock price") | |
| st.write(f"**Difference:** Student-t intervals are {width_difference:.1f}% wider") | |
| # Calculate interval widths | |
| df['Interval Width Normal'] = (df['Upper Bound Normal'] - df['Lower Bound Normal']) / df['Starting Price'] * 100 | |
| df['Interval Width Student-t'] = (df['Upper Bound Student-t'] - df['Lower Bound Student-t']) / df['Starting Price'] * 100 | |
| avg_width_normal = df['Interval Width Normal'].mean() | |
| avg_width_student_t = df['Interval Width Student-t'].mean() | |
| if avg_width_normal > 0: | |
| width_difference = ((avg_width_student_t - avg_width_normal) / avg_width_normal) * 100 | |
| width_diff_text = f"{width_difference:.1f}% wider" | |
| else: | |
| width_difference = 0.0 | |
| width_diff_text = "N/A (normal width is 0)" | |
| st.write("---") | |
| st.subheader("📏 Interval Width Analysis") | |
| st.write(f"**Normal Distribution:** Average interval width: {avg_width_normal:.1f}% of stock price") | |
| st.write(f"**Student-t Distribution:** Average interval width: {avg_width_student_t:.1f}% of stock price") | |
| st.write(f"**Difference:** Student-t intervals are {width_diff_text}") |
🤖 Prompt for AI Agents
In `@pages/Validation.py` around lines 399 - 411, The interval-width comparison
can raise ZeroDivisionError when avg_width_normal is 0; in the block that
computes avg_width_normal, avg_width_student_t and width_difference (variables
and DataFrame columns: df['Interval Width Normal'], df['Interval Width
Student-t'], avg_width_normal, avg_width_student_t, width_difference), add a
guard: if avg_width_normal == 0 (or nearly zero via a small epsilon) set
width_difference to None or a sentinel and render "N/A" instead of computing the
percentage; update the Streamlit writes that output width_difference so they
print "N/A" when width_difference is None (or use conditional formatting) to
avoid division by zero.
| 1. **Confidence intervals may be too wide** - The model is being overly conservative | ||
| 2. **Possible date indexing issue** - Verify that actual prices are being compared at the correct forecast dates | ||
| 3. **Model may be well-calibrated** - But intervals are wider than necessary for risk management | ||
| st.info(f"ℹ️ **Recommendation:** Normal distribution is adequate for this stock/timeframe.") |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, find the file
find . -name "Validation.py" -type fRepository: XCODESSS/Monte-Carlo-Simulation
Length of output: 96
🏁 Script executed:
# Check if the file exists and view the relevant lines around line 420
if [ -f "pages/Validation.py" ]; then
wc -l pages/Validation.py
sed -n '415,425p' pages/Validation.py | cat -n
fiRepository: XCODESSS/Monte-Carlo-Simulation
Length of output: 934
🏁 Script executed:
# Run ruff on the file to check for F541 violations
python -m ruff check pages/Validation.py --select F541 2>&1 || trueRepository: XCODESSS/Monte-Carlo-Simulation
Length of output: 880
Remove unused f-string prefix.
Ruff F541 correctly flags the f-string at line 420 as having no placeholders. The string contains no variable interpolations, making the f prefix unnecessary. Removing it resolves the linting violation.
Proposed fix
- st.info(f"ℹ️ **Recommendation:** Normal distribution is adequate for this stock/timeframe.")
+ st.info("ℹ️ **Recommendation:** Normal distribution is adequate for this stock/timeframe.")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| st.info(f"ℹ️ **Recommendation:** Normal distribution is adequate for this stock/timeframe.") | |
| st.info("ℹ️ **Recommendation:** Normal distribution is adequate for this stock/timeframe.") |
🧰 Tools
🪛 Ruff (0.14.14)
[error] 420-420: f-string without any placeholders
Remove extraneous f prefix
(F541)
[warning] 420-420: String contains ambiguous ℹ (INFORMATION SOURCE). Did you mean i (LATIN SMALL LETTER I)?
(RUF001)
🤖 Prompt for AI Agents
In `@pages/Validation.py` at line 420, The st.info call uses an unnecessary
f-string prefix with no interpolations; update the call in Validation.py (the
st.info("ℹ️ **Recommendation:** Normal distribution is adequate for this
stock/timeframe.") invocation currently written as st.info(f"...")) by removing
the leading f so it becomes a plain string, and then run the linter to confirm
Ruff F541 is resolved.
| # Plot bounds and actual prices (use valid results only) | ||
| dates = pd.to_datetime(df_valid['Test Date']) | ||
|
|
||
| # Confidence interval | ||
| ax.fill_between(dates, df_valid['Lower Bound'], df_valid['Upper Bound'], | ||
| alpha=0.3, color='#4A90E2', label='90% Confidence Interval', zorder=1) | ||
|
|
||
| # Plot bounds | ||
| ax.plot(dates, df_valid['Lower Bound'], color='#E74C3C', linestyle='--', | ||
| linewidth=2, alpha=0.8, label='Lower Bound (5%)', zorder=2) | ||
| ax.plot(dates, df_valid['Upper Bound'], color='#E74C3C', linestyle='--', | ||
| linewidth=2, alpha=0.8, label='Upper Bound (95%)', zorder=2) | ||
|
|
||
| # Plot actual prices | ||
| hits = df_valid[df_valid['Within Bounds'] == True] | ||
| misses = df_valid[df_valid['Within Bounds'] == False] | ||
|
|
There was a problem hiding this comment.
Compare Both mode will crash in the plot due to missing columns.
When distribution == "Compare Both", df_valid does not have Lower Bound, Upper Bound, or Within Bounds, so the plot block raises KeyError. Add a column selector (or plot both intervals).
🧩 Proposed fix (select a plot model; easy to extend for both)
- # Plot bounds and actual prices (use valid results only)
- dates = pd.to_datetime(df_valid['Test Date'])
-
- # Confidence interval
- ax.fill_between(dates, df_valid['Lower Bound'], df_valid['Upper Bound'],
- alpha=0.3, color='#4A90E2', label='90% Confidence Interval', zorder=1)
-
- # Plot bounds
- ax.plot(dates, df_valid['Lower Bound'], color='#E74C3C', linestyle='--',
- linewidth=2, alpha=0.8, label='Lower Bound (5%)', zorder=2)
- ax.plot(dates, df_valid['Upper Bound'], color='#E74C3C', linestyle='--',
- linewidth=2, alpha=0.8, label='Upper Bound (95%)', zorder=2)
-
- # Plot actual prices
- hits = df_valid[df_valid['Within Bounds'] == True]
- misses = df_valid[df_valid['Within Bounds'] == False]
+ # Plot bounds and actual prices (use valid results only)
+ dates = pd.to_datetime(df_valid['Test Date'])
+
+ if distribution == "Compare Both":
+ lower_col = 'Lower Bound Normal'
+ upper_col = 'Upper Bound Normal'
+ within_col = 'Within Bounds Normal'
+ interval_label = 'Normal 90% CI'
+ else:
+ lower_col = 'Lower Bound'
+ upper_col = 'Upper Bound'
+ within_col = 'Within Bounds'
+ interval_label = '90% Confidence Interval'
+
+ ax.fill_between(dates, df_valid[lower_col], df_valid[upper_col],
+ alpha=0.3, color='#4A90E2', label=interval_label, zorder=1)
+
+ ax.plot(dates, df_valid[lower_col], color='#E74C3C', linestyle='--',
+ linewidth=2, alpha=0.8, label='Lower Bound (5%)', zorder=2)
+ ax.plot(dates, df_valid[upper_col], color='#E74C3C', linestyle='--',
+ linewidth=2, alpha=0.8, label='Upper Bound (95%)', zorder=2)
+
+ # Plot actual prices
+ hits = df_valid[df_valid[within_col]]
+ misses = df_valid[~df_valid[within_col]]🧰 Tools
🪛 Ruff (0.14.14)
[error] 631-631: Avoid equality comparisons to True; use df_valid['Within Bounds']: for truth checks
Replace with df_valid['Within Bounds']
(E712)
[error] 632-632: Avoid equality comparisons to False; use not df_valid['Within Bounds']: for false checks
Replace with not df_valid['Within Bounds']
(E712)
🤖 Prompt for AI Agents
In `@pages/Validation.py` around lines 617 - 633, The plotting block assumes
df_valid contains 'Lower Bound', 'Upper Bound', and 'Within Bounds', which
crashes when distribution == "Compare Both"; update the code that builds the
plot (the block using dates = pd.to_datetime(df_valid['Test Date']) and
subsequent ax.fill_between/ax.plot and hits/misses) to first detect distribution
== "Compare Both" (or test for missing columns in df_valid) and either select
one model's result (e.g., choose a column prefix or a selected_model variable)
to create/alias 'Lower Bound'/'Upper Bound'/'Within Bounds' for plotting, or
skip the confidence-interval and bounds rendering when those columns are absent;
ensure you reference df_valid and the plotting variables so later
ax.fill_between, ax.plot and the hits/misses filters always operate on existing
columns.
1 participant
Updated pages/Validation.py to include Student-t distribution validation and a new "Compare Both" mode. This allows users to compare the performance of Normal and Student-t distributions in backtesting. The changes integrate the user-provided logic for displaying comparison results and calculating interval widths.
PR created automatically by Jules for task 3993904861077718799 started by @XCODESSS
Summary by CodeRabbit
Release Notes