[DISCUSSION] Example of property-based testing

[j-bryan]

NOTE: This is for discussion only

What is property-based testing?

• Property-based testing defines behavior that should hold across a range of valid inputs instead of checking only a few hand-selected examples.
• A framework such as Hypothesis generates inputs from declared strategies and evaluates the property repeatedly.
• When it finds a failure, Hypothesis attempts to shrink the input into a smaller, more understandable counterexample.
• In this PR, the test chooses a target IRR, constructs a conventional cashflow stream with that unique IRR, and verifies that DCAF recovers the manufactured result.

Why use property-based testing?

• It explores combinations of rates, project scales, payment distributions, and time horizons that would be impractical to enumerate manually.
• It is particularly useful for mathematical and financial code where invariants, conservation relationships, or independently manufactured
• It can expose edge cases and unstated assumptions even when the implementation is correct. This experiment revealed the distinction between NPV-residual tolerance and rate-error tolerance. @example(...).
• It complements deterministic example tests by covering a wider input space while retaining independently understandable claims.

What are the downsides to property-based testing?

• Failures may reveal ambiguous requirements, numerical tolerances, or invalid assumptions rather than implementation defects, requiring domain expertise to interpret.

• Large sample counts can increase CI time substantially; routine tests need a moderate budget, with larger exploratory campaigns run separately.

• Strategies and generated failures can be less immediately readable than fixed examples, adding some maintenance and debugging complexity.

  Walkthrough of the IRR property test

Property being tested

• For a conventional cashflow stream with one initial investment followed by positive returns, DCAF’s irr() function should recover the stream’s known, unique internal rate of return.
• The expected IRR is chosen before calling DCAF. The test then manufactures a cashflow stream whose IRR is mathematically guaranteed to equal that chosen rate.
• This is stronger than calculating an IRR and then checking that DCAF’s own NPV function returns approximately zero, because that could allow related errors in the IRR and NPV implementations to agree with each other.

Generated inputs

Hypothesis varies three inputs:

• target_rate: the known IRR, ranging from −75% to +100%.
• principal: the initial investment, ranging from $1 to $1 billion.
• allocation_weights: between one and eight positive weights controlling how the investment return is distributed across future years.

Varying these inputs exercises different rates, project scales, payment distributions, cashflow counts, and investment horizons.

Manufacturing a cashflow stream with a known IRR

1. The allocation weights are normalized:

   $$q_i = \frac{\text{weight}_i}{\sum \text{weights}}$$

   The normalized weights are positive and sum to one.

2. The test creates an initial cashflow of $-P$ on January 1, 2025, where $P$ is the generated principal.

3. For each future year i, it creates a positive return:

   $$CF_i = P \cdot q_i \cdot (1 + r)^i$$

   Here, $r$ is the generated target IRR.

4. The dates are placed on January 1 of successive years. Under DCAF’s default actual/actual convention, these are complete year intervals, keeping the manufactured calculation simple and isolating the IRR solver from irregular-date behavior.

Why the target rate is a root

For an arbitrary candidate rate x, the stream’s NPV is:

$$NPV(x) = -P + \sum \left[P \cdot q_i \cdot \frac{(1 + r)^i}{(1 + x)^i}\right]$$

When $x$ equals the generated target rate $r$, the growth and discount factors cancel:

$$NPV(r) = -P + \sum (P \cdot q_i)$$

Because the normalized weights sum to one:

$$NPV(r) = -P + P = 0$$

Therefore, the generated target rate is analytically known to be an IRR of the manufactured stream.

Why the root is unique

• The stream has exactly one sign change: one negative cashflow followed only by positive cashflows.

• For rates greater than −100%, increasing the discount rate strictly reduces the present value of every future inflow while leaving the initial outflow unchanged.

• The NPV function is therefore strictly decreasing over the valid rate domain and can cross zero only once.

• This restriction matters because cashflow streams with multiple sign changes can have multiple IRRs, making “the expected IRR” ambiguous.

The assertion

The test calls:

irr(stream, tol=1e-10)

and compares the result with the independently selected target_rate:

pytest.approx(target_rate, rel=1e-7, abs=1e-7)

The solver tolerance is intentionally tighter than the asserted rate tolerance. DCAF’s tol parameter controls the normalized NPV residual rather than directly specifying an allowable error in the returned rate.

Preserving a discovered case

• A larger exploratory Hypothesis campaign found a cashflow distribution near the original rate-error threshold.
• That case is now included with @example(...), ensuring it runs on every machine and CI invocation regardless of which inputs Hypothesis generates.
• Although the case did not reveal a financially meaningful IRR error, it prompted us to clarify the distinction between residual tolerance and returned-rate accuracy.

What this test does not cover

• Cashflow streams with multiple sign changes or multiple possible IRRs.
• Irregular cashflow dates and detailed day-count-convention behavior.
• Invalid inputs or failure behavior for streams without both inflows and outflows.
• Whether a project’s cashflows are economically modeled correctly before reaching the IRR solver.

This narrow scope is intentional: the test isolates one well-defined mathematical property, gives it an independent analytic oracle, and lets Hypothesis explore the valid input space around that property.

[j-bryan]

@dylanjm @caleb-sitton-inl I'd be curious to get your input on this. I think this style of testing could benefit the robustness of DCAF's calculations, but it would be a fair amount of effort to set up. My idea would be to focus this kind of testing on the metrics calculations since these are well-defined functions with known mathematical properties while also exercising a fair amount of DCAF's under-the-hood CashFlow primitives functionality.

[caleb-sitton-inl]

@j-bryan This sort of testing is definitely best-suited for identifying mathematical edge cases as opposed to logical ones, in my opinion. I think we can handle most logical edge cases with unit tests, and I don't think we could identify a significant number of cases with invariant outcomes resulting from alternative logical routes. Most of the input spaces for such situations will be small enough that unit tests could provide reasonable coverage.

Whether or not we implement property-based testing in DCAF, it would be of great benefit to our test suite to identify some of these cases where independent analytical oracles can confirm correct mathematical behavior. This is something we've been discussing a lot recently. I think that step of identifying independent oracles would likely be the greatest time commitment anyway if we chose to adopt some property-based testing, so attaching the Hypothesis engine would probably be a relatively small marginal commitment. If we're just focusing on mathematical cases, I don't know how many we'll even need. Perhaps a couple dozen?

An uncertainty that does concern me to some extent is the added testing runtime from property-based tests. If the addition of a couple dozen property-based tests would make our test runtime exceed two or three minutes, that could measurably reduce development efficiency. Since DCAF is mostly settled at this point, it may not matter much, but it would be a pretty sharp change from the near-instant pytest suite we have now.

[j-bryan]

@caleb-sitton-inl Regarding runtime, this kind of thing can be configured in Hypothesis. You can create different profiles (e.g. local vs CI testing) which use different numbers of samples by default to control runtimes. From running this on my local machine, the example which is configured to run 10,000 samples (which is total overkill!) runs in about 2 seconds. Dropping that to a more reasonable ~500 samples is trivial in terms of runtime.

I agree that this kind of testing is more likely to catch mathematical errors than logical errors. I'd also add that this kind of testing could help identify numerical precision issues (see issue #4).

[dylanjm]

This is a great idea and based on our discussion something worth adding to DCAF. We will save this for FY27 software hardening and testing.

[j-bryan]

Closing for now. This PR can be reviewed as an example implementation in FY27 when we can actually implement this kind of testing for a broader section of the codebase.