PoP State-Dependence Diagnostic Evidence Report (Step 4A)

This document reports the statistical methodology, empirical scoring results, calendar-block bootstrap inference, recency diagnostics, and decision gate for Step 4A: PoP State-Dependence Diagnostic.

The purpose of Step 4A is to test the fundamental assumption underlying Dynamics v2: \(\Delta_h \sim F_h \quad (\text{unconditional empirical transition bootstrap with sign symmetry})\) against the state-conditioned alternative: \(P(\Delta_i \mid x_o) \propto K(d(x_i, x_o))\) evaluated strictly out-of-sample across 3,610 rolling forecast cases spanning weekly origins from 2014-01-01 to 2026-08-23 across horizons $h \in {7, 14, 28, 56, 84, 112}$.

[!IMPORTANT] Predefined Step 4B Decision Gate: REJECT_STATE_DYNAMICS_KEEP_RC1

• Primary Estimand (Energy Score): Unconditional all-history Dynamics v2 decisively outperforms the state-conditioned 50NN model: \(\text{ES}(\text{Arm A: v2}) = \mathbf{1.26543} \quad \text{vs} \quad \text{ES}(\text{Arm C: 50NN}) = \mathbf{1.39616}\) \(\Delta \text{ES} = \text{ES}_{\text{v2}} - \text{ES}_{\text{50NN}} = \mathbf{-0.13073} \quad (6\text{-month Block Bootstrap } 95\%\text{ CI } [-0.16834, -0.09325], \quad P(\Delta > 0) = 0.0000)\)
• Marginal CRPS: Dynamics v2 is superior on marginal CRPS ($0.31196$ vs $0.34025$, degrading by $+0.02829$ under 50NN).
• Multi-Horizon Consistency: Dynamics v2 beats 50NN in 5 out of 6 horizons ($h=14, 28, 56, 84, 112$).
• Recency Finding: “Nearest historical states” have 69.61% overlap with the 50 most recent transitions (median neighbor age is only 32.0 days). Nearest-neighbor conditioning starves the empirical distribution of sample diversity and collapses into recency bias.
• Directional Finding: Raw signed transitions without sign symmetry perform terribly ($\text{ES} = 1.86674$, $\text{CRPS} = 0.45072$).
• Decision for the tested family: The evaluated state-conditioned alternatives are REJECTED for adoption. Unconditional empirical dynamics with sign symmetry (Dynamics v2) is retained for this comparison. This result does not establish a universal optimum or that all possible dynamics families have been searched.

1. Comparative Diagnostic Arms & Primary Scoring

All four arms were evaluated on 3,610 identical forecast cases using deterministic common random numbers ($M = 1,000$ draws per case). Every sampled transition $\Delta^{(m)}$ is applied to the origin state $x_o$ in CLR space and inverse-transformed to percentage vote shares ($x^{(m)} = \text{clr_to_composition}(\text{clr}(x_o) + \Delta^{(m)})$) before computing proper scores against the realized outcome $x_{o+h}$.

flowchart LR
    subgraph Progression["Model Performance Progression (Lower Energy Score = Better)"]
        Raw["Arm D (50NN Raw)\nES = 1.86674\nCRPS = 0.45072"]
        Recent["Arm B (Recent-50 ±)\nES = 1.44042\nCRPS = 0.34893"]
        NN25["25NN (±)\nES = 1.43079"]
        NN50["Arm C (50NN ±)\nES = 1.39616\nCRPS = 0.34025"]
        NN100["100NN (±)\nES = 1.35393"]
        V2["Arm A (Dynamics v2, All History ±)\nES = 1.26543\nCRPS = 0.31196\n(BEST)"]

        Raw --> Recent --> NN25 --> NN50 --> NN100 --> V2
    end

Key Statistical Findings:

1. Unconditional All-History Leads in This Comparison:
   • Dynamics v2 achieves the lowest Energy Score ($1.26543$) and lowest CRPS ($0.31196$).
   • In this evaluated sequence, the score decreases as neighbor pool size increases from $k=25$ (1.4308) $\to$ $k=50$ (1.3962) $\to$ $k=100$ (1.3539) $\to$ All History (1.2654). This is descriptive evidence for the tested specifications, not a general monotonicity theorem.
   • Restricting transitions to “similar states” starves the empirical distribution of sample diversity, artificially compressing tails and worsening calibration.
2. State Similarity is Mostly Recency:
   • 69.61% of the top 50 nearest neighbors are literally within the top 50 most recent transitions.
   • The median neighbor age is just 32.0 days. Because polling compositions evolve smoothly over time, nearest neighbors are overwhelmingly immediate predecessors rather than genuine historical analogues from prior decades.
3. Directional Drift Fails Completely:
   • Arm D (Raw Transitions) suffers massive score inflation ($\text{ES} = 1.86674$, $\text{CRPS} = 0.45072$).
   • Out-of-sample political opinion movements do not maintain predictable directional inertia; retaining sign symmetry ($\pm \Delta$) is essential.

2. Horizon-by-Horizon Breakdown

In this dataset, at all horizons $\ge 14$ days Dynamics v2 scores better than 50NN; the apparent horizon pattern is descriptive and should not be generalized beyond these tested cases.

3. Predefined Step 4B Gate Evaluation

4. Threshold Starting State Behavior (Descriptive Analysis)

Empirical transition distributions for parties starting in half-open intervals $[2.0, 3.0), \dots, [5.0, 6.0)$ across all 4,361 daily PoP observations are logged in threshold_starting_state_distributions.csv:

Key Takeaway: Threshold crossing probabilities are symmetric and governed entirely by standard diffusive variance ($\sigma \sqrt{h}$) rather than asymmetric tactical threshold drift. Median signed changes are virtually zero across all parties and bins.

5. Scope-limited conclusion

For the evaluated Step 4A alternatives, the state-dependence branch is closed:

========================================================================================
                          FINAL STATISTICAL EVALUATION SUMMARY
========================================================================================
1. Step 2 (Election Days):      Zero post-poll threshold jumps; near-threshold miss = -0.35 pp.
2. Step 3 (SCB Panel):          alpha = -0.0559, placebo = -0.0541 (delta_alpha = -0.0026).
                                -> Tactical voting branch CLOSED (CLOSED_NO_EVIDENCE).
3. Step 4A (State Dynamics):    v2 (1.2654) decisively beats 50NN (1.3962) and Raw (1.8667).
                                -> State-conditioned dynamics REJECTED.
========================================================================================
DECISION:
  Retain ElectionSimulator v1.0-rc1 for the evaluated state-dynamics comparison.
  No claim is made that all statistical development or all possible model families are complete.
========================================================================================

6. Processed Data Artifacts

All Step 4A artifacts are stored under data/processed/pop_state_diagnostics/:

• state_dependence_predictive_evaluation.csv: 3,610 case-by-case paired evaluation records across all 4 arms.
• state_neighbor_diagnostics.csv: 180,500 neighbor audit records logging distance, rank, and age.
• threshold_starting_state_distributions.csv: Empirical crossing rates and moments for half-open bins $[2,3) \dots [5,6)$.
• state_diagnostics_validation_report.json: Complete machine-readable QA report and bootstrap distributions.