Edvin Listo Zec

edvin listo zec

senior ml engineer, phd

Pollofpolls data

Pollofpolls data

This directory contains only data acquisition, normalization, and validation for the Swedish election project. It does not contain forecasting or simulation code.

Snapshot summary

  • Poll of Polls estimates: 4,361 daily observations, 2014-09-15 through 2026-08-23.
  • Reconstructed individual polls: 1,437 polls (14,370 long-format rows), with interview spans from 2009-01-02 through 2026-08-23.
  • Supplementary SwedishPolls dataset: 2,638 polls (26,380 long-format rows), published from 1980-11-30 through 2026-08-26; available interview spans run from 1973-01-29 through 2026-08-23.
  • Unique pollster/interview-span metadata matches: 1,183; unmatched or ambiguous Pollofpolls polls: 254.
  • Supplementary field coverage: publication date on 2,289 polls, interview span on 2,297, sample size on 2,481, and row source references on 461.
  • Parties/categories present in source values: C, FI, KD, L, M, MP, S, SD, V, other.
  • Pollsters present: Demoskop, Indikator, Inizio, Ipsos, Novus, SCB, Sentio, Sifo, Skop, United Minds, YouGov.
  • Supplementary parties reported: C, FI, KD, L, M, MP, S, SD, V.
  • Supplementary pollsters present: Demoskop, Indikator, Infostat, Inizio, Ipsos, Novus, SCB, SVT, Sentio, Sifo, Skop, TV4, United Minds, YouGov.
  • Retrieval method(s) for this raw snapshot: direct_repository_http, first_party_http.

These counts describe the checked-in raw snapshot and are regenerated by the refresh command.

Exact Pollofpolls sources

  • http://pollofpolls.se/ — rendered table of recent individual polls and their interview periods.
  • http://pollofpolls.se/skattat-opinionslage-for-poll-of-polls-fran-och-med-15-september-2014/ — first-party full HTML table of Poll of Polls estimates.
  • http://pollofpolls.se/poll_img/data_table_tot.csv — CSV linked from that table page as “CSV fil för tabellen”.
  • http://pollofpolls.se/poll_img/data_big_2.csv — M; declared by http://pollofpolls.se/opinionsutveckling-for-moderaterna/.
  • http://pollofpolls.se/poll_img/data_big_3.csv — L; declared by http://pollofpolls.se/liberalerna/.
  • http://pollofpolls.se/poll_img/data_big_4.csv — C; declared by http://pollofpolls.se/centerpartiet/.
  • http://pollofpolls.se/poll_img/data_big_5.csv — KD; declared by http://pollofpolls.se/kristdemokraterna/.
  • http://pollofpolls.se/poll_img/data_big_6.csv — S; declared by http://pollofpolls.se/socialdemokraterna/.
  • http://pollofpolls.se/poll_img/data_big_7.csv — V; declared by http://pollofpolls.se/vansterpartiet/.
  • http://pollofpolls.se/poll_img/data_big_8.csv — MP; declared by http://pollofpolls.se/miljopartiet/.
  • http://pollofpolls.se/poll_img/data_big_9.csv — SD; declared by http://pollofpolls.se/sverigedemokraterna/.
  • http://pollofpolls.se/poll_img/data_big_10.csv — FI; declared by http://pollofpolls.se/feministiskt-initiativ/.

Supplementary individual-poll source

  • https://raw.githubusercontent.com/MansMeg/SwedishPolls/master/Data/Polls.csv — the SwedishPolls project poll table.
  • https://raw.githubusercontent.com/MansMeg/SwedishPolls/master/Sources/sources.csv — its poll-level source-reference index.
  • https://github.com/MansMeg/SwedishPolls — documentation, license, and methodology.

SwedishPolls is retained as a separate supplementary dataset. Its support values do not replace Pollofpolls values. A unique normalized-pollster plus exact interview-start/end match may enrich the Pollofpolls row with publication date and sample size; the crosswalk records the match and any party-value differences. The original Pollofpolls source_url and support remain unchanged, while supplementary provenance is stored in dedicated metadata_* fields. SwedishPolls publishes its data under CC0 and code under MIT; its citation requests the repository revision/hash.

Of the matched rows in this snapshot, 122 polls have at least one non-identical party value across the two sources; the largest absolute party difference is 4.5 percentage points. This can reflect source revisions, rounding, FI/Other treatment, or an imperfect chart-based reconstruction. It is surfaced in the crosswalk and is a reason not to combine the two support series silently.

The home and party pages were inspected in normal page source. Each party page calls AmCharts.loadFile with the listed CSV. No endpoint name was guessed. During initial inspection on 2026-08-26 the HTTPS host presented the wrong certificate and a subsequent HTTP request received a transient 455 firewall response; the actual refresh later succeeded over the site’s canonical public HTTP URLs. The fetcher makes at most one live-host attempt after a refusal and opens a circuit breaker if it is blocked. It can then use a pinned public Wayback replay of the same first-party response. The retrieval manifest records both the original Pollofpolls URL and archive replay URL, final URL, capture time, selected HTTP headers, byte length, retrieval time, and SHA-256 hash.

Files and fields

raw/pollofpolls/ stores exact response payloads plus retrieval_manifest.json. Raw payloads are never edited. processed/pollofpolls/ contains:

  • pollofpolls_timeseries.csv: date, M, L, C, KD, S, V, MP, SD, FI, other, source_extra_json, source_url, and retrieved_at.
  • individual_polls.csv: deterministic poll_id, normalized and original pollster, separate interview/publication dates, party, normalized support, exact numeric source_value, reporting status, sample/method, source URL, retrieval time, and separate supplementary-metadata provenance.
  • swedishpolls_individual_polls.csv: separately normalized long-format copy of the supplementary table, including original company/house, approximate-period flag, sample size, source row, and poll-level source references.
  • pollofpolls_swedishpolls_crosswalk.csv: every Pollofpolls poll’s match status, candidate count, linked SwedishPolls ID, and source-value differences.
  • validation_report.json, dataset_summary.json, and normalization_metadata.json.

Dates use ISO YYYY-MM-DD. Support values are percentage points: 28.4 means 28.4%, not 0.284. poll_method remains null because neither acquired table supplies it. Unmatched Pollofpolls publication dates and sample sizes remain null; they are never inferred from another date or a non-unique match.

Historical naming and missingness

Folkpartiet (FP), Folkpartiet Liberalerna, and Liberalerna normalize to canonical L; the source header/label remains in raw data and normalization metadata. Pollster names normalize conservatively while pollster_original preserves the displayed name. For example, TNS-Sifo/Kantar-Sifo map to Sifo, and UM maps to United Minds.

FI begins only where its separate source series exists. Pollofpolls warns that some institutes include FI in Övriga. A displayed FI zero in an individual poll is therefore stored as source_value=0.0, support=null, and support_status=ambiguous_zero_or_included_in_other; it is not treated as a genuine zero. Other absent values remain null. The Poll of Polls CSV does not contain Övriga, so other is null there. Unknown future source categories are retained in source_extra_json rather than discarded.

SwedishPolls has no Övriga/Other column. Its Uncertain field is retained as uncertain_share and is not misclassified as party support. Consequently, a missing supplementary other is null and nine reported party values are only a subtotal.

Individual-poll reconstruction and limitations

The party-chart CSVs repeat each poll’s party value on every day of its interview period. The normalizer joins all nine party files and treats a contiguous, unchanged cross-party vector for one institute as a poll span. Recent homepage rows are merged by normalized pollster and exact interview span; this also supplies Övriga where displayed. This follows the public chart encoding, but two overlapping polls from the same institute cannot both be represented in one CSV column. Historical tracking polls or adjacent polls with an identical full party vector may therefore be merged, truncated, or otherwise ambiguous. No source value is corrected to address that ambiguity.

SwedishPolls documents uneven historical completeness: early data are lower quality, before 2000 the repository currently includes only Sifo, and the post-2008 period is more complete. It also documents that some Ipsos rows were normalized from totals of 101 and that sample size represents respondents rather than everyone contacted. Its source-reference index does not provide a URL for every row. These caveats remain supplementary metadata and are not used to alter Pollofpolls observations.

Each Poll of Polls row date is the model-estimate date; the source says the series ends at the latest included poll’s interview end, rather than its later publication date. The homepage supplies day/month interview spans but no publication date; years are anchored to the preserved response date and cross-year spans are handled explicitly. No date field is substituted for another.

Refresh and tests

From the repository root:

make fetch-pollofpolls
make test-pollofpolls

For a deterministic parse/validation run without network access:

make process-pollofpolls

The refresh stages writes atomically, compares hashes, preserves the original retrieved_at when upstream bytes are unchanged, normalizes both datasets, runs all validations, writes reports, and prints a concise coverage summary. Unit tests use saved fixtures; no normal test makes a live request.