Site Reliability

Community Voting Reliability

Community Voting stays reliable by reducing the risk of overload, catching failures extremely early, recording all decisions and measures, and turning every incident into a lasting safeguard.

This is more a record of decisions and measures than an actionable play. It is meant to provide information on the strategy we take regarding online products’ reliability and 90% of its content is applicable to the entire SIA Website or, by the same measure, to any of our +10 self-managed production apps that “live” on web servers.

Ongoing Site Reliability

Community Voting stays reliable through three continuous processes:

  • Prevent: voting pages are built to stay fast under load, and the server is protected from crawlers, stalled connections and runaway background jobs.
  • Detect: every automated job reports its own success or failure, so problems raise an alert instead of building up quietly.
  • Respond: incidents get a quick fix, a written analysis and a lasting solution, and each one feeds back into better prevention and detection.

Prevent

Platform

Voting pages are built to stay fast when many people vote at once.

Heavy work is done ahead of time: when competition data is refreshed, the person refreshing it rebuilds the stored copy, so no voter waits for it. Pop-ups and reports do only the work they need.

Crawlers that add load without value are turned away before a page is built, and the rest are steered away from pages that never end. Visitors on slow or stalled connections are dropped instead of holding the server’s capacity.

Background jobs such as backups and scheduled tasks run apart from the websites, each with its own memory and time limit, so a runaway job can’t starve voting traffic.

Process

Every change ships as a versioned release with a plain-language description of what changed and why.

Changes are drafted with a coding AI Assistant under the project’s written rules and reviewed by the Systems Designer before release. Server changes come with a step-by-step guide, checks to confirm they worked, and a way to roll back. Larger choices are written down as decision records, with the options that were turned down, so they can be revisited later.

People

The Systems Designer decides and shapes these safeguards and approves changes to the server and the sites.

Detect

Platform

Failures are made visible instead of being allowed to build up.

A health check confirms that the websites’ code and database answer, not just the cached homepage. Scheduled tasks and backups each report to an outside check, which sends an alert when a run fails or doesn’t happen. The uptime monitor watches the websites and the services they rely on. The server records how long each page takes and keeps the network figures needed to trace a slowdown to its cause.

Process

Each automated job is expected to report its own success or failure. A job that fails, stalls or doesn’t start raises an alert instead of passing unnoticed. Slow runs and failed websites are logged with enough detail to act on.

Not every issue is caught by the system. Team members also report problems directly or by email, and IT Support handles these the same way as alerts.

People

IT Support receives the alerts in real time via Slack, email, and uptime apps, and is expected to act on them.

Respond

Platform

Nightly backups of the database, uploaded files and the server’s settings are kept at Amazon, with older versions kept for recovery. A guide covers restoring everything onto a new server, or bringing back a single file or part of the database.

Process

An incident gets a quick fix first, often the same day, to restore service. It is then written up as an analysis of what happened, the evidence and the open items.

The analysis is prepared with a coding AI Assistant from the server’s logs (e.g. access, error, cron logs), and IT Support checks it and acts on it. Where a lasting fix is needed, it is recorded as a decision (i.e. ADR) and delivered in a later release. Each incident also feeds back into prevention and detection: what caused it gets a safeguard, and what hid it gets an alert.

People

IT Support handles incidents, from the first fix through the written analysis.

Record of updates

Every change is recorded in the published record in SIA Build. It’s a running log of each release, describing in plain language what changed and why.

The record is kept up to date as part of each release. The release notes and each plugin’s own notes are written alongside the change itself. The release process then copies them automatically to the published documentation. The overview page shows the ten most recent releases and links to the full history and to each plugin’s page.

Anyone can use the record to check what was done after an incident, to trace when a safeguard was introduced, or to see what’s already in place before planning the next voting season.

How AI is used

An AI coding assistant (e.g. Claude Code, Anthropic or Codex, OpenAI) works on the project alongside the team who designs and maintains it. It drafts code changes, analyses server logs after an incident, and writes the step-by-step guides, decision records and release notes. It follows written project rules covering code style, how changes are documented and how releases are numbered.

People decide what gets changed, review the work, and release it. Release-note titles and change descriptions written with an AI assistant end with [*], so they can be told apart from those written by hand.

Known Limitations

These limits come from deliberate trade-offs between cost, upkeep and how fast recovery needs to be. The decision on backups records that recovery within an hour isn’t needed, and that a second backup system would add cost and upkeep without enough benefit. A lost server is therefore rebuilt from the nightly backups instead.

The complete setup and its management processes are right-sized to our team and can be managed by a small team (as small as one person) holding several roles.

There are relatively simple (depending on domain knowledge) and affordable measures that can be put in place further at the cost of setup complexity.

Server capacity

We use one server and it has a fixed (albeit very large) capacity. No second server takes over when it fails, and capacity doesn’t grow on its own when traffic does. A peak bigger than the server can handle will still slow it down or stop it.

Database is co-located

The database shares the server with the websites as opposed to running on a separate managed database service (e.g. Amazon RDS), so the two compete for the same memory and processing power. There’s no automatic fail-over to a standby database, and the database can’t be scaled up separately from the websites.

Speed to recovery

Recovery takes time and can lose recent data. If the server is lost altogether it’ll be rebuilt from the backups, which takes hours, not seconds. The database is backed up nightly with no record of changes between backups, so up to a day of changes, votes included, could be lost from the server (they remain in reports).

Unpredictable overloads

Floods from bots or attacks can still overwhelm any number of servers. Nothing sits in front of the websites to absorb a sudden flood of automated traffic or an attack before it reaches the server. Blocking is mostly reactive: unwanted crawlers and attacking addresses are blocked once they’ve been spotted, partly automatically and partly by hand. A new crawler or a large, sudden attack can still use up the server’s capacity before it is stopped.

Incidents and Solutions: 2026

Faster pages for voters

  • Venture pop-ups open about twice as fast. They no longer load each venture twice or build a full page behind the pop-up (21.195.0 and 21.197.0).
  • No slow first visit after a reset. Whoever refreshes a competition’s data rebuilds it, so the first voter doesn’t wait (21.196.0 and 21.198.0).
  • The Community Voting report on the country websites loads several times faster (21.201.0).

Capacity kept for real voters

  • Crawlers take up less capacity. TikTok’s crawlers are turned away before a page is built, and other crawlers are told to skip the endless calendar pages (21.207.0 and 21.207.1).
  • Bad mobile connections can’t freeze the server. Unresponsive visitors are dropped after 30 seconds instead of 5 minutes, and more visitors can be served at once (21.207.3 and 21.234.0).
  • Background work is kept apart from the websites. Backups run outside WordPress, and each scheduled task has its own memory and time limit, so a runaway job can’t starve voting traffic (21.230.0 to 21.234.0).
  • Slow and idle visitors no longer tie up the server. The web server moves to a setup where waiting connections don’t take up the capacity that active voters need (21.236.0).

Failures are seen early instead of building up quietly

  • Scheduled tasks can no longer fail silently. They run directly on the server, each website separately with its own limits, and an outside check sends an alert when they fail or don’t run (21.194.0, 21.231.1 and 21.234.0).
  • Uptime monitoring checks that logins and voting work. A new health check confirms the code and database answer, not just the cached homepage (21.234.0).
  • Slowdowns can be traced. The server records how long each page takes and keeps the network figures needed to diagnose a stall (21.207.0 and 21.234.0).
  • Pages switch on their dates without anyone clearing the cache by hand (21.206.0).

Every vote counted and recoverable

  • Votes reach the Airtable Community Voting reports complete. Batches are sized to stay within Airtable’s automation limits, and missed votes can be backfilled for a chosen period (21.204.0 and 21.205.0).
  • Daily vote totals are sent to the Airtable Community Voting reports every night, with midnight votes counted once (21.211.0).
  • The database is backed up nightly to Amazon and we created a restore guide (21.231.0 and 21.232.0).

Abuse closed off

  • No fake accounts through the email sign-up. The loophole used to create accounts while sign-up was switched off is closed (21.201.2).
  • The server no longer opens the activation links in the emails it sends, so accounts can’t be confirmed without the email being opened (21.201.3).

Incident response speed, as a steady process

Each incident in the voting season was diagnosed and addressed within hours or days, and most were written up:

  • **8 Sep: voting didn’t open on schedule.**
    Root cause found and fixed the same day.
  • **9 Sep: a crawler used up the server’s memory.**
    Blocked the same day.
  • **9 Sep: 192 votes missing from the Airtable Community Voting reports.**
    Batch fix and backfill shipped by 17 Sep.
  • **16 Sep: 54 scripted accounts.**
    Loophole closed the same day, written up.
  • **21 Sep: slowdown from crawler traffic and slow connections.**
    Crawler blocks and timing log shipped the same day.
  • **28 Sep: server froze on poor mobile connections.**
    Timeouts and capacity changed the same day, written up.
  • **7 Nov: two freezes caused by the backup.**
    Analysis on 9 Nov, backup switched off before its next run, replacement running by 14 Nov.
  • **19 Nov: HTTPS stalled for 4 minutes.**
    Write-up and first changes the same day, lasting fix in 21.236.0.