Community Voting stays reliable by reducing the risk of overload, catching failures extremely early, recording all decisions and measures, and turning every incident into a lasting safeguard.
This is more a record of decisions and measures than an actionable play. It is meant to provide information on the strategy we take regarding online products’ reliability and 90% of its content is applicable to the entire SIA Website or, by the same measure, to any of our +10 self-managed production apps that “live” on web servers.
Community Voting stays reliable through three continuous processes:
Voting pages are built to stay fast when many people vote at once.
Heavy work is done ahead of time: when competition data is refreshed, the person refreshing it rebuilds the stored copy, so no voter waits for it. Pop-ups and reports do only the work they need.
Crawlers that add load without value are turned away before a page is built, and the rest are steered away from pages that never end. Visitors on slow or stalled connections are dropped instead of holding the server’s capacity.
Background jobs such as backups and scheduled tasks run apart from the websites, each with its own memory and time limit, so a runaway job can’t starve voting traffic.
Every change ships as a versioned release with a plain-language description of what changed and why.
Changes are drafted with a coding AI Assistant under the project’s written rules and reviewed by the Systems Designer before release. Server changes come with a step-by-step guide, checks to confirm they worked, and a way to roll back. Larger choices are written down as decision records, with the options that were turned down, so they can be revisited later.
The Systems Designer decides and shapes these safeguards and approves changes to the server and the sites.
Failures are made visible instead of being allowed to build up.
A health check confirms that the websites’ code and database answer, not just the cached homepage. Scheduled tasks and backups each report to an outside check, which sends an alert when a run fails or doesn’t happen. The uptime monitor watches the websites and the services they rely on. The server records how long each page takes and keeps the network figures needed to trace a slowdown to its cause.
Each automated job is expected to report its own success or failure. A job that fails, stalls or doesn’t start raises an alert instead of passing unnoticed. Slow runs and failed websites are logged with enough detail to act on.
Not every issue is caught by the system. Team members also report problems directly or by email, and IT Support handles these the same way as alerts.
IT Support receives the alerts in real time via Slack, email, and uptime apps, and is expected to act on them.
Nightly backups of the database, uploaded files and the server’s settings are kept at Amazon, with older versions kept for recovery. A guide covers restoring everything onto a new server, or bringing back a single file or part of the database.
An incident gets a quick fix first, often the same day, to restore service. It is then written up as an analysis of what happened, the evidence and the open items.
The analysis is prepared with a coding AI Assistant from the server’s logs (e.g. access, error, cron logs), and IT Support checks it and acts on it. Where a lasting fix is needed, it is recorded as a decision (i.e. ADR) and delivered in a later release. Each incident also feeds back into prevention and detection: what caused it gets a safeguard, and what hid it gets an alert.
IT Support handles incidents, from the first fix through the written analysis.
Every change is recorded in the published record in SIA Build. It’s a running log of each release, describing in plain language what changed and why.
The record is kept up to date as part of each release. The release notes and each plugin’s own notes are written alongside the change itself. The release process then copies them automatically to the published documentation. The overview page shows the ten most recent releases and links to the full history and to each plugin’s page.
Anyone can use the record to check what was done after an incident, to trace when a safeguard was introduced, or to see what’s already in place before planning the next voting season.
An AI coding assistant (e.g. Claude Code, Anthropic or Codex, OpenAI) works on the project alongside the team who designs and maintains it. It drafts code changes, analyses server logs after an incident, and writes the step-by-step guides, decision records and release notes. It follows written project rules covering code style, how changes are documented and how releases are numbered.
People decide what gets changed, review the work, and release it. Release-note titles and change descriptions written with an AI assistant end with [*], so they can be told apart from those written by hand.
These limits come from deliberate trade-offs between cost, upkeep and how fast recovery needs to be. The decision on backups records that recovery within an hour isn’t needed, and that a second backup system would add cost and upkeep without enough benefit. A lost server is therefore rebuilt from the nightly backups instead.
The complete setup and its management processes are right-sized to our team and can be managed by a small team (as small as one person) holding several roles.
There are relatively simple (depending on domain knowledge) and affordable measures that can be put in place further at the cost of setup complexity.
We use one server and it has a fixed (albeit very large) capacity. No second server takes over when it fails, and capacity doesn’t grow on its own when traffic does. A peak bigger than the server can handle will still slow it down or stop it.
The database shares the server with the websites as opposed to running on a separate managed database service (e.g. Amazon RDS), so the two compete for the same memory and processing power. There’s no automatic fail-over to a standby database, and the database can’t be scaled up separately from the websites.
Recovery takes time and can lose recent data. If the server is lost altogether it’ll be rebuilt from the backups, which takes hours, not seconds. The database is backed up nightly with no record of changes between backups, so up to a day of changes, votes included, could be lost from the server (they remain in reports).
Floods from bots or attacks can still overwhelm any number of servers. Nothing sits in front of the websites to absorb a sudden flood of automated traffic or an attack before it reaches the server. Blocking is mostly reactive: unwanted crawlers and attacking addresses are blocked once they’ve been spotted, partly automatically and partly by hand. A new crawler or a large, sudden attack can still use up the server’s capacity before it is stopped.
Each incident in the voting season was diagnosed and addressed within hours or days, and most were written up: