Case study: the automatic updates that kept switching themselves off

Written by Roman Golovanov, an independent engineer who automates the work between a business and the data its partners send it.

Situation. A property listings business for new-build apartments. What it sells is freshness: on the large portals, listings are typed in by hand and go out of date, while here prices and availability come straight from the developers. Each developer publishes a feed, a file at a web address that lists every apartment with its price and status, in one of two formats: the XML format of a major property portal, or the business's own JSON. An importer had been downloading these files and updating the catalogue since 2025.

The request that started this work was one sentence: "I switch automatic updates back on, and they switch off again."

What was actually happening. I read the importer with a single question in mind: how does a developer stop being updated? There were five separate answers.

Any single error, even a feed returning an error page or a short database hiccup, raised a critical alert, and every critical alert switched that developer's updates off. A restart at the wrong moment could drop a developer from the schedule for good: one disappeared during a routine software update and did not come back for two weeks, while the admin panel still showed updates as on. Switching one developer on could knock another out of the queue, because the search for one developer's number also matched any number that contained it. Sometimes switching on did nothing at all while the screen said "Changes saved". And a feed address that stayed broken for about ten hours was dropped permanently.

On the way I found that an empty file would withdraw every complex that developer had, and that buildings were never withdrawn at all, because the step that should have done it failed on every run. The alerts named neither the error nor the feed it came from.

Risk. For this business a stale price means the product itself has failed. An apartment shown as available after it was sold, or at last month's price, is exactly what buyers came here to avoid. The usual sign of trouble was someone noticing that prices on the site had stopped moving.

First, stop the losses

Five small fixes went in on the day of the audit. Alerts lost the power to switch developers off, the search stopped matching the wrong developer, a program that was shut down no longer lost its task, an empty file could no longer withdraw anything, and buildings began to be withdrawn for the first time. That stopped the damage, but the importer was still four loosely connected background programs passing tasks through a queue that could lose them. So I rebuilt it as one pipeline in which every run takes the same steps: download, read, compare with the last accepted version, check, apply.

How the pipeline treats a file

One reader per format. Everything after it works with one internal model, so a new feed format needs a new reader and a dictionary, and the rules stay as they are. Developers also spell the same thing differently: "monolithic-brick" and "brick-monolithic" are the same walls. The dictionary translates these, staff can add a new spelling without waiting for a software update, and a value it does not recognise is counted and reported instead of guessed.

Only what changed is written. Each run applies the difference from the last accepted file, and an unchanged file is not even read. The default is a run every three hours; staff can set each developer anywhere from five minutes to once a day.

A suspicious file never replaces good data. An empty file is never applied. If more than 5% of the records cannot be read, nothing is applied and the site keeps the previous data. If more than 30% of apartments vanish at once, the updates go through but the removals wait until the same apartments are still missing on the third run in a row, or until a manager confirms them with one button.

Apartments and complexes are withdrawn differently. An apartment that drops out of the feed is marked sold on the first run, because it can no longer be bought, and goes back on sale the moment it reappears. A complex has to be missing twice, because withdrawing one by mistake takes every apartment in it off the site.

Where the system stops and people start

People's decisions win. A complex that first appears in a feed waits for a manager to publish it, since the business answers for every property advert on its site. Something a manager blocked or archived is never revived by a feed. Price is the exception: the feed always overwrites it, because the developer's current price is worth more than anyone's earlier edit.

The system can pause a developer, never switch one off. "On" belongs to a person. After 20 failures in a row and two days without a single success, the pipeline pauses that developer with a separate "paused" mark, and it lifts its own pauses once a day. An alert reports a problem and has no power to change anything.

Each failure goes to whoever can fix it. An unreachable feed server is retried after five minutes, fifteen, an hour and three hours, and reported after three failures in a row. A broken file is reported at once, to the business and to the developer. If the search service behind the site is down, the catalogue still updates, the changes wait in a queue and an alert goes out. A run cut short by a crash is repeated: its changes are saved all at once or not at all, and applying them twice gives the same result.

Tradeoffs. Selling an apartment on its first absence means a flaky feed can take one off sale until the next run. Releasing held removals on the third run means that, on the default schedule, a feed that stays broken for about six hours is treated as real sales; those apartments return as soon as the feed lists them again.

Result. The pipeline replaced the old importer in production in September 2026. Nothing in it can switch a developer off. A file that looks wrong is held with a reason in plain words, the kind a manager can forward to the developer as it is.

What this case is about. The first importer was automated, but it failed quietly while the screen said "on", and nobody was checking because the whole point was that nobody had to. Most of the work was deciding, rule by rule, what the system may do alone, what it must hold back, and whom it has to tell. That is the part I spend the most time on in automation work.

Still copying prices, stock or orders from partners' files into your own system? Or running an import that nobody fully trusts? Tell me where the data comes from and where it has to end up.