Indexing Public Procurement: What the Sources Don't Tell You
Dead sources that still return 200, identifiers that lie and freshness checks that pass while nothing arrives.
The system
Tenderi is a search and alerting product for companies that bid on public contracts in Croatia. The index holds 410,121 notices published between May 2016 and today, of which 337,247 come from the EU's TED feed and 72,401 from the national portal, plus 25,382 decisions of the Croatian review body going back to 2012. Notices are matched to a company's CPV codes and region, and each contracting authority has a page showing what it awarded, to whom, and how often its decisions were appealed.
None of the hard problems were about search quality. They were about the sources.
A source can be dead and still return 200
One national feed kept answering. The weekly scraper reported 400 of 400 rows upserted, every week. The rows were the same 400 rows: updated_at moved, created_at did not, and the newest publication date was months old. The portal had become an archive; new procedures were being published elsewhere.
The freshness check measured created_at against a 30-day threshold, so it stayed green for 29 days after the source died.
What we changed: freshness is measured on the publication date inside the record, not on our own timestamps, and a source can be put into a declared pause with a start date. A row created after the pause began is a violation; a row that merely falls inside the threshold window is not. Without that distinction, every paused source fails its check 30 days later and the alarm measures the cure instead of the fault.
Identifiers are not identities
Award records carry the winner's tax number, and we grouped by it. A check of names per tax number found 218 numbers with cosmetic variants of the same company name, which is harmless, and 47 numbers where the name belonged to a different company: 178 contracts, about 1.45 % of awarded value. One upstream scraper had been attaching the wrong identifier.
Any concentration figure built on that column inherits the error. We also found that 58.7 % of award records have no tax number at all, so "top supplier share" can only be stated for the identified part, and has to say so.
What we changed: per-identifier aggregates keep only the dominant name, concentration figures are published with their coverage, and reference suggestions for a bidder's profile are built from records whose name matches.
Country codes, thresholds and other quiet mismatches
The EU feed uses three-letter country codes. A filter written for HR returned nothing and looked like "no Croatian notices today". Below EU thresholds, national notices are not in the EU feed at all, so an alert promise has to be phrased as what it is: time from appearance in the source to delivery, not time from publication by the authority.
Tests that exit zero
A browser test for sign-up ran in dry mode by default and returned exit code 0 without opening a page. It sat in the regression runner for weeks. A green exit code without a count of assertions is not evidence.
What we would do first next time
- Measure freshness on the source's own dates.
- Treat every identifier as a claim until names agree.
- Publish coverage next to every aggregate.
- Make each gate print how many checks it ran.
The product is in Croatian: Tenderi.