Lab notes · 2026-09-06

1 to 33 anycast sites at one cloud provider: a staircase, not a curve

How many anycast sites do you need? Every answer we found was either a vendor's ("more") or an anecdote. So we measured it. One prefix, one ASN, one provider, and the same 1,000 RIPE Atlas probes asking the same DNS question at every stage: 1 site, then 2, 4, 8, 16, and finally all 33 Vultr regions. Total cost of the infrastructure: about 30 dollars.

Read this first. All sites hang off the same upstream (AS20473). We measured anycast inside one cloud provider's topology, not "the internet". For the audience this matters to, budget anycast running exactly like this, it is the relevant setup. It does not generalise to networks with their own transit mix, and the second half of this note shows why.

Median and p90 DNS answer time per stage, 1 to 33 sites, plus the share of probes slower than 150 ms — Median and p90 per stage (left) and the share of the world slower than 150 ms (right). Red: stratified sample. Grey: a plain Atlas sample, which is 58 % European.

The curve

1 site (New Jersey): median 106 ms, a third of the world above 150 ms.
2 sites (+ Frankfurt): 71 ms. Europe drops from 95 to 20 ms. Everyone else is still waiting.
4 sites (+ Singapore, São Paulo): 59 ms. North America gets slower, 38 → 58 ms. More on that below.
8 sites (+ Los Angeles, Tokyo, Sydney, Johannesburg): 31 ms. This is the step that matters: Oceania 222 → 24 ms, Africa 176 → 52, Asia 70 → 33. Eight sites on six continents is the point where it becomes an anycast.
16 sites: 29 ms. Almost nothing.
33 sites: 20 ms, p90 109. The European country sites (Paris, Amsterdam, Madrid, Stockholm, Milan) finally move the median again, because Europe is still the biggest block of probes even after re-weighting.

It is not a diminishing-returns curve. It is a staircase, and each step is exactly as tall as the population behind the new sites. Sites 9 to 16 were mostly second sites on continents that already had one; they bought tail latency (p90 174 → 155) and no median at all.

Heatmap of median answer time per continent and stage — Which stage fixes which continent. Africa and Oceania stop improving after stage 8: sites 9 to 33 are all elsewhere. The Middle East is *best at two sites* and gets worse from there.

Three things the curve hides

1. A plain Atlas sample tells you two sites are enough. RIPE Atlas has 14,900 connected probes; 57 % of them are in Europe, 237 in all of Africa (141 of those in South Africa). With the default "1,000 probes worldwide" selection, stage 2 looks like 38 ms and you conclude Frankfurt plus New Jersey is a global network. With a quota per continent and only dual-stack probes, the same stage is 71 ms. From stage 8 on both samples agree. Before that, the plain sample is simply flattering you.

2. Adding a site can make a continent slower. São Paulo at stage 4 pulled 22 % of North American probes and 10 % of European ones, at 160 ms. Mexico City at stage 16 did the same, catching Deutsche Telekom and Free customers at 130 ms. We turned Mexico City off: the same probes moved to São Paulo. We turned São Paulo off: they went home to Frankfurt, and 11 US probes moved to Santiago. Traceroutes from 120 Telekom probes explained it. Telekom reaches us through Lumen, and Lumen hands traffic to this provider only in Latin America. The magnet is not the site. It is the transit.

3. A region without a site gets worse with every site you add elsewhere. The Middle East was at 55 ms with two sites and 67 ms with 33, because every new Latin American or Indian site gave some Turkish and Gulf networks a "shorter" BGP path away from Frankfurt. Adding Tel Aviv at stage 33 fixed Israel (70 % of Israeli probes, 4 ms) and nothing else.

World map of 1,000 RIPE Atlas probes coloured by the Vultr site that answered them, 33 sites announced — 33 sites, one prefix: each dot is a probe, coloured by the node that answered. Stars are sites. Note the European dots in the Mexico City colour, and Honolulu with zero.

Sites that nobody sees

With all 33 announced, Honolulu answered 0 probes, Osaka 1, Manchester 3, Tel Aviv 2 (in the plain sample), Seoul 0. Past roughly 20 sites, a new site buys latency for its own country and nothing else, and only if that country's transit providers prefer it. Whether they do is not visible from a map. It is visible from a measurement.

Then we broke one on purpose

With all 33 sites up we dropped port 53 on the Frankfurt node and left BGP running. The result: 10 % of probes got no answer, 75 % of German ones, while the median of everyone who still got an answer improved from 20 to 18 ms, because the far-away Frankfurt probes had disappeared from the average. If your monitoring watches median latency, a dead site looks like an optimisation. Only the answer rate and the p90 tell the truth.

For comparison, a clean BGP withdraw of the same site left 0 probes without an answer after four minutes; they moved to Amsterdam, Johannesburg and a second provider's Frankfurt node. RIS Live showed the withdraw took 65 seconds to propagate from this provider's edge, and the re-announce one second. And the return was not symmetric: six German probes that had been in Frankfurt stayed in Mexico City afterwards.

Method, so you can check it

Prefixes 94.249.165.0/24 and 2a03:5840:161::/48, AS218833, announced from Vultr instances running BIRD and Knot. Every node answers with its own NSID.
Per stage: one DNS SOA query with NSID from the same 1,000 Atlas probes, IPv4 and IPv6 separately, UDP and TCP. Snapshots 60 and 240 minutes after each change on the first run of the ladder, 15 minutes on the second (we measured the withdraw wave at 66 s, so 15 is plenty).
Noise floor: two snapshots on unchanged infrastructure differ by ≤1 ms in the median and 24 of 925 probes change node. Everything above that is signal.
Sample: 1,000 probes with working IPv4 and IPv6, quota per continent (Europe 300, North America 250, Asia 200, South America 100, Oceania 60, Africa 50, Middle East 40), cap per country. We ran the whole ladder twice, once with the plain sample and once stratified. Both are in the charts.
What we did not measure: real resolver traffic. Authoritative DNS is asked by resolvers, not by eyeballs; Atlas probes are a proxy. That is the next experiment.

What we would actually run on this provider

Twenty-three sites: eight in Europe, eight in North America, Singapore, Tokyo, Delhi, Sydney, Johannesburg, Tel Aviv, São Paulo. We have not measured that exact set yet; with all 33 announced and the prepends below in place, the same probes saw a median of 18 ms, a p90 of 104 and 5 % of the world above 150 ms. Not Mumbai and not Bangalore (they attract US and Australian traffic at 200+ ms), not Mexico City (Dallas serves Mexico at 28 ms without the European leak), not Honolulu, Osaka, Manchester or Seoul. São Paulo with a 3× prepend towards Lumen, Johannesburg with a 1× prepend towards its South African transit: that sent Telekom and Vodafone customers back to Europe without taking a single route away from anyone. We know that because the first thing we tried, "do not export to Lumen", left eight US probes, Charter and Lumen customers among them, without any route to our prefix while BGP reported everything was fine. Measure after every change, or you will not know you just blackholed Charter.

The lab, its prefixes, etiquette and opt-out are at anycast.org. If you run anycast DNS and want to know where your traffic actually lands, from the same 1,000 vantage points, IPv4 and IPv6: that measurement is what we sell.