Lab notes · 2026-09-13

Prepending one anycast site: the first prepend does the work, the third does nothing

Every anycast operator has done it: a site pulls traffic it should not, somebody adds an AS-path prepend, and the ticket is closed when the graph looks better. How much did the prepend move, where did the traffic go, and did the second and third prepend do anything the first had not? We have a network to find out on. AS218833 announces one /24 and one /48 from 33 sites at one cloud provider plus a lab node in Frankfurt. We picked one site, changed its announcement in five steps, and measured the catchment after every step from the same 1,000 RIPE Atlas probes, IPv4 and IPv6 separately.

Read this first. All 33 sites hang off the same upstream (AS20473), so a prepend at one site is a change inside one provider's topology, and the sites next door are the natural place for the traffic to go. That is exactly the situation of anyone running budget anycast at one cloud, and it does not generalise to a network with its own transit mix. The site we chose is also not a random one: we took the site whose catchment was the most out of place, the one with the largest share of probes from another continent. That was Miami, with 37 of its 47 probes (79 %) in Central and South America. A prepend at a well-placed site would have less to move.

Left: Miami's share of answering probes per stage, IPv4 and IPv6. Right: worldwide median and p90 answer time per stage, flat throughout. — Miami's share of the world per stage (left) and what it cost everyone else (right). The medians do not move; the p90 does not move. One site out of 33 is invisible in the aggregate, which is the point of having 33.

Six stages, two hours

s0, baseline: Miami answers 22 of 914 IPv4 probes (2.4 %) and 27 of 922 IPv6 probes (2.9 %). 49 cases in total.
s1, one prepend: 6 and 3 left. 40 of the 49 have moved.
s2, two prepends: 4 and 2 left. Four more moved.
s3, three prepends: 4 and 2. Identical to s2, probe for probe.
s4, withdrawn: 0 and 0. The last five are gone.
s5, back to baseline: 24 and 29. Miami has everything back, plus a little.

Each stage is the same procedure: change the announcement at Miami, wait 15 minutes for BGP to settle, then one DNS query with NSID from every probe to the IPv4 and the IPv6 address, and compare against the baseline probe by probe. 13 September 2026, 09:27 to 11:36 UTC.

The first prepend does the work

One prepend took 16 of Miami's 22 IPv4 probes and 24 of its 27 IPv6 probes. The second took four more. The third took none: s2 and s3 are the same set of probes on the same sites with the same round trips. Whatever the third prepend was meant to convince had already decided, or does not look at path length at all. The five cases that stayed through three prepends sit in three networks: one is the provider's own AS, where the probe is effectively inside the data centre and no path length will ever make Atlanta closer than the room next door; the other two are a Colombian and a Honduran network that reach Miami over a path they either prefer by policy or have no alternative to. The withdraw moved them, so an alternative exists; they just do not weigh path length against it.

Three maps: at baseline Miami answers 49 probes in Central America, the Caribbean, Colombia and a few in Europe; after three prepends 6 are left; after the withdraw none. — Where Miami's probes went. Orange: answered by Miami. Teal: was on Miami at baseline, answered elsewhere now. Central America, the Caribbean and Colombia are Miami's catchment; a few European probes were there too and left with the prepends.

Where they went: next door, not around the world

The traffic did not scatter. On IPv4, 14 of the 22 went to Atlanta and the rest to Newark, São Paulo and Mexico City, two each. On IPv6, 13 went to Atlanta and 11 to Newark, nowhere else. That is the provider's backbone choosing the next exit for the same upstream, and it is why the medians did not move: the next exit is close. A prepend at a site that has a good neighbour is cheap for the users. A prepend at a site without one would send them across an ocean, which is what the staircase note saw when São Paulo appeared at stage 4.

The two families behave differently and we do not know why. IPv6 gave up 24 of 27 probes at the first prepend and split them evenly between Atlanta and Newark; IPv4 gave up 16 of 22 and sent almost all of them to Atlanta. Same site, same upstream, same probes. The provider routes the two families over different paths internally, and the prepend lands in different places in that topology. We show it; we cannot explain it from the data plane alone.

Stacked bars per stage: of the 22 IPv4 and 27 IPv6 probes on Miami at baseline, how many are on Miami, Atlanta, Newark, São Paulo, Mexico City, Los Angeles or the Frankfurt lab node at each later stage. — The same 49 probes at each stage. Atlanta takes most of them from the first prepend on; the withdraw adds Miami's last five and moves two IPv4 probes to Los Angeles.

What it cost the rest of the world: nothing

Worldwide median 25.5 ms at baseline, between 26.0 and 26.3 ms at every later stage, 26.3 at the control. IPv6 33.3 ms, then 33.3, 33.0, 32.2, 33.4, 33.2. The p90 moved by less than the run-to-run noise. Per continent, nothing moved beyond what the control stage moved on its own (South America's IPv4 median is 26.9 ms at baseline, 28.4 after three prepends and 31.3 at the control, with nothing changed), except Oceania, which has five IPv4 and seven IPv6 probes in this set, all in Australia, none of them ever on Miami, and where one probe's bad round trip is the median. We show Oceania in the chart with its n, and we do not read it.

Median answer time per continent at baseline, after three prepends and after the withdraw, IPv4 and IPv6. All flat except Oceania with five probes. — Median per continent at s0, s3 and s4. One site of 33 leaves no mark on any continent. Oceania is n=5 and n=7; that is jitter, not routing.

The control stage, or how big a change is not a change

Two hours after the baseline, with the same announcement everywhere, we measured again. 97.2 % of IPv4 probes and 98.1 % of IPv6 probes were on the same site as at baseline; 25 and 17 had moved, the median round trip had drifted by 0.01 ms, and not one probe had moved away from Miami. That is the noise floor of this setup: about 25 site changes per family and a millisecond of median drift between two runs that should be identical. Against it, the 16 to 27 probes that a prepend moved away from Miami are a clear signal; the 37 to 44 total site changes per stage are only just above the floor, because most of them are the same background churn the control stage shows. Anyone reading a catchment diff without a control run will see changes that are not there.

What the route collectors saw: one peer

We also watched the control plane. RIPE RIS has 334 IPv4 and 360 IPv6 peers that carry our prefix, 201 and 77 of them through the same upstream. Exactly one of them, a LACNIC peer in Montevideo, had its best path through Miami. It picked up the single prepend one second after we set it, 20473 218833 218833, and at two prepends it left Miami altogether for another exit of the same upstream. A Miami exchange peer switched its path at the first prepend as well, without a prepend ever appearing in it: the upstream had simply chosen another site on its inside. Every other peer already preferred a different site and recorded nothing. The withdraw at stage four produced no withdrawal anywhere in RIS, because 32 sites kept announcing. On IPv6 there is no trace of the experiment at all. Afterwards, 0 of 694 peers carry a prepend and the paths are back to what they were.

That is the useful negative result. A route collector tells you who originates a prefix and how the world reaches your AS. It cannot tell you which of 33 sites answers a given user, and a site that goes dark behind a healthy upstream is invisible to it. The 49 probes above are the only instrument that saw the prepend do anything.

Method, so you can check it

Announcement changes through the same Ansible role that built the sites: bgp_path.prepend(218833) n times in the export filter of the two provider sessions at Miami; for the withdraw, both sessions disabled in BIRD. 15 minutes of convergence after every change, then a lab check that the anycast address still answers, then a RIPE Atlas DNS measurement: one SOA query with the NSID option, UDP, per probe to 94.249.165.53 and 2a03:5840:161::53. The probe set is the same 1,000 probes as in the delegation cost note: drawn per country in proportion to population, only probes whose IPv4 and IPv6 both work, capped by what each country has. Measurements 210914508 to 210925532, public, six pairs. "Site" is the NSID the answer carries. Per probe and family we take the fastest answer with a serial. A stage costs 20,000 credits, the experiment 120,000. Control-plane view: RIS raw MRT dumps of all 23 collectors, 09:00 to 11:45 UTC, parsed with mrtparse. The runner, the BIRD template, the per-stage reports, the RIS events and the summary are in the lab repository; the numbers above were re-derived from the stored answers after the run.

The lab, its prefixes, etiquette and opt-out are at anycast.org. This was our own prefix and nobody's traffic but our probes'. If you run anycast DNS and want the same six-stage look at one of your own changes, before and after, from the same 1,000 vantage points: that measurement is what we sell.

What comes next

The same runner can pull the other levers this provider offers, communities and selective announcement, and measure them the same way. The more useful next step is the one we cannot do at one cloud: the same experiment on a prefix with two transit providers, where a prepend has somewhere to go that is not the same backbone. If you run such a network and are curious, the runner is ours and the probes are free.