Lab notes · 2026-09-26

Drains three and four: this time with real traffic

Withdrawn in one second, nothing left on the drained node, and 159 of 165 source prefixes back home with their first query.

In "Two drains" we followed Atlas probes through a drain and back, and the same ten probe entries never came back, twice. That lab had two sites and nothing on them but our probes. Since 20 September the lab answers real resolver traffic for 78 zones on 34 nodes, so on 23 September we drained two nodes again and watched the traffic as well as the probes.

Both drains ran the same way. Ten minutes of baseline, the routes withdrawn over sessions that stayed up, 15 minutes drained, then the routes back and 90 minutes of active measurement. Drain three was our Vultr node in Frankfurt (fra1), 18:10 to 18:25 UTC. Drain four was our second Frankfurt node, at Virtua (fra1-lab), 22:40 to 22:55 UTC.

The traffic cohort for each drain is every source prefix that reached only the drained node in the six hours before it. For fra1 that is 362 prefixes in 155 ASNs. For fra1-lab it is 111 prefixes in 69 ASNs, with 485 queries in those six hours. The fra1-lab cohort is thin, and most of what follows is about fra1.

What we found

The collectors saw the withdraw within a second at Vultr, and within 11 to 41 seconds at Virtua. RIPE RIS reported the fra1 withdraw 1 second after our action (2 withdraw messages). For fra1-lab the 14 withdraw messages arrived between 11 and 41 seconds, median 11. After the restore, the median announcement came 1 second later for fra1 (157 messages in 30 minutes) and 12 seconds later for fra1-lab (156 messages). We did not measure why the two upstreams differ.
Nothing reached the drained node while it was drained. From the first full minute of the drain to the restore, fra1 answered 0 queries from its cohort and 0 Atlas probes. At fra1-lab one IPv6 probe was still answered 2 seconds after the withdraw, then none. We count the minutes that contain the withdraw or the restore separately, so a query answered just before the switch is not counted as leakage.
159 of 165 prefixes came home with their first query. Of the 362 fra1 prefixes, 165 sent queries after the restore. For 159 of them the very first query after the restore went to fra1 again. 30 minutes after the restore, 5 prefixes in 5 ASNs were still being answered elsewhere.
Two bursts do not change that number. The cohort contains two short bursts from public resolvers, mostly on the registered ASNs of Cloudflare and Google: 80 prefixes that asked in the same minute, once before the drain and once after it. Without those 80 prefixes, 91 of 97 active prefixes went home with their first query, and the 5 prefixes in 5 ASNs are the same.
fra1-lab says the same, on six prefixes. Its cohort was quiet: 6 of 111 prefixes sent queries after the restore, and 6 of 6 went to fra1-lab with the first one. No prefix was still elsewhere 30 minutes later. This is consistent with fra1, not evidence on its own.
Losing one Frankfurt node scatters IPv6 across four continents. 12 Atlas probes saw only fra1-lab over IPv6 before the drain. When it withdrew they went to eight sites: Chicago (ord1) 3, fra1 2, Milan (mxp1) 2, and one each to Johannesburg (jnb1), Atlanta (atl1), Madrid (mad1), Stockholm (stob1) and Singapore (sgp1). All 12 came back after the restore. The node next door in the same city took 2 of 12.
The probes that did return, returned within one measurement interval. Atlas asked every 60 seconds in this phase, so "within a minute" is all we can say. For fra1 that was 5 of 5 probes on IPv4 and 2 of 3 on IPv6. One IPv6 probe was still answered from Amsterdam at the end of the active measurement. For fra1-lab it was 1 of 1 on IPv4 and 12 of 12 on IPv6.

Bar chart of the fra1 cohort's queries per 5 minute window from 17:10 to 20:00 UTC. fra1 carries almost all of them, drops to zero during the drain from 18:10 to 18:25 while a handful appear at other nodes, and carries the cohort again from the first window after the restore. — Queries from the fra1 cohort per 5 minute window, 23 September, 17:10 to 20:00 UTC. Red: answered by fra1, blue: answered by any other node. Grey: drained. Hatched: the two windows that contain the withdraw or the restore. Bars above 40 carry their value on top; three bars are clipped (920, 504 and 67), and 920 and 504 are the two public resolver bursts.

A word on the probe counts. We asked a fixed global set of 60 probes per family, and Frankfurt was the only home in the baseline for very few of them: 5 on IPv4 and 3 on IPv6 for fra1, 1 and 12 for fra1-lab. That is enough to see where probes go, not to put a percentage on it. The story here is the traffic.

What users see is decided by their resolver

We also asked each probe to resolve a name through its own resolver, 126 probe and resolver pairs. Before either drain, only 66 to 68 % of the probes reached the same node through their resolver as they did when asking our address directly. And 9 of 126 pairs before drain three, 12 of 126 before drain four, moved between nodes with nobody touching anything. For a user, the catchment belongs to the resolver, not to the user's own network.

What this does not show

The timings on the control plane compare our clock with RIS's. One announcement after the fra1 restore carries a time 4 seconds before our action, so differences of a few seconds are inside that uncertainty.
The time until a prefix is back at its node in the traffic is not a routing time. It contains the pause until that resolver asks again, which can be half an hour. We report the first query after the restore instead.
The traffic is resolver traffic for parking zones, not end users. Prefixes that did not ask anything after the restore say nothing either way: 197 of the 362 fra1 prefixes were silent.
Two drains, both in Frankfurt, both on one evening, European time. Nothing here says how often a network stays away elsewhere, and nothing says why the 5 prefixes did not come back within 30 minutes.
The Atlas cohorts are 1 to 12 probes. The one IPv6 probe that stayed away from fra1 is one probe, not a rate.

Check it yourself

The two runs are signed trails like the ones behind "Two drains", under keys published before each run. Both are published, log, checkpoints, timestamp tokens and a manifest of file hashes, at /witness/2026-09-23-fra1-t3t7/ and /witness/2026-09-23-fra1-lab-t3t7/, and /witness/ lists every published run with its key. The probe and routing numbers above can be checked against them and against RIPE.

Data: our own nodes on AS218833, drain three (fra1, Vultr) 23 September 2026 18:10:02 to 18:25:01 UTC, drain four (fra1-lab, Virtua) 22:40:02 to 22:55:00 UTC. Traffic after our cleaning rule (our own traffic and scanner-like sources removed), source addresses truncated to /24 and /48. RIPE Atlas measurements 214884611, 214884612, 214884631, 214904926, 214904927, 214904939 (drain three) and 214970143, 214970152, 214970275, 214984643, 214984648, 214984689 (drain four). RIPE RIS Live for the control plane. Trails 2026-09-23-fra1-t3t7 (34,234 entries, 144 checkpoints) and 2026-09-23-fra1-lab-t3t7 (34,187 entries, 142 checkpoints), both verify as PASS. RIPE Atlas data from atlas.ripe.net; where our numbers and theirs disagree, theirs are right.