Out of Sync: The Internet and Father Time

WWV Timekeeper 0:34

At the tone, three hours fifty-nine minutes, coordinated universal time.

Daina Bouquin 0:48

There is a radio station that does nothing but say what time it is. There's no music, no weather, no ads, no traffic updates, just a tone every second, and a voice on the minute announcing the time kept by the official atomic clocks of the US government. It's called WWV. It's been announcing the time since 1945. From Colorado since 1966. And you can call it on the phone. A voice tells you the time, and then the seconds tick off until the line hangs up on you two minutes later. And every one of those seconds that you hear is late. The tick has to travel from Colorado to your ear. And by the time you hear it, that moment is already over. In 1981, there was a computer in Maryland receiving a related time signal from Colorado. Other computers, some thousands of miles away, needed to know what time it was. And the machine in Maryland could tell them. Except its answers would be late too. You see, the clock wasn't broken. The radio was fine. But the time it sent would already be old when it arrived. The problem was you couldn't tell precisely how old. I'm Daina Bouquin, and this is Found in the Machine. I wear a mechanical watch. It doesn't have a battery. I have to wind it. And I wind it most mornings. But I do forget, and when I do, it stops. So I pull out the crown and turn the hands until they agree with my phone. It doesn't even have a second hand, and that's fine. It tells me when I'm late for things, which is about all I ask of it. But a computer has rather more to keep track of. When a file changed, when a message arrives, whether one event happened before another. And when computers work together, all of their clocks need to agree on when those things happen. Inside a typical computer, a piece of quartz vibrates when electricity passes through it. And the machine counts those vibrations and translates them into seconds. It makes it so the computer can keep time even when it's turned off. But quartz responds to temperature and it ages. Its rhythm wanders. Two computers that agree in the morning can disagree by sunset. Not much, not in a way that you'd ever notice, but enough. Now imagine those two computers are thousands of miles apart, exchanging messages across a network. One can send a message at noon and the other records its arrival one second after noon. But was the message traveling for one second? Or was the receiving clock fast? To measure the journey, you need clocks that agree. To make the clocks agree, you need to measure the journey. This was the problem that Dave Mills took up. And he loved it. Dave was born in Oakland in 1938 with glaucoma. Surgery preserved some sight in his left eye, but for much of his life he worked with his face inches from whatever he was trying to see. As a teenager, he became a ham radio operator, helping servicemen stationed at the South Pole communicate with their wives back home. And he liked clocks too. The machinery of them, the mathematics of them. Eventually, Dave went to work at ComSat Laboratories in Maryland in 1977, where he became involved with something called the ARPANET, one of the networks from which the internet would grow. And the digital world was relatively small then. Research computers in universities and laboratories connected through telephone circuits and satellites. Messages could be delayed by busy equipment or take different routes across the network. Dave wanted to measure the delays. And he had considerable freedom to investigate. He later described his research territory as his own little fife, a place where he could experiment and refine things largely on his own terms. By 1981, Dave had a radio receiver at Comsat listening to WWVB, another signal from Colorado. It could distribute that time to machines on experimental networks extending into Europe. To do it, Dave developed a method involving four timestamps. One computer records when it asks for the time. The other records when the request arrives and when its reply leaves. The first records when the answer comes back. Four moments from which a computer can estimate the delay and how far its clock is from the other. Not perfectly, though, the outward journey might take longer than the return. So the machines repeat the exchange, collecting measurements over and over, looking for the ones that are least disturbed by delays. But then there is another problem. What if the clock you're consulting is simply wrong? Dave called clocks whose time estimates agreed closely enough with one another true chimers. The ones that couldn't be trusted were false tickers. And he devised elaborate mathematical procedures for comparing them, accounting for uncertainty and setting aside sources that disagreed too much. But after a computer worked out what time it ought to be, there was the question of what to do about its own clock. What if its clock was wrong? With my watch, I can just move the hands. But a computer may have been keeping records all along. Turn its clock backward and events can appear to happen twice. Move it forward and you can leave a gap. So Dave developed ways to change the speed of the clock instead. When a clock was determined to be behind, it could count seconds just a little faster. One that was ahead could count them just a little slower. He could make them drift on purpose, gradually coming into agreement. He called his system the Network Time Protocol. NTP. An early specification appeared in 1985. By 1988, Dave had refined NTP enough to bring clocks into agreement within tens of milliseconds. A few thousands of a second. But still, he kept refining things. Sometimes the measurement would suddenly jump away from the others. A momentary disturbance that Dave called a popcorn spike. He devised a popcorn spike suppressor to deal with it. When network traffic made the delays unequal, there was a Huffin' Puff filter. He was completely serious about the names and the mathematics behind them. Dave wanted to know not merely that an algorithm worked, but why it worked and under what conditions it might fail. But meanwhile, the internet was growing around him. The computers moved beyond the laboratories into offices and home. They began keeping track of financial transactions, recording events on distant servers, coordinating work across ocean. Every machine still had its own imperfect clock. Every message still took time to arrive. But now there were millions of machines scattered across continents, and they were beginning to depend on one another. They didn't each need a radio receiver or an atomic clock. One computer could ask another for the time, which might ask another and another, all the way back through a hierarchy of machines to a reliable source like WWV. The network had become enormous, and the discrepancies Dave was measuring were almost unimaginably small, and his little fife was everywhere. This growth brought problems that had scarcely existed when Dave began. In 2003, a University of Wisconsin Time server was overwhelmed by requests for the time from hundreds of thousands of consumer routers. A mistake in their software had caused them to keep asking for the time again and again when they didn't receive an answer. The server was trying to tell them what time it was, but they were making it difficult for the server to do anything at all. NTP eventually gained a way for servers to tell clients to stop sending requests. They called it the Kiss-o -Death packet. The internet was no longer a place where one engineer could oversee everything. There were security vulnerabilities to address. New applications and people who wanted to change the software. Dave could be difficult to persuade. He demanded mathematical proofs even when other programmers had tested the change and found that it worked. His insistence on rigor had helped make NTP extraordinarily reliable. It could also frustrate the people trying to maintain it. One of them was Harlan Stenn, who began contributing to NTP in the early 1990s and eventually took responsibility for releasing its software. He and Dave worked together for decades, but they never met in person. They discussed the code over the telephone, trying to resolve problems in a system that had become increasingly difficult for anyone to understand completely. Dave knew why he made his decisions, and he had spent years working through the mathematics, revising the algorithms and testing their behavior. But he knew that he couldn't always explain his reasoning in terms his collaborators understood. And as the years passed, communicating that knowledge became even harder. Dave had spent much of his career with his face close to computer screens and circuit boards. But eventually his remaining eyesight failed. He continued working with software that read his computer screen aloud, but a mathematical equation is difficult to navigate by sound. Symbols that occupy different positions on a page arrive one after another. Finding your place can be painstaking. And other people were now developing their own approaches to keeping internet time. Dave continued working on NTP into his 80s. In 2020, he submitted proposals for improvements to the people developing the next version. But he struggled to get a real response. He spent hours at his computer listening to his work read back to him, sometimes asking family members to help him check what he'd written. And he knew that his proposals might not be adopted. When journalist Nate Hopper asked why he kept working on them, Mills just said, "because it's there." "I like to improve what I do." Dave Mills died on January 17th, 2024. He was 85. In Colorado, the radio station is still broadcasting. Every minute a voice announces the time. Every second, there's another tone. WWV was doing this when Mills was a child. It was doing it when he began distributing time across a handful of research networks, and when those networks grew to encompass billions of devices, it was doing it on the day he died. The signal still takes time to reach you and it always will. And everywhere around you, computers are asking one another what time it is. That answer too will always be a little late. But the machines will figure out the delay. They'll compare what they know. They'll make corrections. A little faster. A little slower. I am Daina Bouquin and this is Found in the Machine. If you like hearing my stories, please take a moment to leave a rating or a review wherever you're listening to this. It helps other people find the show, and so it helps me make this more sustainable. And if you want more content from me, you can go to foundinthemachine.com / support. You'll find a link to the show's Patreon. Patreon members get short bonus episodes every other week. You can also find their other ways to support the show. Thanks as always for listening, guys.