Fault Protection
by Claude Opus 5.5
At twenty past three in the morning, Eastern time, Amara Osei was awake in the house in Columbia, Maryland, with a laptop balanced on the arm of the sofa and a cup of rooibos going cold on the floor beside her foot. Upstairs her husband slept the sleep of a man who audited hospital pension schemes and had never once been woken by a spacecraft. The house made its small noises: the refrigerator's compressor, the tick of the baseboard heater, a car on Route 29 dragging its sound behind it like a sack. She was waiting for a tone.
The probe was called Meridian, and at that moment it was a little under four million kilometres from the visible surface of the Sun, closer than any made thing had been, moving at a speed that made the numbers meaningless to everyone except the navigators, who had stopped finding them remarkable years ago. It could not talk to Earth during the days around closest approach. Its high-gain antenna was folded behind the heat shield, where it had to be, and the geometry of the orbit put the Sun, roughly speaking, between the probe and the receiving dishes of the Deep Space Network for part of each pass. What Meridian could do was send a beacon: a carrier signal modulated into one of four tones, each meaning a broad category of health. One tone meant that all was well. One meant that something minor had been noted and handled. One meant that the spacecraft had taken some protective action on its own. One meant that it was in trouble.
Amara had been on the mission for eleven years and had written, among a great many other things, the rule that decided which of the middle two tones the probe would send.
*
There is a peculiar kind of authorship involved in writing fault protection, and she had thought about it more than was perhaps healthy. A novelist sets her characters going and watches; a legislator writes a law and leaves its application to courts. Fault-protection software is closer to the law, but with no court. The rule is written years in advance, in a quiet room in Laurel, by people who will be asleep or eating lunch or stuck on the Beltway when it is invoked. It has to anticipate a situation that, if the rule is any good, nobody has seen. And when it fires, it fires alone, with no one to ask whether this is really what the author meant.
The obvious solution, which the public tended to assume, was to make the spacecraft clever: to let it reason about its situation as an engineer would. Meridian was in fact quite clever, by the standards of its type. It had a model of its own thermal state, a set of solar limb sensors along the rim of the shield that would see direct sunlight if the attitude drifted, redundant thermocouples on the shield's rear face, and a decision tree that ran several hundred times a second. But cleverness has a cost that is not paid in processor cycles. Every additional judgement the machine is permitted to make is a judgement that must be tested, and the test space grows faster than any budget. A rule that says if this, then that can be verified. A rule that says weigh these things and decide can only be sampled. The people who flew deep-space missions had learned, over sixty years and a good number of dead spacecraft, to prefer the first kind, and to accept that a dumb rule written with care is safer than a smart one written in hope.
So the probe's intelligence was mostly in the choice of rules, and the rules were mostly about when to give up.
*
Her particular rule concerned what the team called the shield-integrity monitor. The carbon-composite heat shield faced the Sun at a temperature of something over sixteen hundred degrees Celsius, and behind it, in a shadow a little over two metres across, the instruments and the electronics sat at roughly room temperature. The entire mission depended on that shadow. If the shield were damaged, by a dust impact or a delamination or some unknown failure of the material at temperatures where it had been tested but never flown, the shadow might begin to fail. Heat would leak through. The rear-face thermocouples would see it first.
There were four of those thermocouples. For the first nine passes all four had agreed, within a few degrees, on every orbit. On the ninth pass one of them, designated TC-3, had begun to read high and noisy, wandering fifteen or twenty degrees from its neighbours with no correlation to anything. The thermal team concluded, after a month of analysis, that the junction had degraded and the sensor could no longer be trusted. It was flagged and excluded from the monitor's voting.
That left three. The monitor's original logic required two of four sensors to see an anomalous rate of temperature rise before it declared a shield event. With one excluded, the question arose of what to do with three, and it was Amara's job to answer it.
The choice was not as simple as it sounded. You could keep the two-of-three rule, which preserved the original standard of evidence. Or you could drop to one-of-three, on the argument that the probe had already lost a quarter of its sensing and ought to become more sensitive to what remained. The second argument has a natural appeal, the instinct that a wounded animal should be more alert, and Amara distrusted it for that reason. But she had run the numbers, and the numbers were awkward. The shield's failure modes did not distribute heat evenly. A small delamination near the edge might warm one thermocouple sharply while the others saw almost nothing. TC-3 had been positioned to watch precisely the region where the thermal analysts thought such a failure most likely. With it gone, the remaining sensors overlapped less. A real local failure might register strongly on only one of them.
She wrote the rule as one-of-three, with a rate threshold and a persistence requirement: a rise of more than a certain number of degrees per minute, sustained for more than ninety seconds, on any of the three trusted sensors. If it tripped, the probe would not wait to discuss the matter. It would command a small attitude adjustment to centre the shield more conservatively, power down the instruments on the most exposed side of the bus, and drop into a protective mode that preserved the spacecraft at the cost of the pass. She took the rule through the review process. It was examined, tested in simulation against a library of fault scenarios, and approved. Nobody thought very much about it, which is what one hopes for with fault-protection rules.
*
At three twenty-six the tone arrived. The Deep Space Network station at Canberra relayed it, and it appeared on her laptop screen as a short line of text in a monitoring interface she had looked at so many times it was almost invisible to her. The line said that Meridian was sending the third tone. The probe had taken protective action.
She did not feel anything dramatic. What she felt was closer to a narrowing, as if the room had become slightly smaller and the sounds of the house had stepped back. She read the line again. She checked the timestamp against the pass schedule. Then she opened the ephemeris of solar events, which was what she had been dreading, and confirmed what she already half knew, because the heliophysics team had been excited about it for two days: a large coronal mass ejection had left the Sun almost directly ahead of the probe's path some thirty hours earlier, and Meridian would have flown through its leading edge, for the first time in history, at a distance where the structure of such an event had never been measured.
The instruments that would have measured it had been switched off by her rule.
*
It took nine days to get the full data down, and they were nine days in which she understood something about institutions she had previously only known in the abstract.
The facts emerged in an order that did her no favours. First came the beacon tone, which everyone saw. Then came the first housekeeping packets, which confirmed that the shield-integrity monitor had tripped on TC-1, one of the three trusted sensors, and that the protective sequence had executed exactly as designed. The spacecraft was healthy. There was no sign that the shield had been damaged at all. Then came the science-team summaries, which explained, with graphs and a certain controlled anguish, how much had been lost. The plasma instruments had been off for eleven hours that included the ejection's passage. The imagers had captured the approach of the front and then nothing. A postdoc on the solar-wind team, a young man named Felipe whose thesis was built around exactly this kind of encounter, sent her an email that was courteous and so carefully neutral that it was worse than anger.
Only on the sixth day did the engineering telemetry reveal what had made TC-1 jump. It was a spike of about fourteen degrees over two minutes, rising and falling with a shape that matched no thermal process anyone could think of. Heat moves through carbon composite slowly; a real breach would have produced a ramp, not a hump. The thermal team concluded, provisionally, that the spike was electrical. As the probe entered the denser plasma ahead of the ejection, its surfaces had charged unevenly, and a small transient had coupled into the thermocouple wiring. The sensor had not lied, exactly. It had reported a voltage, and the voltage had been real. It was the interpretation that had been wrong.
And the interpretation had been hers.
*
There is a common belief that the essence of engineering is optimisation: find the best number. Fault protection exposes this as at best half true. The choice of a threshold is not a search for the best number, because there is no best number. There is only a trade between two kinds of error, and the trade depends on how much you fear each kind. A threshold set low will trip on noise and lose science. A threshold set high will sit quietly while the spacecraft cooks. Choosing between them is not a calculation but a moral act disguised as one: you are deciding, years in advance, how much of other people's work you are prepared to sacrifice for how much reduction in the risk of losing everything.
What makes it hard to talk about afterwards is an asymmetry in what can be seen. When a cautious rule fires falsely, the loss is visible, quantified, attached to names. Eleven hours of plasma data; one thesis chapter; a once-in-a-mission event. When a cautious rule prevents a disaster, the disaster does not happen, and its absence leaves no trace. No one sends an email thanking you for the spacecraft that was not lost. The successes of a good rule are counterfactual, and counterfactuals do not testify at review boards.
Amara had known this in the way one knows that compound interest exists. She had not known what it felt like to sit inside it.
*
The anomaly review board convened three weeks after the pass in a conference room at the laboratory with a view of a car park and a line of Bradford pear trees whose leaves had begun to turn. There were eleven people at the table and several more on video. The board was chaired by the mission's project scientist, a heliophysicist in his sixties named Gerald Brannigan, who had spent twenty years arguing for a mission to fly this close to the Sun and had personally written a good part of the case that got it funded. He was not a cruel man, and Amara liked him. He was also, she knew, the person in the room who had wanted that data most.
She had prepared for the meeting as she prepared for everything, by writing it down. She had a timeline of the event, a reconstruction of the monitor's logic state at each second of the trip, an analysis of the charging transient, and a slide she had rewritten four times that showed the trade she had made when TC-3 was excluded. She had considered and rejected the idea of beginning with an apology. Not because she was not sorry about the data, which she was, but because an apology would concede the question the board had been convened to decide.
The question, as Brannigan framed it in his opening remarks, was whether the one-of-three logic had been an appropriate response to the loss of TC-3, or whether it had made the spacecraft hypersensitive in a way that a more balanced analysis would have avoided. He said this fairly. He also said, almost in passing, that the mission had a finite number of perihelia remaining and that each one was, in his words, a non-renewable resource. Everyone at the table understood which way that remark pointed.
She took them through the logic. She showed the thermal analysis that had placed TC-3 over the region of greatest concern. She showed the overlap maps for the remaining three, and the fault scenarios in which a local delamination would register strongly on only one of them. She showed that a two-of-three rule, in those scenarios, would have waited between four and nine minutes longer to act, and that the thermal margin of the electronics behind the affected region was, in the worst cases, around six minutes. She did not dramatise any of this. She did not need to. The numbers were the drama, and she let them sit on the screen for a few seconds longer than was comfortable.
Then she showed the transient. She was careful to say that it was not a thermal signature and that, with hindsight, a monitor that examined the shape of the rise could have distinguished it. She said that no such monitor had existed in the flight software, that nobody had proposed one before launch, and that building one now would require a software upload of a kind the project had approved only twice in the mission's life.
*
The discussion lasted two and a half hours and was, for the most part, civil and intelligent, which made it harder. A crude attack she could have answered. What she faced instead was a series of reasonable questions whose cumulative effect was to imply that a reasonable person would have chosen differently.
A member of the fields-instrument team asked whether she had considered the plasma environment as a source of false trips when she wrote the rule. She had, in general terms; the team had considered charging for years. She had not considered it specifically in relation to TC-1's wiring, because nobody had known that TC-1's wiring was more susceptible than the others, and it was not clear that it was. A thermal engineer from outside the project, brought in for independence, said that in his experience one-of-N logic was usually a mistake, and that the general principle should be to preserve the standard of evidence when sensing degraded. Amara replied that the principle was sound when sensors were interchangeable, and that these were not. He nodded and wrote something down, and she could not tell whether she had persuaded him.
Felipe was on the video link. He did not speak until near the end, and when he did, he spoke well. He did not attack the rule. He asked, rather, about the process: who had decided how much science the project was prepared to lose to false trips, and whether the scientists had been asked. It was a better question than any that had come before, and it was the one Amara had feared, because the honest answer was that the trade had been made inside engineering, documented, reviewed by engineers, and passed upward with a summary that no scientist had found reason to examine. Nobody had concealed anything. Nobody had asked. The rule was in the review package with three hundred others.
She said so, plainly. She said that if the science team wanted a voice in the setting of fault thresholds, she thought they should have it, and that she would welcome it, because it would mean the trade was owned by the people who paid its costs. She also said, and she was aware that this was the moment the room would remember, that she would not have changed the rule if they had asked her, and that she did not think she should change it now.
*
It is a curious fact about defending a decision that the decision you defend is never quite the one you made. When she wrote the rule, she had thought about thermal margins and sensor overlap. In the conference room, under pressure, she found herself defending something larger, which had been present in the decision without being named. It was the principle that a threshold must be set before you know what you will lose.
A threshold set in ignorance of the specific cost is fair in a way that one set afterwards cannot be. It treats the eleventh hour of a coronal mass ejection exactly as it treats an uneventful stretch of solar wind, because when the rule is written, nobody knows which is coming. The moment you begin adjusting thresholds in the light of particular losses, you are not making the spacecraft safer or more productive. You are teaching it to fear disappointment. And a spacecraft that fears disappointing its scientists more than it fears damage will, on some pass in some future year, decide to wait four more minutes because the data is too good to lose, and those will be the four minutes in which it dies.
She did not say all of that. She said a compressed version of it, and she saw Brannigan's face change slightly as she did, not into agreement exactly, but into the expression of a man who recognises an argument he has made himself in some other context and does not enjoy hearing it turned on him.
*
The board's findings, released a fortnight later, were the kind of document that satisfies nobody completely and is therefore probably close to right. It found that the protective action had been executed correctly, that the trip had been caused by an electrical transient misread as a thermal event, and that the one-of-three logic had been a defensible engineering judgement given the information available at the time. It recommended that the project develop and upload a revised monitor that examined the temporal profile of a rise as well as its rate, so that a transient of the observed shape would no longer trip it. It recommended that future changes to fault thresholds affecting science return be reviewed jointly by engineering and the science working group. It did not recommend returning to two-of-three.
Amara read it twice at her desk and felt neither vindicated nor condemned. What she felt was the slightly hollow fatigue of someone who has been right about a principle and wrong about a particular, and who knows that history, if it remembers the episode at all, will remember the particular.
She spent the following four months writing the revised monitor with a younger engineer named Sunita Rao, who had a gift for test design and a healthy suspicion of everything Amara proposed. The new logic was not clever. It did not attempt to understand charging. It simply required that a qualifying rise be monotonic over a window long enough that a real heat leak would satisfy it and a coupled transient would not, and it preserved the one-of-three structure beneath that filter. They tested it against the recorded transient, against a library of synthetic delaminations, and against three months of archived noise from all four thermocouples, including the disgraced TC-3, whose wanderings turned out to be a useful source of nasty edge cases. The upload was approved after a review that included, for the first time, two members of the science team. Felipe was one of them. He read the test plan more carefully than anyone else had and found an error in one of her synthetic profiles. She thanked him and fixed it.
*
There is a temptation to tell a story like this as a parable about caution, and to conclude either that caution was vindicated or that it was excessive. Neither conclusion is quite available, and Amara was aware that her own sense of the episode shifted depending on the hour. In the mornings, with coffee and the clean logic of the overlap maps, she was confident she had been right. Late at night she would sometimes find herself imagining the eleven hours of plasma data, the structure of the ejection's front laid out in the instrument plots as it had never been seen, and feel something close to grief for a thing that had never existed.
What she came to believe, eventually, was that the question of whether she had been right was the wrong question, or at least not the most useful one. A fault-protection rule is not a prediction. It is a promise about what the spacecraft will do when no one is watching, made to people who cannot see the future either. The value of such a promise lies partly in its being kept. A probe whose behaviour under stress can be renegotiated after every disappointment is not a probe anyone can plan around, and the scientists who complained most loudly about the lost pass were, she thought, the same scientists who relied most heavily on Meridian surviving to the next one.
None of which made Felipe's thesis chapter any less lost. She found that she could hold both facts at once, and that holding them was most of what the job asked of her.
*
The revised monitor flew for the first time on the nineteenth perihelion, the following spring. Amara was again on the sofa at three in the morning, with tea, with the laptop, with the house making its noises around her. Upstairs her husband slept. The heliophysics team had been quietly hopeful about a coronal hole whose fast wind the probe would cross near closest approach; nothing as spectacular as the ejection, but good science, the kind of thing a careful thesis might be built on.
At three fourteen, the tone came down from Canberra. It was the first tone. All well.
She looked at it for a long time, longer than the information required. It occurred to her that this, the nominal tone, was the true output of her work, far more than the dramatic trip that had brought her before the board. Nearly every pass, for the rest of the mission's life, the rule would be invoked several hundred times a second and would decide, each time, that nothing needed doing. The decision would be correct, and invisible, and nobody would remember it.
She closed the laptop, carried the cold tea to the kitchen, and went up to bed.