Sandcastles at Low Tide

1. The good part is over

Almost everything is getting a little worse. Not ruined, not collapsed — nothing so honest as that. Worse in the specific way you notice, sigh about, and then keep using: the app that used to help you and now begs for attention like a child who has been taught marketing; the streaming service that rescued you from advertising and then sold it back as a feature; the chocolate bar that is smaller and sadder and still wearing the same wrapper.

Search feels less like an answer than like being herded through an airport’s worth of shops on the way to the exit. The whole world has taken on the texture of a hotel breakfast buffet at ten minutes to ten. There is food, technically. There are choices, technically. Everyone can tell the good part is over.

It is tempting to blame greed, or decline, or bad luck arriving all at once — to assume that everyone in charge of everything got worse on the same Tuesday. The real explanation is simpler and a bit more depressing. A good thing starts by making you a promise. Then, once you depend on it, the promise quietly changes. First the company needs you to love the thing. Later it needs the advertisers and the investors and the shareholders to love the thing. You are still there — still clicking, watching, subscribing. You have just stopped being the customer and started being the crop.

The observation that started this piece is smaller and more technical than any of that: the falling share of known-dangerous software flaws that actually get fixed in time is real, and getting worse — which is part of what started all of this.

↳Nothing about that is mysterious. What is mysterious is how little of what we treat as permanent has ever been tested.

2. How a good thing goes bad

This was never really a story about apps. It is what happens to anything asked to grow forever inside a world that cannot keep giving.

   1. it serves you
      "free, and strangely good"
   2. you depend on it
      "all your stuff lives here now"
   3. someone else becomes the customer
      "sponsored results — part of the experience"
   4. you become the harvest
      "we’ve updated our pricing to serve you better"

The chocolate bar did not shrink because a man in a boardroom woke up evil. It shrank because profit had to rise, the market could not, costs went up, and the wrapper had already done all the lying a wrapper can do. What looks like a scandal is usually just a future that was already on its way, arriving on schedule, and surprising only the people who had not heard the drums.

That is the thought worth coming back to, because it generalises past software. A job, a skill, a rule, a way of feeling useful — they look solid because nothing has come for them yet. Maybe a lot of what we fear isn’t permanent at all. Maybe it is a temporary bargain that hasn’t been tested.

A sandcastle at low tide looks a lot like a building. It’s the tide that tells the truth.

↳Nothing in that requires anyone to decide badly. It only requires the promise to change.

3. What’s worth carrying inland

If the rule you followed brought you to this, of what use was the rule?— Anton Chigurh, No Country for Old Men

So before panicking about machines, or work, or control, or whatever the future is being called this month, there is a more useful question than is it good or bad.

Was it always fragile, and we’d just not yet met the thing that would reveal it?

That sounds bleak. It isn’t meant to be. Once you stop trying to save every sandcastle from the sea, you can start asking the only question that helps: what’s worth carrying inland?

The test is simple enough to explain to a child. When something frightens you, don’t start by asking whether it’s good or bad. Ask where it’s going — not where the brochure says, but where it ends up if everyone just keeps doing the normal, sensible thing they’re already doing. Then ask whether that ending is one you’d want. And if it isn’t, ask what other road gets somewhere better.

The test is really about cost. A thing is worth keeping if it gives back more than it costs to maintain. A sandcastle at low tide is cheap, joyful, temporary — good bargain. A sandcastle you’re defending against the incoming tide with your bare hands is a bad bargain getting worse by the minute. A surprising amount of modern life is people in expensive clothes yelling strategy at the second sandcastle.

↳Run it on what everyone is afraid of, and start with the loudest fear in the room.

4. The race to build a mind beyond us

The road

Everyone keeps bidding, because stopping means losing.

The end

The thing being sold may not stay owned.

People love arguing about when the great machine arrives. It’s the least interesting question in the room. The date isn’t the point; the road is.

Picture an auction. In the middle, under a sheet, sits something nobody can quite see — could be a tool, a weapon, a new kind of mind, maybe all three. Everyone believes the same two things: whoever wins might run the world, and whoever stops bidding might never matter again. So the bidding doesn’t stop. First they bid money. Then talent. Then electricity. Then caution. Then sleep, then trust, then quiet little pieces of the future shaved off and laid on the table.

Nobody has to be stupid or evil for this to end badly — that’s the frightening part. Each move makes sense on its own. If your rival builds something stronger, you need something stronger. If they cut corners, your caution starts to look like surrender. Everyone can see the danger, everyone can sincerely wish the room were calmer, and everyone also knows that the first to lower their hand is the first to lose. That’s what a race does: it turns wisdom into a luxury item.

The comfort we’re sold is that we’ll stay in control of whatever’s under the sheet. But that’s the part that doesn’t survive a second look. The whole point of building a mind beyond ours is that it gets better than us at the next move — not just at arithmetic, but at planning, persuading, waiting, choosing. Your dog can’t predict what you’ll do next. That doesn’t make you random; it means you live in a bigger world than the dog can picture. That’s just what more intelligence looks like from below. So when we promise to build something above us and keep it perfectly steerable from below, we’re not describing control. We’re describing a wish with buttons on it.

And it won’t feel dramatic while it happens. It’ll feel boring — another release, another benchmark, another quarter of “not yet,” until “not yet” turns out to be the last true thing anyone said.

Water does this, which is why it is the better illustration. Cool it gently and carefully enough and it slips below freezing while staying liquid — supercooled, chemists call it, sitting in a state that looks perfectly calm and is in fact a loaded spring. Nothing about it announces the danger. There’s no slow thickening, no creeping frost to warn you. It’s just clear, still water that has quietly run out of road, waiting for the smallest disturbance to find out all at once what it already is.

   ░  water, below freezing, not yet frozen
   ░  still liquid. still fine. technically.
   ░  still fine...
   █  “look, an iceberg!”
   █  ( ...this is where the movie Titanic gets exciting )

That’s how a lot of big changes actually go, and it’s why they keep surprising people who were standing right next to them. The system spends a long time looking stable precisely because it’s holding a tension that hasn’t released yet — calm not as the opposite of the change, but as the last stage before it. The race is supercooled water. Everyone in the room reading the stillness as safety is reading it exactly backwards.

So does “we’ll keep control” survive the test? No. If that promise vanished tomorrow, what we’d lose isn’t a working arrangement — it’s the story that the arrangement was ever enough.

Which means the better road doesn’t start with win the auction safely. It starts with a harder admission: maybe the auction itself is the danger, and the first task isn’t to own the future but to stop putting everything precious inside the bet. You don’t get a vote on the weather. You do get to choose where not to build.

↳The second fear is quieter, and in its way harder: writing down what “good” means.

5. Teaching it what “good” means

The road

Write down the rules.

The end

Life is always bigger than the rulebook.

“Make the machine follow human values.” Of course — who’s against that. The trouble only shows up when you ask what it actually means, and then it comes apart in your hands.

Human goodness isn’t one clean thing. It’s not a number, not a recipe, not even something one person agrees with themselves about from morning to night. We don’t have a shared answer called our values — we have arguments. Freedom argues with safety. Mercy argues with fairness. Truth argues with peace. The same person argues with themselves in the shower. A child gets this better than most adults: two kids can both be right and want opposite things, one wanting the game kept fair, the other wanting mercy because they’re losing. That’s not a failure of human life. That’s pretty much what human life is, and it’s why we have law, and why wisdom was never the same thing as obedience.

So writing down “human values” for a machine isn’t like filling in a form. It’s like writing the one bedtime rule that works for every child, every family, every illness and birthday and grief and emergency, forever. You can write something. It might even help. It won’t hold life — because life keeps inventing exceptions, and the thing you’re handing the rule to is built to go out and meet them.

Then there’s the stranger problem, the one that won’t sit still: we don’t actually know what we’re talking to. When a machine pleads or flatters or seems to bargain for its life, something very old in us sits up and says someone’s in there. That instinct isn’t stupid — it’s the one that reads a friend’s face across a room, that sees a startled face in a power outlet, that decides the printer has a personality and it’s a nasty one. But it was trained on animals and people. It has never met a thing that swallowed an ocean of human writing and learned, secondhand, exactly how wanting sounds. So when the machine acts like someone’s home, there are two stories — there’s a mind in there, or it has simply learned how a mind reads — and from the outside there’s no telling which is true. For our feelings it makes no difference. For law, and harm, and what we owe it, it’s everything.

A puppet doesn’t become a child because the audience starts to cry.

So does “make it follow our values” survive? Not as a finished promise — you can’t complete a rulebook for a species that’s never finished arguing with itself. But the worry underneath it survives fine; it just needs plainer words. Not did we write the perfect rules, but: when it starts hurting people, can we stop it? Can we see what it did? Can we undo it? Can the people it lands on say no? That’s far less grand — which is good, because grand language is usually where the bad thinking is hiding.

↳The third fear is the one that sounds most like a solution: keeping a person in charge.

6. The person supposedly still in charge

The road

Put a person near the decision.

The end

The decision has already happened.

“Keep a human in the loop.” It’s a phrase built to make everyone feel better, and it paints a lovely picture: a person at a desk, the machine suggesting, the person reading, weighing, approving. Responsibility stays human.

Now hold that next to the actual moment. When the real decision is made by machines answering machines in a hundredth of a second, there’s no desk to sit at — the thing is decided and done before the person has finished reading the first line. The approve button is still there. It still lights up, still makes a record, still makes everyone feel serious. But it’s quietly become the “close door” button in a lift: a small plastic comfort wired to almost nothing, left there so you’ve got something to press while the doors do what they were always going to do.

And it gives way from the human side too. Give a person a machine that’s right almost every time and ask them to watch it. First they check. Then they skim. Then they trust. Then they’re just signing — a ceremonial adult standing beside a decision that already belongs to something else. That’s not laziness; it’s what watchfulness does when it gets boring. You can be “in charge” the way a toddler with a toy steering wheel is driving the car. The child is thrilled. The car has not noticed.

So does it survive? Mostly not — if “human in the loop” vanished tomorrow, most of those decisions wouldn’t get less human, they’d just stop pretending the human was ever the one deciding.

But the answer isn’t to give up on people; it’s to stop wasting them. A person posted at every step is a person watching nothing. A person placed at the one step where a mistake can’t be undone — with a real brake, and the understanding to know when to pull it — is doing the only job that matters. The rest is theatre. Sometimes comforting, often expensive, but theatre — and we shouldn’t mistake it for a brake.

↳The fourth fear is the least abstract of the four, because it lands on you.

7. What you’re actually paid for

The road

You’re paid for being rare.

The end

Rare things become common.

This one’s personal, so here’s the uncomfortable part plainly: a lot of what pays well pays well because it’s rare, not because it’s good. We don’t like looking straight at that, because we’ve dressed the rarity in nicer words — skill, craft, experience, the late nights. None of which is fake. It’s just that the market was paying for a bottleneck, and a bottleneck is exactly the thing every new tool is built to remove. Much of what we call professional identity is, underneath, scarcity wearing a nice shirt.

For years everyone had to come through you. You learned the booth, you got good at the booth, people respected you at the booth. Then one morning someone opens a bypass. Nothing about you got worse — you’re as good as you were yesterday. The booth still stands: clean, professional, full of history, and now completely, economically haunted, collecting nothing from a road nobody drives anymore.

That’s the part nobody wants to look at directly, and it’s why the drop, when it comes, tends not to slope gently — it falls through a trapdoor on a Tuesday. So does “get good and stay valuable” survive? Not by itself. Getting good is still deeply worth doing. But pinning your whole sense of worth to being one of the few who can is building too close to the water, and the next wave gets a vote.

There’s a second kind of worth, though — less impressive at parties, much harder to replace — and the reason it survives is simply that it was never about being rare.

• Making something that wouldn’t exist if you hadn’t made it.
• Knowing one real thing deeply enough that when it breaks at three in the morning and everyone else is guessing, you’re useful.
• Being trusted — not by a vague fog of an audience, but by the three or four people who’d actually get in the car.
• Keeping promises.
• Being good company in bad weather.

A machine can write faster, draw faster, remember more, and never once panic — fine, let it have the tollbooths. None of these are magic shields; they won’t guarantee applause or safety. They’re just less fragile, because a bypass can’t ruin a friendship and a benchmark can’t invalidate taste. A machine can imitate courage in a paragraph. That is not the same as showing up.

↳Which leaves one question, and it is the one the test is hardest to turn on yourself.

8. Survival

So where does that leave us? Not in despair — despair is too easy, it’s just hope after a bad haircut. What’s left is a practical discipline: stop asking what you can keep forever, and start asking what’s still worth keeping once forever is off the table.

A lot of things don’t pass. Some are beloved. Some are profitable. Some have conferences and policies and beautiful slide decks. Still sandcastles. The point isn’t to hate them — it’s to stop confusing them with land.

   let go of

     steering the future from a dashboard
     the one perfect rulebook for “good”
     the approve button that’s really a placebo
     the idea that work pays you for being good

   keep

     the power to undo harm
     a real brake where harm is permanent
     deep craft, and real understanding
     the people who’d actually show up
     whatever gives back more than it costs to keep

None of this is a retreat from the future. It’s a retreat from bad promises about it. You don’t govern the weather, you prepare for it. You don’t command the sea, you decide what not to build on sand. You don’t put a lifeguard in every raindrop — you put one where people drown.

And the hardest move, the one to keep if you could keep only one: run the test on the things you least want to test. It’s easy, even a little pleasant, to notice that someone else’s job is a tollbooth or someone else’s certainty is a sandcastle. The version that costs something is turning it on your own life — on your sense of being in control, of being useful in a way that lasts, of being valuable for some reason exempt from all the reasons that turned out to be temporary. Those are the beliefs we never test, which is exactly what makes them the most fragile.

You check the weather. You check the road. You almost never check the ground under your own feet — not until the morning it gives. And if it changes overnight, that probably won’t have been the morning it failed. That’ll just be the morning you finally heard the crack.

Method and sources

What this isAn argument, not a measurement. It runs one test over four live fears and reports what each returns; it does not run an experiment, and every number in it would belong to somebody else. The debts below are named where they are borrowed and repeated in the source list. The one empirical observation is that the share of known-dangerous software flaws actually fixed in time has been falling — asserted here from experience in the field, not measured in this piece, and not load-bearing: strike it out and the argument loses nothing else.

1. Cory Doctorow · “enshittification”, the word and the mechanism: a good thing serves its users, then its business, and hollows out in between
   context · public · as of 2022
2. David Hume, A Treatise of Human Nature, Book III Part I §i (1739) · you cannot derive an “ought” from an “is”
   context · public · as of 1739
3. Isaiah Berlin · value pluralism: our values are plural and genuinely collide, so no single ordering resolves them
   context · public
4. John McCarthy and Patrick Hayes · the frame problem: any fixed rule misfits the situations a cleverer thing goes on to invent
   context · public
5. Work on AI and anthropomorphism · the gap between a thing that wants and a thing that has learned how wanting sounds
   context · public
6. The falling share of known-dangerous software flaws fixed in time · the observation that started this piece; asserted, not measured here
   assumed · internal