The Biggest Risk Isn't AI

The risk was never that AI would think for you. It's that you'd let it, and slowly forget how to think at all.

In 2023 a New York lawyer filed a brief in federal court citing six cases that did not exist. He had asked ChatGPT for precedent and it gave him case names, citations, and quoted passages from the opinions, all of it in exactly the right shape. At some point he got uneasy and asked whether the cases were real. ChatGPT said they were, and added that they could be found on Westlaw and LexisNexis. Opposing counsel went looking and could not find a single one.

He did ask the question. He just asked it to the thing he was supposed to be checking.

It usually looks much smaller than that. A fabricated statistic doesn't look like anything. That's the problem with it. It arrives in a clean sentence with a specific number attached, sitting in the third paragraph of a summary that's otherwise fine, and it reads exactly like the true sentence next to it. It survives the review. It goes out under someone's name.

Then, weeks later, somebody asks where it came from, and the honest answer turns out to be that nobody ever asked.

That half-second is the whole thing. Not the machine, not the job losses, not any of the futures the headlines keep drafting. A moment where a question should have happened and didn't, repeated a few hundred times a month until it stops feeling like a moment at all.

Passive use is the actual failure mode

A language model will answer almost anything, instantly, in complete sentences. That's the problem. A confident paragraph reads like a correct one, and after enough of them you stop checking which is which.

Use it passively and you hand over three things without noticing: how the problem gets framed, which options get weighed, and what gets decided. Each handoff is too small to feel like one. Add them up over a year and you get someone who can produce answers but can't defend them, and when you ask why, the honest reply is that the model said so.

Nobody decides to become that person. It happens gradually, without a decision point anyone could later point to as the one that mattered.

But some deferring is fine, and pretending otherwise is silly

Here's the part that gets skipped in essays like this one.

You already defer constantly. You don't verify your compiler's output line by line. You don't recompute what the calculator tells you. You don't audit the studies behind a prescription before filling it. Handing off cognitive work is most of what civilization is built on, and the alternative (verifying everything yourself, from first principles, every time) isn't rigor. It's paralysis.

So the useful line runs somewhere other than where these essays usually draw it. It separates the places where a failure would surface on its own from the places where it would sit there indefinitely, looking fine.

Bad compiler output breaks the build. A wrong number from a calculator clashes with the estimate in your head the second time you run it. The fabricated statistic breaks nothing at all. Nothing in the system flags it, because you were the flag.

That's the useful test, and it takes about two seconds to run:

If this is wrong, what tells me?

If the answer is nothing, you have to stay in the loop. If it's the test suite, a source you can open yourself, or your own expertise, go ahead and delegate. Ask a model for boilerplate you'd recognize as broken on sight and delegate freely. Ask it for a claim about a domain you don't know, and you've just made yourself the only check that exists.

Thinking with it looks different from copying from it

Same tool, same window, completely different activity.

Thinking with a model means using it to pressure-test your reasoning rather than skip it. You bring the question and the judgment. It brings speed, range, and infinite patience for being argued with. The useful part usually shows up in the friction between the two.

A few things that work.

Draft something first, then hand it over to be attacked. You're looking for the hole before someone else finds it, while it's still cheap to fix. Ask it to argue the opposite of what you believe, properly, in the strongest form it can manage. Positions that can't survive that were never positions. And make "Where did this come from?" a reflex rather than something you remember on the days that matter.

There's one more that's less obvious. You can now generate ten framings of a problem in the time it used to take to find one. Generating got cheap. Picking never did.

Look at what stays in your hands: the question, the doubt, the call. The model is doing real work in there, but the call is still yours.

It's the same person on two different days

I want to resist the tidy version of this, where there are two kinds of people and you're obviously the good kind. Nobody reads a paragraph about passive AI use and thinks it's about them.

It's the same person on different days. Rested, invested, working on something you'll have to defend, you stay in the loop. Tired, behind, ten tabs deep in something that isn't really yours, you accept the first fluent answer and move on.

I do both.

The proportion is what matters, and the proportion drifts, always in the same direction, because deferring is genuinely easier and the cost shows up much later than the benefit.

The gap this opens between people has nothing to do with intelligence, and less and less to do with access. It comes down to something much smaller: whether you notice the drift while it's happening.

Which is all that literacy ever meant

We call this AI literacy, and the phrase is fine as long as we remember what literacy has always meant. Reading has always included the ability to doubt what you're reading, which is why teaching it takes years and why every generation has to be taught it again.

This is the same skill, applied to a text that writes back. Knowing enough about how the thing works to anticipate where it'll fail. Keeping your own judgment in shape precisely because there's now something eager to spare you the trouble of using it.

That's the bet behind what we're building at LearnlyGo. Everyone is about to have speed. What stays scarce is knowing which answers are worth having.

The biggest risk isn't AI. It's showing up to this world without the habit that makes it yours: catching the half-second before it closes.

Start thinking.