Nobody introduced them

The user in this story is made up, but I suspect you have met them.

They open an agent for the first time and find an empty box and a blinking cursor. Teams call that freedom. To the user it feels like an exam they forgot to study for. Researchers call it the gulf of envisioning: you cannot ask for what you cannot picture.¹

So they play it safe. They ask for a summary of a tab. It works. They close the tab.

A week later they try something bigger and ask the agent to book a dentist appointment. A prompt asks whether the agent may act on their behalf. They click Allow without reading, the way they have clicked through cookie banners for ten years. The agent books 8:00 on a Friday morning, a slot they would never accept. It never asked about their schedule, and it gave them no way to teach it afterwards. They cancel, book the appointment themselves, and from then on use the agent to read articles.

The model was capable and the code ran. The handoff still failed. The user never learned what the agent could safely do, and the agent never learned that this user hates mornings. Two strangers sharing one calendar. Nobody introduced them.

What I keep noticing

I think people hold back from agents less because the agents cannot do the work, and more because neither side has learned the other. If a product introduced both of them, and kept doing it as they changed, I would expect people to delegate more, catch more of what matters, and stay longer.

Delegation is already ordinary. Anthropic’s Economic Index found that the share of Claude.ai conversations where people hand over a whole task, which it calls directive, went from 27% in late 2024 to 39% in August 2025.² Many people are becoming managers of a delegate, and almost nobody is introducing them.

Plenty of controls, little introduction

The user’s click is common. Anthropic reports that people approve 97% of permission prompts in Claude Code, and in a study of 1,053 testers, human review caught only 13.6% of dangerous commands.³ That is one company’s data from one tool, so I hold it loosely. OpenAI raised the same worry in 2023, when it suggested minimum review times so approvals do not become a rubber stamp.⁴ To me a 97% approval rate looks like fatigue.

People say the same thing from their side. In Apple’s study of computer-use agents, one participant put it as “I don’t want you to do anything that’s irreversible, like I don’t want you to hit the purchase buy button.”⁵

What I could not find in the big labs’ public guidance is a description of how a person and an agent get to know each other. Google’s PAIR guidebook comes closest when it says onboarding continues indefinitely,⁶ and Microsoft’s HAX guidelines say to learn from user behavior and to notify people about changes.⁷ Neither describes the introduction itself.

Why a delegate is different

For a long time onboarding only had to teach the person. Software did what it was told, so a spreadsheet never needed to know you hate mornings. An agent is a different kind of thing.

	A tool	A delegate
Who decides how	You, one click at a time	The agent, from a goal you gave it
When it acts	While you watch	Across many steps, often while you are away
What fills the gaps	Nothing, it waits	Its own guess, unless it knows your preferences
What it needs to know about you	Nothing	Your schedule, limits and comfort with risk

A tool needs you to learn it. A delegate also needs to learn you. OpenAI’s paper says an agent needs default common-sense preferences to fill gaps without guidance, such as not spending a person’s money.⁴ The 8 AM booking was a gap-filling guess.

The person needs a memory too

Learning currently runs one way. ChatGPT, Claude and Gemini all have memory now, and saved memories can be viewed and edited. But memory teaches the agent about the user. Nothing teaches the user about the agent: what it can do today, what changed last week, what it did while they were away. Memory also collects facts more than agreements. It may remember the dentist, but “never mornings” only becomes a rule when the two of them settle it together.

I would like the person to get a notebook as well: a card they can reopen that says how the two of them work, a changelog of the agent’s behavior, a weekly receipt, and every rule the agent holds about them in plain language, visible, editable and wipeable in one click.

Once the agent starts learning the person, privacy becomes the risk that can sink the idea. A few rules seem right to me. The agent should say when it infers something, for example “I’ve noticed you skip mornings, do you want that as a rule?” It should keep no hidden profile. And it should treat silence as no.

Moments when both sides learn

A relationship like this takes shape in a handful of moments, and at each one the person and the agent each have something to pick up.

The third moment seems to matter most. People lose confidence in an algorithm faster than in a human who makes the same mistake, which researchers call algorithm aversion,⁸ so I think the first mistake is the real first run. The fifth moment is newer. Bansal and colleagues found that a more accurate model can make a human and AI team perform worse when the update breaks what people had learned to expect.⁹ A silent update is a design problem as much as a release note.

People differ

Apple’s study also found that needs differ by person and by task.⁵ I picture three users on the same agent. A cautious one who keeps their distance and never delegates much. An explorer who skips the tour and hands over a hard task in the first minute, then drops the product when it fails. And an over-truster who approves everything until the agent sends a draft they never read.

They need three different introductions, so whatever the agent does has to adapt. If the over-truster approves in under half a second, they are not reading, and the agent should slow down on anything sensitive.

Somewhere I might start

I am not sure where an introduction should begin. Here are the three places I keep coming back to.

After the first mistake

This is where trust seems to break for good, and it may be cheap to improve because it is mostly a redesign of error states. Own the mistake in plain words, offer an undo, and turn it into a rule in the person’s own words. If I could measure one thing, it would be whether people come back the next day after a failed task.

The first run

Every new person sees it, so it is the widest net. I would want it to end with one real chore done, one boundary the person set, and one capability they did not know existed. A starter task suggested from what is already on screen helps. So does one question instead of a settings page: “What’s one thing I should always check with you first?” takes ten seconds. A plan shown before acting teaches capability while the person stays in charge. And skipping has to stay possible, because anyone who skips can meet the same lessons later, in context. Task variety at day 30 would tell me whether it worked.

A page that belongs to the person

The cheapest of the three. A card about how they work, plus plain change notes after each update. It is content, not model work. I would watch how often the card gets opened and whether “why did it do that?” questions fall.

Where this might be wrong

This is a hypothesis, not a finding. Most of the hard numbers above come from older automation research or from one product, so each belief needs a test and a result that would sink it. If a guided first run and a blank box produce the same task diversity at day 30, I am wrong about under-delegation. If an agent that asks one preference question and shows what it learned does no better on undo rates than one that learns silently, two-way onboarding does not earn its cost. If people dismiss “handle this quietly next time?” after three good runs, earned autonomy is not what they want.

A common objection is that smarter models will make all this unnecessary. I doubt it. A user’s boundaries and their dislike of mornings do not come out of a training run, and Bainbridge argued in 1983 that automation leaves people with less practice, which leaves them less prepared when it hands control back.¹⁰

One observation, mine and not measured: most teams have someone who owns the first five minutes, and almost nobody owns the relationship after that.

I do not have answers to some questions yet. When approval rates rise, how do we tell real confidence from fatigue? Which actions should never be fully autonomous, however much trust is earned? And if the agent keeps learning the person but the person stops learning the agent, when does personalization start to feel like being watched?

If a new person opened your agent today, what would it tell them in the first thirty seconds, and what would it ask?

Footnotes

1. Subramonyam, Pea, Pondoc, Agrawala and Seifert (2023), Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces. ↩

2. Anthropic (September 2025), Economic Index report. The 27% figure is from December 2024 and January 2025 data, the 39% from August 2025. ↩

3. Anthropic (August 2026), Auto mode as default in Claude Code. Vendor data from one tool. ↩

4. OpenAI (2023), Practices for Governing Agentic AI Systems. The paper uses the phrases “rubber stamp” and “default common-sense background preferences”. ↩ ↩²

5. Cheng, Liang, Schoop and Nichols, Apple (2026), Mapping the Design Space of User Experience for Computer Use Agents, IUI 2026. ↩ ↩²

6. Google PAIR, People + AI Guidebook, Mental Models. ↩

7. Microsoft (2019), Guidelines for Human-AI Interaction. ↩

8. Dietvorst, Simmons and Massey (2015), Algorithm Aversion. ↩

9. Bansal et al. (2019), Updates in Human-AI Teams. ↩

10. Bainbridge (1983), Ironies of Automation. ↩