Pete Koomen’s Runabouts

I just finished reading Pete Koomen’s article, AI Horseless Carriages. While I enjoyed it and agree with his view of the problem, I strongly disagree with his proposed solution. 

I recommend reading the entire thing, it’s short and includes nice interactive examples. But put briefly, Koomen makes a few arguments about modern AI writing assistants:

• they generate generic slop
• getting them to output something usable often takes as much work as writing the output yourself
• the solution is to give users more control over the ‘system prompt’ layer of the stack 

He uses Gmail’s AI Assistant to demonstrate his points, with a hypothetical email to his boss as the example:  

Obviously this output is technically correct but not usable. In his formulation of the problem, this is solved by letting him adjust the system prompt:

It's perfect. That was so easy!

Not only is the output better for this particular draft, it's going to be better for every draft going forward because the System Prompt is reused over and over again. No more banging my head against the wall explaining over and over to Gemini how to write like me!

Koomen’s right, this does allow for a tighter match between expectations and output. The mistake he makes is a common one that I see engineers in particular often fall prey to: Other than engineers, no one wants to have to think about how to solve their own problems!

Designing for Non-Engineers

The role of the AI-assistant designer is to think about the users’ needs in advance and to, well, design a solution that gives them the output they’re looking for without them having to teach the AI themselves. Let’s go back to the problem that Koomen calls out with the Gmail assistant:

‍

Problem: inflexible “system prompts” lead to outputs that are poorly suited to specific use cases 

Because the system prompt likely includes information about formality and tone that would most often be appropriate for email in a business context, the output fails when the writer has a use case outside of that context. Now Koomen’s solution: 

‍

Solution: let users act like engineers and rewrite the system prompt

This solution fails to generalize to a broader audience. Koomen includes a set of parameters in his context that are non-obvious to most lay people (age, role, level of busy-ness, rules on punctuation, email length). He also takes for granted the amount of effort the average person is willing to put into a tool vs just doing it themselves. This is the classic engineer -> productization issue in a nutshell.

A better class of solution would make the output flexible with minimal user input, ideally none at all! This is well within reach but requires a design that understands:

1. the parameters of output most likely to change with context (ie: formality, length of email, punctuation, emoji use)
2. the relationship between the context of the request and those outputs (ie: formality varies when emailing my brother vs my lawyer)
3. the smaller subset of parameters a user may want to adjust after the fact without re-rolling their request

A version of the Gmail assistant that understands relevant context and how to apply it on a per-request basis should capture intent with the lowest effort possible. It should understand concepts like intended recipient, what meeting Koomen won’t be coming to, and the relevant day, all from context. This is stuff LLMs are great at! It might look like this:

‍

Input:

Behind the scenes:

Gmail should now a) identify the key levers that would indicate a change in style and b) automatically match those levers to the context based on the immense history available to it.

Output:

Something like the above could reasonably expected to output an email as Koomen would have written it

I would strongly argue that this version of the solution is a much better fit for 99% of Gmail users. It requires less of an internal model of what drives your writing style and, more importantly, much less effort. Those 99% of users would never even look for an ‘advanced system prompt’ dropdown let alone take the time to iterate through approaches, leading to them bouncing off the feature entirely. Underpinning this approach is the fact that Gmail already knows all of the relevant information about how you compose emails, one of the key strategic advantages Google has in the AI tools race. 

Now maybe the solution isn’t perfect, because some aspects of context change in ways that aren’t obvious to the AI. Maybe for today only Koomen is extra sensitive about the fact that he has missed a lot of work lately, and needs to be a bit more formal than he would normally be with Garry. How can we give him the controls he needs to drive the output he wants, as simply as possible? And what if Koomen isn’t conscious of that until he sees the first draft from the assistant and intuitively grasps that it’s not a fit for the current context - how can he change it without having to rewrite the prompt and start over?

‍

Post Draft Refinement:

In our hypothetical, Koomen wants to make his email longer and more formal, without substantively changing the content of the text. Turns out, Gmail already understands this concept and has a secondary option to change the text after the fact, across just these parameters.

This isn’t the worst, but playing with them doesn’t really get us to our desired output. The challenge here is that these options are pretty flawed:

• ‘formalize’ is a one-directional button, there is no ‘informalize’
• each option is a boolean, it’s either ‘formalized’ or not, you can’t keep clicking to steadily increase/decrease to taste 
• ‘shorten’ still fails to remove most of the fluff, making it of questionable value

I’d probably aim for something more like the following:

As the designer the job here is realizing that context should be used first without needing user input and that after context, length and formality are the correct parameters to expose to users. You could use another dialog box and have the user type out a request to "make this less formal and shorter" but you end up with the request taking longer than just doing it yourself. Asking users to do MORE work to increase the quality of the output is going in exactly the wrong direction and fails to take into consideration what AI is best at. 

Horseless Carriage -> Runabout Buggy -> Modern Auto

Now I’ve been a little unfair to Koomen. He is right to call out the weaknesses of the first generation of AI assistants, and his analogy of horseless carriages is a good one.

The ‘old world thinking’ that gave us the original horseless carriage was swapping a horse out for an engine without redesigning the vehicle to handle higher speeds. What is the old world thinking constraining these AI apps?

The modern automobile arose in fits and starts from the original horseless carriage with major steps in between, like mass-production occurring first on the Oldsmobile Model R “runabout” model that shockingly still resembled a carriage, followed by the much more famous Model T that popularized the form factor we recognize today.  

I argue that AI assistants with editable system prompts are the intermediary “runabouts” connecting the past to the future. System prompts and Input Prompts are remnants of a computing era that still required a human pilot. The promise of AI agents is removing the human from the loop entirely. Just like Waymos won’t need steering wheels eventually, writing assistants won’t need editable system prompts. I’ll end with Koomen’s words, which I agree with wholeheartedly:

AI-native software should maximize a user's leverage in a specific domain. An AI-native email client should minimize the time I have to spend on email. AI-native accounting software should minimize the time an accountant spends keeping the books

Well designed AI will do just that, and it will never ask a user to consider a system prompt along the way.

‍

If this type of thinking is interesting to you and you want to apply it to the future of user-generated video games, check out what we’re up to at Liminal. We believe that AI can expand human creativity and bring communities closer together and we are leveraging this type of design to unlock the next generation of legendary storytellers.

‍