inferring preference

Let's talk about something exciting in AI: human-computer interfaces, and how do we build systems that ethically infer user preferences?

I was trying to do this and instantly ran into the profiling pitfall. I don’t want to build a system that’s used to read people’s minds faster than they can themselves. We already know how that turned out. But after thinking about it, and seeing what the big folk have been talking about, it turns out there’s a good solution if we focus on the data and outcomes in the system.

I’ll start with a preference of mine; I’ve noticed that I learn more from what I dislike than from what I like. It seems that reactive negative feelings toward something enables me to go further because I want to improve on it and do better, it’s similar to how much of the content people consume online targets base emotions like rage. So when brainstorming something I want to write, I ask an LLM for a draft of an idea I have, I give it the raw brain dump and let it roll. Most of the time, I don't like the draft, so I say what's wrong, then the next one is wrong in a different way, so I point to that. Four or five iterations later, the thing is much simpler than where I started and it's mine, because the AI didn’t write it, but seeing everything I didn't want made me refine what I did want.

Since time immemorial, we’ve been used to write by editing our own drafts, but AI enables me to get variations of my ideas that I fully expect to reject. It's like reading a lot of authors to find out what you don't want to sound like, except every version is aimed at the exact thing you're trying to say. You couldn't do that this fast before.

I’m not saying this is a universal writing tip, but it works for me because, as I said, I learn faster from what I dislike. It's a trait that took me a long time to notice, and I only noticed it by paying close attention across a lot of sessions.

So that’s when I asked myself: how do you build a system that accelerates that noticing, without the system building a psychological profile of the person?

A system watching my sessions should not conclude "Joshua learns through dislike." It’s making a claim about me that I’m not sure is okay for software to reach it on its own, even with my consent, because it can read me faster than I read myself, and it could influence me faster than I can tell, involuntarily.

However, the system could conclude that sessions that go reject-refine-reject produce better output than sessions that don't. This claim is purely about the work, you can verify it, measure it, act on it, I mean it’s data, right there, telling you to show more variations earlier and make rejection cheap.

So I have the system review the outcomes and compare them to stated goals, metrics. Then I look at that improvement and notice it points to a gap in myself. Sometimes I don’t notice it, but because I’m always working in the system, and that system improvement is enforced by repetition, it improves me. What’s important is that I do the profiling on myself, if any.

Another example, say I have writing skills and preferences, but the system notices failures in written outputs relating a lack of audience considerations in my preferences. The system can see that gap in its own rule set, see that outputs aren't landing, and tell me: your preferences are missing audience. That suggestion improves the system, and prompts me to learn something about writing for an audience, I set better practices, the system gets better inputs, and it’s free to find the next gap. The quality of my work can be judged, it doesn’t need to judge me as a person, this is the essential boundary.

Over time, you accumulate instances, you could consider each instance of a pattern to be an “upvote” on its feature request, you gate those votes, say, if the patterns is shown 3+ times, the system considers it an “inferred preference” and uses it for its next applicable output. The output then carries provenance and links to all the preferences that were used to create it, including any inferred preferences. When the user reviews that output, their approval of the output becomes approval of the inferred preferences, which then graduate to pure user preferences, during the review, they can also reject any inferred preference. This has the benefit that it’s ambient, the user doesn’t need to manually go and approve each preference every time, they can keep doing their usual work and at some point that work will become approval, rejection, refinement, etc. of those preferences.

This is a self-improving system that evolves both software and people.