What Has ChatGPT Become?
For the first time since its debut in 2022, a large trove of user data from ChatGPT has been made available to researchers, who attempted to answer a straightforward — but contentious and heavily obfuscated — question: What are most people doing with ChatGPT, most of the time?
There are a few major caveats here. OpenAI’s own researchers worked on the paper with Harvard economist David Deming under the auspices of the National Bureau of Economic Research (in other words, the company is comfortable with this paper’s findings). Additionally, the research, which is mostly an attempt to classify and sort a large set of messages, was done substantially by OpenAI’s own models, which both automated the process and, the researchers say, helped preserve user anonymity. “No one looked at the content of messages while conducting analysis for this paper,” the researchers claim, although they did validate their automations against human analysis used in smaller previous studies we’ve talked about here before. The researchers’ methodology is both fascinating and could insufficiently but not entirely inaccurately be characterized as “we asked ChatGPT.”
With all that in mind, here are a few of the researchers’ findings: as of July, “about 70% of ChatGPT consumer queries were unrelated to work,” with non-work queries increasing faster. The three most common topics of conversation are what the company and researchers call Practical Guidance, Writing, and Seeking Information (which the company says “appears to be a very close substitute for web search), with Computer Programming and Relationship and Personal Reflections making up a small percentage of messages. Within the category of work, writing assistance is by far the main use case, with text modification or editing, rather than composing from scratch, making up a majority of requests.
Most interesting is the paper’s attempt to categorize what “kind” of output users are looking for: About 49% of messages are users asking ChatGPT for guidance, advice, or information (Asking), 40% are requests to complete tasks that can be plugged into a process (Doing), and 1% are messages that have no clear intent (Expressing). Asking messages have grown faster than Doing messages over the last year and are rated higher quality using both a classifier that measures user satisfaction and direct user feedback.
None of this will be too surprising to anyone who has been using tools like ChatGPT for more than a few months, or even to people who are merely aware of how their friends, family, and coworkers seem to be chatbotting. (While the paper describes widespread uses that are compatible with “doing your homework for you,” its design allows it to generally steer around the topic.) It is, however, slightly out of step with some of the more provocative messaging that companies like OpenAI have at times engaged in or at least indulged. The paper describes a powerful, versatile tool that people use for a wide variety of assistive tasks — something akin to a search engine with powerful automated productivity software attached to it — rather than an increasingly autonomous human-like entity. It describes a technology that’s become less alien, not more, to its users, and notes their evolving expectations. “Within work usage, we find that users currently appear to derive value from using ChatGPT as an advisor or research assistant, not just a technology that performs job tasks directly,” the paper concludes, arguing that it “likely” improves worker output by offering “decision support,” or advice.
The research brings to mind another paper, published last year in Science, in which an interdisciplinary group of researchers argues that LLMs are better understood as “cultural and social technologies” than in terms of intelligence and autonomy, making comparisons to search, bureaucracy, and even the written word — systems that share the trait of “allowing humans to take advantage of information other humans have accumulated” in new and transformative ways.
The picture that emerges from this data matches this thesis pretty closely: ChatGPT, for many of its users, is a way to access, remix, summarize, retrieve, and sometimes reproduce information and ideas that already exist in the world; in other words, they use this one tool much in the way that they previously engaged with the entire web — arguably the last great “cultural and social technology” — and through a similar routine of constant requests, consultations, and diversions. One doesn’t get the feeling from this research that we’re careening toward uncontrollable superintelligence, or even imminent invasion of the workforce by agentic AI bots, but it does suggest users are more than comfortable replacing and extending many of their current online interactions — searching, browsing, and consulting with the ideas of others — with an ingratiating chatbot simulation.