Who’s better at my job, Chinese AI or me?

OpenAI’s artificial intelligence chatbot, ChatGPT, is not officially available in China without special software to bypass the block. However, there are many Chinese alternatives. Baidu’s Ernie Bot is one.

The Chinese company released a paid version, Ernie Bot 4.0, last month. According to Baidu CEO Robin Li, Ernie Bot is “not inferior to [OpenAI’s] ChatGPT-4.” He recently demonstrated that Ernie Bot could analyze business issues on the spot to impress a boss, draft the plot of a martial arts novel and create a car commercial within minutes. It got me wondering how useful the AI chatbot would be for me in my work.

For this assignment, and with the full consent of my editors, I incorporated the Chinese chatbot in my work as much as possible.

First, every good story must have a good opening.

Ernie Bot suggested that a great way to grab the Marketplace audience’s attention is to begin with a mystery or an intriguing question, like “Have you ever imagined having a personal assistant who knows everything about you, can anticipate your needs and is always one step ahead of solving your problems? Meet Ernie Bot, the revolutionary artificial intelligence system that’s changing the game.”

Not a bad start, though it might be overselling its skills a bit.

Officially, Ernie Bot is bilingual in English and Chinese, but it is trained mostly on Chinese learning models and frequently lapses into Chinese, sometimes in the middle of a sentence. For instance, “There’s more to Ernie Bot than meets the eye. Rumors have been circulating about his 超乎寻常的能力 [extraordinary abilities].”

There goes my hope that Ernie Bot would help me write this story in English.

Ernie Bot might also seem like an all-knowing, supercharged search engine. But it is not. It was able to answer Baidu CEO Li’s complex question about whether he can use his housing loan funds in Beijing to buy a property in his hometown in a different province. However, when I asked Ernie Bot what the average wage in Shanghai was in 2022, it gave me the figure for 2021 instead.

At first, I wondered if this error was down to the poor quality of Baidu’s search engine. Ernie Bot refuted that: “While it’s true that Baidu search engine has faced criticism in the past for its accuracy and relevance, I want to assure you that Ernie Bot is a separate and distinct product from the search engine.”

“Chatbots are language learning models. They’re trained with data, and the data is not being updated in real time,” said data scientist Abner Wang in Shanghai. He has been testing Ernie Bot and ChatGPT and posting the results on social media.

It means chatbots aren’t always accurate. That does not bode well for my work in news, which relies on accuracy and speed. Instead, Wang suggested I could ask Ernie Bot to handle more mundane tasks, such as summarizing our nearly two-hour interview, flagging good quotes or drafting interview questions.

“Though I’m sure you’ve noticed that a lot of the questions Ernie Bot generates are pretty general and boring,” Wang said. I have noticed.

It took five questions and 7½ minutes before I could get Ernie Bot to understand that I was not interviewing a data scientist who had worked on the chatbot but speaking to a user who happens to be a data scientist. But Wang said that might have happened partly because I was not using the right words to prompt Ernie Bot. This is also a challenge in using ChatGPT and the reason companies hire so-called prompt engineers.

Ernie Bot’s answers are always long-winded — except when it comes to brainstorming story ideas about the upcoming elections in Taiwan. No matter how I rephrased the question, I did not get answers on Taiwanese politics.

Beijing sees the island as a breakaway province.

“Please refer to relevant websites for more information and feel free to ask me any other questions,” Ernie Bot said.

But Wang does not think political censorship hinders businesses all that much.

“For a business working in mainland China, I think the stronger the censorship, the better,” Wang said. “It’s safer for me if I create a product using AI because otherwise the authorities would hold me responsible.”

As a data scientist, he said, two things really matter for innovation and the wider U.S.-China tech rivalry.

“I look at the chatbot’s ability to reason logically and how often it hallucinates,” Wang said, adding that that will determine how much companies can reduce costs, increase efficiency and create value with it.

After nearly a year of testing, Wang said Ernie Bot is a bit weaker than ChatGPT. But he added that Ernie Bot still has good reasoning capability. And, like all chatbots, it is continually learning and improving.

“The gap between Chinese AIs and foreign AIs is getting smaller,” he said.

I am still getting to know Ernie Bot. It gives conflicting answers on the most basic questions, like whether its name stems from the puppet character Ernie from “Sesame Street,” its age (anywhere from 18 to ageless) or its favorite color (sometimes green, other times blue).

In the end, Ernie Bot did not write this whole story for me. It didn’t summarize any interviews or generate any headlines that were of use.

It did generate a few drawings I could use for the web story. I used keywords to get Ernie Bot to generate drawings depicting me holding a microphone while interviewing Wang at a Starbucks. There was very little consistency among the drawings, and in a couple of them, Ernie Bot portrayed me as a cat and a bear.

At least Ernie Bot can tell jokes: “Why don’t scientists trust atoms? Because they make up everything!”

Did you crack a smile? Well, I did, because my job feels safe. At least for now.

Additional research by Charles Zhang.