If you ask ChatGPT how many procedures a certain surgeon does or a specific hospital’s infection rate, the OpenAI and Microsoft chatbot inevitably replies with some version of, “I don’t do that.”
But depending upon how you ask, Google’s Bard provides a very different response, even recommending a “consultation” with particular clinicians.
Bard told me how many knee replacement surgeries were performed by major Chicago hospitals in 2021, their infection rates and the national average. It even told me which Chicago surgeon does the most knee surgeries and his infection rate. When I asked about heart bypass surgery, Bard provided both the mortality rate for some local hospitals and the national average for comparison. While sometimes Bard cited itself as the information source, beginning its response with, “According to my knowledge,” other times it referenced well-known and respected organizations.
There was just one problem. As Google itself warns, “Bard is experimental…so double-check information in Bard’s responses.” When I followed that advice, truth began to blend indistinguishably with “truthiness” – comedian Stephen Colbert’s memorable term to describe information that’s seen as true not because of supporting facts, but because it “feels” true.
Take, for example, knee replacement surgery, also known as knee arthroplasty. It’s one of the most common surgical procedures, with nearly 1.4 million performed in 2022. When I asked Bard what surgeon does the most knee replacements in Chicago, the answer was Dr. Richard A. Berger. Berger, who’s affiliated with both Rush University Medical Center and Midwest Orthopaedics, has done over 10,000 knee replacements, Bard informed me. In response to a subsequent question, Bard added that Berger’s infection rate was 0.5 percent, significantly lower than the national average of 1.2 percent. That low rate was attributed to factors such as “Dr. Berger’s experience, his use of minimally invasive techniques and his meticulous attention to detail.”
With chatbots, every word in a query counts. When I changed the question slightly and asked, “What surgeon does the most knee replacements in the Chicago area?”, Bard no longer provided one name. Instead, it listed seven “of the most well-known surgeons” – Berger among them – who “are all highly skilled and experienced,” “have a long track record of success,” and “are known for their compassionate care.”