Why does AI lie?

This blog can be considered a warning.

Or a head's up.

Or maybe something more.

We recently had a strange run-in with AI (Google's Co-Pilot for those who want to know) that left us more than scratching our head. It left us a little frightened. Not of AI, mind you, but the people who use AI to make their decisions. This is a story of how AI made mistake after mistake on a simple query. Nothing life threatening, mind you, but the story should make you stop and consider who you use this tool.

Our boss (John Boyanoski, who is not a fan of AI) recently wondered which MLB catcher had caught the most 300-game winners during their career. He knew Carlton Fisk has caught two, but wondered if anyone had more. So, like he has done since the early 2000s, he used a search engine. Google.

Seeing an AI answer, he clicked on that.

Here is the answer below.

John decided to keep going and asked for some clarity. By asking a few follow-up questions. Seeing some answers that didn't make sense, but figured the AI engine was finding All-Star games and exhibition games in the list, he asked this:

Without going into a lot of details, John knew these weren't the right answers, but refined his question some more.

It then got worse. The AI swore each time it had found the right answer. John would correct it. It would try again. John would correct it. See below for some examples.

Finally, he had enough (It wasn't Pat Borders either) and asked the following:

That should scare you. That should make you question those people who swear by AI as the answer for everything.

It should also make you wonder why the AI programmers from multi-billion dollar corporations are training it to lie (or as it stated a plausible sounding answer)?

Does that mean these mechanical marvels they taut as the future of the economy, really just being propped by lies?