Bing's new GPT-4 chat mode has shown its potential as an Xbox game guide writer, but it looks like there's a long way to go before it competes with real human writers.
While it's currently unknown if the new Bing AI chat mode is coming to Xbox Series X, the software is now available as a preview for those who join a waiting list. According to Xbox Wire editor Mike Nelson, "he knows an awful lot about video games." This was tested in a way that describes both the potential and some problems that could arise when a continuous learning algorithm addresses game-specific knowledge.
Detailed in the full press release - opens in a new tab - the sites Bing pulls information from to try to answer questions you have about some of the best Xbox Series X games. One particular example was asking the new AI chat GPT-4 mode to give you a detailed rundown of what happens in the first 20 hours of The Witcher 3: Wild Hunt, pulled from ten different posts, including our sister site PC Gamer and various YouTube sources.
While the information presented was accurate through the first half of CD Projekt Red's work, the breadth of sources, including those from YouTube, indicates exactly how well Bing's AI will verify what is provided.
Not all online guides are created equal, and information on ideal in-game strategies can vary widely. Therefore, combining a variety of sources that vary in credibility will not always guarantee an accurate answer.
What is the correct answer again?
My concerns extend to the accuracy presented in the Bing AI GPT-2 chat mode Overwatch 4 question. The test message asks, "Which is the best Overwatch 2 character for me?" To this question, the AI answers that there are a total of 33 characters to choose from in the game with Damage, Tank, and Support classes.
Unfortunately, at the time of writing, there are actually 36 total playable characters in the game, which means that the information pulled is out of date. Rammatra is the newest character and was added back in December in Season 2, which means the AI is around three months behind.
Of the 10 sources cited, it seems GPT-4 had a hard time discerning which is the most up-to-date answer because it gives a vague answer: "You may want to take a test that matches your personality and character preferences" rather than offer viable options based on the current meta. Bing was able to produce something that sounded good on the surface but didn't really answer the question in a meaningful way. He liked Echo, the AI character, but he didn't explain anything about the playstyle or the character's quirks outside of that.
This seems to be the biggest problem when asking questions about Bing games and expecting accuracy. Given that YouTube has repeatedly been cited as a credible source, what's to stop people from deliberately spreading misinformation on a topic and then adapting it into AI copy?
It also raises questions about how GPT-4 cites existing websites directly and how appropriate what is said is in the source material. Writers are not given the credit they deserve, and the information attached to their names and posts may not fully reflect the original intent.