Despite the hype surrounding its launch, recent third-party testing revealed that Google’s new Gemini Pro LLM falls slightly behind free OpenAI models like GPT-3.5 on most benchmarks. Gemini solved a slightly smaller number of problems compared to GPT-3.5, spanning across academics, reasoning, math, and other tasks.
Longer queries posed a challenge, as the system struggled to handle them effectively. Additionally, there was a tendency for aggressive content blocking and a bias towards certain end choices. Gemini’s proficiency in non-English language generation was remarkable, but the implementation of content filtering led to the blocking of responses in 10 language pairs. The claims made in the paper were challenged by a Google representative, who highlighted the company’s own research and emphasized the upcoming release of Gemini Ultra in early 2024.
Will Gemini receive coal in its stocking this holiday season? The spotlight is on Google’s flagship AI launch, and the results of this study clearly indicate that the tech giant is falling behind. If Gemini Ultra doesn’t present notable upgrades in the beginning of next year, Google might face some unpleasant consequences.
