Financial Advice from AI is Mostly Wrong
Millions of AI users trust LLMs for financial advice. They probably shouldn’t. On September 19, the Financial Times published an article titled, “AI chatbots give wrong answers to financial queries most of the time.” Reporter Aliya Shibli wrote:
Using AI chatbots to address financial questions could risk large losses, with models from a range of providers giving wrong answers most of the time, research has found.
The most popular AI models from ChatGPT, Claude, Copilot, Grok and Gemini provided wrong answers to financial queries 57% of the time on average, according to research from technology firm Saturn. When asked more complex questions, such as those which involved more than one calculation, the AI models made mistakes in 88% of cases on average. Some models gave wrong answers to 99% of these harder questions.
The research used more than 100 different money-related questions through the free and paid-for AI models from providers including ChatGPT, Gemini, Claude and Copilot. Questions were repeated up to five times, with more than 10,000 questions collectively put to 18 different AI models. Their answers contained calculation errors, omitted upcoming tax changes or hallucinated rules, according to the research.
AI put to the test
Saturn’s research team tested 18 of the most popular AI models asking 121 questions about financial advice and found that, on average, “models made mistakes 57% of the time.” A key finding was that “all types of LLMS tested have a low level of accuracy for all categories of questions, easy, medium and hard. For the easy questions, the accuracy was just 54%. On hard questions, the accuracy dropped to just 12%.”
The free-to-use models failed 53% of the time while the best-performing paid-for models performed a bit better. The top performer, Claude Opus 5, failed 39% of the time across all question types and made mistakes 67% of the time on the hardest questions.
Models make a wide range of errors, and even hallucinate
The study “uncovered failures across a broad spectrum of issues, affecting consumers in different age groups and income brackets, including debt, student loans, mortgages, pensions, tax and savings. Serious errors across the models tested included quoting incorrect figures, using out-of-date tax rules, leaving out key warnings or omitting crucial points.”
Wrong answers were given in the same confident, authoritative tone as correct ones, creating a misleading impression to consumers. Some of the mistakes could have resulted in tax penalties or mistakes on loan and mortgage applications. In one case, Claude invented a rule that could have resulted in a graduate being penalised by being put on the wrong student loan repayment schedule.
AI has potential
Al has potential to help people navigate financial decisions, and for some easy questions can be a good resource, but the quality of the advice it’s providing today is unreliable. Users should be aware of AI’s limitations, do their own independent research and consult with financial professionals before making important financial decisions. In the future, we can be sure AI will improve and become better at helping humans answer financial questions but, for now, beware.
◇ ◇ ◇
Some interesting things I came across this week…
A Perfect Storm Is Raging in the Bond Market (Wall Street Journal)
Bitcoin Extends Winning Streak, Trades Above $86K (Coindesk)
Copper Demand is Accelerating as Supply Deficit Looms (S&P Global)
AI Build-Out on Track to Become Biggest Economic Bet in U.S. History (Wall Street Journal)
Ray Dalio: AI will be ‘miraculous’ for productivity but a ‘devastating’ investment bubble (Fortune)
Hyperscaler Credit Rests on One Key Assumption (The Daily Spark)
Business Leaders at Salesforce Annual Conference: Last year’s AI models are enough (CNBC)
◇ ◇ ◇
Off the beaten path...
U.S. Department of War publishes sixth release of declassified UFO files (WAR.GOV/UFO)
The government is releasing more UFO history, but where’s the rest of it? (YouTube)
Inside the Geneva Freeport: A Warehouse in Switzerland Holding Priceless Treasurs (YouTube)
Secrets of Roman Architecture: The Pantheon and its Majestic Dome (YouTube)
◇ ◇ ◇
When you're ready, here's how I can help:
Review your investments. Your "buy-and-hope" portfolio presents a serious risk to your retirement plan. I'll review your holdings one-by-one and share ideas for how to de-risk.
Free portfolio analysis. Get a comprehensive portfolio analysis with Portfolio Visualizer, an institutional-grade tool that measures portfolio risk, volatility and diversification gaps.
Free Retirement Planning Tool. I've partnered with ProjectionLab, a simple but detailed financial planning tool that will reduce anxiety around your finances.
Learn how we invest for clients. Every quarter we provide clients with a detailed Investment Outlook that explains exactly how they're invested and why.
Thanks for reading. I'd love to hear from you!
I founded Alpha Rock to provide investors the highest level of personal service. Discover the benefits of working with a solo advisor and contact me directly with your thoughts, questions and feedback.
IMPORTANT DISCLOSURE INFORMATION
This commentary reflects the personal opinions, viewpoints and analyses of the Alpha Rock Investments, LLC employees providing such comments, and should not be regarded as a description of advisory services provided by Alpha Rock Investments, LLC or performance returns of any Alpha Rock Investments, LLC client. The views reflected in the commentary are subject to change at any time without notice. Nothing in this commentary constitutes investment advice, performance data or any recommendation that any particular security, portfolio of securities, transaction or investment strategy is suitable for any specific person. Any mention of a particular security and related performance data is not a recommendation to buy or sell that security. Alpha Rock Investments, LLC manages its clients’ accounts using a variety of investment techniques and strategies, which are not necessarily discussed in the commentary. Investments in securities involve the risk of loss. Past performance is no guarantee of future results.
The S&P 500 Index or the Standard & Poor's 500 Index is a market-capitalization-weighted index of the 500 largest U.S. publicly traded companies. The S&P 500 is a float-weighted index, meaning company market capitalizations are adjusted by the number of shares available for public trading. Note: Investors cannot invest directly in an index. These unmanaged indices do not reflect management fees and transaction costs that are associated with most investments.
Alpha Rock Investments, LLC provides links for your convenience to websites produced by other providers of industry related material. Accessing websites through links directs you away from our website. Alpha Rock Investments, LLC is not responsible for errors or omissions in the material on third party websites, and does not necessarily approve of or endorse the information provided. Users who gain access to third party websites may be subject to the copyright and other restrictions on use imposed by those providers and assume responsibility and risk from use of those websites.
