In total, the team evaluated more than 700 hypotheses and asked the same question 10 times for each one to measure consistency.

Accuracy Results and Limits of AI Performance

When the experiment was first conducted in 2024, ChatGPT answered correctly 76.5% of the time. In a follow-up test in 2025, accuracy rose slightly to 80%. However, once the researchers adjusted for random guessing, the results looked far less impressive. The AI performed only about 60% better than chance, a level closer to a low D than to strong reliability.

The system had the most difficulty identifying false statements, correctly labeling them only 16.4% of the time. It also showed notable inconsistency. Even when given the exact same prompt 10 times, ChatGPT produced consistent answers only about 73% of the time.

Inconsistent Answers Raise Concerns

“We’re not just talking about accuracy, we’re talking about inconsistency, because if you ask the same question again and again, you come up with different answers,” said Cicek, an associate professor in the Department of Marketing and International Business in WSU’s Carson College of Business and lead author of the new publication.

“We used 10 prompts with the same exact question. Everything was identical. It would answer true. Next, it says it’s false. It’s true, it’s false, false, true. There were several cases where there were five true, five false.”

AI Fluency vs. Real Understanding

The findings, published in the Rutgers Business Review, highlight the importance of using caution when relying on AI for important decisions, especially those that require nuanced or complex reasoning. While generative AI can produce smooth, convincing language, it does not yet demonstrate the same level of conceptual understanding.

According to Cicek, these results suggest that artificial general intelligence capable of truly “thinking” may still be further away than many expect.

“Current AI tools don’t understand the world the way we do — they don’t have a ‘brain,'” Cicek said. “They just memorize, and they can give you some insight, but they don’t understand what they’re talking about.”

Study Design and Methods

Cicek worked with co-authors Sevincgul Ulu of Southern Illinois University, Can Uslay of Rutgers University, and Kate Karniouchina of Northeastern University.

The team used 719 hypotheses from scientific studies published in business journals since 2021. These types of questions often involve nuance, with multiple factors influencing whether a hypothesis is supported. Reducing such complexity to a simple true or false judgment requires careful reasoning.

The researchers tested the free version of ChatGPT-3.5 in 2024 and the updated ChatGPT-5 mini in 2025. Overall, performance remained similar across both versions. After adjusting for random chance, which gives a 50% probability of a correct answer, the AI’s effectiveness was only about 60% above chance in both years.

Key Weakness in AI Reasoning

The results point to a fundamental limitation of large language model AI systems. Although they can generate fluent and persuasive responses, they often struggle to reason through complicated questions. This can lead to answers that sound convincing but are actually incorrect, Cicek said.

Why Experts Urge Caution With AI

Based on these findings, the researchers recommend that business leaders verify AI-generated information and approach it with skepticism. They also emphasize the need for training to better understand what AI systems can and cannot do effectively.

Although this study focused specifically on ChatGPT, Cicek noted that similar experiments with other AI tools have produced comparable outcomes. The work also builds on earlier research pointing to caution around AI hype. A 2024 national survey found that consumers were less likely to purchase products when they were marketed with a focus on AI.

“Always be skeptical,” he said. “I’m not against AI. I’m using it. But you need to be very careful.”

The world, called L 98-59 d (an exoplanet, meaning it orbits a star outside our Solar System), circles a small red star about 35 light-years from Earth. Data from the James Webb Space Telescope (JWST) and ground based observatories revealed something unusual. For a planet about 1.6 times the size of Earth, it has a surprisingly low density and an atmosphere rich in hydrogen sulfide.

A Planet That Defies Classification

Until now, scientists would have grouped a planet like L 98-59 d into one of two categories. It could be a rocky “gas-dwarf” with a hydrogen dominated atmosphere, or a water rich world covered by deep oceans and ice.

New evidence shows it fits neither category. Instead, L 98-59 d appears to belong to a completely different class of planet dominated by heavy sulfur compounds.

A Global Magma Ocean Beneath the Surface

To understand this unusual world, researchers from the University of Oxford, the University of Groningen, the University of Leeds and ETH Zurich used advanced computer simulations to trace its evolution from shortly after formation to today, spanning nearly five billion years. By combining telescope observations with detailed models of planetary interiors and atmospheres, they were able to infer what is happening deep inside the planet.

Their findings suggest that L 98-59 d has a mantle made of molten silicate, similar to lava on Earth. Beneath its surface lies a vast magma ocean extending thousands of kilometers deep. This enormous reservoir allows the planet to trap large amounts of sulfur within its interior over long periods of time.

The magma ocean also helps maintain a thick hydrogen rich atmosphere that contains sulfur bearing gases such as hydrogen sulfide (H2S). Normally, radiation from the host star would gradually strip these gases away into space through X-ray driven processes.

Sulfur Cycling Between Interior and Atmosphere

Over billions of years, ongoing chemical exchanges between the molten interior and the atmosphere have shaped the planet’s current appearance. These interactions explain the unusual signals detected by telescopes.

Researchers suggest that L 98-59 d may be the first identified example of a broader population of gas rich sulfur dominated planets that sustain long lived magma oceans. If that is the case, it points to a much wider variety of planetary types across the galaxy than previously recognized.

Lead author Dr. Harrison Nicholls (Department of Physics, University of Oxford) said: “This discovery suggests that the categories astronomers currently use to describe small planets may be too simple. While this molten planet is unlikely to support life, it reflects the wide diversity of the worlds which exist beyond the Solar System. We may then ask: what other types of planet are waiting to be uncovered?”

How Sulfur Shapes the Atmosphere

JWST observations from 2024 detected sulfur dioxide along with other sulfur gases high in the upper atmosphere of L 98-59 d. According to the team’s models, these gases form when ultraviolet radiation from the host star, the red dwarf L 98-59, drives chemical reactions.

At the same time, the magma ocean below acts as a massive storage system for volatile materials, absorbing and releasing gases over billions of years after the planet formed. This combination of deep interior storage and ultraviolet driven chemistry explains the planet’s distinctive properties.

Simulations indicate that L 98-59 d likely formed with a large supply of volatile material and may once have resembled a larger sub-Neptune type planet. Over time, it cooled, lost part of its atmosphere, and became smaller.

Scientists note that magma oceans are thought to be the initial state of all rocky planets (including the Earth and Mars). Studying these environments on distant worlds can provide insight into the earliest stages of our own planet’s history.

Reconstructing Alien Worlds With Models

Co-author Professor Raymond Pierrehumbert (Department of Physics, University of Oxford) said: “What’s exciting is that we can use computer models to uncover the hidden interior of a planet we will never visit. Although astronomers can only measure a planet’s size, mass and atmospheric composition from afar, this research shows that it is possible to reconstruct the deep past of these alien worlds — and discover types of planets with no equivalent in our own Solar System.”

JWST is already delivering a growing stream of data, and future missions such as Ariel and PLATO are expected to expand that dataset even further. The research team plans to apply their models to these observations using machine learning to map the diversity of planets beyond our Solar System and link them to their early development.

By doing so, scientists hope to better understand how planets form and evolve, and to identify which types of worlds might be capable of supporting life.

Dr. Richard Chatterjee (University of Leeds/ University of Oxford) said: “Our computer models simulate various planetary processes, effectively enabling us to turn back the clock and understand how this unusual rocky exoplanet, L 98-59 d, evolved. Hydrogen sulfide gas, responsible for the smell of rotten eggs, appears to play a starring role there. But, as always, more observations are needed to understand this planet and others like it. Further investigation may yet show that rather pungent planets are surprisingly common.”