Australian aged care providers are increasingly exploring artificial intelligence to predict health risks such as strokes and diabetes complications among residents. However, new research highlights serious flaws in many AI-driven clinical prediction models, raising concerns about patient safety and the reliability of tools entering the sector.
A study published in BMC Medicine on 4 June 2026 examined two popular public datasets on Kaggle used to build prediction models for stroke and diabetes. Researchers found major deficiencies in data provenance, the essential information detailing where, when, why, and how data were collected.
Both datasets scored zero out of nine on key TRIPOD+AI checklist items related to data provenance. There was no verifiable information on data origins, collection methods, or authenticity.
The study authors, including Alexander D Gibson and Adrian G Barnett, concluded that the datasets “have no reliable provenance of authenticity and should not be used for informing research or practice.”
Exploratory analyses revealed troubling irregularities. The diabetes dataset, which contained exactly 100,000 patients, showed implausible patterns such as only 18 discrete values for blood glucose and HbA1c levels, unusual spikes in BMI and age distributions, and duplicated observations.
The stroke dataset also displayed abnormal truncations and shifts in key variables, including blood glucose levels. These features suggest the data may be synthetic or fabricated, yet the datasets have been downloaded hundreds of thousands of times.
From these datasets, the researchers identified 125 peer-reviewed clinical prediction model studies. Many made practical recommendations for real-world use. For the stroke dataset, 67 per cent of articles suggested clinical applications, while 80 per cent of diabetes-related articles did the same.
Three models showed evidence of potential use in clinical practice, and one was cited in a medical device patent. Collectively, the articles have received thousands of citations, including references in 86 review papers.
“Prediction models based solely on inauthentic or unreliable datasets should never be used to directly inform decisions on patient care,” the authors stated.
In aged care, where residents are often frail and managing multiple chronic conditions, the stakes are particularly high.
Stroke and diabetes prediction tools could influence decisions about monitoring, medication, hospital transfers, or end-of-life care. Flawed models risk denying necessary treatments or leading to unnecessary interventions, potentially causing harm to vulnerable older people.
Australian aged care providers piloting AI solutions for early risk detection should proceed cautiously. Many models rely on publicly available datasets without rigorous checks. The study notes that poor data provenance is not limited to these two examples and likely affects other data repositories.
Only a small fraction of the 125 studies properly addressed ethics, with just three per cent stating that ethical approval had been obtained and seven per cent noting that it was not required. Most provided no ethical statement at all.
This is concerning even for publicly available data, as the original collection methods remain unknown.
The researchers recommend mandatory data provenance reporting requirements for journals, publishers, and data repositories. This includes clear details on data collection purpose, sources, locations, dates, funders, and measurement definitions.
They urge repositories such as Kaggle to implement stricter requirements and encourage journals to reject or flag articles using unreliable data.
For aged care providers and clinicians, the advice is to scrutinise AI tools carefully. Published models should not automatically be assumed to be reliable. Providers should prioritise tools with transparent data sources, external validation, and adherence to standards such as TRIPOD+AI.
Pre-registered studies and open data sharing can help improve transparency, but basic quality checks remain essential.
The study highlights a broader issue in the rush to adopt AI. While well-developed models hold promise for improving outcomes in aged care, including better chronic disease management, unreliable models risk wasting resources and eroding trust.
As the sector moves further into AI adoption, Australian providers, regulators, and researchers must demand higher standards of data integrity to protect residents.
The full study is available open access in BMC Medicine. Aged care organisations should consider these findings when evaluating or developing predictive AI tools.
