AI is actually supposed to support us in everyday life – but in this case it makes our lives more difficult. (Symbolic image, source: Adobe Stock – ipopba)
In recent weeks there has been a lively discussion about the use of AIs in scientific studies. Also the magazine 404Media reported on March 18, 2024 about questionable ChatGPT phrases from more reputable publishers.
Of course, artificial intelligences are intended to make our everyday lives easier – but what harm do they cause if they are used in scientific work and studies without being checked? A search for clues.
Why AI-generated texts are a problem in studies
Artificial intelligence has largely been developed to support our everyday lives. ChatGPT can, for example, write email texts for us. However, there is also the option of giving the AI the following task: “Write me a scientific study about earthworms”.
The AI then makes use of it existing sources from the Internet or your learned database – so you can no new knowledge in the classic research sense to generate or own conduct research. The AI therefore only reproduces existing or learned knowledge.
In other words, it is someone else’s, stolen knowledge and can potentially contain errors.
According to the magazine 404Media, several X-postings on this topic (formerly Twitter) are currently going viral on Twitter, criticizing this situation:
Link to Twitter content
This is what the author of this post writes:
Things get even worse. If you search Google Scholar for “my last knowledge update” or “I don’t have access to real-time data,” tons of AI-generated papers pop up.
We tested it too. At Google Scholar we get around 188 results that match the sentence As of my last knowledge update
include. This is a typical ChatGPT phrase that is intended to make it clear to the user where the AI’s knowledge limit is.
Because: For a long time, ChatGPT used a fixed offline knowledge pool with a time frame and only later became directly connected to the Internet.
Important: Given the corresponding number of studies, we were not able to check whether the content was actually generated. However, Google Scholar alone contains over 170 studies that were published using the classic AI phrase after 2020.
It is therefore very likely that at least some sections with the corresponding formulations were generated with AI support.
The following restrictions apply: Some studies themselves are about AI and those formulation aids. On the other hand, it can also be assumed that the number of unreported cases will be significantly higher. The corresponding AI phrase trace can of course be erased later.
However, many authors and publicists don’t even seem to make the effort. 404Media gave another Twitter example here:
Link to Twitter content
Here is a possible introduction to your topic
is even included here as an AI sentence right at the beginning of the published study.
The problem: According to 404Media, these are often smaller ones Paper Mills
, i.e. magazines with low editorial standards. In some cases, quick publication is promised in exchange for money and the content is little or not checked at all.
But also large magazines like the renowned Nature
fight against the flood of bogus studies. According to their statements, around 2023 alone 10,000 studies withdrawn or rejected.
Here’s how you can check a study with three simple tricks
Of course, AI phrases aren’t the only indication of a flawed study. If you need exact sources for your own scientific work, a presentation or something similar or are not sure whether you can trust what is written in a study, there are three simple tricks:
Trick 1: Check sample size
Sample size means the number of study participants. For example, the more patients have taken part in a clinical trial, the more accurately a result can be determined. This number can usually be found in the results section.
So if you read in the results of a study that only 8 patients tested a new inhaler, the significance is logically lower than with a sample size of 400 or more.
Trick 2: How high is the impact factor of the publishing medium?
Important: The impact factor (IF) is not a direct quality feature. The number indicates how often the articles and publications of a medium are cited in other scientific publications per year.
In practice, however, the IF is often used by governments, libraries or universities to assess the medium. As a rule, it is enough if you, for example Impact-Factor Nature Magazin
googled. The rule of thumb: The higher the value, the better. Values above 10 are considered outstanding.
For those interested: One The Helmholz Institute has more detailed instructions published.
Trick 3: Check the authors’ Hirsch factor
The Hirsch factor or index, similar to the impact factor for media, says how often an author was recited in another academic work. A high H-index means that the author has been considered a trustworthy source by many scientists and the author’s work is therefore often recited. This is what it looks like:
Here you can see the famous physicist Stephen Hawking as an example. You can see its high h-index in the picture on the right. (Source: Screenshot Google Scholar)
That’s how it’s done: Go for it Google Scholar. Here you can search for the scientist in the search field, for example Stephen Hawking. Then click on the profile. Then Google Scholar should give you data about the exact number of recitations as well as the H-index on the right.
Of course, these are also just a few straws of security. Unfortunately, you are never completely immune to counterfeits. But with the right tricks, we can at least do something about it.
Have you already noticed similar problems? Do you know any other tricks for classifying studies? Then feel free to write it in the comments.
Table of Contents
