Like the idea of shorter papers - I can definitely agree with your observation that a big part of most papers could be scrapped without a big problem.
Regarding your last point: I agree that we need to get better at fraud detection, however I am not sure if harsh, public punishment is the way to go about it. As you mentioned there is a lot of external pressure on (young) scientists to engage in "subtle fraud" and also, I believe that a lot of research mistakes do happen not with bad intentions but due to (sloppy) mistakes. In consequence, I was wondering whether it could be maybe more effective to push for a 'failure culture' in science and change the incentive system? This would entail e.g.:
- making it acceptable for someone to correct their own mistakes - if a scientist finds a mistake in his/her own study and corrects it, they should be congratulated and not shamed
- incentivize people to replicate/check each others work - checking other people's work currently isn't rewarded at all in the academic incentive system
- incentivize metastudies and aggregation of multiple studies (which would bolster the current evidence and make it harder for individual fraudsters to go unrecognized)
What do you think about that - curious to hear your opinion!
I think bounties or rewards for discovering fraud are a great idea! But I also think punishments are needed too, more so than we have now.
I suspect the problem of fraud, especially subtle fraud, is larger than generally recognized. Certainly people like Elizabeth Bik are finding a lot of it, and I've heard several different anecdotes about it now.
I tried to replicate some work a few years ago which I'm pretty sure was fraudulent. The person I believe committed the fraud also lied on their CV and website numerous times and went on to become a professor. I warned the hiring committee about the lies, which were quite egregious, but they got hired anyway.
While it might be hard to distinguish in practice, I believe there is an important distinction to make between fraud and mistakes. I think there should be hard punishment if people behave fraudulently. However, if it happens unintentionally and e.g. somebody publicly admits their mistakes and corrects them later on, I believe it would overall be more beneficial if these people are not punished - otherwise we (as it is currently the case) would disincentivize people to speak up publicly if they discover mistakes in their work.
I certainly agree that there is probably a big number of fraudulent work that remains undiscovered and I definitely see a need for more investigations in this area. Maybe it needs something like 'white hat hackers' for science that try to replicate (at least the data analysis part of) studies and uncover fraud cases. I would definitely be up for starting something like this!
I agree it's hard to distinguish fraud from mistakes or sloppiness. That's why I think the lab has to be locked down and a forensic team needs to be sent in to try to determine what actually happened. This has the secondary effect of incentivizing greater rigor, which is desperately needed since nobody wants their lab to be shut down for a few days for forensic teams to analyze everything. Anyone who reports fraud that is later verified should get a monetary reward.
I know this all sounds incredibly harsh and draconian but I think these are the measures we need, or at least we need to try for some time and see how it goes. I suspect a lot of "sloppiness" is actually deliberate fraud or fudging of data. The only way to figure out how much that is the case is to do forsenic investigations, I think.
I personally have failed to replicate a former lab-member's published results and could find no evidence on any computers that they actually trained the AI models they said they trained. It was pretty clearly fraud. So this entire matter hits home for me. The episode was frustrating for me, not only because I wasted time but because the person in question also did several other unethical things such as lying on their CV and website, and went on to receive a reward for the fraudulent work and become an assistant professor. I have spoken with at least two other people who have had similar experiences - they failed to replicate a former lab member's work and concluded it was deliberate fraud.
It's unfortunately tough to be a young academic these days -- lot of time spent trying to get positions, get grants and turn out papers.
I saw an interview of Peter Higgs a few years ago.... one sec..... ahh here it is:
"Peter Higgs, the British physicist who gave his name to the Higgs boson, believes no university would employ him in today's academic system because he would not be considered "productive" enough.
The emeritus professor at Edinburgh University, who says he has never sent an email, browsed the internet or even made a mobile phone call, published fewer than 10 papers after his groundbreaking work, which identified the mechanism by which subatomic material acquires mass, was published in 1964.
Speaking to the Guardian en route to Stockholm to receive the 2013 Nobel prize for science, Higgs, 84, said he would almost certainly have been sacked had he not been nominated for the Nobel in 1980. Edinburgh University's authorities then took the view, he later learned, that he "might get a Nobel prize – and if he doesn't we can always get rid of him"....."
Great quote! Thanks Howard! I think it would be interesting and survey how many publications the most important physicists of the 20th century wrote. If I recall, some of them published surprisingly few works, but what they did publish was of stunning quality and originality. The stat that comes to mind is that Richard Feynman only published 37 papers in his entire career!
Like the idea of shorter papers - I can definitely agree with your observation that a big part of most papers could be scrapped without a big problem.
Regarding your last point: I agree that we need to get better at fraud detection, however I am not sure if harsh, public punishment is the way to go about it. As you mentioned there is a lot of external pressure on (young) scientists to engage in "subtle fraud" and also, I believe that a lot of research mistakes do happen not with bad intentions but due to (sloppy) mistakes. In consequence, I was wondering whether it could be maybe more effective to push for a 'failure culture' in science and change the incentive system? This would entail e.g.:
- making it acceptable for someone to correct their own mistakes - if a scientist finds a mistake in his/her own study and corrects it, they should be congratulated and not shamed
- incentivize people to replicate/check each others work - checking other people's work currently isn't rewarded at all in the academic incentive system
- incentivize metastudies and aggregation of multiple studies (which would bolster the current evidence and make it harder for individual fraudsters to go unrecognized)
What do you think about that - curious to hear your opinion!
I think bounties or rewards for discovering fraud are a great idea! But I also think punishments are needed too, more so than we have now.
I suspect the problem of fraud, especially subtle fraud, is larger than generally recognized. Certainly people like Elizabeth Bik are finding a lot of it, and I've heard several different anecdotes about it now.
I tried to replicate some work a few years ago which I'm pretty sure was fraudulent. The person I believe committed the fraud also lied on their CV and website numerous times and went on to become a professor. I warned the hiring committee about the lies, which were quite egregious, but they got hired anyway.
While it might be hard to distinguish in practice, I believe there is an important distinction to make between fraud and mistakes. I think there should be hard punishment if people behave fraudulently. However, if it happens unintentionally and e.g. somebody publicly admits their mistakes and corrects them later on, I believe it would overall be more beneficial if these people are not punished - otherwise we (as it is currently the case) would disincentivize people to speak up publicly if they discover mistakes in their work.
I certainly agree that there is probably a big number of fraudulent work that remains undiscovered and I definitely see a need for more investigations in this area. Maybe it needs something like 'white hat hackers' for science that try to replicate (at least the data analysis part of) studies and uncover fraud cases. I would definitely be up for starting something like this!
I agree it's hard to distinguish fraud from mistakes or sloppiness. That's why I think the lab has to be locked down and a forensic team needs to be sent in to try to determine what actually happened. This has the secondary effect of incentivizing greater rigor, which is desperately needed since nobody wants their lab to be shut down for a few days for forensic teams to analyze everything. Anyone who reports fraud that is later verified should get a monetary reward.
I know this all sounds incredibly harsh and draconian but I think these are the measures we need, or at least we need to try for some time and see how it goes. I suspect a lot of "sloppiness" is actually deliberate fraud or fudging of data. The only way to figure out how much that is the case is to do forsenic investigations, I think.
I personally have failed to replicate a former lab-member's published results and could find no evidence on any computers that they actually trained the AI models they said they trained. It was pretty clearly fraud. So this entire matter hits home for me. The episode was frustrating for me, not only because I wasted time but because the person in question also did several other unethical things such as lying on their CV and website, and went on to receive a reward for the fraudulent work and become an assistant professor. I have spoken with at least two other people who have had similar experiences - they failed to replicate a former lab member's work and concluded it was deliberate fraud.
Good ideas.
It's unfortunately tough to be a young academic these days -- lot of time spent trying to get positions, get grants and turn out papers.
I saw an interview of Peter Higgs a few years ago.... one sec..... ahh here it is:
"Peter Higgs, the British physicist who gave his name to the Higgs boson, believes no university would employ him in today's academic system because he would not be considered "productive" enough.
The emeritus professor at Edinburgh University, who says he has never sent an email, browsed the internet or even made a mobile phone call, published fewer than 10 papers after his groundbreaking work, which identified the mechanism by which subatomic material acquires mass, was published in 1964.
Speaking to the Guardian en route to Stockholm to receive the 2013 Nobel prize for science, Higgs, 84, said he would almost certainly have been sacked had he not been nominated for the Nobel in 1980. Edinburgh University's authorities then took the view, he later learned, that he "might get a Nobel prize – and if he doesn't we can always get rid of him"....."
Howard (Toronto)
Great quote! Thanks Howard! I think it would be interesting and survey how many publications the most important physicists of the 20th century wrote. If I recall, some of them published surprisingly few works, but what they did publish was of stunning quality and originality. The stat that comes to mind is that Richard Feynman only published 37 papers in his entire career!