Bug In Python Script May Have Affected Hundreds Of Studies
A set of Python scripts popularly used to analyze chemistry data has had a long-standing bug in that it had assumptions about the order in which filenames in a directory would be returned https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/. The code worked correctly on Windows and an older version of macOS, but discrepancies crept in on Linux and on newer macOS.
The fix was to sort the returned names into a consistent order before processing. I just note this Twitter comment from one of the original authors:
When I wrote the scripts 6 years ago, the OS was able to handle the sorting.
That would never have been true on Linux. There have never been any guarantees about ordering of directory entries on POSIX-compliant systems. Note the docs https://docs.python.org/3/library/glob.html#glob.glob clearly say:
Whether or not the results are sorted depends on the file system.
One of the researchers who found (and fixed) the bug says
... he hopes the paper will get scientists to pay more attention to the computational side of experiments in the future.
A set of Python scripts popularly used to analyze chemistry data has had a long-standing bug in that it had assumptions about the order in which filenames in a directory would be returned https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/. The code worked correctly on Windows and an older version of macOS, but discrepancies crept in on Linux and on newer macOS.
The fix was to sort the returned names into a consistent order before processing. I just note this Twitter comment from one of the original authors:
When I wrote the scripts 6 years ago, the OS was able to handle the sorting.That would never have been true on Linux. There have never been any guarantees about ordering of directory entries on POSIX-compliant systems. Note the docs https://docs.python.org/3/library/glob.html#glob.glob clearly say:
Whether or not the results are sorted depends on the file system.One of the researchers who found (and fixed) the bug says
... he hopes the paper will get scientists to pay more attention to the computational side of experiments in the future.
The upside: the script was made available with the study. Lots of studies cannot be replicated due to data and/or scripts/programs not being made available as well. Hence the "open science" movement (https://en.wikipedia.org/wiki/Open_science).
Cheers, Peter
On Wed, 16 Oct 2019 11:42:59 +1300, Peter Reutemann wrote:
The upside: the script was made available with the study. Lots of studies cannot be replicated due to data and/or scripts/programs not being made available as well. Hence the "open science" movement (https://en.wikipedia.org/wiki/Open_science).
Yup. It seems so obvious to me (and to lots of others) that when you publish the results of scientific research, you have to make available notes, raw data and other stuff that would be necessary for others to replicate your results. This would have to include any essential processing software. Otherwise how can we trust them?
Here https://www.theregister.co.uk/2019/10/15/bug_python_scripts/ is a further article, which includes a copy of the buggy function. How depressing is it that all it does is collect the items of a list, only to append them to another list ...
The upside: the script was made available with the study. Lots of studies cannot be replicated due to data and/or scripts/programs not being made available as well. Hence the "open science" movement (https://en.wikipedia.org/wiki/Open_science).
Yup. It seems so obvious to me (and to lots of others) that when you publish the results of scientific research, you have to make available notes, raw data and other stuff that would be necessary for others to replicate your results. This would have to include any essential processing software. Otherwise how can we trust them?
To people working with and on open source projects, this is second nature.
However, a lot of academics are not necessarily experts in programming and other tools that they require in order to analyze data. I've encountered it before that people didn't want other people to see their "ugly" code (can't say that I'm proud of all the code that I've written over the years!). In other instances, they don't like being told that there is a bug. Admitting mistakes is hard, but reality is that they happen. Rather than seeing it as criticism, it should be viewed as encouragement to improve things. I mean, I'm well aware of the fact that my (large) software projects still contain lots of bugs and I'm glad when somebody stumbles upon one and reports it. :-)
Here https://www.theregister.co.uk/2019/10/15/bug_python_scripts/ is a further article, which includes a copy of the buggy function. How depressing is it that all it does is collect the items of a list, only to append them to another list ...
Oh dear... It was probably a fluke on their system back then that, when they checked, the files got returned in a (seemingly?) ordered fashion.
Cheers, Peter
On Wed, 16 Oct 2019 12:09:14 +1300, Peter Reutemann wrote:
However, a lot of academics are not necessarily experts in programming and other tools that they require in order to analyze data.
Some worthwhile reader comments on this sort of thing in that Register article. One point made was that, if the quality of your scientific results depend so crucially on the quality of this code, then the code is very much a part of your science and has to be treated as such.
However, a lot of academics are not necessarily experts in programming and other tools that they require in order to analyze data.
Some worthwhile reader comments on this sort of thing in that Register article. One point made was that, if the quality of your scientific results depend so crucially on the quality of this code, then the code is very much a part of your science and has to be treated as such.
I totally agree.
Cheers, Peter
On Wed, 16 Oct 2019 12:14:23 +1300, I wrote:
Some worthwhile reader comments on this sort of thing in that Register article.
Including this one https://forums.theregister.co.uk/forum/all/2019/10/15/bug_python_scripts/#c_3893686 which mentions a similar situation that arose with a piece of software called FreeSurfer, which in 2012 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234 was discovered to produce results that varied by OS platform and version. Also talks about two different meanings of “Wiener filter”, both of which are implemented in (different parts of) SciPy.
On Wed, 16 Oct 2019 11:42:59 +1300, Peter Reutemann wrote:
The upside: the script was made available with the study. Lots of studies cannot be replicated due to data and/or scripts/programs not being made available as well. Hence the "open science" movement (https://en.wikipedia.org/wiki/Open_science).
Nice discussion here https://arstechnica.com/science/2019/10/a-tale-of-two-retractions/ about what happens to their articles reporting on research results which are subsequently retracted. Turns out there’s a whole “Retraction Watch” website that reports on retracted papers, but it doesn’t always make clear why they were withdrawn. This information, too, can be scientifically useful, as guides on how not to make the same mistakes again.
participants (2)
-
Lawrence D'Oliveiro -
Peter Reutemann