At the 2021 SORTEE conference this past summer, I gave a talk about my experience conducting a reproducibility audit and cleaning up a colleague’s workflow as she prepared to publish her M.S. thesis research.
Here, I’m embedding the video of the talk, and also reproducing it as a blog post (slides and captions) in case you’d rather read than listen/watch.
The slide descriptions are a pretty direct transcription of what I said during the talk, so if the style seems strange for a blog post, that’s why.
Scroll down to watch the video!
Slides and descriptions
Hello everyone! Thank you for being here. My name is Kaija Gahm, and I’m a data manager at the Cary Institute of Ecosystem Studies. Late last year, my colleague Chelsea had just finished her Master’s thesis, and my supervisor asked me to help her to organize her code and analysis into a reproducible workflow that we could reference in the manuscript that we were putting together.
In this talk, I’m going to be speaking about my experience implementing data management and reproducibility tools retroactively, for an existing project, even if best practices weren’t used from the beginning. I’ll also discuss organizing and making sense of someone else’s analyses, approaching the project as an outsider.
A little bit of background on the project: It consisted of some pretty typical ecological data––measurements of lake dissolved organic carbon, morphometric landmarks of bluegill sunfish, and a bunch of statistical analyses and figures.
Some of the challenges I knew I was going to face were that the data had largely been edited manually in Excel, or copied and pasted from other files with uncertain origin.
There were many file versions, and it wasn’t really clear how the data and code tied together.
The goal of my work was to compile all of this data and code into a reproducible analysis. I knew that we were going to be storing the data on Figshare; all of the analyses would be conducted in RStudio and stored on GitHub. The data and code could be then referenced in the final manuscript.
To begin my work, I asked Chelsea to start by uploading all of her files to GitHub.
I think GitHub is a very important tool, not just technically, for version control, but also psychologically. If everything was being version controlled, then I would have the power to edit and delete without being burdened by the idea that I might be deleting something very important. I knew that as long as everything was on GitHub, I could always walk the repository back to its original version.
My first step in organizing this repository was to set it up as an RStudio Project, which means that it would open a fresh R session for everyone who downloaded the code, and all file paths could be written relative to the root directory.
I also implemented renv, which is a great package that allows for managing package versions. This was particularly important in the case of this analysis, because some of the code depended on an old version of the geomorph package for analyzing morphometric landmarks. I personally wasn’t very familiar with the package, so even if I had wanted to update the code to the newest version, I wouldn’t really have been able to. Using renv allowed me to run the code with the old version of the package, and it ensures that anyone who wants to run our analysis in the future will be able to restore the packages from the lockfile and immediately proceed without affecting packages on their own computer.
My next step was to organize the data.
The first principle is that raw data is sacred and should not be modified. So that means no copying and pasting.
Derived data should then be fully reproducible and, by extension, disposable. I should be able to just delete all of the derived data, and then recreate it by re-running the scripts in the pipeline.
Now, that might be a bit of a pain because some of the analyses might take a long time to run, but it would be possible in a fully reproducible analysis.
And my job approaching this project was to distinguish between data that was raw––like maybe it was created manually or it was exported from a program somewhere else––versus data that could be regenerated from the analysis code.
This is a list of all the data files that I was originally presented with in the GitHub repository, and it was pretty overwhelming. I had to sort through all of those and figure out which were raw, which were derived, and where they all went in the pipeline.
And in order to do this, I asked the key question, ‘Where did this come from?’ For each file, I wanted to know, how was it generated? Can I trace the origin of all the files and their columns?
I started out by making a spreadsheet listing out each file, and this seemed like a good idea at the time, but it ended up being pretty overwhelming, and it was hard for Chelsea to go through and answer my questions, especially because I was attacking all the files at once, without regard for their importance or the order that they might go in. I was just trying to get my head around it.
Chelsea and I decided that a better approach would be to start with the most recent and up-to-date analysis file; in our case, called ReviewApril2020.
Then, I could look at all of the data files that were read in as inputs. It was much easier, with this reduced set of files, to determine…
…which ones were raw data and which ones were derived data. So, my next step was, for each of those derived data files, to figure out how to re-create it from the data in the database, using R scripts, so that the process would be fully reproducible.
To do that, I created one R script for each of the derived data files, and I used that R script to read in data from the database stored on Figshare and modify it to generate these ‘FINAL’ csv files. In the background, I kept the original data and used it as a comparison, just to make sure I was accurately recreating that original data.
My next step was to organize the code in the analysis files. I had to start by reorganizing a lot of file paths; they had been written as absolute paths, and I rewrote them as relative paths using the wonderful here package.
I clarified a lot of the code by renaming the variables with descriptive names and adding clear comments.
I also organized the code to make sure it was easy to read and follow.
And to do that, I made heavy use of the outline panel within RStudio, which you can access by clicking on this sort of list icon at the top right corner of the script pane.
You’ll notice that this ‘Load packages’ header corresponds to a line in this document outline, and that actually happens automatically within RStudio.
You can create these header lines by clicking Command + Shift + R on a Mac, and they automatically get added to the outlines.
And then you can also see that I have a lot of packages loaded at the top here, a lot of comments, and everything is just a little bit cleaner and easier to read.
I wrote a README to document this project, including an abstract to introduce people to the project, instructions for how to download the data from Figshare and how to run the project, restoring all the packages from the renv lock file, and also information about the contributors.
Finally, I wanted to emphasize that I had to go over the same scripts multiple times, and sometimes doing a first cleanup of the code helped me realize that reorganizing the code in different way would actually be clearer, and so I ended up doing what felt like the same work twice. I think that that re-tidying––it shouldn’t be dismissed as being inefficient or not doing it right the first time, because in my case, I had to take a lot of time to just get my head around how the code was organized before I could make informed decisions about how to clean it up.
So, I’ve talked about a lot of tools that I used in creating this reproducible pipeline, from Figshare to RStudio and RStudio Projects, GitHub for version control, Google Drawings in my case for creating this flowchart, and renv for organizing packages. And this might be overwhelming if you’ve never used these tools before and it feels like you have a lot of things to learn if you want to start implementing reproducible workflows.
But I want to emphasize that there are smaller things that you can do to move in the right direction, even if you can’t jump right into using best practices from the get-go.
For example, don’t be afraid to tell stories with your comments. I like to be very conversational; I like to say ‘Okay, now we’re going to do X’, ‘I noticed this about the output, and that causes me to make this decision about the model’
For example here, I wrote, ‘To make these calculations easier, I’m going to….’ do XYZ, and then I have some lines of code.
Now, you’ll notice here that I only have about four lines of code, and a LOT of lines of comments, and not all of my scripts look like that. But I just want to get the point across that it’s really okay to write essays in your code and to have your code comments be informal, conversational and very descriptive.
Something that Chelsea did in her raw code files that I found very helpful when I went through and organized them, was to keep track, in the comments, of package versions and decisions that were made in the analyses, and file paths. So for example, I already discussed how we had to use an older version of the package geomorph, and conveniently, Chelsea had recorded the version number of the older version that she was using, so it was much easier for me to then restore that version when I started using renv.
You’ll notice that this script is far from being perfect; it’s still using absolute file paths, it hasn’t yet been edited to use the here package, but even without using all best practices in reproducibility, just having these introductory comments is really helpful.
I want to conclude this talk by just emphasizing that we really need to teach workflow in graduate programs, undergraduate programs, whenever we teach coding. I would argue that, for biologists, workflow, organization, and reproducibility are some of the things that take up the greatest proportion of our research time. They are actually more valuable skills sometimes than the actual statistical analysis. They are also not intuitive.
I think it can be easy, as a student, to assume that if you’re just an organized person, you will magically know how to organize your code. And then it’s easy to get frustrated when that doesn’t come naturally and when no one has really taught you what the best practices are.
I think this resource, written by Jenny Bryan and Jim Hester, ‘What they forgot to teach you about R’, is a great example of this. This resource focuses on a lot of the workflow tactics that I’ve been talking about here, like a project-oriented workflow, and it’s titled ‘What they forgot to teach you about R’. And I think for many students, that’s exactly true. They do forget to teach you the workflow stuff, and that’s sometimes the most important part.
Here’s a list of resources that I put together related to the various tools for reproducibility that I’ve talked about in this presentation
And I would be happy to take any questions at the Q&A session right after this talk. Thank you very much.
Hello everyone! Thank you for being here. My name is Kaija Gahm, and I’m a data manager at the Cary Institute of Ecosystem Studies. Late last year, my colleague Chelsea had just finished her Master’s thesis, and my supervisor asked me to help her to organize her code and analysis into a reproducible workflow that we could reference in the manuscript that we were putting together.
In this talk, I’m going to be speaking about my experience implementing data management and reproducibility tools retroactively, for an existing project, even if best practices weren’t used from the beginning. I’ll also discuss organizing and making sense of someone else’s analyses, approaching the project as an outsider.
A little bit of background on the project: It consisted of some pretty typical ecological data––measurements of lake dissolved organic carbon, morphometric landmarks of bluegill sunfish, and a bunch of statistical analyses and figures.
Some of the challenges I knew I was going to face were that the data had largely been edited manually in Excel, or copied and pasted from other files with uncertain origin.
There were many file versions, and it wasn’t really clear how the data and code tied together.
The goal of my work was to compile all of this data and code into a reproducible analysis. I knew that we were going to be storing the data on Figshare; all of the analyses would be conducted in RStudio and stored on GitHub. The data and code could be then referenced in the final manuscript.
To begin my work, I asked Chelsea to start by uploading all of her files to GitHub.
I think GitHub is a very important tool, not just technically, for version control, but also psychologically. If everything was being version controlled, then I would have the power to edit and delete without being burdened by the idea that I might be deleting something very important. I knew that as long as everything was on GitHub, I could always walk the repository back to its original version.
My first step in organizing this repository was to set it up as an RStudio Project, which means that it would open a fresh R session for everyone who downloaded the code, and all file paths could be written relative to the root directory.
I also implemented renv, which is a great package that allows for managing package versions. This was particularly important in the case of this analysis, because some of the code depended on an old version of the geomorph package for analyzing morphometric landmarks. I personally wasn’t very familiar with the package, so even if I had wanted to update the code to the newest version, I wouldn’t really have been able to. Using renv allowed me to run the code with the old version of the package, and it ensures that anyone who wants to run our analysis in the future will be able to restore the packages from the lockfile and immediately proceed without affecting packages on their own computer.
My next step was to organize the data.
The first principle is that raw data is sacred and should not be modified. So that means no copying and pasting.
Derived data should then be fully reproducible and, by extension, disposable. I should be able to just delete all of the derived data, and then recreate it by re-running the scripts in the pipeline.
Now, that might be a bit of a pain because some of the analyses might take a long time to run, but it would be possible in a fully reproducible analysis.
And my job approaching this project was to distinguish between data that was raw––like maybe it was created manually or it was exported from a program somewhere else––versus data that could be regenerated from the analysis code.
This is a list of all the data files that I was originally presented with in the GitHub repository, and it was pretty overwhelming. I had to sort through all of those and figure out which were raw, which were derived, and where they all went in the pipeline.
And in order to do this, I asked the key question, ‘Where did this come from?’ For each file, I wanted to know, how was it generated? Can I trace the origin of all the files and their columns?
I started out by making a spreadsheet listing out each file, and this seemed like a good idea at the time, but it ended up being pretty overwhelming, and it was hard for Chelsea to go through and answer my questions, especially because I was attacking all the files at once, without regard for their importance or the order that they might go in. I was just trying to get my head around it.
Chelsea and I decided that a better approach would be to start with the most recent and up-to-date analysis file; in our case, called ReviewApril2020.
Then, I could look at all of the data files that were read in as inputs. It was much easier, with this reduced set of files, to determine…
…which ones were raw data and which ones were derived data. So, my next step was, for each of those derived data files, to figure out how to re-create it from the data in the database, using R scripts, so that the process would be fully reproducible.
To do that, I created one R script for each of the derived data files, and I used that R script to read in data from the database stored on Figshare and modify it to generate these ‘FINAL’ csv files. In the background, I kept the original data and used it as a comparison, just to make sure I was accurately recreating that original data.
My next step was to organize the code in the analysis files. I had to start by reorganizing a lot of file paths; they had been written as absolute paths, and I rewrote them as relative paths using the wonderful here package.
I clarified a lot of the code by renaming the variables with descriptive names and adding clear comments.
I also organized the code to make sure it was easy to read and follow.
And to do that, I made heavy use of the outline panel within RStudio, which you can access by clicking on this sort of list icon at the top right corner of the script pane.
You’ll notice that this ‘Load packages’ header corresponds to a line in this document outline, and that actually happens automatically within RStudio.
You can create these header lines by clicking Command + Shift + R on a Mac, and they automatically get added to the outlines.
And then you can also see that I have a lot of packages loaded at the top here, a lot of comments, and everything is just a little bit cleaner and easier to read.
I wrote a README to document this project, including an abstract to introduce people to the project, instructions for how to download the data from Figshare and how to run the project, restoring all the packages from the renv lock file, and also information about the contributors.
Finally, I wanted to emphasize that I had to go over the same scripts multiple times, and sometimes doing a first cleanup of the code helped me realize that reorganizing the code in different way would actually be clearer, and so I ended up doing what felt like the same work twice. I think that that re-tidying––it shouldn’t be dismissed as being inefficient or not doing it right the first time, because in my case, I had to take a lot of time to just get my head around how the code was organized before I could make informed decisions about how to clean it up.
So, I’ve talked about a lot of tools that I used in creating this reproducible pipeline, from Figshare to RStudio and RStudio Projects, GitHub for version control, Google Drawings in my case for creating this flowchart, and renv for organizing packages. And this might be overwhelming if you’ve never used these tools before and it feels like you have a lot of things to learn if you want to start implementing reproducible workflows.
But I want to emphasize that there are smaller things that you can do to move in the right direction, even if you can’t jump right into using best practices from the get-go.
For example, don’t be afraid to tell stories with your comments. I like to be very conversational; I like to say ‘Okay, now we’re going to do X’, ‘I noticed this about the output, and that causes me to make this decision about the model’
For example here, I wrote, ‘To make these calculations easier, I’m going to….’ do XYZ, and then I have some lines of code.
Now, you’ll notice here that I only have about four lines of code, and a LOT of lines of comments, and not all of my scripts look like that. But I just want to get the point across that it’s really okay to write essays in your code and to have your code comments be informal, conversational and very descriptive.
Something that Chelsea did in her raw code files that I found very helpful when I went through and organized them, was to keep track, in the comments, of package versions and decisions that were made in the analyses, and file paths. So for example, I already discussed how we had to use an older version of the package geomorph, and conveniently, Chelsea had recorded the version number of the older version that she was using, so it was much easier for me to then restore that version when I started using renv.
You’ll notice that this script is far from being perfect; it’s still using absolute file paths, it hasn’t yet been edited to use the here package, but even without using all best practices in reproducibility, just having these introductory comments is really helpful.
I want to conclude this talk by just emphasizing that we really need to teach workflow in graduate programs, undergraduate programs, whenever we teach coding. I would argue that, for biologists, workflow, organization, and reproducibility are some of the things that take up the greatest proportion of our research time. They are actually more valuable skills sometimes than the actual statistical analysis. They are also not intuitive.
I think it can be easy, as a student, to assume that if you’re just an organized person, you will magically know how to organize your code. And then it’s easy to get frustrated when that doesn’t come naturally and when no one has really taught you what the best practices are.
I think this resource, written by Jenny Bryan and Jim Hester, ‘What they forgot to teach you about R’, is a great example of this. This resource focuses on a lot of the workflow tactics that I’ve been talking about here, like a project-oriented workflow, and it’s titled ‘What they forgot to teach you about R’. And I think for many students, that’s exactly true. They do forget to teach you the workflow stuff, and that’s sometimes the most important part.
Here’s a list of resources that I put together related to the various tools for reproducibility that I’ve talked about in this presentation
And I would be happy to take any questions at the Q&A session right after this talk. Thank you very much.
Video
For the conference, I embedded captions in the video using iMovie, which was a pretty clunky process but was all I had access to at the time. There are some glitches and at one point I accidentally inserted the caption for a previous slide. On the whole, I hope the captions are better than nothing.
Resources
The resource links from the second to last slide didn’t translate well to this format, so here they are:
Happy Git with R (Jenny Bryan and Jim Hester)
Project-oriented workflow with RStudio projects (Jenny Bryan)
Using the here package
How to use renv
Keeping a Paper Trail: Data Management Skills for Reproducible Science(Kate Laskowski)
Get an embed link for a Google drawing so you can put it into e.g. a README
Using the `tree` command in the terminal to make a file tree
The document outline in RStudio
Headers and comments in RStudio (Y. Wendy Hyunh)
What they forgot to teach you about R (Jenny Bryan and Jim Hester)
Finally, you can download the video and the annotated slides from OSF here.