We’ve now gone through all the steps of creating a more reproducible research project and workflow. The question though is, can we recreate our thought process and analyses? As we talked about at the beginning, in many labs the actual test of this comes when you, or another lab member, go back to a project a few weeks or a few months later and try to figure out what you did. We can’t really wait for our own memories to decay, so we’ll use each other as stand-ins for our future selves who have probably forgotten a lot of details. So, go ahead and pair up with the group beside you, and share the URL for your projects. Look over each others projects. Take note of things they did well and things they could potentially do better.
Activity: Look over another groups project and critique the transparency/documentation
Alright, hopefully you guys had time to get through each others projects. To kick things off I’ll say a few things about my own project that I did well, and a few things that I could have done better.
-
Done Well
- All my data, code, and materials are in one central place so I haven’t left any files out that could get lost
- I’ve done at least a bit of code commenting so I have information about what I did in plain english
- I documented the basic ideas and rationals for the project so I will be aware if this changes overtime
-
Could do better
- My data files are SPSS files, so not everyone will be able to use them easily (and I might not be able to use them in the future). It would be be great if I also uploaded the data files as csv files so that I or anyone else would always be able to use them.
- I have a private data subcomponent because, presumable, my IRB said that I couldn’t share the de-anonymized data. However, I don’t make a note of that anywhere. I could improve my documentation by noting that in the wiki.
For the projects you looked over, what were other things that you noticed people doing well? Were there things you noticed that might have helped increase the documentation or transparency of the project?
Activity: Q&A depending on the feedback that comes in above.
We covered a lot of information today, but you should think about this all on a continuum; reproducible research practices aren’t an all or nothing game. Even small improvements to organization and documentation can help increase the transparency and reproducibility of research. Some people will go for the full workflow overhaul straight out of the gate, others will make a series of small, incremental changes to their research processes. Both of these approaches are fine, it’s all about what works best to integrate the ideas we’ve talked about into your own workflow. That integration process will be different for different researchers. This process is also sometimes the hardest; figuring out how to apply examples to your particular issues can be challenging. So, before I open it up to questions I wanted to mention where you can get help after this workshop.
- If you want to review any of the workshop content, it is all available here:
- If you have specific questions, the Center for Open Science provides completely free, one-on-one stats and methods consulting over email and google hangouts
- If you have questions about the OSF, there is a helpdesk manned by a real person who can help you through any issues or questions you might have