The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of Jupyter Notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourages poor coding practices and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we analyzed 1.4 million notebooks from GitHub. Based on the results, we proposed and evaluated Julynter, a linting tool for Jupyter Notebooks. Papers: PIMENTEL, J. F.; MURTA, L.; BRAGANHOLO, V.; FREIRE, J.; A large-scale study about quality and reproducibility of jupyter notebooks. In: International Conference on Mining Software Repositories (MSR), 2019, Montreal, Canada. PIMENTEL, J. F.; MURTA, L.; BRAGANHOLO, V.; FREIRE, J.; Understanding and Improving the Quality and Reproducibility of Jupyter Notebooks. Empirical Software Engineering, 2021 (in press) This repository contains three files: db2020-09-22.dump.gz sample.tar.gz julynter_reproducility.tar.gz Reproducing the Notebook Study The db2020-09-22.dump.gz file contains a PostgreSQL dump of the database, with all the data we extracted from notebooks. For loading it, run:

gunzip -c db2020-09-22.dump.gz | psql jupyter

Note that this file contains only the database with the extracted data. The actual repositories are available in a google drive folder, which also contains the docker images we used in the reproducibility study. The repositories are stored as content/{hash_dir1}/{hash_dir2}.tar.bz2, where hash_dir1 and hash_dir2 are columns of repositories in the database. For scripts, notebooks, and detailed instructions on how to analyze or reproduce the data collection, please check the instructions on the Jupyter Archaeology repository (tag 1.0.0) The sample.tar.gz file contains the repositories obtained during the manual sampling. Reproducing the Julynter Experiment The julynter_reproducility.tar.gz file contains all the data collected in the Julynter experiment and the analysis notebooks. Reproducing the analysis is straightforward: Uncompress the file: $ tar zxvf julynter_reproducibility.tar.gz Install the dependencies: $ pip install julynter/requirements.txt Run the notebooks in order: J1.Data.Collection.ipynb; J2.Recommendations.ipynb; J3.Usability.ipynb. The collected data is stored in the julynter/data folder. Changelog 2019/01/14 - Version 1 - Initial version
2019/01/22 - Version 2 - Update N8.Execution.ipynb to calculate the rate of failure for each reason
2019/03/13 - Version 3 - Update package for camera ready. Add columns to db to detect duplicates, change notebooks to consider them, and add N1.Skip.Notebook.ipynb and N11.Repository.With.Notebook.Restriction.ipynb.
2021/03/15 - Version 4 - Add Julynter experiment; Update database dump to include new data collected for the second paper; remove scripts and analysis notebooks from this package (moved to GitHub), add a link to Google Drive with collected repository files

The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of Jupyter Notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we analyzed 1.4 million notebooks from GitHub.Paper: https://2019.msrconf.org/event/msr-2019-papers-a-large-scale-study-about-quality-and-reproducibility-of-jupyter-notebooksThis repository contains two files:dump.tar.bz2jupyter_reproducibility.tar.bz2The dump.tar.bz2 file contains a PostgreSQL dump of the database, with all the data we extracted from the notebooks.The jupyter_reproducibility.tar.bz2 file contains all the scripts we used to query and download Jupyter Notebooks, extract data from them, and analyze the data. It is organized as follows:analyses: this folder has all the notebooks we use to analyze the data in the PostgreSQL database.archaeology: this folder has all the scripts we use to query, download, and extract data from GitHub notebooks.paper: empty. The notebook analyses/N12.To.Paper.ipynb moves data to itIn the remaining of this text, we give instructions for reproducing the analyses, by using the data provided in the dump and reproducing the collection, by collecting data from GitHub again.Reproducing the AnalysisThis section shows how to load the data in the database and run the analyses notebooks. In the analysis, we used the following environment:Ubuntu 18.04.1 LTS
PostgreSQL 10.6
Conda 4.5.11
Python 3.7.2
PdfCrop 2012/11/02 v1.38First, download dump.tar.bz2 and extract it:

tar -xjf dump.tar.bz2

It extracts the file db2019-03-13.dump. Create a database in PostgreSQL (we call it "jupyter"), and use psql to restore the dump:

psql jupyter < db2019-03-13.dump

It populates the database with the dump. Now, configure the connection string for sqlalchemy by setting the environment variable JUP_DB_CONNECTTION:

export JUP_DB_CONNECTION="postgresql://user:password@hostname/jupyter";

Download and extract jupyter_reproducibility.tar.bz2:

tar -xjf jupyter_reproducibility.tar.bz2

Create a conda environment with Python 3.7:

conda create -n analyses python=3.7conda activate analyses

Go to the analyses folder and install all the dependencies of the requirements.txt

cd jupyter_reproducibility/analysespip install -r requirements.txt

For reproducing the analyses, run jupyter on this folder:

jupyter notebook

Execute the notebooks on this order:Index.ipynbN0.Repository.ipynbN1.Skip.Notebook.ipynbN2.Notebook.ipynbN3.Cell.ipynbN4.Features.ipynbN5.Modules.ipynbN6.AST.ipynbN7.Name.ipynbN8.Execution.ipynbN9.Cell.Execution.Order.ipynbN10.Markdown.ipynbN11.Repository.With.Notebook.Restriction.ipynbN12.To.Paper.ipynbReproducing or Expanding the CollectionThe collection demands more steps to reproduce and takes much longer to run (months). It also involves running arbitrary code on your machine. Proceed with caution.RequirementsThis time, we have extra requirements:All the analysis requirements
lbzip2 2.5
gcc 7.3.0
Github account
Gmail accountEnvironmentFirst, set the following environment variables:

export JUP_MACHINE="db"; # machine identifierexport JUP_BASE_DIR="/mnt/jupyter/github"; # place to store the repositoriesexport JUP_LOGS_DIR="/home/jupyter/logs"; # log filesexport JUP_COMPRESSION="lbzip2"; # compression programexport JUP_VERBOSE="5"; # verbose levelexport JUP_DB_CONNECTION="postgresql://user:password@hostname/jupyter"; # sqlchemy connectionexport JUP_GITHUB_USERNAME="github_username"; # your github usernameexport JUP_GITHUB_PASSWORD="github_password"; # your github passwordexport JUP_MAX_SIZE="8000.0"; # maximum size of the repositories directory (in GB)export JUP_FIRST_DATE="2013-01-01"; # initial date to query githubexport JUP_EMAIL_LOGIN="[email protected]"; # your gmail addressexport JUP_EMAIL_TO="[email protected]"; # email that receives notificationsexport JUP_OAUTH_FILE="/oauth2_creds.json" # oauth2 auhentication fileexport JUP_NOTEBOOK_INTERVAL=""; # notebook id interval for this machine. Leave it in blankexport JUP_REPOSITORY_INTERVAL=""; # repository id interval for this machine. Leave it in blankexport JUP_WITH_EXECUTION="1"; # run execute python notebooksexport JUP_WITH_DEPENDENCY="0"; # run notebooks with and without declared dependneciesexport JUP_EXECUTION_MODE="-1"; # run following the execution orderexport JUP_EXECUTION_DIR="/home/jupyter/execution"; # temporary directory for running notebooksexport JUP_ANACONDA_PATH="/anaconda3"; # conda installation pathexport JUP_MOUNT_BASE="/home/jupyter/mount_ghstudy.sh"; # bash script to mount base direxport JUP_UMOUNT_BASE="/home/jupyter/umount_ghstudy.sh"; # bash script to umount base direxport JUP_NOTEBOOK_TIMEOUT="300"; # timeout the extraction# Frequenci of log reportexport JUP_ASTROID_FREQUENCY="5";export JUP_IPYTHON_FREQUENCY="5";export JUP_NOTEBOOKS_FREQUENCY="5";export JUP_REQUIREMENT_FREQUENCY="5";export JUP_CRAWLER_FREQUENCY="1";export JUP_CLONE_FREQUENCY="1";export JUP_COMPRESS_FREQUENCY="5";export JUP_DB_IP="localhost"; # postgres database IP

Then, configure the file ~/oauth2_creds.json, according to yagmail documentation: https://media.readthedocs.org/pdf/yagmail/latest/yagmail.pdfConfigure the mount_ghstudy.sh and umount_ghstudy.sh scripts. The first one should mount the folder that stores the directories. The second one should umount it. You can leave the scripts in blank, but it is not advisable, as the reproducibility study runs arbitrary code on your machine and you may lose your data.ScriptsDownload and extract jupyter_reproducibility.tar.bz2:

tar -xjf jupyter_reproducibility.tar.bz2

Install 5 conda environments and 5 anaconda environments, for each python version. In each of them, upgrade pip, install pipenv, and install the archaeology package (Note that it is a local package that has not been published to pypi. Make sure to use the -e option):Conda 2.7

conda create -n raw27 python=2.7 -yconda activate raw27pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Anaconda 2.7

conda create -n py27 python=2.7 anaconda -yconda activate py27pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Conda 3.4It requires a manual jupyter and pathlib2 installation due to some incompatibilities found on the default installation.

conda create -n raw34 python=3.4 -yconda activate raw34conda install jupyter -c conda-forge -yconda uninstall jupyter -ypip install --upgrade pippip install jupyterpip install pipenvpip install -e jupyter_reproducibility/archaeologypip install pathlib2

Anaconda 3.4

conda create -n py34 python=3.4 anaconda -yconda activate py34pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Conda 3.5

conda create -n raw35 python=3.5 -yconda activate raw35pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Anaconda 3.5It requires the manual installation of other anaconda packages.

conda create -n py35 python=3.5 anaconda -yconda install -y appdirs atomicwrites keyring secretstorage libuuid navigator-updater prometheus_client pyasn1 pyasn1-modules spyder-kernels tqdm jeepney automat constantly anaconda-navigatorconda activate py35pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Conda 3.6

conda create -n raw36 python=3.6 -yconda activate raw36pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Anaconda 3.6

conda create -n py36 python=3.6 anaconda -yconda activate py36conda install -y anaconda-navigator jupyterlab_server navigator-updaterpip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Conda 3.7

conda create -n raw37 python=3.7 -yconda activate raw37pip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

Anaconda 3.7When we executed the experiments, the anaconda package for Python 3.7 was not complete. So, we attempted to install all Anaconda 3.x dependencies manually

conda create -n py37 python=3.7 anaconda -yconda activate py37conda install -y _ipyw_jlab_nb_ext_conf alabaster anaconda-client anaconda-navigator anaconda-project appdirs asn1crypto astroid astropy atomicwrites attrs automatconda install -y babel backports backports.shutil_get_terminal_size beautifulsoup4 bitarray bkcharts blaze blosc bokeh boto bottleneck bzip2conda install -y cairo colorama constantly contextlib2 curl cycler cythonconda install -y defusedxml docutils et_xmlfile fastcache filelock fribidiconda install -y get_terminal_size gevent glob2 gmpy2 graphite2 greenletconda install -y harfbuzz html5lib hyperlink imageio imagesize incremental isortconda install -y jbig jdcal jeepney jupyter jupyter_console jupyterlab_launcher keyring kiwisolverconda install -y libtool libxslt lxml matplotlib mccabe mkl-service mpmath navigator-updaterconda install -y nltk nose numpydoc openpyxl pango patchelf path.py pathlib2 patsy pep8 pkginfo ply pyasn1 pyasn1-modules pycodestyle pycosat pycrypto pycurl pyflakes pylint pyodbc pywaveletsconda install -y rope scikit-image scikit-learn seaborn service_identity singledispatch spyder spyder-kernels statsmodels sympyconda install -y tqdm traitlets twisted unicodecsv xlrd xlsxwriter xlwt zope zope.interfaceconda install -y sortedcollections typed-astpip install --upgrade pippip install pipenvpip install -e jupyter_reproducibility/archaeology

StopwordsUse nltk to download stopwords:

conda activate py36python -c "import nltk; nltk.download('stopwords')"

Everything should be set to run right now.ExecutingIn this step, we recommend using the py36 environment to orchestrate the execution. We designed the scripts for Python 3.6, and if they are correctly configured, it can invoke the other environments.

conda activate py36

If you want to extend the execution to more environments, configure the environments on the file archaeology/config.py.For querying and downloading repositories from github, run on the jupyter_reproducibility/archaeology directory:

python s0_repository_crawler.py

For extracting data from the repositories and notebooks, run on this order:

python s1_notebooks_and_cells.pypython s2_requirement_files.pypython s3_compress.pypython s4_markdown_features.pypython s5_extract_files.pypython s6_cell_features.pypython s7_execute_repositories.pypython p0_local_possibility.pypython p1_notebooks_and_cells.pypython p2_sha1_exercises.py

Alternatively, execute the following script that orchestrates all the executions and notifies when they finish:

python main_with_crawler.py

If some script fails to process all repositories/notebooks/cells, use the option "-e" to rerun it and force the re-extraction.After this process, refer to the Reproducing the Analysis Section for analyzing the collected data.Changelog2019/01/14 - Version 1 - Initial version
2019/01/22 - Version 2 - Update N8.Execution.ipynb to calculate rate of failure for each reason
2019/03/13 - Version 3 - Update package for camera ready. Add columns to db to detect duplicates, change notebooks to consider it and add N1.Skip.Notebook.ipynb and N11.Repository.With.Notebook.Restriction.ipynb.

The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of Jupyter Notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourages poor coding practices and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we analyzed 1.4 million notebooks from GitHub. Based on the results, we proposed and evaluated Julynter, a linting tool for Jupyter Notebooks. Papers: PIMENTEL, J. F.; MURTA, L.; BRAGANHOLO, V.; FREIRE, J.; A large-scale study about quality and reproducibility of jupyter notebooks. In: International Conference on Mining Software Repositories (MSR), 2019, Montreal, Canada. PIMENTEL, J. F.; MURTA, L.; BRAGANHOLO, V.; FREIRE, J.; Understanding and Improving the Quality and Reproducibility of Jupyter Notebooks. Empirical Software Engineering, 2021 (in press) This repository contains three files: db2020-09-22.dump.gz sample.tar.gz julynter_reproducility.tar.gz Reproducing the Notebook Study The db2020-09-22.dump.gz file contains a PostgreSQL dump of the database, with all the data we extracted from notebooks. For loading it, run:

gunzip -c db2020-09-22.dump.gz | psql jupyter

Note that this file contains only the database with the extracted data. The actual repositories are available in a google drive folder, which also contains the docker images we used in the reproducibility study. The repositories are stored as content/{hash_dir1}/{hash_dir2}.tar.bz2, where hash_dir1 and hash_dir2 are columns of repositories in the database. For scripts, notebooks, and detailed instructions on how to analyze or reproduce the data collection, please check the instructions on the Jupyter Archaeology repository (tag 1.0.0) The sample.tar.gz file contains the repositories obtained during the manual sampling. Reproducing the Julynter Experiment The julynter_reproducility.tar.gz file contains all the data collected in the Julynter experiment and the analysis notebooks. Reproducing the analysis is straightforward: Uncompress the file: $ tar zxvf julynter_reproducibility.tar.gz Install the dependencies: $ pip install julynter/requirements.txt Run the notebooks in order: J1.Data.Collection.ipynb; J2.Recommendations.ipynb; J3.Usability.ipynb. The collected data is stored in the julynter/data folder. Changelog 2019/01/14 - Version 1 - Initial version
2019/01/22 - Version 2 - Update N8.Execution.ipynb to calculate the rate of failure for each reason
2019/03/13 - Version 3 - Update package for camera ready. Add columns to db to detect duplicates, change notebooks to consider them, and add N1.Skip.Notebook.ipynb and N11.Repository.With.Notebook.Restriction.ipynb.
2021/03/15 - Version 4 - Add Julynter experiment; Update database dump to include new data collected for the second paper; remove scripts and analysis notebooks from this package (moved to GitHub), add a link to Google Drive with collected repository files