STEP 2: REVIEWING MATERIALS

Assessing Data for Sharing (2.2)

Examples of Data Restrictions

Return to SPPQ Journal Verification Framework

File Organization (2.1)

The final review of materials is a necessary check to ensure that your verification submission package is complete and ready for processing. When reviewing your materials, please ensure that:

All necessary files are accounted for in the submission package. This includes all data (original and analysis), code (cleaning and/or analyses), and documentation (README and codebook)
All files are documented in the README. This is a good opportunity to review the contents of your documentation one more time
All analysis datasets and their variables are described in your codebook(s)
Extraneous materials have been removed from the submission package. Only those files necessary for understanding and running the analyses should be submitted to SPPQ Dataverse
Code properly calls correct filenames when run. Check script for typos that may cause errors
Manuscript and appendices are not included in your verification submission package within the SPPQ Dataverse. These are submitted to the Odum Verification Team by the SPPQ Editor and should not exist within your SPPQ Dataverse record

Once you have completed the above checklist, please move to the next section of this guide Assessing Data for Sharing (2.2)

Assessing Data for Sharing (2.2)

As part of the final review, it is important to verify any copyright, data use agreement terms, or license restrictions on all original source data used in your analyses. If your submission contains data that you did not collect on your own, then you must explicitly cite these data in your README. Further, for all original source data, you must ensure that you are permitted to share the data files within the SPPQ Dataverse. Some original source data will not be shareable due to licensing restrictions, government restrictions, or sensitivity. In these cases, authors must contact SPPQ Editors to inform them of the specific data files that cannot be shared within the SPPQ Dataverse. Authors must provide the SPPQ Editors with the following:

Full data citation
Description of how these data are used in their analysis and which tables and/or figures are generated using these data
Whether or not the data can be shared with the Odum Verification Team for verification only

The Editors will determine if an exemption is appropriate or if another sharing method may be feasible.

For example, in some instances, original source data producers are fine with sharing the data with the Odum Verification Team specifically for verification. In these cases, the author will submit the data file(s) to the Editors via email, or other secure transfer method. Authors are also required to include the following information in their README:

Full data citation of original source data not included in the SPPQ Dataverse record
Reason for why these data are not included in the SPPQ Dataverse record
Instructions for secondary users on how and where to obtain these original source data

The Odum Verification Team will conduct the standard verification workflow for the submission. Once the verification is successful, the Odum Verification Team will delete the restricted/proprietary data from their servers. Additionally, the final published version of the SPPQ Dataverse record will include an updated data verification statement denoting that certain files are not accessible via the SPPQ Dataverse due to data use agreement, Terms of Use, or licensing restrictions. A data availability statement will be added to the Terms of Use tab of the SPPQ Dataverse record describing where and how users may request access to these files.

If the Editors decide an exemption is appropriate, the Odum Verification Team will verify all tables and figures not impacted by these original source data. The verification statement and Terms of Use will be updated to reflect the approved exemption upon publication of the SPPQ Dataverse record.

Examples of Data Restrictions

Reuning, Kevin, 2019, “Replication Data for: Mapping Influence: Partisan Networks Across the United States, 2000 to 2016”, https://doi.org/10.15139/S3/9JJNBV, UNC Dataverse, V1, UNF:6:J9LNFrDW5nOeanmYf1CxVA== [fileUNF]
Mancinelli, Abigail, 2022, “Replication Data for: Does Public Financing Motivate Electoral Challengers?”, https://doi.org/10.15139/S3/HQFBAY, UNC Dataverse, V1, UNF:6:7b6tFw3VMC8fsZQQ/1uVRg== [fileUNF]
Perez, Vanessa, 2022, “Replication Data for: How the American Public Perceived Electoral Competition in the States During the Pre-Poll Era: A Prediction Market Data Analysis of the 1896 Presidential Election”, https://doi.org/10.15139/S3/X6BYHS, UNC Dataverse, V1

Please move to the next section of this guide Run on Clean Instance (2.3)

Run on Clean Instance (2.3)

Verifiers and other users will attempt to reproduce the same computing environment/workspace you used in your analyses. This may be difficult, even when the environment is specified, due to differences in machines, operating systems, software packages, and potentially complex dependencies between them. For example, users may encounter deprecated commands, out-of-date packages, and differences in the underlying dependency packages. Therefore, it is important to re-run the replication code on a clean instance of the environment/workspace so that it can be replicated by others.

One option is to remove any existing packages and reinstall only those packages that are necessary for the replication (this is relatively easy to do in Stata; see below). Alternatively, If that is not feasible, you should fully specify the workspace, including all packages and package versions.

💡 Tip: Remove existing packages/dependencies to run on a clean instance

Stata Example:

** Print list of installed packages
ado

** Uninstall packages (by # in list)
ado uninstall [5]
ado uninstall [4]
ado uninstall [3]
ado uninstall [2]
ado uninstall [1]

** Re-install the packages necessary for replication
ssc install texdoc
ssc install coefplot
ssc install rdrobust

💡 Tip: Specify package versions installed into workspace

R Example:

## Print version of R
print(version)

## Print all loaded packages (and package version)
print(sessionInfo())

In some instances, a package used in the analyses may not be available in a stable repository like CRAN. Sometimes packages are housed on a researcher’s professional website and that website may disappear or be inaccessible in the future. This can make the stability of accessing that package unreliable. If this is the case with a package used in the analysis, we recommend the following:

Ask the original package creator if you may share the package in the SPPQ Dataverse dataset record. If they agree, submit the package(s) as part of your verification submission package.
If they do not agree, send the package(s) in a zip to the SPPQ Editors to be shared with the Odum Verification Team for the purposes of verification only. Provide detailed instructions for secondary users on how to access the package(s) from the original producer(s) website.

Please move to the next section of this guide Compare Outputs (2.4)

Compare Outputs (2.4)

Often you may have to make edits to the replication code that are not reflected in the current manuscript version, or vice versa. Therefore, it is important to re-run the replication package (from start to end) to double check that the output matches the current version of the manuscript. The replication code should properly reproduce all tables, figures, and analytical results in the primary article. By confirming this before submission to the SPPQ Dataverse, you may catch and correct any discrepancies, and prevent additional rounds of review by the Odum Verification Team.

If any of the tables or figures were manually manipulated outside of the code, instructions must be provided within the verification submission package that describe each manual step and any tools used to format the tables and/or figures. The Odum Verification Team should be able to walk through each step and achieve the same formatted output present in the manuscript.

Please move to the next section of this guide Generate Log Files (2.5)

Generate Log Files (2.5)

When the Odum Verification Team is unable to replicate the results, it is often helpful to check the session log file from when you initially ran the replication package on your local machine. Therefore, we recommend that you create a text log file to store your results and include this with the verification submission package.

💡 Tip: Generate log files in Stata and R

Stata Example:

log using  “$path/ajps_results”
// …
// [run analysis code]
// …
log close

R Example:

# Install logr package
install.packages("logr")
library(logr)

# Open the log
log_open("ajps_results.log")

# Print text to the log
log_print("Here is a test log:")

## Estimate regression model
X <- c(6, 10, 2, 4, 6)
Y <- c(82, 88, 56, 64, 77)
df <- data.frame(X, Y)
ols <- lm(Y ~ X, data = df)

# Print results to the log
log_print(ols)

# Close the log
log_close()

Once you have completed reviewing your materials, please move to the next step of this guide Step 3: Using SPPQ Dataverse

SPPQ Journal Verification Policy

Step 1: Preparing Materials

Step 3: Using SPPQ Dataverse

Step 4: Contacting SPPQ Editor

Resubmitting for Verification

Common Issues

Frequently Asked Questions

Templates