Call for Artifacts

Traditionally, only papers are published. However, claims and results described in a paper often originate from artifacts not present in the paper. Artifacts are any additional material that substantiates the claims made in the paper, and ideally makes them fully replicable. For some papers, these artifacts are as important as the paper itself because they provide crucial evidence for the quality of the results.

The goal of TAP artifact evaluation is twofold. On the one hand, we want to encourage authors to provide more substantial evidence to their papers and to reward authors who create artifacts. On the other hand, we want to simplify the independent replication of results presented in the paper and to ease future comparison with existing approaches. Artifact submission is optional for TAP 2019. However, all authors of all accepted papers for TAP 2019 are encouraged to submit an artifact for evaluation.

Artifacts of interest include (but are not limited to):

Important Dates

Artifact Evaluation

All artifacts are evaluated by the artifact evaluation committee. Each artifact will be reviewed by at least two committee members. Reviewers will read the accepted paper and explore the artifact to evaluate how well the artifact supports the claims and results of the paper. The evaluation is based on the following questions.

The artifact evaluation is performed in the following two phases.

  1. In the test phase, reviewers check if the artifact is functional, i.e., they look for setup problems (e.g., corrupted, missing files, crashes on simple examples, etc.). If any problems are detected, the authors are informed of the outcome and asked for clarification. The authors will get 96 hours to respond to the reviews in case problems are encountered.
  2. In the assessment phase, reviewers will try to reproduce any experiments or activities and evaluate the artifact w.r.t the questions detailed above.

Papers with accepted artifacts that are publicly available will receive an artifact-evaluation badge on the first page and may extend their paper with an additional appendix of up to 2 pages.

Artifact Submission

An artifact submission should consist of

The artifact evaluation chairs will download the .zip file and distribute it to the reviewers. Please also look at the Artifact Packaging Guidelines section for detailed information about the contents of the submission.

Additionally, the abstract including the URL of the download link, as well as the SHA256 checksum of your .zip file, and the .pdf file of your paper must be submitted to EasyChair.

http://www.easychair.org/conferences/?conf=tap2019

We need the checksum to ensure the integrity of your artifact. You can generate the checksum using the following command-line tools.

If you cannot submit the artifact as requested or encounter any other difficulties in the submission process, please contact the artifact evaluation chairs prior to submission.

Artifact Packaging Guidelines

We expect that authors package their artifact (.zip file) and write their instructions such that the artifact evaluation committee can evaluate the artifact within a virtual machine provided by us. The TAP 2019 virtual machine was created with VirtualBox 6.0.8 and consists of an installation of Ubuntu 19.04 with Linux 5.0.0 and the following notable packages.

The VM has a user tap2019 with password tap2019. The root user has the same password.

The artifact evaluation committee will be instructed not to download software or data from external sources. Any additional software required by your artifact must be included in the .zip file and the artifact must provide instructions for the installation. To include an Ubuntu package in your artifact submission, you can create a .deb file with all the necessary dependencies from inside the VM. Reviewers can then install them by using sudo dpkg -i <.deb file>. You can create the necessary .deb files for example as follows.

In case you think the VM is improper for evaluation of your artifact, please contact the artifact evaluation chairs prior to artifact submission.

Your artifact .zip file must contain the following elements.

Publication of Artifacts

The artifact evaluation committee uses the submitted artifact only for the artifact evaluation. It may not publicize the artifact or any parts of it during or after completing evaluation. Artifacts and all associated data will be deleted at the end of the evaluation process. However, to get the artifact badge the accepted artifact must be publicly available. We encourage the authors of artifacts to make their artifacts also permanently available, e.g., on ZENODO or figshare.com, and refer to them in their papers via a DOI or link to the published artifact.

Artifact Evaluation Committee

Chairs
Committee