Call for Artifacts

Traditionally, only papers are published. However, claims and results described in a paper often originate from artifacts not present in the paper. Artifacts are any additional material that substantiates the claims made in the paper, and ideally makes them fully replicable. For some papers, these artifacts are as important as the paper itself because they provide crucial evidence for the quality of the results.

The goal of TAP artifact evaluation is twofold. On the one hand, we want to encourage authors to provide more substantial evidence to their papers and to reward authors who create artifacts. On the other hand, we want to simplify the independent replication of results presented in the paper and to ease future comparison with existing approaches.

Artifacts of interest include (but are not limited to):

Artifact submission is optional for TAP 2020. However, all authors of all accepted papers for TAP 2020 are encouraged to submit an artifact for evaluation. Additionally, badges shown on the title page of the corresponding paper give you credit for good artifact submissions.

Important Dates

Artifact Evaluation

All artifacts are evaluated by the artifact evaluation committee. Each artifact will be reviewed by at least two committee members. Reviewers will read the accepted paper and explore the artifact to evaluate how well the artifact supports the claims and results of the paper. The evaluation is based on the following questions.

The artifact evaluation is performed in the following two phases.

  1. In the test phase, reviewers check if the artifact is functional, i.e., they look for setup problems (e.g., corrupted, missing files, crashes on simple examples, etc.). If any problems are detected, the authors are informed of the outcome and asked for clarification. The authors will get 3 days to respond to the reviews in case problems are encountered.
  2. In the assessment phase, reviewers will try to reproduce any experiments or activities and evaluate the artifact w.r.t the questions detailed above.

Awarding

We award up to two badges that are granted independently of each other. For artifacts that are successfully evaluated by the artifact evaluation committee we either grant the functional or reusable badge. Each successfully evaluated artifact receives at least the functional bade. The reusable badge is granted to artifacts of very high quality. Independently, artifacts that are publicly available under a DOI receive an availability badge. Authors may use all granted badges on the title page of the respective paper.

Additionally, papers with successfully evaluated artifacts may extend their paper with an additional appendix of up to 2 pages.

Artifact Submission

An artifact submission should consist of

The artifact evaluation chairs will download the .zip file and distribute it to the reviewers. Please also look at the Artifact Packaging Guidelines section for detailed information about the contents of the submission.

The abstract including the URL of the download link, as well as the SHA256 checksum of your .zip file, and the .pdf file of your paper must be submitted to EasyChair.

https://easychair.org/conferences/?conf=tap2020

We need the checksum to ensure the integrity of your artifact. You can generate the checksum using the following command-line tools.

If you cannot submit the artifact as requested or encounter any other difficulties in the submission process, please contact the artifact evaluation chairs prior to submission.

Artifact Packaging Guidelines

We expect that authors package their artifact (.zip file) and write their instructions such that the artifact evaluation committee can evaluate the artifact within a virtual machine provided by us. Only submit the required files to replicate your results in the provided virtual machine. Do not submit a virtual machine image in the .zip file. AEC members will copy your .zip file into the provided virtual machine.

TAP 2020 Virtual Machine

The TAP 2020 virtual machine was created with VirtualBox 6.0.16 and consists of an installation of Ubuntu 19.04 with Linux 5.0.0-38 and the following notable packages. To switch between Java versions, you can use sudo update-alternatives --config java. The default Java version is Java 1.8.

The VM has a user tap2020 with password tap2020. The root user has the same password.

The final version of the VM will be available on ZENODO.

In order to save space, the VM does not have an active swap file. Please mention in your submission if you expect that a swap file is needed. You can activate swap for the running session using the following commands.

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

The artifact evaluation committee will be instructed not to download software or data from external sources. Any additional software required by your artifact must be included in the .zip file and the artifact must provide instructions for the installation. To include an Ubuntu package in your artifact submission, you can create a .deb file with all the necessary dependencies from inside the VM. Reviewers can then install them by using sudo dpkg -i <.deb file>. You can create the necessary .deb files for example as follows.

In case you think the VM is improper for evaluation of your artifact, please contact the artifact evaluation chairs prior to artifact submission.

Artifact Content

Your artifact .zip file must contain the following elements.

Publication of Artifacts

The artifact evaluation committee uses the submitted artifact only for the artifact evaluation. It may not publicize the artifact or any parts of it during or after completing evaluation. Artifacts and all associated data will be deleted at the end of the evaluation process. We encourage the authors of artifacts to make their artifacts also permanently available, e.g., on ZENODO or figshare.com, and refer to them in their papers via a DOI. All artifacts for which a DOI exists that is known to the artifact evaluation committee are granted the availability badge.

Artifact Evaluation Committee

Chairs
Committee