dev:regtesting
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
dev:regtesting [2014/06/26 10:28] – Update link krack | dev:regtesting [2021/06/02 13:11] – tmueller | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ==== CP2K Regression Testing ===== | + | ====== CP2K Regression Testing |
- | === Regression Testing Script === | + | CP2K comes with over 3000 test input files (located in [[src> |
- | CP2K comes with over 2400 test input files (located in '' | + | ==== Dashboard ==== |
+ | A number of regtests are run automatically by various members of our community. The results of these tests are collected centrally at the [[http:// | ||
- | The regression test suite is run using the '' | + | ==== Code Coverage ==== |
+ | We aim that the regression test suite covers all the functionality of CP2K. For this purpose we regularly create [[http://www.cp2k.org/ | ||
- | In addition, when new features are added to CP2K, please add new tests which cover those features and commit them to SVN along with the code. | + | ===== How does it work? ===== |
- | === Automated Regression Tester === | + | The regression test suite is run using the [[src> |
+ | It performs the following tasks: | ||
+ | * performs a realclean build of the source | ||
+ | * executes a list of tests | ||
+ | * compares the results (outputs) with those of the last known result (reference) | ||
+ | * produces a summary | ||
- | An automated regression test service is available at http:// | ||
- | We plan to extend | + | ===== Running |
- | === CP2K Code Coverage | + | ==== Step 0: make based testing ==== |
- | We aim that the regression test suite covers all the functionality of CP2K. | + | * If you are able to build and run cp2k on the local machine |
+ | * Be careful about the value of '' | ||
+ | * If this fails (e.g. on batch systems), continue with Step 1 else go to interpretation. | ||
- | === Results of the Automatic Regression Tester | + | ==== Step 1: Preparation ==== |
- | * [[http://sourceforge.net/p/cp2k/ | + | * Decide on a directory for doing the regtest, there will be plenty of files in this dir (after a while) so make it something like '' |
+ | * Clone a version of cp2k into '' | ||
+ | * Set up the arch files so that you can cleanly build cp2k (test this) | ||
- | [[http://cp2k.web.psi.ch/ | + | ==== Step 2: Running ==== |
+ | < | ||
+ | $ tools/regtesting/do_regtest | ||
+ | Usage: do_regtest [OPTION] | ||
+ | Run the CP2K regression test suite | ||
+ | Example: do_regtest | ||
- | === Latest commits === | + | General: |
+ | -h, -help, --help | ||
+ | -c, -config FILE read any of the following configuration switches from FILE. | ||
+ | -cp2kdir PATH | ||
+ | -dirout PATH root path of output folder (default: current working directory). | ||
+ | -mpiexec EXE name of the executable to run mpi-ranks. Default: mpiexec. | ||
- | {{rss>http://sourceforge.net/ | + | Build: |
+ | -version VERSION | ||
+ | -arch ARCH ARCH passed to make. Default: Linux-x86-64-gfortran. | ||
+ | -quick | ||
+ | -nobuild | ||
+ | |||
+ | Runtime: | ||
+ | -mpiranks NRANKS | ||
+ | -ompthreads NTHREADS | ||
+ | -maxtasks NPROCS | ||
+ | -maxbuildtasks NPROCS | ||
+ | -jobmaxtime SECONDS | ||
+ | -farming | ||
+ | |||
+ | Testing: | ||
+ | -noreset | ||
+ | -shard ISHARD NSHARDS | ||
+ | -skiptest | ||
+ | -skipunittest | ||
+ | -skipdir TESTDIR | ||
+ | -restrictdir TESTDIR | ||
+ | -retest | ||
+ | |||
+ | Exit codes: | ||
+ | 0 clean exit with testing | ||
+ | 3 | ||
+ | 4 build errors | ||
+ | 5 | ||
+ | 6 | ||
+ | 7 | ||
+ | 8 | ||
+ | |||
+ | For more information visit: < | ||
+ | </ | ||
+ | |||
+ | ==== Step 3: Interpretation ==== | ||
+ | A test results can be any of the following: | ||
+ | ^ Test Result | ||
+ | | '' | ||
+ | | '' | ||
+ | | '' | ||
+ | | '' | ||
+ | |||
+ | The last two outcomes generally mean that a bug has been introduced, which requires investigation. | ||
+ | Since regtesting only yields information relative to a previously known result, it is most useful to do a regtest before and after you make changes. To allow per-test numerical difference higher than that set as a default, add third column in appropriate TEST_FILES file with a relative value of the difference. | ||
+ | |||
+ | |||
+ | ===== Adding and Resetting Tests===== | ||
+ | The test-suite is fully controlled by the following files in the [[src> | ||
+ | |||
+ | ^ File Name ^ Content | ||
+ | | '' | ||
+ | | '' | ||
+ | | '' | ||
+ | | '' | ||
+ | |||
+ | ===== Run with sbatch ===== | ||
+ | |||
+ | The '' | ||
+ | |||
+ | What you need: | ||
+ | |||
+ | * '' | ||
+ | * a CP2K source tree with a built CP2K | ||
+ | |||
+ | ==== Instructions ==== | ||
+ | |||
+ | The way the regtest script works is that it goes through all the directories (for example '' | ||
+ | |||
+ | Also, '' | ||
+ | |||
+ | Append the following to your '' | ||
+ | |||
+ | <code bash> | ||
+ | CP2K_BASE_DIR="/ | ||
+ | CP2K_TEST_DIR=" | ||
+ | # CP2K_REGTEST_SCRIPT_DIR="" | ||
+ | |||
+ | CP2K_ARCH=" | ||
+ | CP2K_VERSION=" | ||
+ | |||
+ | # the following is the default, adjust if you want to run single tests with more than 2 ranks/ | ||
+ | NTASKS_SINGLE_TEST=2 | ||
+ | NNODES_SINGLE_TEST=1 | ||
+ | SRUN_CMD=" | ||
+ | |||
+ | # the following should be sufficiently generic: | ||
+ | |||
+ | mkdir -p " | ||
+ | cd " | ||
+ | |||
+ | cp2k_rel_dir=$(realpath --relative-to=" | ||
+ | # srun does not like `-np`, override the complete command instead: | ||
+ | export cp2k_run_prefix=" | ||
+ | |||
+ | " | ||
+ | -arch " | ||
+ | -version " | ||
+ | -nobuild \ | ||
+ | -mpiranks ${NTASKS_SINGLE_TEST} \ | ||
+ | -ompthreads ${OMP_NUM_THREADS} \ | ||
+ | -maxtasks ${SLURM_NTASKS} \ | ||
+ | -cp2kdir " | ||
+ | |& tee " | ||
+ | |||
+ | # the above will output both to the slurm-*.out as well as a log file, | ||
+ | # if you want only the log file replace the `|& tee` with a `>&`. | ||
+ | |||
+ | # More options: | ||
+ | # -farming | ||
+ | # -retest | ||
+ | </code> | ||
+ | |||
+ | A complete '' | ||
+ | |||
+ | <code bash> | ||
+ | #!/bin/bash -l | ||
+ | #SBATCH --time=01: | ||
+ | #SBATCH --nodes=4 | ||
+ | #SBATCH --ntasks-per-node=128 | ||
+ | #SBATCH --cpus-per-task=2 | ||
+ | #SBATCH --ntasks-per-core=1 | ||
+ | |||
+ | # More SBATCH options: | ||
+ | # If you need 512GB memory nodes (otherwise only 256GB guaranteed): | ||
+ | # #SBATCH --mem=497G | ||
+ | # To run on the debug queue (max 10 nodes, 30 min): | ||
+ | # # | ||
+ | |||
+ | set -o errexit | ||
+ | set -o nounset | ||
+ | set -o pipefail | ||
+ | |||
+ | export MPICH_OFI_STARTUP_CONNECT=1 | ||
+ | export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK} | ||
+ | export OMP_PROC_BIND=close | ||
+ | export OMP_PLACES=cores | ||
+ | |||
+ | source " | ||
+ | |||
+ | module load cpeGNU | ||
+ | module load \ | ||
+ | cray-fftw \ | ||
+ | ELPA/ | ||
+ | libxsmm/1.16.1 \ | ||
+ | libxc/5.1.3 \ | ||
+ | Libint-CP2K/ | ||
+ | gcc/ | ||
+ | |||
+ | # Let the user see the currently loaded modules in the slurm log for completeness: | ||
+ | module list | ||
+ | |||
+ | CP2K_BASE_DIR="/ | ||
+ | CP2K_TEST_DIR=" | ||
+ | |||
+ | CP2K_ARCH=" | ||
+ | CP2K_VERSION=" | ||
+ | |||
+ | NTASKS_SINGLE_TEST=2 | ||
+ | NNODES_SINGLE_TEST=1 | ||
+ | SRUN_CMD=" | ||
+ | |||
+ | # the following should be sufficiently generic: | ||
+ | |||
+ | mkdir -p " | ||
+ | cd " | ||
+ | |||
+ | cp2k_rel_dir=$(realpath --relative-to=" | ||
+ | # srun does not like `-np`, override the complete command instead: | ||
+ | export cp2k_run_prefix=" | ||
+ | |||
+ | " | ||
+ | -arch " | ||
+ | -version " | ||
+ | -nobuild \ | ||
+ | -mpiranks ${NTASKS_SINGLE_TEST} \ | ||
+ | -ompthreads ${OMP_NUM_THREADS} \ | ||
+ | -maxtasks ${SLURM_NTASKS} \ | ||
+ | -cp2kdir " | ||
+ | | ||
+ | </ | ||
+ | |||
+ | ==== Test parallelization via Farming ==== | ||
+ | |||
+ | Since the startup time of '' | ||
+ | |||
+ | Simply add the option '' | ||
+ | |||
+ | ==== Minimal directory setup ==== | ||
+ | |||
+ | If you want to test a precompiled executable there is a minimal directory layout you have to reproduce to run the regtest: | ||
+ | |||
+ | * '' | ||
+ | * '' | ||
+ | * '' | ||
+ | |||
+ | An example if your HPC center uses EasyBuild to provide the CP2K package: | ||
+ | |||
+ | <code> | ||
+ | cp2k-prebuilt | ||
+ | ├── data -> /apps/ | ||
+ | ├── exe | ||
+ | │ └── prebuilt -> / | ||
+ | └── tests | ||
+ | </ | ||
+ | |||
+ | and then update the variables as follows: | ||
+ | |||
+ | <code bash> | ||
+ | CP2K_BASE_DIR="/ | ||
+ | CP2K_TEST_DIR=" | ||
+ | CP2K_REGTEST_SCRIPT_DIR="/ | ||
+ | |||
+ | CP2K_ARCH=" | ||
+ | CP2K_VERSION=" | ||
+ | </ |
dev/regtesting.txt · Last modified: 2023/10/19 14:03 by krack