The test suite
Reliability in the face of complexity is the real challenge of software development, as discussed in this paper.
Download and explode the test cases
The input files are in the directories below tsuite in the distribution. The output is in this ftp directory.
There are four directories below tsuite in the distribution. These contain different types of tests. Those in the auto directory are run every night in Lexington. These are a large number of simulations that test some aspect of the physics. The tests in the slow directory take about three days and are continuously running on machines here. The directories under programs contain programs that call Cloudy as a subroutine. Finally the crash directory contains a series of tests which should cause the code to crash if it has been compiled correctly.
Each directory contains several perl scripts and html files. The readme_*.htm files describe the contents of each directory. The file doc_tsuite.htm documents the purpose of each of the tests. It could be studied to see how to set up the code.
The tests should not be placed in a directory which contained an older version until the old version is first deleted. The names of the test cases change from time to time and placing the current version of the suite on top of an old one will not overwrite all the old ones. This will cause problems when the test suite is run. Either start with a new directory or first delete the contents of the directory.
Running the test cases
The test suite includes Perl scripts to run all the test cases and then check for problems. These are files that have names *.pl.
The runall.pl script will run all the input files. If you built the code using make in the source directory then you do not need to change runall.pl. If you built the code using one of the makefiles in the sys_XXX directories beneath source then you can execute the script giving the name of the directory as an argument. So, if the code was built in sys_icc then you would enter runall.pl sys_icc.
If you have built the code some other way then you will need to edit runall.pl so that the string "$exe" points to your executable for Cloudy.
The test suite takes about half a day with a fairly modern CPU so you might want to do this overnight.
Running the test suite on multiple cores
If you have a multi-core system, you could consider using the script run_parallel.pl instead of runall.pl to run the test suite. It will use multiple cores to run several test suite cases in parallel, thus considerably reducing the amount of wallclock time needed to run the test suite. For this to work, the GNU version of make (called gmake) needs to be installed on your computer. Other versions of make should also work, provided they support the "-j" flag to enable parallel execution. If you don't want to use gmake, you need to edit the script to change the name from "gmake" to the correct name of the make utility (usually "make"). Most modern UNIX systems (including Linux) will have gmake installed though. Contact your system administrator if you are in doubt. The syntax for calling run_parallel.pl is the same as described above for runall.pl, except that it takes an optional second argument setting the number of cores that should be used. By default, run_parallel.pl will use all available cores on your system (provided it can determine that number - this doesn't work on a Mac). If you want to use less cores, or the number was determined incorrectly, simply add the number of cores that should be used as a second parameter. Some typical examples are:
run_parallel.pl (cloudy.exe in source directory, use all available cores)
run_parallel.pl sys_icc 3 (cloudy.exe in sys_icc directory, use 3 cores)
run_parallel.pl . 2 (cloudy.exe in source directory, use 2 cores)
The "." in the last example needs to be used if you built Cloudy in the source directory and you want to override the default number of cores. Providing a higher number of cores than are physically available will not speed up execution (the parallel make is nearly 100% efficient), but will increase the risk of running out of memory. If you built Cloudy in a non-standard location, you need to provide the full path to the binary, e.g.
run_parallel.pl /path/to/cloudy.exe
BEWARE The run_parallel.pl script will clean out the test suite (i.e., it calls clean_tsuite.pl) before running the test suite to remove any files that are not part of the distribution. Make sure that you saved any relevant output from a previous run of the test suite before executing run_parallel.pl! Don't put any of your own files in the test suite directories, create a separate directory for that.
Starting with the C08 release there are 3 incarnations of the run_parallel.pl script: tsuite/auto/run_parallel.pl and tsuite/slow/run_parallel.pl run the auto or slow test suite only, while tsuite/run_parallel.pl runs both test suites combined.
Cleaning up the test suite directory
If you want to clean up the test suite directory after a run, you can type the command clean_tsuite.pl, which will delete all the test suite output. This is useful if diskspace is at a premium. The test suite output is surprisingly bulky! The command only works in the local directory, so you need to change directory to the test suite directory first before entering the command.
Starting with the C08 release there are 3 incarnations of the clean_tsuite.pl script: tsuite/auto/clean_tsuite.pl and tsuite/slow/clean_tsuite.pl clean the auto or slow test suite only, while tsuite/clean_tsuite.pl cleans both test suites.
The "assert" commands within the test suite
A large code must be completely checked every time anything is changed. The code uses extensive self-checking during a calculation to insure that the results are valid. The assert commands in the input files tell the code what answer to expect based either on analytical or previous results. This allows the code to confirm that it has found the correct answer. A distinctive string is printed if the right answer is not obtained. The Perl script checkall.pl will examine all output files to search for failed asserts, other problems, or crashes. These would indicate a problem.
The assert commands have nothing to do with the simulation or the astrophysics, but instead provide a way for the code to implement these automatic self-checks. The Perl script tests_remove_asserts.pl will remove all the asserts if you wish to reuse these scripts for other purposes.
Check the results
Problems will be announced at the end of the calculation by a line that includes the string PROBLEM, DISASTER or botched asserts. If the code crashes then the normal end-of-calculation string will not be printed.
The test distribution includes the Perl script checkall.pl that will check for problems once the test cases have been computed. Run this script and notice what it says. If you use run_parallel.pl this script will be called automatically after the test suite finished.
The stellar atmosphere test cases stars_*.in require the compiled stellar atmosphere files. These will not succeed if you did not install the stellar atmospheres. This is OK if you don't need those. Starting with the C08 release, the checkall.pl script will quietly skip sims that failed because the stellar atmosphere file was not installed and list them in the file skip.txt instead.
Next step, apply any HotFixes.
Return to the StepByStep instructions.
Return to nublado.org
