No description
This is an initial attempt to introduce some mechanisms to identify racy testcases present in our testsuite. As can be seen in previous discussions, racy tests are really bothersome and cause our BuildBot to pollute the gdb-testers mailing list with hundreds of false-positives messages every month. Hopefully, identifying these racy tests in advance (and automatically) will contribute to the reduction of noise traffic to gdb-testers, maybe to the point where we will be able to send the failure messages directly to the authors of the commits. I spent some time trying to decide the best way to tackle this problem, and decided that there is no silver bullet. Racy tests are tricky and it is difficult to catch them, so the best solution I could find (for now?) is to run our testsuite a number of times in a row, and then compare the results (i.e., the gdb.sum files generated during each run). The more times you run the tests, the more racy tests you are likely to detect (at the expense of waiting longer and longer). You can also run the tests in parallel, which makes things faster (and contribute to catching more racy tests, because your machine will have less resources for each test and some of them are likely to fail when this happens). I did some tests in my machine (8-core i7, 16GB RAM), and running the whole GDB testsuite 5 times using -j6 took 23 minutes. Not bad. In order to run the racy test machinery, you need to specify the RACY_ITER environment variable. You will assign a number to this variable, which represents the number of times you want to run the tests. So, for example, if you want to run the whole testsuite 3 times in parallel (using 2 cores), you will do: make check RACY_ITER=3 -j2 It is also possible to use the TESTS variable and specify which tests you want to run: make check TEST='gdb.base/default.exp' RACY_ITER=3 -j2 And so on. The output files will be put at the directory gdb/testsuite/racy_outputs/. After make invokes the necessary rules to run the tests, it finally runs a Python script that will analyze the resulting gdb.sum files. This Python script will read each file, and construct a series of sets based on the results of the tests (one set for FAIL's, one for PASS'es, one for KFAIL's, etc.). It will then do some set operations and come up with a list of unique, sorted testcases that are racy. The algorithm behind this is: for state in PASS, FAIL, XFAIL, XPASS...; do if a test's state in every sumfile is $state; then it is not racy else it is racy (The algorithm is actually a bit more complex than that, because it takes into account other things in order to decide whether the test should be ignored or not). IOW, a test must have the same state in every sumfile. After processing everything, the script prints the racy tests it could identify on stdout. I am redirecting this to a file named racy.sum. Something else that I wasn't sure how to deal with was non-unique messages in our testsuite. I decided to do the same thing I do in our BuildBot: include a unique identifier in the end of message, like: gdb.base/xyz.exp: non-unique message gdb.base/xyz.exp: non-unique message <<2>> This means that you will have to be careful about them when you use the racy.sum file. I ran the script several times here, and it did a good job catching some well-known racy tests. Overall, I am satisfied with this approach and I think it will be helpful to have it upstream'ed. I also intend to extend our BuildBot and create new, specialized builders that will be responsible for detecting the racy tests every X number of days. 2016-03-05 Sergio Durigan Junior <sergiodj@redhat.com> * Makefile.in (DEFAULT_RACY_ITER): New variable. (CHECK_TARGET_TMP): Likewise. (check-single-racy): New rule. (check-parallel-racy): Likewise. (TEST_TARGETS): Adjust rule to account for RACY_ITER. (do-check-parallel-racy): New rule. (check-racy/%.exp): Likewise. * README (Racy testcases): New section. * analyze-racy-logs.py: New file. |
||
---|---|---|
bfd | ||
binutils | ||
config | ||
cpu | ||
elfcpp | ||
etc | ||
gas | ||
gdb | ||
gold | ||
gprof | ||
include | ||
intl | ||
ld | ||
libdecnumber | ||
libiberty | ||
opcodes | ||
readline | ||
sim | ||
texinfo | ||
zlib | ||
.cvsignore | ||
.gitattributes | ||
.gitignore | ||
ChangeLog | ||
compile | ||
config-ml.in | ||
config.guess | ||
config.rpath | ||
config.sub | ||
configure | ||
configure.ac | ||
COPYING | ||
COPYING.LIB | ||
COPYING.LIBGLOSS | ||
COPYING.NEWLIB | ||
COPYING3 | ||
COPYING3.LIB | ||
depcomp | ||
djunpack.bat | ||
install-sh | ||
libtool.m4 | ||
ltgcc.m4 | ||
ltmain.sh | ||
ltoptions.m4 | ||
ltsugar.m4 | ||
ltversion.m4 | ||
lt~obsolete.m4 | ||
MAINTAINERS | ||
Makefile.def | ||
Makefile.in | ||
Makefile.tpl | ||
makefile.vms | ||
missing | ||
mkdep | ||
mkinstalldirs | ||
move-if-change | ||
README | ||
README-maintainer-mode | ||
setup.com | ||
src-release.sh | ||
symlink-tree | ||
ylwrap |
README for GNU development tools This directory contains various GNU compilers, assemblers, linkers, debuggers, etc., plus their support routines, definitions, and documentation. If you are receiving this as part of a GDB release, see the file gdb/README. If with a binutils release, see binutils/README; if with a libg++ release, see libg++/README, etc. That'll give you info about this package -- supported targets, how to use it, how to report bugs, etc. It is now possible to automatically configure and build a variety of tools with one command. To build all of the tools contained herein, run the ``configure'' script here, e.g.: ./configure make To install them (by default in /usr/local/bin, /usr/local/lib, etc), then do: make install (If the configure script can't determine your type of computer, give it the name as an argument, for instance ``./configure sun4''. You can use the script ``config.sub'' to test whether a name is recognized; if it is, config.sub translates it to a triplet specifying CPU, vendor, and OS.) If you have more than one compiler on your system, it is often best to explicitly set CC in the environment before running configure, and to also set CC when running make. For example (assuming sh/bash/ksh): CC=gcc ./configure make A similar example using csh: setenv CC gcc ./configure make Much of the code and documentation enclosed is copyright by the Free Software Foundation, Inc. See the file COPYING or COPYING.LIB in the various directories, for a description of the GNU General Public License terms under which you can copy the files. REPORTING BUGS: Again, see gdb/README, binutils/README, etc., for info on where and how to report problems.