No description
Find a file
Sergio Durigan Junior fb6a751f5f Improve analysis of racy testcases
This is an initial attempt to introduce some mechanisms to identify
racy testcases present in our testsuite.  As can be seen in previous
discussions, racy tests are really bothersome and cause our BuildBot
to pollute the gdb-testers mailing list with hundreds of
false-positives messages every month.  Hopefully, identifying these
racy tests in advance (and automatically) will contribute to the
reduction of noise traffic to gdb-testers, maybe to the point where we
will be able to send the failure messages directly to the authors of
the commits.

I spent some time trying to decide the best way to tackle this
problem, and decided that there is no silver bullet.  Racy tests are
tricky and it is difficult to catch them, so the best solution I could
find (for now?) is to run our testsuite a number of times in a row,
and then compare the results (i.e., the gdb.sum files generated during
each run).  The more times you run the tests, the more racy tests you
are likely to detect (at the expense of waiting longer and longer).
You can also run the tests in parallel, which makes things faster (and
contribute to catching more racy tests, because your machine will have
less resources for each test and some of them are likely to fail when
this happens).  I did some tests in my machine (8-core i7, 16GB RAM),
and running the whole GDB testsuite 5 times using -j6 took 23 minutes.
Not bad.

In order to run the racy test machinery, you need to specify the
RACY_ITER environment variable.  You will assign a number to this
variable, which represents the number of times you want to run the
tests.  So, for example, if you want to run the whole testsuite 3
times in parallel (using 2 cores), you will do:

  make check RACY_ITER=3 -j2

It is also possible to use the TESTS variable and specify which tests
you want to run:

  make check TEST='gdb.base/default.exp' RACY_ITER=3 -j2

And so on.  The output files will be put at the directory
gdb/testsuite/racy_outputs/.

After make invokes the necessary rules to run the tests, it finally
runs a Python script that will analyze the resulting gdb.sum files.
This Python script will read each file, and construct a series of sets
based on the results of the tests (one set for FAIL's, one for
PASS'es, one for KFAIL's, etc.).  It will then do some set operations
and come up with a list of unique, sorted testcases that are racy.
The algorithm behind this is:

  for state in PASS, FAIL, XFAIL, XPASS...; do
    if a test's state in every sumfile is $state; then
      it is not racy
    else
      it is racy

(The algorithm is actually a bit more complex than that, because it
takes into account other things in order to decide whether the test
should be ignored or not).

IOW, a test must have the same state in every sumfile.

After processing everything, the script prints the racy tests it could
identify on stdout.  I am redirecting this to a file named racy.sum.

Something else that I wasn't sure how to deal with was non-unique
messages in our testsuite.  I decided to do the same thing I do in our
BuildBot: include a unique identifier in the end of message, like:

  gdb.base/xyz.exp: non-unique message
  gdb.base/xyz.exp: non-unique message <<2>>

This means that you will have to be careful about them when you use
the racy.sum file.

I ran the script several times here, and it did a good job catching
some well-known racy tests.  Overall, I am satisfied with this
approach and I think it will be helpful to have it upstream'ed.  I
also intend to extend our BuildBot and create new, specialized
builders that will be responsible for detecting the racy tests every X
number of days.

2016-03-05  Sergio Durigan Junior  <sergiodj@redhat.com>

	* Makefile.in (DEFAULT_RACY_ITER): New variable.
	(CHECK_TARGET_TMP): Likewise.
	(check-single-racy): New rule.
	(check-parallel-racy): Likewise.
	(TEST_TARGETS): Adjust rule to account for RACY_ITER.
	(do-check-parallel-racy): New rule.
	(check-racy/%.exp): Likewise.
	* README (Racy testcases): New section.
	* analyze-racy-logs.py: New file.
2016-03-05 20:43:40 -05:00
bfd Automatic date update in version.in 2016-03-06 00:00:18 +00:00
binutils [ARM] Build attributes for ARMv8.1-A AdvSIMD 2016-03-04 14:16:48 +00:00
config Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
cpu Correct fr30 comment 2016-03-03 12:55:30 +10:30
elfcpp Remove typename from Mips64_rela_data 2016-01-12 12:08:06 -08:00
etc PR external/{16327,16328}: Remove etc/configure.texi and etc/standards.texi. 2014-06-27 11:33:25 +02:00
gas [ARM] Build attributes for ARMv8.1-A AdvSIMD 2016-03-04 14:16:48 +00:00
gdb Improve analysis of racy testcases 2016-03-05 20:43:40 -05:00
gold Fix datestamps on ChangeLog entries to read 2015 instead of 2016. 2016-03-04 14:19:12 -08:00
gprof Regen configure 2016-01-17 12:28:14 +10:30
include Fix datestamps on ChangeLog entries to read 2015 instead of 2016. 2016-03-04 14:19:12 -08:00
intl Regen intl/configure 2015-08-31 12:53:36 +09:30
ld Fix a ChangeLog entry 2016-03-04 06:48:01 -08:00
libdecnumber Remove leading/trailing white spaces in ChangeLog 2015-07-24 04:16:47 -07:00
libiberty Sync libiberty with GCC. 2016-01-28 21:44:42 +01:00
opcodes Regenerate or1k opcodes file 2016-03-03 00:23:31 +10:30
readline Revert "Sync readline/ to version 7.0 alpha" 2015-07-25 15:57:00 -04:00
sim Fix bugs in the simulation of the AArch64's ADDP, FADDP, LD1, CCMP and CCMP instructions. 2016-03-03 15:22:53 +00:00
texinfo
zlib Import zlib 1.2.8 with local change merged in. 2015-11-25 15:14:47 -08:00
.cvsignore add autom4te.cache to .cvsignore 2007-02-13 15:25:58 +00:00
.gitattributes Add a .gitattributes file for use with git-merge-changelog 2014-07-25 18:07:23 -04:00
.gitignore Import changes made to files shared with the FSF GCC project. 2016-01-11 11:06:56 +00:00
ChangeLog Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
compile Update from upstream Automake 2014-11-16 13:43:48 +01:00
config-ml.in Sync toplevel files with GCC 2015-07-27 07:49:05 -07:00
config.guess Import changes made to files shared with the FSF GCC project. 2016-01-11 11:06:56 +00:00
config.rpath Remove freebsd1 from libtool.m4 macros and config.rpath. 2011-02-13 21:00:14 +00:00
config.sub Import changes made to files shared with the FSF GCC project. 2016-01-11 11:06:56 +00:00
configure Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
configure.ac Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
COPYING 2005-07-14 Kelley Cook <kcook@gcc.gnu.org> 2005-07-14 01:24:56 +00:00
COPYING.LIB 2005-07-16 Kelley Cook <kcook@gcc.gnu.org> 2005-07-16 02:41:34 +00:00
COPYING.LIBGLOSS 2013-01-07 Jeff Johnston <jjohnstn@redhat.com> 2013-01-07 21:39:26 +00:00
COPYING.NEWLIB 2013-10-01 Jeff Johnston <jjohnstn@redhat.com> 2013-10-01 18:14:04 +00:00
COPYING3 * COPYING3: New file. Contains version 3 of the GNU General Public License. 2007-07-17 13:50:23 +00:00
COPYING3.LIB * COPYING3: New file. Contains version 3 of the GNU General Public License. 2007-07-17 13:50:23 +00:00
depcomp Update from upstream Automake 2014-11-16 13:43:48 +01:00
djunpack.bat * djunpack.bat: Use ".." quoting in Sed command, for the sake of 2009-03-27 13:37:09 +00:00
install-sh Update from upstream Automake 2014-11-16 13:43:48 +01:00
libtool.m4 Sync top-level btool.m4 with GCC 2016-01-12 08:44:52 -08:00
ltgcc.m4 * libtool.m4: Update to libtool 2.2.6. 2008-09-29 15:28:14 +00:00
ltmain.sh PR target/59788 2014-02-06 11:01:57 +01:00
ltoptions.m4 Sync Libtool from GCC. 2010-01-09 21:11:44 +00:00
ltsugar.m4 * libtool.m4: Update to libtool 2.2.6. 2008-09-29 15:28:14 +00:00
ltversion.m4 Sync Libtool from GCC. 2010-01-09 21:11:44 +00:00
lt~obsolete.m4 Sync Libtool from GCC. 2010-01-09 21:11:44 +00:00
MAINTAINERS Update description of ownership of files in include/ 2014-11-04 16:14:14 -08:00
Makefile.def Sync top-level Makefile.def with GCC 2016-01-12 08:34:40 -08:00
Makefile.in Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
Makefile.tpl Sync top level files with gcc. 2016-02-10 10:54:29 +00:00
makefile.vms 19990502 sourceware import 1999-05-03 07:29:11 +00:00
missing Update from upstream Automake 2014-11-16 13:43:48 +01:00
mkdep * mkdep: New file. 1999-08-08 17:46:02 +00:00
mkinstalldirs Update from upstream Automake 2014-11-16 13:43:48 +01:00
move-if-change Update `move-if-change' from gnulib 2014-11-16 17:04:02 +01:00
README 19990502 sourceware import 1999-05-03 07:29:11 +00:00
README-maintainer-mode Cleanups after the update to Autoconf 2.64, Automake 1.11. 2009-08-22 17:08:11 +00:00
setup.com 2009-09-01 Tristan Gingold <gingold@adacore.com> 2009-09-01 13:38:26 +00:00
src-release.sh fix gdb version parsing in src-release.sh 2016-01-17 10:01:55 +04:00
symlink-tree 2005-07-14 Kelley Cook <kcook@gcc.gnu.org> 2005-07-14 01:24:56 +00:00
ylwrap Update from upstream Automake 2014-11-16 13:43:48 +01:00

		   README for GNU development tools

This directory contains various GNU compilers, assemblers, linkers, 
debuggers, etc., plus their support routines, definitions, and documentation.

If you are receiving this as part of a GDB release, see the file gdb/README.
If with a binutils release, see binutils/README;  if with a libg++ release,
see libg++/README, etc.  That'll give you info about this
package -- supported targets, how to use it, how to report bugs, etc.

It is now possible to automatically configure and build a variety of
tools with one command.  To build all of the tools contained herein,
run the ``configure'' script here, e.g.:

	./configure 
	make

To install them (by default in /usr/local/bin, /usr/local/lib, etc),
then do:
	make install

(If the configure script can't determine your type of computer, give it
the name as an argument, for instance ``./configure sun4''.  You can
use the script ``config.sub'' to test whether a name is recognized; if
it is, config.sub translates it to a triplet specifying CPU, vendor,
and OS.)

If you have more than one compiler on your system, it is often best to
explicitly set CC in the environment before running configure, and to
also set CC when running make.  For example (assuming sh/bash/ksh):

	CC=gcc ./configure
	make

A similar example using csh:

	setenv CC gcc
	./configure
	make

Much of the code and documentation enclosed is copyright by
the Free Software Foundation, Inc.  See the file COPYING or
COPYING.LIB in the various directories, for a description of the
GNU General Public License terms under which you can copy the files.

REPORTING BUGS: Again, see gdb/README, binutils/README, etc., for info
on where and how to report problems.