old-cross-binutils/gdb/testsuite/gdb.perf/lib/perftest/reporter.py

86 lines
2.8 KiB
Python
Raw Normal View History

# Copyright (C) 2013-2016 Free Software Foundation, Inc.
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# Text reports are written here.
# This is the perftest counterpart to gdb.sum.
SUM_FILE_NAME = "perftest.sum"
# Raw data that went into the report is written here.
# This is the perftest counterpart to gdb.log.
LOG_FILE_NAME = "perftest.log"
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
class Reporter(object):
"""Base class of reporter to report test results in a certain format.
Subclass, which is specific to a report format, should overwrite
methods report, start and end.
"""
def __init__(self, append):
"""Constructor of Reporter.
attribute append is used to determine whether to append or
overwrite log file.
"""
self.append = append
def report(self, *args):
raise NotImplementedError("Abstract Method:report.")
def start(self):
"""Invoked when reporting is started."""
raise NotImplementedError("Abstract Method:start.")
def end(self):
"""Invoked when reporting is done.
It must be overridden to do some cleanups, such as closing file
descriptors.
"""
raise NotImplementedError("Abstract Method:end.")
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
class TextReporter(Reporter):
"""Report results in a plain text file 'perftest.log'."""
def __init__(self, append):
super (TextReporter, self).__init__(Reporter(append))
self.txt_sum = None
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
self.txt_log = None
def report(self, test_name, measurement_name, data_points):
if len(data_points) == 0:
self.txt_sum.write("%s %s *no data recorded*\n" % (
test_name, measurement_name))
return
average = sum(data_points) / len(data_points)
data_min = min(data_points)
data_max = max(data_points)
self.txt_sum.write("%s %s %s\n" % (
test_name, measurement_name, average))
self.txt_log.write("%s %s %s, min %s, max %s, data %s\n" % (
test_name, measurement_name, average, data_min, data_max,
data_points))
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
def start(self):
mode = "a+" if self.append else "w"
self.txt_sum = open (SUM_FILE_NAME, mode);
self.txt_log = open (LOG_FILE_NAME, mode);
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
def end(self):
self.txt_sum.close ()
Perf test framework This patch adds a basic framework to do performance testing for GDB. perftest.py is about the test case, testresult.py is about test results, and how are they saved. reporter.py is about how results are reported (in what format). measure.py is about measuring the execution of tests by a collection of measurements. In V5: - Simplify perftest.exp. In V4: - Rename MeasurementCPUTime to MeasurementCpuTime, - Add 'pass' in empty method, - Simplify string comparison in perftest.exp. - Rename GDB_PERFORMANCE to GDB_PERFTEST_MODE and rename GDB_PERFORMANCE_TIMEOUT to GDB_PERFTEST_TIMEOUT. In V3, there are some changes, - Add wall time measurement, cpu time measurement and vmsize measurement. - Rename SingleStatisticTestCase to TestCaseWithBasicMeasurements, which measures cpu time, wall time, and memory (vmsize). - GDB_PERFORMANCE=run|compile|both to control the mode of perf testing. - New GDB_PERFORMANCE_TIMEOUT to specify the timeout. - Split proc prepare to proc compile and startup. - Disable GC while doing measurements. In V2, there are several changes to address Doug and Sanimir's comments. - Add copyright header and docstring in perftest/__init__.py - Remove config.py. - Fix docstring format. - Rename classes "SingleVariable" to "SingleStatistic". - Don't extend gdb.Function in class TestCase. Add a new method run to run the test case so that we can pass parameters to test. - Allow to customize whether to warm up and to append test log. - Move time measurement into test harness. Add a new class Measurement for a specific measurement and a new class Measure to measure them for a given test case. - A new class ResultFactory to create instances of TestResult. - New file lib/perftest.exp, which is to do some preparations and cleanups to simplify each *.exp file. - Skip compilation step if GDB_PERFORMANCE_SKIP_COMPILE is set. gdb/testsuite/ 2013-11-06 Yao Qi <yao@codesourcery.com> * lib/perftest.exp: New. * gdb.perf/lib/perftest/__init__.py: New. * gdb.perf/lib/perftest/measure.py: New. * gdb.perf/lib/perftest/perftest.py: New. * gdb.perf/lib/perftest/reporter.py: New. * gdb.perf/lib/perftest/testresult.py: New.
2013-09-24 09:56:26 +00:00
self.txt_log.close ()