Execute an example program as a test suite. More...

#include "example-as-test.h"

Inheritance diagram for ns3::ExampleAsTestSuite:

Collaboration diagram for ns3::ExampleAsTestSuite:

Public Member Functions
	ExampleAsTestSuite (const std::string name, const std::string program, const std::string dataDir, const std::string args="", const TestDuration duration=QUICK, const bool shouldNotErr=true)
	Constructor. More...

Public Member Functions inherited from ns3::TestSuite
	TestSuite (std::string name, Type type=UNIT)
	Construct a new test suite. More...

TestSuite::Type	GetTestType ()
	get the kind of test this test suite implements More...

Public Member Functions inherited from ns3::TestCase
	TestCase (const TestCase &)=delete

virtual	~TestCase ()
	Destructor. More...

std::string	GetName () const

TestCase &	operator= (const TestCase &)=delete

Additional Inherited Members
Public Types inherited from ns3::TestSuite
enum	Type { ALL = 0 , UNIT , SYSTEM , EXAMPLE , PERFORMANCE }
	Type of test. More...

Public Types inherited from ns3::TestCase
enum	TestDuration { QUICK = 1 , EXTENSIVE = 2 , TAKES_FOREVER = 3 }
	How long the test takes to execute. More...

Protected Member Functions inherited from ns3::TestCase
	TestCase (std::string name)
	Constructor. More...

void	AddTestCase (TestCase *testCase, TestDuration duration=QUICK)
	Add an individual child TestCase to this test suite. More...

TestCase *	GetParent () const
	Get the parent of this TestCase. More...

bool	IsStatusFailure () const
	Check if any tests failed. More...

bool	IsStatusSuccess () const
	Check if all tests passed. More...

void	SetDataDir (std::string directory)
	Set the data directory where reference trace files can be found. More...

void	ReportTestFailure (std::string cond, std::string actual, std::string limit, std::string message, std::string file, int32_t line)
	Log the failure of this TestCase. More...

bool	MustAssertOnFailure () const
	Check if this run should assert on failure. More...

bool	MustContinueOnFailure () const
	Check if this run should continue on failure. More...

std::string	CreateDataDirFilename (std::string filename)
	Construct the full path to a file in the data directory. More...

std::string	CreateTempDirFilename (std::string filename)
	Construct the full path to a file in a temporary directory. More...

Detailed Description

Execute an example program as a test suite.

You can use this TestSuite to add an example to the test suite with checking of the example output (std::out and std::err). This is an alternative to adding an example using the examples-to-run.py file. The key difference between the two methods is what criteria is used to for success. Examples added to examples-to-run.py will be run and the exit status checked (non-zero indicates failure). ExampleAsTestSuite adds checking of output against a specified known "good" reference file.

Warning: If you are thinking about using this class, strongly consider using a standard test instead. The TestSuite class has better checking using the NS_TEST_* macros and in almost all cases is the better approach. If your test can be done with a TestSuite class you will be asked by the reviewers to rewrite the test when you do a pull request.

Test Addition

To use an example program as a test you need to create a test suite file and add it to the appropriate list in your module CMakeLists.txt file. The "good" output reference file needs to be generated for detecting regressions.

Let's assume your module is called mymodule, and the example program is mymodule/examples/mod-example.cc. First you should create a test file mymodule/test/mymodule-examples-test-suite.cc which looks like this:

#include "ns3/example-as-test.h"
static ns3::ExampleAsTestSuite g_modExampleOne("mymodule-example-mod-example-one",
    "mod-example", NS_TEST_SOURCEDIR, "--arg-one");
static ns3::ExampleAsTestSuite g_modExampleTwo("mymodule-example-mod-example-two",
    "mod-example", NS_TEST_SOURCEDIR, "--arg-two"); 

The arguments to the constructor are the name of the test suite, the example to run, the directory that contains the "good" reference file (the macro NS_TEST_SOURCEDIR is normally the correct directory), and command line arguments for the example. In the preceding code the same example is run twice with different arguments.

You then need to add that newly created test suite file to the list of test sources in mymodule/CMakeLists.txt. Building of examples is an option so you need to guard the inclusion of the test suite:

if (bld.env['ENABLE_EXAMPLES']):

module.source.append('model/mymodule-examples-test-suite.cc')

Since you modified a CMakeLists.txt file you need to reconfigure and rebuild everything.

You just added new tests so you will need to generate the "good" output reference files that will be used to verify the example:

./test.py --suite="mymodule-example-*" --update

This will run all tests starting with "mymodule-example-" and save new "good" reference files. Updating the reference file should be done when you create the test and whenever output changes. When updating the reference output you should inspect it to ensure that it is valid. The reference files should be committed with the new test.

Test Verification

You can run the test with the standard test.py script. For example to run the suites you just added:

./test.py --suite="mymodule-example-*"

This will run all mymodule-example-... tests and report whether they produce output matching the reference files.

Writing good examples for testing

When setting up an example for use by this class you should be very careful about what output the example generates. For example, writing output which includes simulation time (especially high resolution time) makes the test sensitive to potentially minor changes in event times. This makes the reference output hard to verify and hard to keep up-to-date. Output as little as needed for the example and include only behavioral state that is important for determining if the example has run correctly.

Definition at line 209 of file example-as-test.h.

Constructor & Destructor Documentation

◆ ExampleAsTestSuite()

ns3::ExampleAsTestSuite::ExampleAsTestSuite	(	const std::string	name,
		const std::string	program,
		const std::string	dataDir,
		const std::string	args = `""`,
		const TestDuration	duration = `QUICK`,
		const bool	shouldNotErr = `true`
	)

Constructor.

Parameters

[in]	name	The test case name, typically the program name and summary of the arguments, such as `my-example-foo`
[in]	program	The actual example program names, such as `my-example`
[in]	dataDir	The location of the reference file. This is normally provided by the symbol `NS_TEST_SOURCEDIR` in the `module-examples-test-suite.cc` file. The reference file should be named after the test case name, for example `my-example-foo.log`. If you use the `--update` argument to `test.py` or `test-runner` the reference file will be created with the correct name.
[in]	args	Any additional arguments to the program.
[in]	shouldNotErr	Whether an error return status should be considered a test failure. This is useful when testing error detection which might return a non-zero status. The output (on `std::cout` and `std::cerr`) will be compared to the reference logs as normal.
[in]	duration	Amount of time this test takes to execute (defaults to QUICK).

The documentation for this class was generated from the following file:

src/core/model/example-as-test.h

Public Member Functions

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ ExampleAsTestSuite()