RobotTestingFramework 2.0.1
Robot Testing Framework
Loading...
Searching...
No Matches
Using plug-in to develop test cases
Author
Ali Paikan


Description

Test cases can be written as dynamic loadable libraries (e.g., .so, .dll) to be loaded and executed as separate plug-ins. This allows to have a simple test runner which loads and run the tests at run time without any needs to recompile the test runner. Using a simple example, we show how this can be done.

Note
The source code of the tutorial can be also found in the 'examples/plugin' folder within the Robot Testing Framework project directory. To compile it, you just need to enable the BUILD_EXAMPLES=ON flag of the CMake (if it is not already enabled) and build the Robot Testing Framework.


Requirements

You do not need any external library to use the plug-ins. All you need is to simply enable the plug-in system in the Robot Testing Framework and recompile it if it is not already enabled (it should be enabled by default).

   $ cd robot-testing-framework/build
   $ cmake ../ -DENABLE_PLUGIN=ON


Writing the plug-in

Create a class which inherited from the TestCase class. Override the run() method and/or setup(int argc, char** argv) , tearDown() if needed. The parameters set for the test case using robottestingframework::TestCase::setParam are parsed into standard `(int argc, char**argv)` format and are passed to the setup method. The original string parameter is also accessible using robottestingframework::TestCase::getParam. Here is an example:

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#ifndef ROBOTTESTINGFRAMEWORK_MYTEST_H
#define ROBOTTESTINGFRAMEWORK_MYTEST_H
class MyTest : public robottestingframework::TestCase
{
public:
MyTest();
bool setup(int argc, char** argv) override;
void tearDown() override;
void run() override;
};
#endif // ROBOTTESTINGFRAMEWORK_MYTEST_H
The base class to implememnt a test case.
Definition TestCase.h:42
virtual void tearDown()
tearDown is called after the test run
virtual void run()=0
run is called by the TestCase class if setup is successfull;
virtual bool setup(int argc, char **argv)
setup is called before the test run.

Implement the test body in the run() method and add use the 'PREPARE_PLUGIN(...)' macro to prepare the plug-in. The macro implements the necessary functionalities to make your test class be accessible from a shared library.

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include "MyTest.h"
using namespace robottestingframework;
MyTest::MyTest() :
TestCase("MyTest")
{
}
bool MyTest::setup(int argc, char** argv)
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::setup...");
return true;
}
void MyTest::tearDown()
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::teardown...");
// assert an arbitray error for example.
ROBOTTESTINGFRAMEWORK_ASSERT_ERROR("this is just for example!");
}
void MyTest::run()
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("testing integers");
ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(2 < 3, "is not smaller");
int a = 5;
int b = 3;
ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(a < b, Asserter::format("%d is not smaller than %d.", a, b));
}
#define ROBOTTESTINGFRAMEWORK_PREPARE_PLUGIN(classname)
Definition Plugin.h:30
#define ROBOTTESTINGFRAMEWORK_ASSERT_ERROR(message)
Assertion with throwing error exception.
Definition TestAssert.h:90
#define ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(condition, message)
Conditional failure report.
Definition TestAssert.h:149
#define ROBOTTESTINGFRAMEWORK_TEST_REPORT(message)
Reporting a message to the TestResult.
Definition TestAssert.h:131


Writing the test runner

Note
Basically you do not need to develop a test runner. There is the robottestingframework-testrunner utility (see Running test case plug-ins using robottestingframework-testrunner) already implemented in Robot Testing Framework which does the job for you. However, the following example is just for your reference and to understand better how a test plug-ing can be loaded and executed from the code.

Your plug-in is ready and can be compiled and built. Before doing, that we show how to use robottestingframework::DllPluginLoader to develop a simple test runner for loading the plug-in and running the test. It is quite simple. First create an instance of robottestingframework::plugin::DllPluginLoader class and call its open() method with the plug-in library (.dll, .so, ...) filename as its paramter. The robottestingframework::plugin::DllPluginLoader loads the library and return a pointer to the robottestingframework::TestCase implemented into the plug-in. If the plug-in cannot be loaded or it is not an Robot Testing Framework plug-in file, the robottestingframework::plugin::DllPluginLoader::open() returns a null pointer and the error message can be gotten using robottestingframework::plugin::DllPluginLoader::getLastError(). When you have an instance of your test case, it can be used to run the test as usual (see Some examples if you are not familiar with running a test case). In the following example we use a robottestingframework::TestRunner to execute our test:

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include <stdio.h>
using namespace robottestingframework;
int main(int argc, char* argv[])
{
if (argc < 2) {
printf("Usage: %s <plugin file name>\n", argv[0]);
printf("for example: %s libmytest.so\n", argv[0]);
return 0;
}
// load the test case plugin
printf("Loading the plugin... \n");
TestCase* test = loader.open(argv[1]);
if (test == NULL) {
printf("%s\n", loader.getLastError().c_str());
return 0;
}
// create a test listener to collect the result
ConsoleListener listener(false);
// create a test result and add the listeners
TestResult result;
result.addListener(&listener);
// create a test runner and run the test case
TestRunner runner;
runner.addTest(test);
runner.run(result);
return 0;
}
class ConsoleListener listens to any messages reported by the tests during the test run,...
The TestResult class is used to deliver the test results including any error and failures produced by...
Definition TestResult.h:44
void addListener(TestListener *listener)
Adding a new listener.
The TestRunner class runs the tests added as TestCase or TestSuite.
Definition TestRunner.h:42
void run(TestResult &result)
Run all the tests in the list.
void addTest(Test *test)
Adding a new test.
The DllPluginLoader loads an test case plug-in and gives the direct access to the TestCase.
TestCase * open(const std::string filename) override
open Loads a test case plugin
std::string getLastError() override
getLastError gets the last error if any.


Build and run the test

Now you can compile and build the plug-in and the runner. There is a CMake file in the examples/plugin folder which helps you to compile and build your test plugin and the simple test runner. Make sure that RobotTestingFramework_DIR environment variable is correctly set to point your Robot Testing Framework build or installed directory so that CMake can find the required modules to configure the project.

# Robot Testing Framework
#
# Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
cmake_minimum_required(VERSION 3.5)
find_package(RobotTestingFramework COMPONENTS DLL)
add_library(mytest MODULE MyTest.cpp MyTest.h)
target_link_libraries(mytest RobotTestingFramework::RTF
RobotTestingFramework::RTF_dll)
add_executable(simple_run run.cpp)
target_link_libraries(simple_run RobotTestingFramework::RTF
RobotTestingFramework::RTF_dll)


Now you can build and run the test as follows:

$ cd examples/plugin; mkdir build
$ cd build; cmake ../; make;
$ ./simple_run libmytest.so
 Loading the plugin...
 Starting test runner.
 Test case MyTest started...
 [INFO]  (MyTest) reports: running MyTest::setup...
 [INFO]  (MyTest) reports: testing integers
 [FAIL]  (MyTest) checking (a<b): 5 is not smaller than 3.
 [INFO]  (MyTest) reports: running MyTest::teardown...
 [ERROR] (MyTest) asserts error with exception: this is just for example!
 Test case MyTest failed!
 Ending test runner.