Specifying Test Settings¶
This section gives a detailed description of how to specify settings for regression tests. It also describes the relation of test settings to test cases.
Basics¶
Test settings are specified as class attributes inside test classes, described in section Creating a New Test. The general form is
settings_name = TestSettings(
arg1=value1,
arg2=value2,
# ...
)
The name can by any valid Python identifier, written conventionally in snake_case
. The arguments and values specified in the initializer of the used settings class define the parameters to be used for decompilations. For example, you may specify the input file or the used architecture. The selected arguments and values are then used to create arguments for the decompiler. For example, the following settings specify the input file and prescribe the use of the x86
architecture:
settings = TestSettings(
input='file.exe',
arch='x86'
)
From the above settings, the following retdec-decompiler
argument list is automatically created:
retdec-decompiler file.exe -a x86
For a complete list of possible arguments to the initializer, see the description of DecompilerTestSettings
.
Every argument can be either a single value or a list of values. When you specify a single value, it will be used for all decompilations. However, if you specify multiple values, a separate decompilation is run for all of them. For example, consider the following test settings:
settings = TestSettings(
input='file.exe',
arch=['x86', 'arm']
)
For such settings, the following two decompilations are run:
retdec-decompiler file.exe -a x86
retdec-decompiler file.exe -a arm
That is, the regression tests framework runs a single decompilation for every combination of the values specified in the settings.
Test Cases and Their Creation¶
A test case is an instance of your test class with an associated decompilation. Recall that a test class is a class that inherits from Test
(see section Creating a New Test). As described above, when you specify settings with some arguments having multiple values (e.g. several architectures), a separate decompilation is run for all of them. For every decompilation, a test case is created, and all its test methods are called.
For example, consider the following test:
class Sample(Test)
settings = TestSettings(
input='file.exe',
arch=['x86', 'arm']
)
def test_something1(self):
# ...
def test_something2(self):
# ...
For this test, the following two test cases are created:
Sample (file.exe -a x86)
Sample (file.exe -a arm)
The two test methods are then called on each of them:
Sample (file.exe -a x86)
test_something1()
test_something2()
Sample (file.exe -a arm)
test_something1()
test_something2()
Classes Having Multiple Settings¶
You may specify several settings in a test class. This is handy when you want to use different settings for some decompilations. For example, consider the following class:
class Sample(Test)
settings1 = TestSettings(
input='file1.exe',
arch=['x86', 'arm']
)
settings2 = TestSettings(
input='file2.elf',
arch='thumb'
)
# Test methods...
We want to decompile file1.exe
on x86
and arm
, and file2.elf
on thumb
. From this test class, the following three test cases are created:
Sample (file1.exe -a x86)
Sample (file1.exe -a arm)
Sample (file2.elf -a thumb)
Arbitrary Parameters for the Decompiler¶
If you look at the complete list of possible arguments (DecompilerTestSettings
), you see that not all retdec-decompiler
parameters may be specified as arguments to TestSettings
. The reason is that retdec-decompiler
provides too many parameters and their support in the form of arguments would be cumbersome. However, it is possible to specify arbitrary arguments that are directly passed to the retdec-decompiler
via the args
argument:
class Sample(Test):
settings = TestSettings(
input='file.exe'
arch='x86',
args='--select-decode-only --select-functions func1,func2'
)
These settings result into the creation of the following decompilation:
retdec-decompiler file.exe -a x86 --select-decode-only --select-functions func1,func2
In a greater detail, the args
argument is taken, split into sub-arguments by whitespace, and passed to the retdec-decompiler
. The argument list internally looks like this:
['file.exe', '-a', 'x86', '--select-decode-only', '--select-functions', 'func1,func2']
Hint
When it is possible to specify a retdec-decompiler
parameter in the form of a named argument (like architecture or endianness), always prefer it to specifying raw arguments by using the args
argument. That is, do not write
class Sample(Test):
settings = TestSettings(
input='file.exe'
args='-a x86' # Always prefer using arch='x86'.
)
The reason is that named arguments are less prone to changes in retdec-decompiler
. Indeed, when such an argument changes in retdec-decompiler
, all that has to be done is changing the internal mapping of named arguments to retdec-decompiler
arguments. No test needs to be changed.
If you want to specify separate arguments for several decompilations for single settings, place them into a list when specifying the settings. For example, consider the following test class:
class Sample(Test):
settings = TestSettings(
input='file.elf'
arch='x86',
args=[
'--select-decode-only --select-functions func1,func2',
'--select-decode-only --select-functions func3'
]
)
It results into these two decompilations:
retdec-decompiler file.elf -a x86 --select-decode-only --select-functions func1,func2
retdec-decompiler file.elf -a x86 --select-decode-only --select-functions func3
You can also specify multiple settings, as already described earlier in this section.
Specifying Input Files From Directory¶
When there is a lot of input files, the directory structure may become less readable (the test.py
file is buried among input files). In a situation like this, you may put the input files into a directory (e.g. input
) and use files_in_dir()
to automatically generate a list of files in this directory:
settings = TestSettings(
input=files_in_dir('inputs')
)
You can also specify which files should be included or excluded:
settings = TestSettings(
input=files_in_dir('inputs', matching=r'.*\.exe', excluding=['problem-file.exe'])
)