Tester namespace¶

This namespace provided a number of object predefined testing object to help define a given type of test. Custom test object can be defined by the user inline of a given test for via the extension API. It is recommend if a custom test is to be used by for than one test that it should be provided via the extension API to avoid maintaining issue with keeping multiple copies in sync.

Most testing properties of provide a default tester object that will accept a raw value such as a string or number. however it common to want to provide a different type of test or a more complex test set. Using the objects in the tester namespace can make it easy to customize or compose the needed logic

Logical Test¶

These provide the common equal, less than, etc logic that are needed for basic value testing.

class Equal(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value is equal to the expected value

Parameters:

value (Any) – The value we are testing for
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class NotEqual(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value does not match the expected value

Parameters:

value (Any) – The value we are testing for
test_value (Optional[Any]) – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class LessThan(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value is less than the expected value

Parameters:

value (Any) – The value we are testing for
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class LessEqual(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value is less than or equal to the expected value.

Parameters:

value (Any) – The value we are testing for
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class GreaterThan(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value is greater than the expected value

Parameters:

value (Any) – The value we are testing for
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class GreaterEqual(value, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the value is greater than or equal to the expected value

Parameters:

value (Any) – The value we are testing for
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

Content Test¶

These provide easy way to test the content of a file or text stream.

class ContainsExpression(regexp, description, killOnFailure=False, description_group=None, reflags=0)¶

Test that the content contains the provided expression.

If re.M is provided as part of the reflags argument the whole string will be tested with re.search. If re.M is not provided the content will be broken up into separate line and the expression is tested on each line with re.search.

Parameters:

regexp (str) – The expression to test for
reflags (int) – Optional set of re flags to control behavior of the expression.
kill_on_failure – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (str) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class ExcludesExpression(regexp, description, killOnFailure=False, description_group=None, reflags=0)¶

Test that the content does not contain the provided expression.

If re.M is provided as part of the reflags argument the whole string will be tested with re.search. If re.M is not provided the content will be broken up into separate line and the expression is tested on each line with re.search.

Parameters:

regexp – The expression to test for.
reflags – Optional set of re flags to control behavior of the expression.
kill_on_failure – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’.
description – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”.

class GoldFile(goldfile, test_value=None, kill_on_failure=False, normalize_eol=True, case_insensitive=False, description_group=None, description=None)¶

Tests if a file matches expected results via comparing a file or stream output with the content of the gold file. The gold file can contain wild card statement of {} or `` to tell the testing logic that differences in the area are to be ignored.

Parameters:

goldfile (str) – the gold file to be used for the test. Assumed to relative to the location of the test file.
test_value (Optional[str]) – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
normalize_eol (bool) – If True will normalize the \\r\\n sequences to \\n to help with common differences between different operating system such as Unix or Windows systems. If False no normalizing will happen.
case_insensitive (bool) – If True, the content of both the gold file and the stream will be compared case-insensitively. If False, gold comparison will be case-sensitive.
description_group (str) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

Examples

Test a gold file with new normalization of new line of /r/n to /n

tr.Processes.Default.stdout= Testers.GoldFile("gold/exists.gold", normalize_eol=True)

class ZipContent(includes=None, excludes=None, kill_on_failure=False, description_group=None)¶

Tests that a compressed archived contains or excludes a give entry. Supported archive types are:

.bz2
.tar.gz
.tgz
.tar.bz2
.tbz
.tb2
.zip

Parameters:

includes (Optional[List[str]]) – A list of one more path relative of from the archive root of the entity to test to exist.
excludes (Optional[List[str]]) – A list of one more path relative of from the archive root of the entity to test to not exist.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – Optional value used to help provide better context in the test message.

Examples

Test if a zip file contains a certain files.

contains = ['lorem.txt', 'lorem3/lorem.txt', 'lorem2.txt']
content_tester = Testers.ZipContent(includes=contains)
t.Disk.File("lorem.zip", exists=True, content=content_tester)

Another example of the above using the File object directly

tar = t.Disk.File("lorem.tar.gz")
contains = ['lorem.txt', 'lorem3/lorem.txt', 'lorem2.txt']
tar.content = Testers.ZipContent(includes=contains)

class FileContentCallback(callback, description, killOnFailure=False, description_group=None)¶

Allow for defining custom content tests for a file or stream object

Parameters:

callback – The callback function.
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
description – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”
kill_on_failure – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’

Callback Interface:

def callback(data:str) -> Optional[str]:
    return errorMessage

data: is file contents

returns: either a string describing what’s wrong with the test if it fails or ‘’ or None if the test succeed.

File System¶

Test file system for state.

class FileExists(exists, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the file exists or does not exist on Disk.

Parameters:

exists (bool) – Test that is the file exists if True, else non-existence.
test_value (Optional[str]) – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure (bool) – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group (Optional[str]) – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description (Optional[str]) – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

class DirectoryExists(exists, test_value=None, kill_on_failure=False, description_group=None, description=None)¶

Test that the file exists or does not exist on Disk.

Parameters:

exists – Test that is the directory exists if True, else non-existence.
test_value – The runtime value we will test. This is normally a string that is used to reference the eventinfo object for a runtime value. However it might be a user defined value, such as a path to a file. It can also be a function that will be called to return the expected content to test against.
kill_on_failure – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’
description – This is what we are testing such as “Testing return code is 5” or “Checking file file X exists”

Composeable¶

Often it is useful to compose a set of test. These objects allow testing for a set of conditions.

class Any(*lst)¶

Passes if any of the provided Tester passes. Provided in as a Global to help with ease of use.

Parameters:: *lst – The function list of all Testers provided

Example

Test is the return code is 0 or 5

tr.Processes.Default.ReturnCode = Any(0,5)

Test that the stdout stream of process match one of two possible gold files

tr.Processes.Default.Streams.stdin = All("gold/hello1.gold", "gold/hello2.gold")

class All(*lst)¶

Passes if all of the provided Tester pass. This can also be achieved in many cases by just using the += operator on the testable property. Provided in as a Global to help with ease of use.

Parameters:: *lst – The function list of all Testers provided

Example

Test is the File object

f = Test.Disk.File("foo.txt")
f.Content = All(
                Testers.ContainsExpression("value1", 'Has value1')
                Testers.ContainsExpression("value2", 'Has value2')
                )
# this is the same as saying...
f.Content = Testers.ContainsExpression("value1", 'Has value1')
f.Content += Testers.ContainsExpression("value2", 'Has value2')

Test that the stdout stream of process match a gold files and does not contain an expression. In this case the gold file might have a wild card for a data section, which normally can be ignored, unless it contains a certain values.

tr.Processes.Default.Streams.stdin = All(
    "gold/hello1.gold",
    Testers.ExcludesExpression("bad value", "Does not contain 'bad value'")
    )

class Not(testobj)¶

Results in the negation of the Test as passing. The Python not operator ! cannot be used as it results in immediate result. This allows for a delay result needed for composing more complex tester logic. Provided in as a Global to help with ease of use.

Example

Test is the return code is a non-zero value

tr.Processes.Default.ReturnCode = Not(0)

Lambda¶

When something custom is needed there is a general lambda tester to make it easy to test about anything

class Lambda(func, kill_on_failure=False, description_group=None)¶

Allow providing to python function to:

Provide a custom test
Custom logic to execute as part of an AuTest event object being called.

Parameters:

func – The callback function to call
kill_on_failure – Setting this to True will kill the test from processing the rest of the test run and any existing item in the event queue for the current scope. This should only be used in cases when a failure mean we really need to do a hard stop. For example need to stop because the test ran to long.
description_group – This is extra information about the file, process, etc that might be useful to give the test more context, should be in form of ‘Type: name’, ie ‘Process: proc1’

Callback Interface:

callback(eventinfo)

callback(eventinfo,tester)

Where:

eventinfo: Object related to the event or test property the tester is mapped to.

tester: Reference to the Lambda object. Can be used for better control setting state.

The callback is expected to return a tuple of (result:resulttype, description:str, message:str) where:

resulttype: is the ????

description: A string describing what the Lambda is testing

message: A string describing the result of the test.

Example

This can also be use to help with advance Flow control by reacting to the eventinfo object.

def StopProcess(event,time):
    if event.TotalRunTime > time:
        event.object.Stop()
    return 0, "this is a test", "this is what happened"


t = Test.AddTestRun()
# Maps call back to the Running Event
# This will stop the test in after it run for at least 1 second.
t.RunningEvent.Connect(
    Testers.Lambda(
        lambda ev: StopProcess(ev,1)
    )
)

Define a custom test.

t = Test.AddTestRun("Get data of Python version")
t.Processes.Default.Command = "python --version"
t.ReturnCode = 0

path = t.Processes.Default.Streams.All.AbsPath


def custom_test(event, tester):
    with open(path) as f:
        data1 = f.read()
    with open(tester.GetContent(event)) as f:
        data2 = f.read()
    if data1 == data2:
        return (True, "Check that versions match", "Python versions did matched")
    else:
        return (False, "Check that versions match", "Python versions did not matched")


t = Test.AddTestRun("Do it again")
t.Processes.Default.Command = "python --version"
t.ReturnCode = 0
t.Processes.Default.Streams.All.Content = Testers.Lambda(custom_test)

Tester namespace¶

Logical Test¶

Content Test¶

File System¶

Composeable¶

Lambda¶

AuTest

Navigation

Related Topics