Output Stream Policy for Tests

- -

Overview of the `IO` Object

- -

Each test function is given an IO object, which provides - methods for inspecting stdout and stderr output - streams, programmatically adding data to the stdin input stream, - and clearing output streams as needed. Although the IO object is - optional, it is available for cases where I/O validation or cleanup is - essential to the test.

- -

Purpose

- -

Each test function is responsible for managing any output generated - on stdout or stderr by the function under test - (fut). TestBench will automatically clear the streams before - each test begins and will check them after the test completes, treating any - remaining output as unintended and marking the test as a failure. This policy - ensures that tests intentionally handle output by either validating, - clearing, or ignoring it, thereby maintaining a clean and predictable testing - environment.

- -

Policy Guidelines

1. Define an Output Handling Policy:

Every test should have a defined policy for how it handles output generated by the fut. There are three primary approaches:

Validation: Check the fut output and confirm its correctness.
Intentional Ignoring: If output validation isnât relevant to the test, the output should still be acknowledged and cleared to avoid unintended failures.
Mixed Policy: A test can validate specific output while ignoring others, as long as any remaining output is cleared before the test returns.

2. When to Validate Output:

If the test expects specific output from the fut, it should retrieve and check the content on stdout and stderr using methods like io.get_out_content() or io.get_err_content(). The test passes if the actual output matches the expected content.
After validating, the test should clear the output buffers (io.clear_buffers()) if further output handling is not needed to avoid residual content.

3. When to Ignore Output:

If the test does not require output verification, it should acknowledge the output by clearing the streams before returning.
This approach signals to TestBench that any output generated was intentionally disregarded and avoids marking the test as failed.

4. Failure Due to Residual Output:

No Defined Policy: If a test leaves output on the streams without a clear handling policy (validation or intentional clearing), TestBench will flag this as a failure.
Ensuring Clean Tests: To avoid unexpected failures, verify that each test has no residual output before returning by either validating or clearing output streams.

- -

Example Scenarios

1. Output Validation:

public static Boolean test_with_output_verification(IO io) {
-  System.out.println("Expected output");
-  String output = io.get_out_content();
-  boolean isCorrect = output.equals("Expected output");
-  io.clear_buffers(); // Clear remaining content if not needed
-  return isCorrect;
-}

2. Ignoring Output:

public static Boolean test_without_output_verification(IO io) {
-  System.out.println("Output not needed for this test");
-  io.clear_buffers(); // Clear output since itâs intentionally ignored
-  return true;
-}

- -

Summary

Each test should manage its output streams with an intentional policy:

Validate output if it is relevant to the test.
Acknowledge and clear output if it is not relevant.
Avoid residual output to prevent TestBench from marking the test as failed.

This approach ensures that tests remain clean and focused on their primary objectives without unintended side effects from unhandled output.

Overview of the IO Object

Each test function is given an IO object, which provides + methods for inspecting stdout and stderr output + streams, programmatically adding data to the stdin input stream, + and clearing output streams as needed. Although the IO object is + optional, it is available for cases where I/O validation or cleanup is + essential to the test.

Purpose

Each test function is responsible for managing any output generated + on stdout or stderr by the function under test + (fut). TestBench will automatically clear the streams before + each test begins and will check them after the test completes, treating any + remaining output as unintended and marking the test as a failure. This policy + ensures that tests intentionally handle output by either validating, + clearing, or ignoring it, thereby maintaining a clean and predictable testing + environment.

Policy Guidelines

1. Define an Output Handling Policy:

Every test should have a defined policy for how it handles output generated by the fut. There are three primary approaches:

Validation: Check the fut output and confirm its correctness.
Intentional Ignoring: If output validation isnât relevant to the test, the output should still be acknowledged and cleared to avoid unintended failures.
Mixed Policy: A test can validate specific output while ignoring others, as long as any remaining output is cleared before the test returns.

+ +

2. When to Validate Output:

If the test expects specific output from the fut, it should retrieve and check the content on stdout and stderr using methods like io.get_out_content() or io.get_err_content(). The test passes if the actual output matches the expected content.
After validating, the test should clear the output buffers (io.clear_buffers()) if further output handling is not needed to avoid residual content.

+ +

3. When to Ignore Output:

If the test does not require output verification, it should acknowledge the output by clearing the streams before returning.
This approach signals to TestBench that any output generated was intentionally disregarded and avoids marking the test as failed.

+ +

4. Failure Due to Residual Output:

No Defined Policy: If a test leaves output on the streams without a clear handling policy (validation or intentional clearing), TestBench will flag this as a failure.
Ensuring Clean Tests: To avoid unexpected failures, verify that each test has no residual output before returning by either validating or clearing output streams.

Example Scenarios

1. Output Validation:

public static Boolean test_with_output_verification(IO io) {
+  System.out.println("Expected output");
+  String output = io.get_out_content();
+  boolean isCorrect = output.equals("Expected output");
+  io.clear_buffers(); // Clear remaining content if not needed
+  return isCorrect;
+}

+ +

2. Ignoring Output:

public static Boolean test_without_output_verification(IO io) {
+  System.out.println("Output not needed for this test");
+  io.clear_buffers(); // Clear output since itâs intentionally ignored
+  return true;
+}

Summary

Each test should manage its output streams with an intentional policy:

Validate output if it is relevant to the test.

Acknowledge and clear output if it is not relevant.

Avoid residual output to prevent TestBench from marking the test as failed.

This approach ensures that tests remain clean and focused on their primary objectives without unintended side effects from unhandled output.

Informed Spot Checking

Output Stream Policy for Tests

Overview of the IO Object

Purpose

Policy Guidelines

Example Scenarios

Summary

Output Stream Policy for Tests

Overview of the IO Object

Purpose

Policy Guidelines

Example Scenarios

Summary

Overview of the `IO` Object

Overview of the `IO` Object