Output Stream Policy for Tests

+ +

Overview of the `IO` Object

+ +

Each test function is given an IO object, which provides + methods for inspecting stdout and stderr output + streams, programmatically adding data to the stdin input stream, + and clearing output streams as needed. Although the IO object is + optional, it is available for cases where I/O validation or cleanup is + essential to the test.

+ +

Purpose

+ +

Each test function is responsible for managing any output generated + on stdout or stderr by the function under test + (fut). TestBench will automatically clear the streams before + each test begins and will check them after the test completes, treating any + remaining output as unintended and marking the test as a failure. This policy + ensures that tests intentionally handle output by either validating, + clearing, or ignoring it, thereby maintaining a clean and predictable testing + environment.

+ +

Policy Guidelines

1. Define an Output Handling Policy:

Every test should have a defined policy for how it handles output generated by the fut. There are three primary approaches:

Validation: Check the fut output and confirm its correctness.
Intentional Ignoring: If output validation isnât relevant to the test, the output should still be acknowledged and cleared to avoid unintended failures.
Mixed Policy: A test can validate specific output while ignoring others, as long as any remaining output is cleared before the test returns.

2. When to Validate Output:

If the test expects specific output from the fut, it should retrieve and check the content on stdout and stderr using methods like io.get_out_content() or io.get_err_content(). The test passes if the actual output matches the expected content.
After validating, the test should clear the output buffers (io.clear_buffers()) if further output handling is not needed to avoid residual content.

3. When to Ignore Output:

If the test does not require output verification, it should acknowledge the output by clearing the streams before returning.
This approach signals to TestBench that any output generated was intentionally disregarded and avoids marking the test as failed.

4. Failure Due to Residual Output:

No Defined Policy: If a test leaves output on the streams without a clear handling policy (validation or intentional clearing), TestBench will flag this as a failure.
Ensuring Clean Tests: To avoid unexpected failures, verify that each test has no residual output before returning by either validating or clearing output streams.

+ +

Example Scenarios

1. Output Validation:

public static Boolean test_with_output_verification(IO io) {
+  System.out.println("Expected output");
+  String output = io.get_out_content();
+  boolean isCorrect = output.equals("Expected output");
+  io.clear_buffers(); // Clear remaining content if not needed
+  return isCorrect;
+}

2. Ignoring Output:

public static Boolean test_without_output_verification(IO io) {
+  System.out.println("Output not needed for this test");
+  io.clear_buffers(); // Clear output since itâs intentionally ignored
+  return true;
+}

+ +

Summary

Each test should manage its output streams with an intentional policy:

Validate output if it is relevant to the test.
Acknowledge and clear output if it is not relevant.
Avoid residual output to prevent TestBench from marking the test as failed.

This approach ensures that tests remain clean and focused on their primary objectives without unintended side effects from unhandled output.

White Box Testing

+ +

Introduction

+ +

Testing centers around three key components: the test + bench, the test functions (or tests), and + the functions under test. In most cases, the + developer provides the functions under test. When this tool is used, Mosaic + supplies the test bench. This leaves the tester with the role of creating and + running the tests. Often, of course, the tester role and the developer role are + performed by the same person, though these roles are distinct.

+ +

The term function refers to any program or + circuit where outputs are determined solely by inputs, without internal + state being kept, and without side effects. All inputs and outputs are + explicitly defined. By definition, a function returns a single result, but + this is not a very strong constraint because said single result can be a + collection, such as a vector or set.

+ +

We need this precise definition for a function to make meaningful + statements in this document, but the Mosaic TestBench can be used with + tests designed to evaluate any type of subroutine. A later chapter will + cover testing stateful subroutines, provided that I get around to writing it.

+ +

There is also a nuanced distinction between function + in singular and plural forms, because a collection of functions can be viewed as + a single larger function with perhaps more inputs and outputs. Hence, when a test + is said to work on a function, we cannot conclude that it is a single function + defined in the code.

+ +

A test must have access to the function under test so that it can supply + inputs and harvest outputs from it. A test must also have a + failure detection function that, when given + copies of the inputs and outputs, will return a result indicating if a + test failed or not. Ideally, the failure detection function is accurate, + or even perfect, as this reduces missed failures and minimizes the need + to verify cases that it has flagged as failures.

+ +

The testerâs goal is to identify failures, + observable differences between actual outputs and expected outputs. Once a + failure is identified, a developer can investigate the issue, locate + the fault, and implement corrections as + necessary. While Mosaic aids in failure detection, it does not directly + assist with debugging.

+ +

Unstructured Testing

+ +

Unstructured testing is at the base of all testing strategies. The following are some + examples of approaches to unstructured testing. The Mosaic TestBench is agnostic + to the approach used for unstructured testing, rather this section is about writing + the test code that the TestBench will call.

+ +

Reference Value based testing

+ +

In reference value-based testing, an ordering + is assigned to the inputs for + the function under test, as well as to + its outputs. With this ordering, the function + under test can be said to receive an input + vector and to return an actual output vector.

+ +

In this testing approach, a Reference Model is also used. + When given an input vector, the Reference Model will produce + a corresponding reference output vector that follows the + same component ordering as the actual output vector from the + function under test.

+ +

The failure detection function then compares each + actual output vector with its respective reference output vector. If they do + not match, the test is deemed to have failed.

+ +

The Reference Model is sometimes referred to as the golden + model, and said to produce golden values. However, this + terminology is often an exaggeration, as testing frequently reveals inaccuracies + in reference values.

+ +

Thus, in reference value-based testing, the failure detection function + relies on a comparison between the actual and reference output vectors. Its accuracy + depends directly on the accuracy of the Reference Model.

+ +

Property Check Testing

+ +

Property check testing is an alternative to + reference value-based testing. Here, rather than comparing the actual + outputs to reference outputs, the actual output is validated against + known properties or expected characteristics.

+ +

For example, given an integer as input, a function that squares this + input should yield an even result for even inputs and an odd result for odd + inputs. If the output satisfies the expected property, the test passes; + otherwise, it fails. This approach allows testing of general behaviors + without specific reference values.

+ +

Spot Checking

+ +

With spot checking, the function under test is checked against one or + two input vectors.

+ +

Moving from zero to one, i.e. trying a program for the first time, + can have a particularly high threshold of difficulty. A tremendous + around is learned during development if even one tests passes for + a function.

+ +

Sometimes there are notorious edge cases. Zeros and one off the + end of arrays come to mind. Checking a middle value and the edge + cases is often an effective test.

+ +

It takes two points to determine a line. In Fourier Analysis, + it takes two samples per period of the highest frequency component + to determine an entire wave form. There is only so much a piece of + code can do different if it works at the edge cases and in between. + It is because of this effect that ad hoc testing has produced so + much working code. +

+ +

Spot checking is particularly useful during development. It is the + highest leverage testing return for low investment. High investment is + not approrpiate for code in development that is not stable, and is open to + being refactored. +

+ +

1. Running the Tests

To run all tests and gather results, follow these steps:

Ensure the environment is clean by running clean_build_directories.

Make sure no old files are hanging about by running clean_build_directories.

Run make to compile the project and prepare all test class shell wrappers.

Run run_tests to run the tests. Each test class will output its results, identifying tests that failed.

diff --git a/tester/javac/Test_MockClass.java b/tester/javac/Test_MockClass.java new file mode 100644 index 0000000..64bc962 --- /dev/null +++ b/tester/javac/Test_MockClass.java @@ -0,0 +1,98 @@ +/* -------------------------------------------------------------------------------- + Integration tests directly simulate the use cases for TestBench. + Each test method validates a specific feature of TestBench ,including pass, + fail ,error handling ,and I/O interactions. +*/ + +import java.util.Scanner; +import com.ReasoningTechnology.Mosaic.IO; +import com.ReasoningTechnology.Mosaic.TestBench; + +public class Test_MockClass{ + + public class TestSuite{ + + public TestSuite() { + // no special initialization of data for this test + } + + public Boolean test_failure_0(IO io){ + return false; + } + + // returns a non-Boolean + public Object test_failure_1(IO io){ + return 1; + } + + // has an uncaught error + public Boolean test_failure_2(IO io) throws Exception { + throw new Exception("Intentional exception for testing error handling"); + } + + // extraneous characters on stdout + public Boolean test_failure_3(IO io) throws Exception { + System.out.println("Intentional extraneous chars to stdout for testing"); + return true; + } + + // extraneous characters on stderr + public Boolean test_failure_4(IO io) throws Exception { + System.err.println("Intentional extraneous chars to stderr for testing."); + return true; + } + + public Boolean test_success_0(IO io){ + return true; + } + + // pushing input for testing + + public Boolean test_success_1(IO io){ + io.push_input("input for the fut"); + + Scanner scanner = new Scanner(System.in); + String result = scanner.nextLine(); + scanner.close(); + + Boolean flag = result.equals("input for the fut"); + return flag; + } + + // checking fut stdout + public Boolean test_success_2(IO io){ + System.out.println("fut stdout"); // suppose the fut does this: + String peek_at_futs_output = io.get_out_content(); + Boolean flag0 = io.has_out_content(); + Boolean flag1 = peek_at_futs_output.equals("fut stdout\n"); + io.clear_buffers(); // otherwise extraneous chars will cause an fail + return flag0 && flag1; + } + + // checking fut stderr + public Boolean test_success_3(IO io){ + System.err.print("fut stderr"); // suppose the fut does this: + String peek_at_futs_output = io.get_err_content(); + Boolean flag0 = io.has_err_content(); + Boolean flag1 = peek_at_futs_output.equals("fut stderr"); + io.clear_buffers(); // otherwise extraneous chars will cause an fail + return flag0 && flag1; + } + + } + + public static void main(String[] args) { + Test_MockClass outer = new Test_MockClass(); + TestSuite suite = outer.new TestSuite(); // Non-static instantiation + + /* for debug + IO io = new IO(); + io.redirect(); + suite.test_success_2(io); + */ + + int result = TestBench.run(suite); // Pass the suite instance to TestBench + System.exit(result); + } + +} diff --git a/tester/javac/Test_TestBench.java b/tester/javac/Test_TestBench.java index 848fae3..a4ae469 100644 --- a/tester/javac/Test_TestBench.java +++ b/tester/javac/Test_TestBench.java @@ -23,6 +23,7 @@ public class Test_TestBench { // Tests if a method with an invalid return type is identified as malformed by TestBench public static Boolean test_method_is_wellformed_1(IO io) { + System.out.println("Expected output: Structural problem message for dummy_invalid_return_method."); try { Method invalidReturnMethod = Test_TestBench.class.getMethod("dummy_invalid_return_method", IO.class); return Boolean.FALSE.equals(TestBench.method_is_wellformed(invalidReturnMethod)); @@ -64,9 +65,9 @@ public class Test_TestBench { if (test_run_test_0(io)) passed_tests++; else { System.out.println("test_run_test_0"); failed_tests++; } // Summary for all the tests - System.out.println("Total tests run: " + (passed_tests + failed_tests)); - System.out.println("Total tests passed: " + passed_tests); - System.out.println("Total tests failed: " + failed_tests); + System.out.println("Test_TestBench Total tests run: " + (passed_tests + failed_tests)); + System.out.println("Test_TestBench Total tests passed: " + passed_tests); + System.out.println("Test_TestBench Total tests failed: " + failed_tests); return (failed_tests > 0) ? 1 : 0; } diff --git a/tester/jvm/Test_Mosaic.jar b/tester/jvm/Test_Mosaic.jar index ff370e33eb262f42d9e689e10cd291ab9f6cd2bc..73b55655feba138c3172d6497c1feedc45646574 100644 GIT binary patch delta 3877 zcmai1XH-+$)(uUA^qU}EK{^C!0wE$ruk;c?I-v+i2c;L`NEK<)iy%!5(t?11fzUf5 zB}nhR7Zs$#2flB-d*6LO-r7IrT6^rV&e&({G1r{keEWAPkSavPP!Jdl2Dx#W$5J4O zE^b!6})``lAfr}jj(BI<5MgGXj)y-7R*~0#TgSo3K$7QMQZtG@s+v2h= zP8fBCC{Vww@1IQ!33g=YrXhE62#lntY;2?_AYcMtxk(K9qa(X$nBCjF*8bO1)_uR1 z3u1tr38aAf(Ff(Jp9-4!2A%i+K))!!Z2%j)yU$P4wlWU)XH5Yji<7ag+dZ6}F4CR6 z==g4-LgD0@`*HA+3YXYJ;^a77kCm7=dt(|Yh6CGW#OTaFa93%}OV97x+E0@>rCnmO zu6AT*E3sD8L)x4KGo^l2D=b=iSPT|rB^Oa|>EqWN_-j=K|C3BK`k5ade@{@*29UHH zLj%TrsrDut^{_BCM1)qKxZ{|MxV}lu7N=ihTwwPni18vvCvsL?}0%^-t4_lii;2A7`+*_{2jkvb=W3A*ynAwJRbB};=eCRp-+6z|i zBQ%@d^MTUZrNN-Dq5Wb(bo8OSQi%cb&0(@X$x&91`V45O571(FOoa=4s_|qnQ`(i- zI`3(*O)V2@enX%9$_ieqs8UKWLN6C*(sz1ac&9cHa=;%T7{=fB)Sa0I$X)iE{4D*h zBv>uW+#de*V^H7%4)Qrv%A!Di)nOy>R5_YenE&2+(flDJKd+CM@Iyfft*x@qgxT=V+GVUK^ zoK(1X%G%<#d<>>Lea>qu6qVNV4!iKz(AgPL>fH4q$S*R4s;?DA4M%xv!|(yT zmAmK~;zpr!xW=I>@3~Yf-SgBr{=|%uL-`ok6@MMeaOc4>&*y?8J00n&--BGvV>54H zRfg5Rmm^vnE0*ahtk>f)AF^t(%S@_?;zkgYX;ffT6jJU^-V~i9pi@OCsRwmi#9)B< zbsN4`&cS08Hs5Et;GW<}x-V{1C*3EpO3L&$Sx)rY(pOqwfT4QkO{L6sjh1OoNrk~X zgRJ5Ysa9k;cz2R}8h+fworw)be8KLzy>Y0UuF3}X!c|K{DC7~O61Q5Yp#PC~xPEFd znki{sNiG6Mvyuc1Cxow&4FWoSx^Z>Y-6YMVcXj(JbA6+dzGC<6W5r?lH6bBc`Ux0r zMS(|2w+s0~#cKJ&UMdw_iNlOTKlI$Es_7uisYOkCvcAenmL{$VAiecxg8Oy@s%6Ue z&okm%IH*rSqPXFz8C9RocaC;YFc8i|Og#iS5y@_hyH%K^S21;A$7ELK@!0Fyvyehud>H)^e;A2IblB>>_rOhl89uax*~N+>KM#p*MYSh*I~0@^fBWmO^tU z2q)#j6%k-ngz09V$0FtC{z?i1EIyS5Hh|SAcBt0aRdg^y5mXHjM&g^;5cKaNnjMzG zHJvZ}75FC%ca6n-t{TC+;WX79PlKg2Js^ub?M)UpN^EOXI4mOz=NxXwGxH3Hi6WnT z*)V&u{stW&GX5)Z^o+@I{PFCER)6lwp-=sYQf@%;Y1|!%WlC=Dwew;d1U#kj=OZPO9icwe7dc83le@=3dFs*`*0+wvg0 zvk3*bLg3HbRIX<7$!>h}#JPVkk98BGzH9Vp`eWxk>}V_WH0&INRJjT!mRvFsBLRVM zWK^JwjlXeqd^Y$x@Neq<7vZ|cm})>Ysat^oC!^}3lfp;DG{_bK*7f?9my9umB&*T9b?LxA{s|fY%--rBAk=cz`%eIj?3M{4d z({1t^r_zyN*~<4CpCn5V8jO~bSg9$^TwKnI-8eQ(v9Ug?dE0Ruw=Pa%v2v7+tdjXe zHQ|hbe4cGs9}$w1siZ$5mmT-LS62N*zS0UtUg!^(cKtOqeb`eac(h&BdrpLF&Jn5ix>lWP?#&D1isroOWPY!ZoQe5nmSNKyatNZO>yq=Bv zWeG6?k~+jheAjSeQ}2yFmYxhWP);1%JkIyZQ}7E z7U=YlZ7-?lPu^RqR#TT%z+Y)0xwqGpKF{!^Lg=n3v*4t}P-LLh$X-+Kynffm zh^7FsW)XW=wvR&JX6Fm6RCG@7Tk>Da3GSzS$)t|eLA~l8peig_O;ZOkh_cLX4x4e{ zG=D{{t)=$^4rCvDMFqh`XqK|t_a{mA&d;{UEGCjrq!P;^XNHCS`Xz2=q|N;XOxbgx z)Z;_(C7d%k)SgBjDdPrm&Wss(cC+^~SR4Dx%nk1iwG39dyQqCfZl` z#Jy+4hW?OxhjIB5oi@&`w0y^yg~~MVA<8=HD#E+X{YDwam2CuNX$*(2!t0i6Y9N0x^4D`WA8dJ7-e8(&uMg(Exa(~xq;DZC9{IJSh7aELX}hEk>T35{I~rllH~BRV?DGyBBNbQILs8`k zo>JFfTYht223L=gtZ?|IU9(R2#DqAhgX@ktR*gu0rZ*I5q2)hsiyd3nO!Olb^ z!~}798Eof-kD0If8vcd^F~yH6MacUh@})OY7Us^&*6y@N9F+Mfo} za>W&$;|{HPZxpe03(Lc0rEnZ7ge!TOFn02od8C(lH2ieeC42xZ8J9lmK<^jm3VrdD zrZv2c0N;4v069`g>l@8m8I=%U`Px?ICrEvFBuby}`Q-Idxwr;u-e{NLDu8~T-dn!1 zlO;QOJ7nPp%RRF~ez|hGX1hp&>U`u_As~w2l5yh~;hp zMWv1}t?(>fymUV%Mqh-+O8Xu~7 zwceU0*rw$b5fDu`51Ed^0jH4xSkW}*v6-D|MH%@zv_!||%$3^L4|^PbEJdKV9nzw4 zHjLq_8vQRrWqf;t9D+l$mzgKWsAQIlpURwgerg%!m^*(_xtrGkdiYp1X2#D=`a)5t zxo_ifylIlec3CsRnJ?u z;Y}};%76B{ zP&I-VKJuR}{(i^*Y+*(n{}FsW{t!y8aA{YMzip3@T>OktG7#wU%72CJ_5c5tj+?E+ Yzr+(#h2+ZR0n&>lelar(nJ%mU10;tX!~g&Q delta 1606 zcmY+EYdF&j9LFbiY{`)2EZa`fPDL&&F_L7t94T@ciMdB)Xu~4>t0CPKdt8o;L=+um z?g|mPZ8e8txfEmOnlNT=Cth`aZ$97W`99D0_wKV~$WoU*XC)~G2SFeZkXNL8lI$_b zO`Pu7L@7dovu!I(N|s7(20BUp06=s?_Uww!)`XfTAMl91qRd0|BV0sQiHxJS&Ct>} z(aN5#rhk!egx|(N-sUo}2WhdfV7(O=8?&E>;?|Jhu=qbEYU176=s1{h01J7&7| ze8ysbYqE#yYd>b_IPvBqF-u#AWoJ!M??COjG#`~ABi^{>uf@M^Xke-how+n0%w*Y> zh|UZCphV;9R5A0p^Y{SOXKFc>BjO43+`7k&27V3p_+q#NI(QIDTpKu&KYS6l6PC(% zfIBT5DPEl*+am6jKP?7^M;|+0$!#|pjks%a-l@DDJj9NAuG}qC+J1Z6S7938A&e{c z#>2lBi#APk(LBS3TUb#Ux&_}1*6loxLynJ%%!b}(@T$xGB{fj~hKhw|HUu8N%L#4@ zZRvycL)!08kh_c`>I)scah!)xoV*)Xdf!kSdCI{lpYbUb1Q?UqPH{0kxIjNbKV+>S z^V-=ny|agWDJFs<-3g)aG%^+2TUIWwo^zvjE~`K7`aJ$fpeVn>{$}ljX;R_e$CFgP zlnJT^E7S;7S9kr)`mX&h1~4Mf_V{ULh}urEVO3mN&G+~0#kN$M`FTQ*>p!*|ANcw& zPm$scv=*j90cdSsm135nS>unf!LIk74Nn5>PdRIR(I^Szk$Zf>Hi$@Tnci+&y%RUs zYIAx;QCo(u-f6gXEdYt~!MuN(zlE$d=ZZy}XU3L7{FD{%2S+%|u(xnPxsLno!0>t=`|^=uoRp?%c${KsbwUMwVw;5BuDCETM)w_OFG74yC4^MfI79H z!`e#^NykJlVB7uQmEYCL)a~4t$$R5TI_RhYd)TS%C5>Oy!fQ{Zaa5vF7ZcKU^Lt_V zmos!uvnJlX0>YTg<4`d{M)_l`@w1UjI}O{O6PwHhI*!EaQOJ^anos9QO&_K)6?wOz zX%z{zaR@J>8a>2s;3IPO9l)0Sr)H87-#tC2{Ib%+4VN@UVpx&nnc%&e3HR2W1MmrE z$(UIJGaa>(SnTq^KtDR1UMEZb`Y<9P%If1jG3AyX;_5+iIQ`vgqIOVq46{#Y*Ck9? z-$&bHJOl~EKCaKgib9P=vAMp+BDojcm3heqWTlo6P6 zptP;SoAwCR_qnp=Sr`qjlUL2nr^IJdnlaPEB>K@dg5nYkkYGaOL`b=a+wKy#{vyi^ zRX78r&kxq6&!McOAz6&Zrf@I_gxmoF{ZHVf;YudpEmPMZ^|vTQQnwh4(%a%Ql)l1G zlRN)90jK{!DbZ8VNR`cIrTcp?_~$rR1_Y7={d#9$bPjqqOd2E&dL{t^B`R)~{{f(7 B2KE2| diff --git a/tester/shell/Test_MockClass b/tester/shell/Test_MockClass new file mode 100755 index 0000000..2e4f2a7 --- /dev/null +++ b/tester/shell/Test_MockClass @@ -0,0 +1,2 @@ +#!/bin/env bash +java Test_MockClass diff --git a/tester/tool/shell_wrapper_list b/tester/tool/shell_wrapper_list index 3b46b8d..99bf5de 100755 --- a/tester/tool/shell_wrapper_list +++ b/tester/tool/shell_wrapper_list @@ -9,5 +9,5 @@ if [ "$ENV" != "$env_must_be" ]; then fi # space separated list of shell interface wrappers -echo Test0 Test_Util Test_IO Test_TestBench +echo Test0 Test_Util Test_IO Test_TestBench Test_MockClass -- 2.20.1