06 Testing and Debugging

Testing and debugging

Foundation due 10/26 at 1:00 pm

Deep Dive due 11/09 at 1:00 pm

Last day for on-time checkoff: 11/10

About

In this lab, you will come up with test cases to test your lab 5 code, based on a reference state machine. You will run these test cases and use the tests to debug your code. You will then be asked to reflect on the assumptions we made in testing. By the end of this lab, you will have seen how SM-driven design influences unit testing for an embedded project. You will also have gained experience with property-based testing.

Lab 6 Rubric

Resources

Materials

Same circuit as for Lab 5.

Steps

Make sure that you still have your circuit and code from Lab 5, and follow these steps to set your code up for testing:
1. In snake.h, add the following line to the top of the file:
```
 #define TESTING
```
2. In lab5.ino, add the following lines to the end of your setup() function:
```
 testAll();
 while(true);
```
3. Compile your code (Sketch -> Verify/Compile or the checkmark button next to the upload button). Your Arduino does not need to be plugged in to complete this step. If there are compilation errors, double-check the steps above.
Run the test cases you created in the foundation:
1. Read the testTransition function in snakeTests.ino and discuss how it works with your partner.
2. Work with your partner to combine your test case tables. For most transitions and non-transitions, choose one partner’s test case to keep. For four transitions of your choice, include both tests, such that you have 22 tests in total. To create more rows in the testing table, highlight the two sheet rows that make up one transition entry. Copy the rows. Right-click on the next blank row and select “Paste special -> Paste format only”) Make sure that the tests are numbered from 0 to 21, in order, in column A.
3. Follow the instructions at the end of the foundation to turn your test cases into lines of Arduino code. Remember that your spreadsheet shouldn’t have any invalid input errors nor blank cells in order for this code to work. Paste the generated lines from cell B1 into snakeTests.ino under the REPLACE THE FOLLOWING 4 LINES WITH YOUR TEST CASES comment. You will probably have to remove some quotation marks that Google Sheets automatically adds on to copied cells.
4. Connect your Arduino, open the Serial Monitor, and upload and run the code. If you get to “All tests passed!”, congratulations. You’ll get checked off in the next step, but be prepared for your TA to ask you conceptual questions about your test cases. Otherwise, debug your code, based on which transition fails. You may find that there is a flaw in your implementation, or you may find that your test case made an incorrect assumption. In the second case, you should correct your assumptions (asking a TA if need be), modify the test case, and re-parse your test cases as in step 4 above.
All because your state machine matches the SM specification doesn’t mean that the entire code works. In particular, we also want to test the helper functions. Because the lab is time-limited, we’ll explore one way of testing one of the helper functions you wrote in lab 5 (turn).
1. One way to test the turn function is to exhaustively test all 25 input combinations and write an assertion for each one. While this would achieve full path coverage through the code, it would be pretty tedious to express. On the other hand, we could write a few test cases and call it a day, but this might miss bugs. We’ll walk a middle ground by implementing lightweight property-based testing. We’ll loop through all 25 input combinations, but instead of manually typing in the expected outcome of each call to turn, we’ll express three properties that we want to hold. Paste the following code into snakeTests.ino somewhere after the o2s function:
```
void turnAssert(orientation in1, orientation in2, bool expected) {
if (turn(in1, in2) != expected) {
   char sToPrint[200];
   sprintf(sToPrint, "TEST FAILED: turn(%s, %s) returned %s (expected %s)", o2str(in1), o2str(in2), expected ? "F" : "T", expected ? "T" : "F");
   Serial.println(sToPrint);
   while(true);
}
}
/*
* Test all possible combinations of the turn function against the following properties:
* - turn(a, a) should return false
* - turn should return false when either input is NONE
* - turn(a, b) should return the same value as turn(b, a)
*/
void testTurns() {
const orientation orientations[] = {UP, RIGHT, DOWN, LEFT, NONE};
for (orientation o1 : orientations) {
   for (orientation o2 : orientations) {
      if (o1 == o2) { // turn(a,a) should return false
      turnAssert(o1, o2, false);
      } // test more properties below
   }
}
Serial.println("All turn property tests passed!");
}
```
2. Study how the property *turn(a, a) should return false is implemented. Write similar tests for the other two properties.
3. With your partner, discuss what other property/ies would be useful for testing this function (you don’t have to implement them). Similarly, what property could you use to test newPos? (Hint: mathematically, what would we expect to be different between the inputs of newPos and the outputs?)
4. In between the testAll()/while(true) lines of lab5.ino, call testTurns() and verify that the tests still pass, and debug them if they don’t. Once all tests are passing, get checked off by a TA.
Comment out the #define TESTING line in snake.h; and remove the testAll()/while(true) lines in lab5.ino. If you’re using a different code/circuit combination, you might have to re-calibrate your controller, as in lab 5. Run your code and verify that the game plays as intended. If not, discuss what might have gone wrong and work on debugging your test cases and code (note that testTransition does not test that you called the outputs correctly, so even if your tests are correct and all pass, you should double-check that the outputs are being set correctly in each transition). Once the game is working, get it checked off by a TA.

Cell M12 should be 95, not 65!
Turn in your work:
1. Turn in the code (entire Arduino project zipped up) on the Lab 6: Code Gradescope assignment
2. Fill out the lab 6 exit form
3. INDIVIDUALLY, complete the Lab 6 Deep Dive assignment on Gradescope.