CSCI 240 - Testing Assignment

Final Compilation due Tues Nov 20 at the start of class

The Assignment

For this assignment, you will be developing and implementing a strategy for testing ANOTHER TEAM's system. This will give you a chance to practice working with a "legacy" system, applying testing methodologies, and broaden your experience of the technologies involved in Vi-Char.

This assignment is divided into 4 parts, some of which you will complete individually and some of which you will complete with your Testing Group (which is different from your normal team). Each part will be done in order, building on the previous work.

Part 1: Exploring The Code Base (due Fri 11/02)

The first part of the assignment involves working with legacy systems. You'll be given access to another team's code and documentation. Your goal is to try to understand their system using only the documentation they provide. For this part of the assignment, you can ONLY use the documentation they provide--asking a member of the developing team for help or an explanation will be considered cheating.

During this part of the assignment, you will perform the following activities:

Install and run the system you are testing, following the documented instructions
Consider a list of features (functionality) that may need to be tested. Consider features from a "black-box" point of view (e.g., how can we verify the system). Note that this list of features will act as a draft of your "testing plan."
Evaluate the system you are testing. How well does it conform to the existing documentation? How easy is it to understand (with allowances for the complexities of the technology; certain systems rely on complex libraries--are you still able to understand what this system is doing)? How well did the development team follow their own coding and design guidelines?

You'll start this process on your own, write a copy of a short document (2-3 pages) describing your work. Upload this document to Moodle by normal class time on Mon 11/05. This document should answer the following questions, each in a paragraph or so:

Were you able to install and run the system? Why or why not?
Does the system conform to the specificatin? In what ways does or doesn't it match the documentation?
Is the implementation (code) easy to understand? Why or why not?
What might you have changed about how the system was organized/
List three (3) features or functions of this system that should be tested for defects and robustness.

You will be handing in a copy of this write-up as well (also on Fri 11/02); it will be graded as a Quiz.

On the Friday this is due, you'll be meeting with other reviewers of the same system. This will be your Testing Group. You should compare notes about how things work and what functionality needs to be tested, begining to compile a group Testing Plan.

Part 2: Code Review

On Tues 11/06, you'll be doing a pair-based informal code review. You'll be getting together with a developer of the system you're testing. The developer will give you a verbal walk-through of the code, explaining what each method/component/etc. does and how it works.

Remember to ask lots of questions! Inquire as to how things work, why the system is designed the way it is, and what kinds of tests the system might support/require. Keep an eye out for any potential coding faults (e.g., methods that could be refactors, loops that end poorly, etc.) But overall, your goal is to improve your understanding of how the system you are testing works so that you can better perform tests on it!
Also take lots of notes--you'll want to include elements of this code review in your final document (see Part 4).

From this point on, it is acceptable to discuss and coordinate with the developers of your testing system about how things work and any other work you may need to do.

Part 3: Perform Testing

In your Testing Group, you will peform tests--both functional and unit tests--on the system (which you should now have a sufficient understanding of). Remember that everyone needs to take part in the testing process--if possible, have each person take the lead on a different test.

Your primary concern is functional tests--executing your test plan to see if there are any failure points. What functinality are you testing? How will you know if the system passes or not?
You should attempt to run Unit Tests on your testing system. Luckily, most programming languages and packages have an xUnit-style system that you should be able to use.
- Mobile: Android also has extensive testing capabilities (such as JUnit) built in to the sdk.
- AR: Vuforia may also be able to use the Android testing framework
- Motion: the Microsoft SDK may have Unit Tests built in, or you may consider something like NUnit
- Game: There are also a couple of different test frameworks that work with Unity, SharpUnit was recommended as one good option.
- Web: Node.js has a number of unit test frameworks; a quick search pulls up node-unit as a common option.
I acknowledge that this may be incredibly difficult for certain systems (because we're using an eclectic mix of hardware and software); if you can't get unit tests implemented, make sure you can clearly document either why an existing testing framework won't suffice.
You will have some time in class (Mondays & Fridays primarily) to work on this testing and coordinate with other teams as needed, but most of the work will need be done outside of class.

Part 4: Write up Testing Results (due Tues Nov 20)

As a group, you will compile your analysis, reviews, test results, etc. into a single testing report to give back to the developing team. This document should have a format similar to the following¹:

Cover Page: Include a cover page with the name of the system you are testing and the names of all the Testing Team members.
Introduction: As with previous documents, include a few sentences about the project so we know what the document is concerning. Also make sure to identify what software system is to be tested, as well as the risks and impacts introduced by this software. This should be about a page.
Features to be Tested: List what features and functionality you tested.
- Give a name to each feature that you are testing (for easy identification purposes)
- Describe each feature in non-technical language that lay users could understand.
- Detail the Pass/Fail Criteria for each feature to be tested. How will you know if a feature works or not? At the unit level, this may be "all test cases completed," while at a higher level this might be "A specified number of attempts completed without errors and a percentage with minor defects."
- Detail the Testing Methodology: e.g., how are you going to practically apply the pass/fail criteria (e.g., "have one person hit the same button 10 times to see if it worked").
Test Results: Include a table of whether each feature passed or failed testing. You can also include any notes about tests may have failed (e.g., "caused system to crash") to help the developers fix the fault in the future. Note that the system does not need to pass all the tests at this point.
References: Include annotated links to any tools or frameworks that you used to perform your testing.

This final document will be due on Tues Nov 20 (before Thanksgiving break); in class, your group will share back the results of your testing.

¹ Although your document will not exactly follow the standard, you may be interested in the IEEE Standard 829 for Software Test Documentation.

Submitting

Part 1 is due Mon 11/05 at the start of class; upload a pdf copy to the appropriate Moodle dropbox.

Part 4 is due Tues 11/20 at the start of class; upload a pdf copy to the appropriate Moodle dropbox.