CS 240 - Testing Assignment

Due Fri Nov 15 at 11:59pm

Overview

For this assignment you will be testing ANOTHER TEAM's project code. This will give you a chance to practice working with "legacy" systems (i.e., maintaining code you didn't write). This may seem like an annoying change of focus (and will draw time away from your own implementation efforts), but it will produce a better overall project and reflects what "real" programming is like--it's more common to fix bugs that other people make than to fix your own!

There will be three steps to this assignment:

A code walkthrough to understand the system you will be testing
Analyzing and debugging the code using a static analysis tool
Writing a small number of unit tests for the code

Most of this work (Steps 1 and 3 in particular) will be completed during lecture, though you will need to write up your results outside of class.

In the end, you will creating a short report summarizing and describing your findings from these activities. While you will be doing some of the works in groups in class, your final written report should be completed individually.

Objectives

Practice performing a code walkthrough
Practice using static analysis tools (linters) to debug programs
Practice writing and running unit tests

Details

1. Code Walkthrough

The first step of this assignment will be completing a code review (really a code walkthrough) of another team's implementation.

On Mon 11/11 during class, you will be getting together with an assigned member of another team. You will each give a verbal walkthrough of your codebase to this team member, explaining what each function/component/etc. does and how it works. In short, you will be introducing your partner to your code and making sure they understand the overall structure and flow of the implementation.
- Remember to ask lots of questions! Inquire as to how things work, why the system is designed the way it is, and what kinds of tests the system might support/require. Keep an eye out for any potential coding faults (e.g., methods that could be refactored, loops that end poorly, documentation that is lacking, etc.)--any issues that you find should be recorded for future improvement. Overall, your goal is to gain a basic understanding of how the system works so that you will be able to run tests on it!
- Be sure and take lots of written notes; you will need to use these later. Sketches of the program flow, notes about what does what, notes about what should be fixed or improved, etc. You will need to be able to detail the system on your own later!
Once you have completed the walkthrough, you will need to write up a short summary of your findings. This summary should include a brief overview of the code's structure (maybe 5-10 sentences explaining the system's architecture and implementation). You should also mention any problems or concerns that you identified during the walkthrough. What seemed odd/confusing/really-shouldn't-work-but-seems-to-somehow? The more "bugs" you can identify at this stage, the betetr!
- As with the rest of this assignment, your report will be given to the team whose system you are testing. Thus you should write your report as an informational document for them--give a summary of how you understand the system, and then write them a nice explanation of what you think works and what may need to be improved. You're basically writing a review of their software!
- You should write this up Monday evening after the walkthrough, while it is still fresh in your head!

2. Static Analysis

The next step of the assignment will be to use a static analysis tool (a linter) to analyze and debug the code you are testing--this will be your partner's code, not yours!

You should take their code and run it through one of the many existing JavaScript lint programs, such as JSLint, JSHint, or a similar program. Be sure and read the documentation of these tools for details about how they work!
- Note that while you can use the online tools, it may be easier (and more useful in the future) to install one of these programs directly into your IDE. See here for details.
- For groups whose code is not primarily in JavaScript (e.g., Sharing), you may need to find an alternate linting tool. For example, there are a number of PHP lint programs (including an online system).
After you have run the code through the static analysis tool, look at the results! You will need to analyze and consider what the errors mean. Are they reporting stylistic problems? Do they indicate actual bugs? You will need to synthesize the results into a shorter list of common problems and warnings that point to actual bugs.
For stylistic or simple syntactical errors, you should fix the code and then re-run the static analysis tool--your goal is to get the number of errors to be as small as possible!
You will need to write down the results of this process for your testing report. Do not just copy and paste the output from the linter. Instead, you should provide a summary of the errors you found and fixed. Again, this is a report you will give to the other team to explain the changes that you made.
- If you choose not to respond to particular lint warnings, be sure and explain why!
Note that you will need to do this on your own outside of class; I recommend doing it before class on Wed 11/13, so that these simple bugs are already caught and fixed.

3. Unit Testing

For the last part of this assignment, you will be writing a small number of unit tests for the code your are testing in order to check its function, and to provide the beginnings of a testing suite to the other team.

On Wed 11/13 you be getting together with other students who are testing the same module (they will not necessarily be from your team, but you will all be testing the same code). As a group you will need to come up with an extensive list of functional tests--tests that the program works as specified. You should think of these primarily as unit tests: what functions or interactions can you programmatically test and evaluate that they work?
- Be sure to consider what functionality you are testing, and how you will know if the system passes or fails the test!
Once you have a list of unit tests, you will need to divide these tests among the group so everyone has unique tests to work on (this will avoid duplicating effort). Note that each person will need at least two (2) functions to write unit tests for.
- If you have problems coming up with 10 different functions to test, let me know!
Finally, each person will need to modify the code and add in their assigned unit tests. I recommend using QUnit as a testing framework for JavaScript; this will let you easily write assert and test functions. However, you are also welcome to use another testing framework (let me know which!) or "roll your own" tests.
- You will need to both write and run the unit tests! Be sure to check if the program passes the tests or not!
- Note that you do not need to fix the program so that it passes all the unit tests--that is work for the development team!
In your testing report, write a brief (1-2 sentence) description of which pieces of functionality you tested. Also include a lists of tests you ran (i.e., what inputs/outputs did you check?), and whether the program passed or failed those tests. In the end, your goal is to state with confidence whether the program passes or fails at including the functionality you are testing.

The Report

You will need to compile the write-ups of the above 3 activities into a single report. This report should include the following:

A summary of the program's structure and details of any problems identified during the walkthrough
A summary of the results of your static analysis, as well as a description of changes you made or bugs you caught
A description of the functions you wrote unit tests for and whether or not the functions passed those tests.

This report need not be very long--each section may be only a page or two. (You should write this report in $L a T e X$ for practice!) Be sure and proofread your work--after all, you want to look like you did a professional testing job before you tell other people what mistakes they made!

When finished, the report should be submitted to the dropbox on Moodle. This is different from our normal process of uploading to github; I will grade the reports and then put organized versions of them on github for the team members to review.

However, you will need to push the changes you've made to the code out to github, so that the development team is able to utilize your unit tests (and include the fixes you implemented based on the linter!)

Remember to pull changes in to your local repository first so you can resolve any merge conflicts locally!

Submitting

As mentioned above, your report should be uploaded to the Moodle dropbox by midnight on Friday 11/15. You will also need to push out your code changes to the master github repository.

Grading

Your testing assignment will be graded on approximately the following criteria:

Includes a summary of the code walkthrough and a list of issues identified [25%]
Includes a summary of the results of your static analysis [25%]
Includes a description of the unit tests you wrote and the pass/fail status of the program [30%]
Your code with embedded unit tests has been uploaded to github [10%]
Document is polished and properly formatted, with clear and proofread writing [10%]