MIPSUnit::MSpec

MIPSUnit::MSpec is a unit testing framework for MIPS assembly code. In particular, users describe the desired behavior of the assembly code under test (i.e., student assignments) using an RSpec-like syntax. MSpec converts those specs (RSpec uses the term "specification" or spec instead of "test") into assembly code that detects and reports errors in the code under test.

Example Workflow

Suppose you have assigned the following to your assembly language class:

Begin by writing an RSpec-like spec file that describes the expected behavior of in_range:

Next, run in_range_spec.rb through MSpec and redirect the output to a file:

in_range_spec.asm contains assembly code that will execute in_range with the given parameters and report whether $v0 has the expected value. Use a MIPS simulator such as MARS or SPIM to run the tests.

Download these files and try it yourself:

• in_range_spec.rb
• in_range_correct.asm
• in_range_error.asm
• in_range_spec.asm (You may want to generate this file yourself.)

Spec Basics

We expect that the vast majority of test cases can be written by users who don't know any Ruby or RSpec by simply applying the patterns in Examples 1, 2 and 3. However, for those who prefer to thoroughly understand the code they write (as opposed to copying and modifying examples), this section explains the basics of MSpec syntax.

Symbols

In Ruby, tokens that begin with a colon (i.e, ":in_range", and ":v0") are Symbols. For purposes of using MSpec, you can think of a symbol as a unique name. MSpec uses symbols to specify, among other things, functions, registers, and labels. (In most cases, we could have chosen to implement MSpec using Strings instead of Symbols; however, we chose to follow the idiomatic Ruby convention of Symbols where possible.)

describe

Tests (i.e., "specs") are contained in describe blocks. In most cases, a file contains a describe block for each function tested. The parameter to the describe function is usually the name of the function written as a symbol (i.e., a colon followed by the name of the function to be tested). See Example 1.

it

Each spec is defined by an "it" block. The it block can contain any arbitrary Ruby code; but, it typically follows the pattern demonstrated in Example 1 (three lines: calls to set, call, and verify).

data

The data method creates and initializes a label in the assembly file's .data section. For example,

data(:my_array => [:word, 2, 4, 6, 8, 10])

creates this line in the assembly file's .data section

my_array__unique_id: .word 2 4 6 8 10

These labels can be defined either in the it block, or the describe block. Like variables in most programming languages, data labels are visible within their enclosing scope. (Notice the use of :array1 in two specs in Example 2.) The array containing the desired values may also contain MIPS data size predicates, such as :byte, :half, :word, :ascii, and :asciiz. Notice the use of Ruby symbols (as opposed to the MIPS "dot" syntax). (As with set, the "=>" notation is used because the parameter to data is a Ruby Hash.)

set

The set method sets the specified registers to the specified values. For example

set(:t0 => 3, :s1 => 4)

sets register $t0 to 3 and register $s1 to 4. (The "=>" notation is used because the parameter to set is a Ruby Hash.) Similarly,

set(:t3 => :array1)

sets register $t3 to the address corresponding to label array1. (See the data section and Example 2.)

call

The call method calls the function under test with the given parameters. call can take up to four parameters, which become the values of $a0 through $a3. (The name of the function under test is passed as a parameter to describe. See Example 1. See also call_by_name.)

verify

The verify method verifies the values of the specified registers. For example,

verify(:v0 => 3, :v1 => 4)

verifies that $v0 has the value 3 and $v1 has the value 4. (As with set, the "=>" notation is used because the parameter to verify is a Ruby Hash.)

verify_memory

The method verify_memory(observed, expected) verifies that the observed parameter refers to a memory location containing the expected sequence of values.

Limitations

We designed MSpec to minimize the restrictions placed on the those writing the assembly code (i.e., the students). To the extent possible, we avoided reserving registers, labels, or memory locations for MSpec use. As a result, the current version of MSpec has a few, minor limitations:

• The verify methods use $at. Therefore, you cannot use verify to directly verify the value of $at. Also, calling verify or verify_memory will modify the value of $at.
• MSpec reports only the first error. When MSpec detects a spec failure, it reports that failure and quits. (The process of reporting errors changes several registers, thereby making any other tests unreliable.)
• MSpec does not re-set the CPU state before each test. Consequently
  1. You cannot make assumptions about the initial value of any register.
  2. Changes made to data labels by one spec will be visible in other specs.

The "Advanced Techniques" section below contains workarounds for most of these limitations.

Advanced Specs

The great thing about spec files being Ruby code is that you can use all of the flexibility of Ruby to help make your spec files concise and easily maintainable. Here are a few of the techniques I find most useful:

Helper methods

Each it block of code runs as if it were an instance method of a class defined by the describe block. Consequently, you can add additional instance methods to the describe block that can be called by the it blocks. One common use of this feature when teaching an assembly language class is to write a helper method to calculate the expected answer. (See the nck_spec example below.)

Use a loop to generate multiple tests

In Java's JUnit, individual tests are statically defined methods. In contrast, MSpec tests are generated dynamically by executing the it method in a describe block. Thus, you can generate multiple tests inside a loop. This is particularly useful when writing exhaustive tests. (Although exhaustive tests aren't common in industry, they are useful for thoroughly testing student code.) Example 4 below uses a nested loop to test every possible input for "n choose k" for k ≤ 20.

before blocks

If you have some code you want run at the beginning of every spec in a describe block, you can put it in a before block. (See the in_range_s_unchanged example below.)

Instance variables

Each describe block corresponds to a class, and the code in it and before blocks behave like instance methods of that class. Thus, you can introduce and use instance variables. Example 5 below illustrates how before blocks and instance variables can be used together.

data vs. data!

The data method generates assembly code that allocates and initializes a single region of memory. (Specifically, it generates a single line in the .data section.) This is the desired behavior when specifying a string or array that will be used as a read-only input by several different specs. When the function under test does not modify any data memory passed as a parameter, it is important to put the data statements in the describe block and not in the before block. Placing them in the before block will needlessly (and wastefuly) create a separate label for each test. For this reason, calling data in a before block will generate a warning.

However, because MSpec does not re-set the processor state between tests, if a shared data label is passed as a parameter to a function that modifies the memory (e.g., a sort function), successive tests will not receive the expected input. (They will instead receive the result of the previous test.) In this case, each test needs its own copy of the input. One way to generate one copy per test is to call data! (notice the exclamation point) in the before block. Calling data! will suppress the warning that appears when calling data from withing a before block.

If the user wishes to share a data input among several, but not all, tests in a describe block, she can

• Nest describe blocks, placing the desired label definition in a before block in the inner describe.
• Use a before block to define instance variables, but actually call data only in the desired it blocks.

@output

@output is the variable that contains a handle to the output file. You can use it to add arbitrary assembly to your test. For example, if you really wanted to verify the value of $at at the end of a function, you could do this:

We don't use these features often; but, they are occasionally useful:

call_by_name

The call_by_name method allows you to call an arbitrary function. (In contrast, call calls the function specified by the describe block.

Nesting describe blocks

MSpec allows you to nest describe blocks the same way you can in RSpec. Nesting describe blocks allows you to logically organize specs into groups and provide a scope for helper methods and data labels. Note: Only one describe block in a particular chain may take a label as a parameter (otherwise, the function to be called by call would be ambiguous). The remaining describe blocks instead take Strings as parameters that are used to build the description of the spec that is printed in the event of a failure.

Other Restrictions

We took care to limit the number of rules users had to follow; but there are a few:

• Do not give helper methods the same name as methods in the TestFactory class. If you do, you will replace the MSpec code with your code.
• Do not give your instance variables names identical to TestFactory instance variables, including
  • @output
  • @description
  • @local_labels
• Do not attempt to verify the value of $at using the verify method.
• Do not call data from within a before block. If you need test to have a separate copy of a data label, use data!

FAQ

Why does MSpec report only the first failure?

MSpec generates an assembly file that is then executed by MARS or SPIM together with the code under test. To report all failures, MSpec would have to store the list of failures somewhere in the simulated memory space --- the same memory space to which the code under test has access. Thus,

• The list of failures would be susceptible to corruption by buggy code.
• The code that stores the list of failures would have to take great care not to disturb the state of the CPU (or at least restore any changes made). We simply haven't yet had time to write this code.

Why does MSpec generate an assembly file instead of interacting directly with MARS?

Instead of having MSpec generate an assembly file, we could have used JRuby to directly drive MARS. This approach would have made it easier to (1) reset the simulated CPU between tests, and (2) report the complete list of failures (as opposed to the first failure only). However, it would have required that all users install Ruby and JRuby. In contrast, having MSpec generate an assembly file, only the spec writer (i.e., the professor) must have Ruby installed. Those who actually run the tests (i.e., the students) need only have the MSpec-generated assembly file and MARS or SPIM. (While installing Ruby is simple in theory, I did not want to be responsible for helping 40 students a semester install Ruby on their Windows machines.)

Why the two-part name?

Two reasons: First MSpec is the name of another testing framework. Second, we are considering adding other frameworks to the MIPSUnit suite. For example, MIPSUnit::MUnit would be a JUnit-based testing framework that would interact directly with MARS, thereby avoiding the limitations of MSpec's assembly-based implementation.

Contact Info