Question

I’m building a Ruby command-line program, and using Cucumber and Aruba to test it. Aruba includes some really handy matchers, so I can test output with a few lines in the .feature file:

When I run `myprogram`
Then it should pass with:
  """
  my program output
  """

The problem is that my program may contain dozens or even hundreds of lines of output; putting all that in the .feature file will make it harder to read and navigate (and is kind of obnoxious). What’s the recommended way to test the output in such a case?

Answer 1

The short answer is: you should not do that.

Cucumber tests are supposed to be user-facing and readable. They describe features. A user isn’t going to care if error output matches some known value byte for byte.

You need to ask yourself: What am I testing? Is the answer the error message? Probably not. You’re testing some functionality in your application. If all you really want to ensure it fails, then what you want in your Cucumber scenario is the following line:

Then the exit status should not be 0

This assumes the script follows the standard convention that a non-zero exit status signals an error.

If your scenario requires that there be a certain message in the output, you can add it:

Then it should fail with
"""
Some error message
"""

but this need not be the entire output, only a partial match. (Note that there’s a “it should fail with exactly:” defined in aruba, but I don’t recommend using it.)

Edit: You’ve changed your example to be testing for a pass rather than a fail, but my basic advice is the same:

1. Separate the specifics of the output from the logic of the scenario. Using the example in the comments, if you have a test that confirms you can output a single user-generated comment, and another test that confirms you have output the correct 100 comments, then it is sufficient, you don’t need to have 100 comments’ worth of output in the Cucumber scenario.
2. Keep the Cucumber scenarios written from the user’s perspective. Each scenario should test something that will be important to the user. Try to keep them minimal by removing anything that leaks in from the implementation when the user wouldn’t care.
3. Use the built-in Aruba constructs that test for partial matches to achieve this. Look for key words or phrases in the output. Not only will the Cucumber tests be easier to read, they’ll be more robust and immune to unrelated output changes.