Testing Lies

I was recently invited to be a guest speaker at the 2014 German Perl Workshop and I had a fantastic time. Not only was the workshop well-organized and with great food, but it was also my favorite Perl Workshop to date. Amongst other things, one of the organizers offered me his spare bedroom instead of a hotel room and it made for a very pleasant change (though the hotels are great, too, as there are usually groups of people I know who hang out at the hotels).

This time I gave a talk on a subject I have long wanted to speak about: the lies we tell about testing. Below is the video of my talk.


The video has a bit of a break partway through due to a medical emergency which occurred (everything turned out OK), but it certainly made it one of the more difficult situations to recover from in a talk. I’ve been speaking at conferences for years and nothing like this has happened before.

Regarding the video, you may notice a slide on all pairs testing where I refer to a hypothetical 10 argument subroutine as having 10 billion possible combinations of arguments. I didn’t explain that well enough because we’re also talking about taking multiple combinations of one argument, two arguments, and so on, up to ten arguments, and also testing each of the possible interesting values for each argument. I should have been more clear.

There’s much more I could have said in the talk, but as usual, there’s only so much time available. However, it boils down to this:

  • Don’t trust snake-oil salesmen who use words like “always” or “never”
  • You want to break your code with tests
  • Don’t blindly trust your tools
  • Code coverage of unit tests can be very misleading
  • Writing more tests does not guarantee better software
  • Claiming that tests are documentation is often a stretch
  • Testing needs to be focused more on business needs
  • TDD enthusiasts need to calm down
  • The Perl community isn’t stealing enough testing ideas from others
  • 100% code coverage is useless if customer’s don’t use your software (A/B testing, anyone?)

Hire us to fix your test suite (or write it), or to teach you how to effectively test Perl applications.

This entry was posted in General. Bookmark the permalink.

4 Responses to Testing Lies

  1. Nate Custer says:

    Thanks for the talk, there are some really helpful reminders here. About a year ago I moved from a perl testing job to php testing. If you are looking for a list of good tools to consider stealing from, check out:


    They do a soft BDD style testing, where they use an API that looks a lot like writing in english to write your tests in. Since its actually code, you can handle things in code when that complexity is required. You can see a simple example here:


    • Nat, that looks pretty interesting (though I have to admit that PHP 5.3.0′s decision to use the backslash as a namespace separator gives me the willies). Thanks for the link. I’ll have to look around and see if Perl has anything like that yet.

  2. Jarkko Hietaniemi says:

    “You want to break your code with tests”

    Also known as “negative testing”. Ride your implementation to hell and back. Abuse the API.
    - no arguments / one million arguments
    - undef, zero, -1, one million, NaN
    - if you have an exact limit N, try at N-1, N, and N+1
    - empty string, UTF-8 string, Latin-1 string, one megabyte string
    - nonexistent files, unreadable files, unwritable files, unwritable directories, directories when files expected, and vice versa

    • Jarkko: absolutely. While I understand that due to time constraints, it’s not always possible to do that for every bit of code, it should at least be done for the major features.

Comments are closed.