exupero's blog
RSSApps

Exploratory testing on pull requests

When I create a pull request, I try to include in the description steps for manually testing the code changes, which encourages reviewers to explore and exercise the code in ways I possibly haven't thought of. I tend to use the following structure:

  1. What steps put the app in a state where it can be tested? Reviewers are usually familiar with the app in general, so step-by-step instructions don't need to be detailed, but it can help to assume that reviewers aren't familiar with the specific feature that's changed and may need some orientation. I think of the steps I had to go through when manually testing the feature, then describe them at a high level. That's helpful not only for reviewers but also for my future self should I ever come back to the PR.
  2. What action(s) should be taken? Make sure a reviewer knows what to do once the app is in the desired state. Again, I try to describe actions at a very high level and not to be too prescriptive, since I want reviewers to execute actions differently than I did to increase their chances of uncovering edge cases. It helps to trigger their creativity by suggesting they perform actions in different orders, with different values, etc.
  3. What should happen? Given some action, I ask reviewers to verify certain results by checking the UI, the database, or an external API call.
  4. What should not happen? This is usually only necessary for bugfixes, where previously something erroneous happened and now it shouldn't. It's helpful to reviewers to tell them exactly what they shouldn't see.

Mostly I've thought of exploratory testing in terms of clicking around in a UI, but lately I've been doing backend work that's not yet represented in the UI, which makes exploratory testing harder for reviewers. In those cases, I help reviewers set up state, perform actions, and/or check the resulting state by providing some code. They can tweak the code and run it to try different scenarios. Often it's relatively easy for me to provide that code, as I can mine scratch files I wrote while making the changes.

Outlining these testing steps also reminds me to keep pull requests focused. It's much harder to write good testing instructions when a PR has accumulated a lot of unrelated work.