Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design [Paperback]

A few years back, I read "How to Break Software" by James Whittaker. I liked it. It wasn't wonderful, but it had a good batch of practical, useful tips. Then I read "How to Break Software Security" and "How to Break Web Software". I liked them as well, but not as much. Still, I figured I'd read James Whittaker's newest book "Exploratory Software Testing". Sadly, the downward progression of his writing continues. This book is by far the worst of the bunch.

Chapter 1 - "The Case for Software Quality" is nothing more than "software is terrific, but it has bugs". That's it, nothing more here.

Chapter 2 - "The Case for Manual Testing" talks a bit about testing, and tries to define exploratory testing. Whittaker's definition has apparently caused some controversy among some well-known practitioners of exploratory testing, so here is his perhaps unique definition:

"When the scripts are removed entirely (or as we shall see in later chapters, their rigidness relaxed), the process is called exploratory testing."

Whittaker then divides exploratory testing into two sections. Exploratory testing in the small is that which guides the tester to make small, distinct decisions while testing. Exploratory testing in the large guides the tester in how an application is explored more than how a specific feature is tested.

Chapter 3 - "Exploratory Testing in the Small" was, to me, the only useful chapter in the whole book. Here Whittaker offers practical advice with examples for thinking about constructing test data, software state, and test environment.

Chapter 4 - "Exploratory Testing in the Large" is where Whittaker dives into what appears to be the point of the whole book - his Tourist Metaphor. Apparently this is a big hit at Microsoft, but I found it pointless. Think about every type of testing you have ever performed. Now try to torture it into a phrase that ends with the word Tour. There you go - that's the chapter.

Just to give you a flavor, here's a list of all these Tours, and their variations:

The Guidebook Tour

Blogger's Tour

Pundit's Tour

Competitor's Tour

The Money Tour

Skeptical Customer Tour

The Landmark Tour

The Intellectual Tour

Arrogant American Tour

The FedEx Tour

The After-Hours Tour

Morning-Commute Tour

The Garbage Collector's Tour

The Bad-Neighborhood Tour

The Museum Tour

The Prior Version Tour

The Supporting Actor Tour

The Back Alley Tour

Mixed-Destination Tour

The All-Nighter Tour

The Collector's Tour

The Lonely Businessman's Tour

The Supermodel Tour

The TOGOF Tour

The Scottish Pub Tour

The Rained-Out Tour

The Couch Potato Tour

The Saboteur Tour

The Antisocial Tour

Opposite Tour

Crime Spree Tour

Wrong Turn Tour

The Obsessive-Compulsive Tour

Perhaps the idea of calling UI Testing a Supermodel Tour appeals to you, and will make for a richer, more productive set of tests. I don't get it. I just don't see any value here. Doesn't testing have enough variation in language and definitions already, without adding this silliness?

Chapter 5 - "Hybrid Exploratory Testing Techniques" tells us that it's acceptable to combine scenario testing with exploratory testing. Then it spends time rehashing each of the tours from Chapter 4 and tries to suggest a side trip for each.

Chapter 6 - "Exploratory Testing in Practice" presents essays written by several Microsoft testers describing how they each used one or more of the tours in a testing situation. It appears as if Whittaker instructed his charges to write a "What I did this summer"-style essay, in the form of "How I used Tours to do my testing".

Chapter 7 - "Touring and Testing's Primary Pain Points" tries to tell us (in a few paragraphs) how to avoid five pain points - Aimlessness, Repetiveness, Transiency, Monontony, and Memorylesness. There's little real instruction here. For example, we are told that in order to avoid repetitiveness, we must know what testing has already occurred, and understand when to inject variation. Uhm, ok.

Chapter 8 - "The Future of Software Testing" has nothing at all to do with the other chapters, or exploratory testing. It's basically Whittaker's gee-whiz vision of what might be possible (some day) in the future. Perhaps. Whittaker has given this talk in several webinars - it's simply rehashed here.

Since these chapters take up only 136 pages, and obviously aren't enough to fill out a real book, three unrelated appendices are bolted on. A few pages about Testing as a career, and a bunch of pages lifted directly from Whittaker's blogs fill out the book to over 200 pages.

If you really want to learn about Exploratory Testing, this is probably not the place. "Exploratory Software Testing" is fluff - stretched and tortured out barely to book-length. There's not much in the way of learning here.

And if Microsoft testers are really instructed to "Tell me what kind of testing you did today, and make sure it ends with the word Tour", then I feel very sorry for them.

Some of the first testing books I read were James Whittaker's How to Break Software series. Those books, like this one, are laid out in a practical manner with each chapter focused on a specific attack or approach making them easy to read, reference and apply. Perfect for learning. I picked up this book a few years ago when I started questioning the way I was testing. The material was new to me and made me ask what is exploratory testing and what does touring have to do with it?

According to Whittaker (pg. 16) exploratory testing (E.T.) is testing where scripts or rigidity have been removed (paraphrasing). Whittaker explains his terms "E.T. in the small", decisions made where the scope of the testing is small and "E.T. in the large", decisions made when the scope of testing is large (small might be a screen in an application while large is the whole application). At the end of chapter 3 he mentions E.T. can be done in a way that allows test planning and execution to be completed simultaneously which is one of E.T.s most important aspects and simplest definitions. Touring (as in a tour guide or sight-seeing) becomes a metaphor for and a way to structure E.T.

There are eight chapters in the book plus a number of appendices. In the first few chapters Whittaker discusses what he sees as the case for software quality (the context of the book), introduces E.T. and explains how he uses it, in the small and the large. The next four chapters cover tours he and others have come up with. The last chapter is about how Whittaker sees the future of testing or at least how he did at the time of publishing.

The first appendix, A, is one of the most important parts of the book: building a successful career in software testing. Whittaker talks about how he got into testing and gives some advice on "getting over the hump" to be a better tester. Appendix A is short but worth reading. The rest of the appendices are old blog posts from his Microsoft days.

As a beginner I found this book much more valuable than I do now several years later. I understand E.T. is an approach to testing that can but doesn't necessarily include tours or scripts. It isn't just manual testing either. For reference Michael Bolton (the testing expert - Google developsense) has some good posts in what E.T. is not.

As you might guess from the title of this book it does not do a proper job explaining E.T. in a way that one can use it, aside from following the tour metaphor. In fact after reading it again this book seems to say to the reader: these tours are the best, don't you agree? It's important to understand exploratory testing is about the way you work, and the extent to which test design, test execution, and learning support and reinforce each other.

According to James Bach the term "exploratory testing" was coined and first published by Cem Kaner and has been worked on by Bach, Whittaker and Kaner over the last decade. It seems a bit odd that in a book about E.T. Whittaker never mentions their work and provides no references for the reader to follow up. Apparently Whittaker thinks the easiest way to explain E.T. is through testing tours (hence the book) while Bach has a more direct explanation of (Google this phrase) "what constitutes exploratory testing". I found Bach's post more informative, applicable and frankly cheaper than Whittaker's Exploratory Software Testing book.

Exploratory Software Testing offers a limited metaphor for understanding exploratory testing. It isn't as practical as Whittaker's previous books because you can't apply the teachings very well without fully understanding what E.T. is and how tours fit in. If you only want ideas on how Microsoft's testers used the touring metaphor to "perform" exploratory testing then you'll get four chapters of information otherwise Exploratory Software Testing is worth skipping.

I happened to review a copy of the book pre release. All I could say is that this book is full of new ideas regarding exploratory testing. The concept of tours is a revolutionary one, and provides a well defined structure to how we test. These concepts form a basic building block for a extremely comprehensive and solid test plan, paving way for automation. What is also impressive is the simplicity of the book. Easy and interesting to read, this book fills your mind with new ideas, which will change the way you think on approaching a test problem. 5 stars.

› Go to Amazon.com to see all 5 reviews  3.6 out of 5 stars