Start reading Beautiful Testing: Leading Professionals Reveal How They... on your Kindle in under a minute. Don't have a Kindle? Get your Kindle here or start reading now with a free Kindle Reading App.

Deliver to your Kindle or other device


Try it free

Sample the beginning of this book for free

Deliver to your Kindle or other device

Anybody can read Kindle books—even without a Kindle device—with the FREE Kindle app for smartphones, tablets and computers.
Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice)

Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice) [Kindle Edition]

Adam Goucher , Tim Riley
4.5 out of 5 stars  See all reviews (2 customer reviews)

Print List Price: £32.50
Kindle Price: £24.71 includes VAT* & free wireless delivery via Amazon Whispernet
You Save: £7.79 (24%)
* Unlike print books, digital books are subject to VAT.


Amazon Price New from Used from
Kindle Edition £24.71  
Paperback £32.50  
Kindle Daily Deal
Kindle Daily Deal: At least 60% off
Each day we unveil a new book deal at a specially discounted price--for that day only. Learn more about the Kindle Daily Deal or sign up for the Kindle Daily Deal Newsletter to receive free e-mail notifications about each day's deal.

Product Description

Product Description

Successful software depends as much on scrupulous testing as it does on solid architecture or elegant code. But testing is not a routine process, it's a constant exploration of methods and an evolution of good ideas.

Beautiful Testing offers 23 essays from 27 leading testers and developers that illustrate the qualities and techniques that make testing an art. Through personal anecdotes, you'll learn how each of these professionals developed beautiful ways of testing a wide range of products -- valuable knowledge that you can apply to your own projects.

Here's a sample of what you'll find inside:

  • Microsoft's Alan Page knows a lot about large-scale test automation, and shares some of his secrets on how to make it beautiful
  • Scott Barber explains why performance testing needs to be a collaborative process, rather than simply an exercise in measuring speed
  • Karen Johnson describes how her professional experience intersected her personal life while testing medical software
  • Rex Black reveals how satisfying stakeholders for 25 years is a beautiful thing
  • Mathematician John D. Cook applies a classic definition of beauty, based on complexity and unity, to testing random number generators

All author royalties will be donated to the Nothing But Nets campaign to save lives by preventing malaria, a disease that kills millions of children in Africa each year.

This book includes contributions from:

  • Adam Goucher
  • Linda Wilkinson
  • Rex Black
  • Martin Schröder
  • Clint Talbert
  • Scott Barber
  • Kamran Khan
  • Emily Chen
  • Brian Nitz
  • Remko Tronçon
  • Alan Page
  • Neal Norwitz
  • Michelle Levesque
  • Jeffrey Yasskin
  • John D. Cook
  • Murali Nandigama
  • Karen N. Johnson
  • Chris McMahon
  • Jennitta Andrea
  • Lisa Crispin
  • Matt Heusser
  • Andreas Zeller
  • David Schuler
  • Tomasz Kojm
  • Adam Christian
  • Tim Riley
  • Isaac Clerencia

About the Author

Tim Riley is the Director of Quality Assurance at Mozilla. He has tested software for 18 years including everything from spacecraft simulators, ground control systems, high security operating systems, language platforms, application servers, hosted services and open source web applications. He has managed software testing teams in startups to large corporations consisting of 3 to 120 people in size and in up to 6 countries. He has a software patent for a testing execution framework which matches test suites to available test systems. He enjoys being a breeder caretaker for Canine Companions for Independence ( along with live and studio sound engineering.

Adam Goucher has been testing software professionally for over ten years. In that time he has worked with start-ups, large multi-nationals and ones in between in both traditional and agile testing environments. A believer in the communication of ideas big and small, he writes frequently at and teaches testing skills at a Toronto area technical college. In his off hours he can be found either playing or coaching box lacrosse - and then promptly applying lessons learned to testing. He is also an active member of the Association for Software Testing.

Product details

  • Format: Kindle Edition
  • File Size: 3101 KB
  • Print Length: 352 pages
  • Page Numbers Source ISBN: 0596159811
  • Simultaneous Device Usage: Unlimited
  • Publisher: O'Reilly Media; 1 edition (14 Oct 2009)
  • Sold by: Amazon Media EU S.à r.l.
  • Language: English
  • Text-to-Speech: Enabled
  • X-Ray:
  • Average Customer Review: 4.5 out of 5 stars  See all reviews (2 customer reviews)
  • Amazon Bestsellers Rank: #460,256 Paid in Kindle Store (See Top 100 Paid in Kindle Store)
  •  Would you like to give feedback on images?

More About the Authors

Discover books, learn about writers, and more.

Customer Reviews

3 star
2 star
1 star
4.5 out of 5 stars
4.5 out of 5 stars
Most Helpful Customer Reviews
3 of 3 people found the following review helpful
4.0 out of 5 stars Beautiful 26 Mar 2010
Some titles serve to inspire the readers to delve into the book - Beautiful Testing certainly did it for me. This is a collection of 23 essays by several contributors who reflect on the various aspects of testing including the people involved, the tools and processes.

The preface sets the scene by equating beauty of testing to fun, challenging, engaging, experiential, thoughtful and valuable. This theme runs across the essays that follow, with emphases varying on what is beautiful (and what is not!) when it comes to software testing.

There are some excellent pieces here: chapter two manages to discuss the various stakeholders involved in testing projects followed by simple defect detection metrics. The chapter is insightful and succinct. Chapter three has a focus on building open source tester communities. The authors describe their experiences highlighting the challenge of recruiting and managing volunteers. Chapter five is another gem, with the author making a case for fuzz testing, presenting a convincing case for it.

The most interesting part of the book - for me at least - is the `Bug Management' section in chapter six. Exploring the history of bugs the section describes, with an illustration, the very first bug report in the history of computers. The analysis of the terminology of bug that follows provides a lesson central to the book: that the severity and size of the defect is often underestimated by the use of the term.

As is inevitable with such books, some contributions would be of interest to only a narrow set of readers. A good example is chapter ten, which is a thorough treatment of random number generators, albeit in the context of testing. The book could also have benefited from a helpful testing-related glossary at the end.
Read more ›
Comment | 
Was this review helpful to you?
5.0 out of 5 stars Fantastic 12 July 2010
Brilliant opening and the book just gets better.

For those of us in the industry longer than 5 years it just solidifies testing and methodologies established and learnt over the years. For those new to testing a fantastic insight in to the world of testing.
Comment | 
Was this review helpful to you?
Most Helpful Customer Reviews on (beta) 4.7 out of 5 stars  10 reviews
8 of 8 people found the following review helpful
5.0 out of 5 stars Real Case Studies For Real Testers 27 Jun 2010
By Michael Larsen - Published on
Many of us who test as a dedicated profession realize two things. First, what we do requires a lot of emotional brain labor, and it isn't easy to do. Second, we realize that there are a lot of people who do not share that view. Oftentimes, we have to go to great lengths to explain what testing provides for an organization and that (zoinks!), it's not just the testers responsibility to focus on quality. In fact it's the entire organizations role to focus on quality (repeat after me: testers cannot assure what is not there. Developers can create quality product and we can assure that the quality initiatives that are in place are effective, or we can show where quality issues are. We cannot manufacture quality out of thin air. Either it is already there, or it is not).

Beautiful Testing takes a great approach here, in that each chapter is written as a standalone case study. Each chapter likewise has a different author(s). Some of the details are very familiar to every tester, and some situations are unique challenges that many of us may not have faced yet . The first part of the book deals with testing as a people issue, and focuses on tester attributes and abilities. The second section of the book deals with test processes and procedures, and real world examples of those procedures. Part three deals with testing tools and how to make the most of them in real world environments and with real testing challenges.

What's more, each author agrees to donate their portion of royalties for the books to a charitable cause. In this case, the charitable cause is "Nothing but Nets" a program to distribute mosquito nets in Africa to help stem the tide of malaria infections.

Chapter 1 : Was It Good for You? (Linda Wilkinson)

This chapter leads off the book and gives a great introduction to the mindset of a tester, and the reason and rationale they use to help a company get the most out of their software development time. It makes a clear case that "not just anyone can test" (or at least not do so and do it well), and it helps identify the areas testers really care about.

Chapter 2 : Beautiful Testing Satisfies Stakeholders (Rex Black)

There are many stakeholders that have a say and a personal vested interest in our testing being done well and providing a lot of information to help make good decisions. Those stakeholders range from customers, vendors and users, but also include such entities as law enforcement, elected officials, company shareholders and all of the other key contributors to any project (PM's, developers, software developers, and yes, even our fellow testers).

Chapter 3 : Building Open Source QA Communities (Martin Schröder & Clint Talbert)

Using the example of Open Source projects, getting a community involved in the efforts will help get people excited about applications and give those who are part of that community a desire and drive to see it succeed. My own experience with this has been with the Selenium Users Group here in San Francisco. While I find using the tool itself to be interesting, getting involved with and getting to know others that are also actively involved gives me extra energy and motivation to learn and practice more so I can likewise share with the broader community.

Chapter 4 : Collaboration Is the Cornerstone of Beautiful Performance Testing (Scott Barber)

Scott shares some of his insights into the development of his approach to performance testing, and the idea that performance testing challenges can be tackled via collaboration with other groups.

Chapter 5 : Just Peachy: Making Office Software More Reliable with Fuzz Testing (Kamran Khan)

Fuzzing is described as a technique where deliberately corrupt data is entered into your application to see how the system reacts to the inputs (for good or ill). Kamran uses Excel as an example application and demonstrates using tools that fuzz input and data values.

Chapter 6 : Bug Management and Test Case Effectiveness (Emily Chen & Brian Nitz)

Emily and Brian share bug management techniques and methods defining defects as relates to their involvement with Bugzilla and the OpenSolaris Desktop development team.

Chapter 7 : Beautiful XMPP Testing (Remko Tronçon)

Remko walks through examples and issues faced with testing the Extensible Messaging and Presence Protocol (XMPP) and describes his approach to creating Unit Tests for testing protocol interactions.

Chapter 8 : Beautiful Large-Scale Test Automation (Alan Page)

Alan walks the user through an example of test automation on a grand scale, and shows that many of the approaches and methods that are used for small scale automation projects work the same way for large automation, but the scale is totally different. This chapter helps a lot in showing neophyte testers that the steps from one world to another need not be so scary.

Chapter 9 : Beautiful Is Better Than Ugly (Neal Norwitz, Michelle Levesque & Jeffrey Yasskin)

Python has made its way from an interesting yet obscure language back in the 90's to becoming one of the go-to languages of the web and testing today. Testing an entire development scripting language puts a whole new area and emphasis on testing and stability.

Chapter 10 : Testing a Random Number Generator (John D. Cook)

Here's a great example of taking an application that can be tested in a number of ways, and the correctness or incorrectness can be difficult to pin down.

Chapter 11 : Change-Centric Testing (Murali Nandigama)

Murali demonstrates a call system and makes the case that, instead of testing everything over and over again, make a series of tests that will focus on the change. By using a change-centric testing approach, the number of tests run nightly can be reduced dramatically.

Chapter 12 : Software in Use (Karen N. Johnson)

Karen describes the feeling and the responsibility of testing equipment that works in a Hospital's Intensive Care Unit, the very definition of Mission Critical. This one hit close to home, as it described a situation my Dad (a retired physician) faced a number of years ago with a program and a glitch that almost cost patient's lives in an infant ICU. Karen describes the process, ups and downs, and resolutions related to, in her words, working on a product that really matters.

Chapter 13 : Software Development Is a Creative Process (Chris McMahon)

Chris makes the case (and a really compelling one) that developing and testing software is artistic work. Evaluating software quality is evaluating art, and that, when we recognize the artistic aspect of creating software, Beautiful Testing becomes a reality.

Chapter 14 : Test-Driven Development: Driving New Standards of Beauty (Jennitta Andrea)

Jeanette introduces the idea of the Diderot Effect and relays it to test driven development and the unintended consequences of upgrading just one area of a process and thinking that it's all done. To embrace the beauty of TDD, all aspects of the role and purpose of testing and embracing TDD have to be applied. Requirements, system design, he very act of writing code, the pace of work and the level of engagement of the testers involved all face changes when TDD becomes part of the landscape.

Chapter 15 : Beautiful Testing As the Cornerstone of Business Success (Lisa Crispin)

Anyone familiar with Agile Testing will notice the Mind-map that leads off everything, and gives a clear picture of the ideas that Lisa wishes to impart. The take away is clear, testing is part of the overall process of development, and testing is a process at every stage of development. Testing drives development, and development is not complete until tested.

Chapter 16 : Peeling the Glass Onion at Socialtext (Matthew Heusser)

Matt makes the point that, in mathematics, often the simplest solution is the most beautiful solution, and the same holds true for testing. Through examples at Matt's company, Socialtext, he shows how they do not just test to show that they have done testing, but that the solution they have developed fits what their customers want to see.

Chapter 17 : Beautiful Testing Is Efficient Testing (Adam Goucher)

Efficiency and focusing on how to get the best bang for your buck requires setting some parameters, using some tools to help focus on the goal, and making a mindmap to capture test ideas and methods. Adam uses the mnemonic SLIME to help organize his approach ((Security, Languages, RequIrements, Measurement, Existing).

Chapter 18 : Seeding Bugs to Find Bugs: Beautiful Mutation Testing (Andreas Zeller & David Schuler)

Andreas and David discuss the idea of mutation tests, and the tool Javalanche to perform those tests.

Chapter 19 : Reference Testing As Beautiful Testing (Clint Talbert)

An inside look at how Mozilla tests the variety of products in the Mozilla portfolio, and how they create tests and their reference points. Their goal is to encourage people to get involved and test in the way that is the most simple, direct and easy to understand way possible.

Chapter 20 : Clam Anti-Virus: Testing Open Source with Open Tools (Tomasz Kojm)

A look under the hood at an open source product (Clam Anti-Virus, a tool I actively use and wholeheartedly endorse, by the way) , and all of the open source tools used to test it, along with the testing strategies used.

Chapter 21 : Web Application Testing with Windmill (Adam Christian)

Adam provides a quick tutorial in how to set up and use the Windmill web testing tool and a quick way to implement automated testing for web applications.

Chapter 22 : Testing One Million Web Pages (Tim Riley)

Tim describes the Spider and Sisyphus projects at Mozilla and how they use the framework to test huge numbers of pages and web sites.

Chapter 23 : Testing Network Services in Multimachine Scenarios (Isaac Clerencia)

Isaac describes the ANSTE test tool and how it is used at his company, eBox, to test environments with multiple and varying machines.

Bottom Line:

Not every section will be relevant to every tester, and I found a few of the sections not immediately applicable to my current situations, but each section gives the reader a greater appreciation of the testing process in their respective sections. The multi-writer style for each chapter makes the book very engaging, and allows the reader to skip to the sections that matter the most to them. There is something for everyone in the testing process here, from technical testers with deep programming knowledge to relatively new testers without specific development backgrounds. Agile and traditional development methodologies will find value in these chapters, and overall it's a fun read. If you are looking to put a little more elegance and art into your testing life, Beautiful Testing has a lot to offer.
8 of 9 people found the following review helpful
5.0 out of 5 stars Excellent read for both big and small concepts 27 Nov 2009
By James Holmes - Published on
This is a great book for testers, leads, and managers to read to get a better picture of where your testing process can bring value to your work. A few sections of this book didn't get me much value, but the vast majority of the book left me frantically scratching notes and folding corners of pages over. I read the book over a weekend and came away with a large number of major additions to my QA roadmap I use at work.

Kamran Khan's chapter on fuzz testing reinforced my ideas that choking your system with invalid parameters and input data is a tremendous way to shore up that system's stability. I also really enjoyed Lisa Crispin's and Alan Page's separate chapters, both of which emphasized value-driven, sensible approaches to test automation.

If you want an amazing story around how testing can directly impact the lives of those around you, read Karen Johnson's chapter "Software in Use." Johnson ties a visit to an Intensive Care Unit to work she'd done on equipment in that ICU - it's rare anyone sees that practical a link to work we do in this industry.

Other highly worthwhile chapters include the piece on Python's development process, the overview on TDD, Mozilla's regression testing philosophy, and others. The Python chapter, in particular, is a tremendous testament to how a rigorous testing philosophy can guarantee very solid releases even with a broad, distributed team of varying skills.

As my examples above point out, there's a great amount of broad-stroke value in the book; however, a wealth of smaller, critical points abound in various chapters as well. Some weren't phrased exactly like this, but I've taken away these other concept as well:

* Track the source of your bugs (test plans, exploratory, developer, etc.) and pay special attention to bugs found by customers. These "escapees" point to areas to shore up in your test plan.
* Mindmaps are a great way to brainstorm out your test plan or test areas.
* Use small tools like fuzzers to help create your baseline input data.
* 100% passing rates for your automated tests isn't reasonable. Investigating 100% of your failing tests to determine whether the specific failure matters is reasonable. (I already firmly believed this, but it was nice to see in print!)
* Using image comparison to check formatting.

This is one of the better books I've read this year, and it's absolutely worth adding to your shelf.
5 of 5 people found the following review helpful
5.0 out of 5 stars Beautiful case studies and essays 28 July 2010
By N. Martin - Published on
As a software developer, I see the importance of good testing on a day to day basis. It is the essential piece of the puzzle that ensures that your efforts bare fruit in terms of business value. However, I also see it as the most neglected and poorly implemented aspect of producing software - at least with some of the projects and environments I worked on (and, sadly, I certainly haven't always been an exception).

Having read this book my drive to improve the testing practices, procedures, and tools has been dramatically invigorated. Of particular interest to me, because they tie more directly with the work I do on a day to day basis, were the chapters on agile development, continuous integration, automated testing, and the associated tools. Chapter 14, "Test-Driven Development: Driving New Standards of Beauty" and "Beautiful Testing as the Cornerstone of Business Success" were truly awe-inspiring. They drew a picture in my mind of the perfect development environment, and even give somewhat of a road-map to get there.

Some of the chapters were excellent for their high level breadth. Chapter 2, "Beautiful Testing Satisfies Stakeholders" and chapter 6, "Bug Management and Test Case Effectiveness" gives you a good sense of the value proposition of a good testing foundation.

Others were a real joy to read (well...I guess if you're a geek). Chapter 1 was an entertaining, yet insightful, view into the mind of a tester. Chapter 12 was dramatic and really made you feel the weight of the pressure to make sure the testing was impeccable. One of my favorite chapters was "Software Development is a Creative Process", chapter 13. This chapter distilled how I feel about development.

There were a few chapter that didn't give me much value. Many of the chapters were very narrowly focused on a specific technology or project, and didn't do enough, in my opinion, to relate the chapter to the larger scope. Chapter 21, "Web Application Testing With Windmill" read almost like marketing material for the product. I also didn't get much out of "Beautiful XMPP Testing" (chapter 6) and "Testing a Random Number Generator" (chapter 10). These chapter, however, really didn't detract from the overall superb quality of the writing or high value of the information contained in this book - not to mention that it is probably a matter of perspective.

If you are involved in any aspect of software development, you should consider this required reading. The knowledge and insight taken out of this book will certainly make me more effective professionally.
3 of 3 people found the following review helpful
3.0 out of 5 stars Ask Felgall - Book Review 15 Dec 2011
By Stephen Chapman - Published on
I found this book to be a rather interesting mix of chapters that were exteremely interesting an enlightening as well as some that were of no interest whatever. Of course which chapters I found interesting and which chapters others find interesting could be completely different. The book should have something in it of interest to anyone who has any sort of interest in the testing process.

The one theme that does run throughout the book is how testing is an art rather than a science (but then so is coding). For those who haven't had much exposure to testing it will demonstrate how there is much more to testing than just defining a few basic tests and running them.

This book isn't going to teach someone how to test but it may help expand your view of testing. While those who wrote the chapters do not all agree on exactly what testing is they all consider it to be of much wider scope than some people from outside of testing have considered it to be (a lot of people think testing and debugging are the same - which is completely false). That much is clear from some of the chapters that talk about where testers have become more involved earlier in the process than they had been previously and the impact that had on how they were able to handle the testing.
2 of 2 people found the following review helpful
4.0 out of 5 stars Understanding testers and testing 24 Aug 2011
By Yuriy Zubarev - Published on
The book features 23 essays from leading testers, all providing different takes on what it means for testing to be beautiful. It was especially beneficial for me in these 3 areas:

1) Understanding a mindset of a tester. The first essay "Was it Good for You?" was revealing and brilliant. After reading the book I'm in a much better position to find and recognize great testers to work with, and also be more helpful to them.

2) "Test-Driven Development: Driving New Standards of Beauty" essay went beyond red-green-refactor cycle and exposed TDD in much broader context. The end result and organizational benefits associated with TDD were really transformational and beautiful as shown in this chapter.

3) I'm interested in software architecture theory and practice, and reading "Peeling the Glass Onion at Socialtext" came as a pleasant surprise. Among many other things, the author presented an argument that the "... testing group has the capability of serving - in the best possible way - as architects of the product." It's an angle on architecture that I never encountered before, and it made sense in a given context.

Overall, I learned some really useful insights about testing and testers and I'm glad I picked up this book.
Were these reviews helpful?   Let us know
Search Customer Reviews
Only search this product's reviews

Customer Discussions

This product's forum
Discussion Replies Latest Post
No discussions yet

Ask questions, Share opinions, Gain insight
Start a new discussion
First post:
Prompts for sign-in

Search Customer Discussions
Search all Amazon discussions

Look for similar items by category