I was lucky enough to be a reviewer on Specification By Example by Gojko Adzic, and the final version was recently released to print by Manning. And I was stoked to see not only my name in the acknowledgements, but that my quote made it to the cover of the book. The following is my brief review and notes from the book.

Review

“I love this book. This is testing done right.” That is my quote on the back cover of the book, and I meant every word of it. Having been a quality advocate in the agile space for a few years now, this is the first book I have read in a long time which had me nodding my head all of the way through, as it resonated with my ideas on how development teams need to reconsider specifications and testing.

The book starts out by summarising why specification by example is so important and outlines some key patterns for success and then, through examples throughout the book, steps through the patterns pointing out the warning signs along the way. The key steps are to ensure the culture is fit, then approach specification in a collaborative manner, use examples and automate and finally evolving a living document / specification.

I really appreciated the fact that the examples were not just the run of the mill greenfield Java web applications that are used in most books. There is a good sampling of different organisations, most of which are using this technique on existing legacy applications on a variety of different platforms. The book is an easy read for the entire team, which means it can (and should) be required reading for the developer, tester, analyst and project manager. I have encouraged many of my teams to take a look at the book, and a couple of my colleagues have indicated this book helped convince and reinforce why this approach is so valuable.

My only concern when reviewing was the fact that the title of this book may not standout to testers and developers (not perhaps as much as Acceptance Test Driven Development or ATDD might). Currently the community has a number of similar approaches with similar names, although I must acknowledge that the specification by example tag has grown on me over the last few months.

The book does not expend much effort talking about tools in this space, by design, I think this fact makes the book more readable and accessible to a wider audience, but that said it suggests to me that there is still a gap for a good text that matches specification by example to particular tools like Concordion, Fitnesse and the like.

Overall, this book is a definite must read for any teams (particularly agile teams) who are trying to balance or find a decent approach to specifications and testing. It is a good balance of patterns and real case studies on how testing and specifications should be approached in an agile world. It would make my list of Top 5 must read testing books and Top 10 must read agile books. And now I know what the proper name is for the cats eyes that are embedded in the freeway!

Finally, I had some other suggestions for summaries for the book that did not make it to cover, but they are just as relevant of my feelings about the book:

• “One of the best Agile related books I have ever read. Buy it, read it, recommend it to your colleagues.”
• “This book sums up the right way to attack requirements and testing while delivering to your customer. A must read for all agile teams.”
  
• “I loved this book. I could not stop raving about it to my colleagues. It’s testing done right”
  

Summary

Here are my key notes from the book:

• a people problem, not a technical one
• building the product right and building the right product are two very different things, we need both to be successful
• living documents – fundamental – a source of information about system functionality that is as reliable as the programming language code but much easier to access and understand
• allows easier management of product backlogs
• proceed with specifications only when the team is ready to start implementing an item eg. at the start of an iteration
• derive scope from goals – business communicate the intent and team suggest a solution
• verbose descriptions over-constrain the system – how something should be done rather than just what is to be done
• traditional validation – we risk introducing problems if things get lost in translation between the business specification and technical automation
• an automated specification with examples, still in a human readable form and easily accessible to all team members, becomes an executable specification
• tests are specifications, specifications are tests
• consider living documentation as a separate product with different customers and stakeholders
• may find that Specification By Example means that UAT is no longer needed
• changing the process – push Specification By Example as part of a culture change, focus on improving quality, start with functional test automation, introduce a new tool, use TDD as a stepping stone
• changing the culture – avoid agile terminology, management support, Specification By Example a better way to do UAT, don’t make automation the end goal, don’t focus on a tool, leave one person behind to migrate legacy scripts (batman), track who is/isn’t running automated tests, hire someone who has done it before, bring in a consultant, introduce training
• dealing with signoff and tracebility – keep specifications in a version control system, get signoff of living documentation, get signoff on scope not specifications, get signoff on slimmed down use cases, introduce use case realisations
• warning signs – watch out for tests that change frequently, boomerangs, test slippage, just in case code and shotgun surgery
• F16 – asked to be built for speed but real problem was to escape enemy combat – still very successful 30+ years later
• scope implies solutions – work out the goals and collaborately work out the scope to meet goals
• people tell you what they think they need, and by asking them ‘why’ you can identify new implicit goals they have
• understanding why something is needed, and who needs it, is crucial to evaluating a suggested solution.
• discuss, prioritise and estimate at goals level for better understanding and reduced effort
• outside-in design – start with the outputs of the system and investigate why they are needed and how the software can provide them (comes from BDD)
• one approach is to get developers to write the “I want” part of the storycard
• when you don’t have control of scope – ask how something is useful, ask for an alternative solution, don’t only look at lowest level, deliver complete features
• collaboration is valuable – big all team workshops, smaller workshops (three amigos), developers and analysts pairing on tests, developers review tests, informal conversations
• business analysts are part of the delivery team, not customer representatives
• right level of detail is picking up a card and saying ‘I’m not quite sure’, it pushes you to have a conversation
• collaboration – hold introductory meetings, involve stakeholders, work ahead to prepare, developers and testers review stories, prepare only basic examples, overprescribing hinders discussion
• one of the best ways to check if the requirements are complete is to try to design black-box test cases against them. If we don’t have enough information to design good test cases, we definitely don’t have enough information to build the system.
• feature examples should be precise (no yes/no answers, use concrete examples), realistic (use real data, get realistic examples from customers), complete (experiment with data combinations, check for alternate ways to test) and easy to understand (don’t explore every combination, look for implied concepts)
• whenever you see too many examples or very complicated examples in a specification, try to raise the level of abstraction for those descriptions
• illustrate non-functional requirements – get precice performance requirements, use low-fi prototypes for UI, use the QUPER model, use a checklist for discussions, build a reference example for things that are hard to quantify (such as fun) to compare against
• good specifications – should be precise and testable, not written as a script, not written as a flow
• watch out for descriptions of how the system should work, think about what the system should do
• specifications should not be about software design – not tightly coupled with code, work around technical difficulties, trapped in user interface details
• specifications should be self explanatory – descriptive title and short paragraph of the goal, understood by others, not over-specified, start basic and then expanded
• specifications should be focussed – use given-when-then, don’t explicitly detail all the dependencies, put defaults at the technical layer but don’t rely on them
• define and use an ubiquitous language
• starting with automation – try a small sample project, plan upfront, don’t postpone or delegate, avoid automating existing manual scripts, gain trust with UI tests
• managing test automation – don’t treat as second-grade code, describe validation. don’t replicate business logic, automate along system boundaries, don’t check business logic through the UI
• automating user interfaces – specify interaction at a higher level (logging rather than filling out the login page), check UI functionality with UI specifications, avoid record and playback, setup context in a database
• test data management – avoid using pre-populated data, use pre-populated reference data, pull prototypes from the database,
• Bott’s Dott’s are the lane markers on the roads that alert you when you move out of your lane, continuous integration has that function in software, run it with Specification By Example and you have continuous validation
• reducing unreliability – find most annoying thing and fix it, identify unstable tests, setup dedicated validation environment, automated deployment, test doubles for external systems, multi-stage validation, execute tests in transactions, run quick checks for reference data, wait for events not elapsed time, make asynchronous processing optional, don’t use specification as an end to end validation
• faster feedback – introduce business time, break long tests into smaller modules, avoid in-memory databases for testing, separate quick and slow tests, keep overnight tests stable, create a current iteration pack, parallelise test runs
• managing failing tests – sometimes you can’t fix tests – create a known regression failures pack, automatically check disabled tests
• easy to understand documentation – avoid long specifications, avoid lots of small specifications for a single feature, look for higher level concepts, avoid technical automation concepts
• consistent documentation – evolve an ubiquitous language, use personas, collaborate on defining language, document building blocks
• organize for easy access – by stories, functional areas, UI navigation routes, business processes, use tags instead of URLs