Episode 127 – Storming DD’s with Paul Rayner

The Agile Revolution Podcast

Craig chats with Paul Rayner, a BDD and DDD expert who helps people bridge the gap of collaborative design between developers and business representatives, at YOW! West in Perth, and two old friends talk about the following:

* Agile Alliance Functional Testing Tool workshop (2011 in Salt Lake City)
* “Domain Driven Design” (Eric Evans) and “Working Effectively with Legacy Code” (Michael Feathers)
* The heart of DDD is about developing a rich model to allow you to deal with complex business domains
* Domain Driven Design Europe conference
* Design done well should pay off immediately, as well as in the medium to long term as well
* There are lot of overlaps between DDD and BDD, particularly the use of an ubiquitous language, BDD is a test first way to drive out your domain model
* YOW! West Keynote “EventStorming”
* Given When Then has…

View original post 208 more words

AAFTT Workshop 2012 (Dallas)

Agile AllianceThe Agile Alliance Functional Testing Tools Workshop (AAFTT), was one again held this year the day before the Agile 2012 conference in Dallas. Despite there being only a small group there this year, the discussion was still open and free flowing under the facilitation of Matt Barcomb and the organisation of Joseph Wilk and Elisabeth Hendrickson.

From Agile 2012

We created an agenda for the day:

From Agile 2012
From Agile 2012

Here are my notes from the day:

Enabling Non-Programmers

George Dinwiddie led this session which turned into a lively discussion! I had proposed what I thought was a related session on Specification By Example and had combined them, but the conversation never really had a chance of getting onto that topic!

From Agile 2012
  • George expects the business people to be able to read and understand the tests
  • non-programmers should not be writing automation, it is the programmers responsibility
  • wants to be able to extract working tests into a step definition rather than needing to rewrite in Ruby (George Dinwiddie)
  • there is a difference between a specification and testing (Christian Hassa), this is a fundamental shift
  • building a DSL – talk about terminology and how we explore our domain – essential step
  • you don’t create a DSL, you build it
  • not a problem with the toolset but our training in thinking in a procedural way rather than an example way of thinking (Corey Haines
  • testers new to automation create large scripts because it’s their only hope in creating some sort of repetition (@chzy), it does not take a lot of effort and most business people are open to working this way
  • enable non-programmers by getting them to come work with us every day (Woody Zuill)
  • George is helping people make a transition, don’t want people to throw away what they have,
  • ideal is not to have step definitions call step definitions, Cucumber community is becoming a community of programmers and are moving away from this
  • Robot Framework is more keyword driven, more aligned to non-programmers, you can also make a mess, “it is a double edged sword” (Elisabeth Hendrickson)
  • testers like to test the negative cases, should they be expressed at a high level or expressed as a unit test by pairing developers and testers
  • if you are testers and you cannot write simple Ruby scripts, then you have no place on my team (Corey Haines), this opinion is probably shared by the Cucumber community (George disagreed…)
  • need to use the same design patterns in both Robot and Cucumber (@chzy)
  • in an environment that is test centric and BDD, Cucumber is the tool (usually environments with little to no QA),  in a business centric environment where you an get the business involved Robot Framework is your tool
  • Corey works in environments where there is very few Cucumber specifications per scenario, backed by lots of unit tests
  • Cucumber came out of environments where the team is predominantly developers, hence the desire to drill down to Ruby code sooner
  • at a large household name company – theyexpect testers to be more technical, happening more in the industry, eliminated the role of tester due to different pay grades (@chzy)
  • moving traditional organizations to a collaborative way of working is hard (@chzy)
  • wants simple refactorings that are are a bridge from one place to another (George Dinwiddie)
From Agile 2012
From Agile 2012

Not Testing

Joseph Wilk led this discussion on thoughts that are coming from the Lean Startup movement.

From Agile 2012
  • at a startup Joseph was at, tests were taking up to 8 hours to run and costs for distributed architecture was high
  • Forward Internet (London) – let developers do what they want – by not testing they could be faster and more interactive than their competitors – did testing in Production, a risk that sometimes things could fail – testing should not block deployment
  • in some situations it is just worth hacking it out, particularly in a lean startup
  • if it is faster to rewrite rather than maintain it, then don’t write tests (Fred George via Corey Haines)
  • a big question of this is the skill level of your developers – do you have the skill level to make the choice to not do it (Corey Haines), primary impact of success is the skill level of your developers
  • cost of failure?
  • complexity is in the eye of the beholder
  • Etsycheck error rates in Production (and decide whether to roll back or not)
  • Scribd – were having trouble with test speed and found out the developers were scared of breaking the PDF (which is the heart of the business) – they separated the PDF out to speed up development (so developers weren’t worried about breaking it)
  • quick delivery – need the quick feedback cycle to make this work, simulate production
  • need effective tests – small suite of tests that are 5-10 minutes long
  • test what you are most scared of
  • Silicon Valley’s issue is hiring – Facebook is stealing developers from Google because they hire good people and enable them to just hack it out
  • 2 software industries – small companies and large corporations, very different worlds
  • question everything – can only do this if you have experienced it before and understand it
  • need a model to help others adopt this
From Agile 2012

What Are The Better Ways To Specify Tests With Large Test Data

I unfortunately did not get to this session as it was running at the same time as the No Testing session, but here is the output from that session.

From Agile 2012
From Agile 2012
From Agile 2012

Deliberate Test Practice

Brandon Leiran led this session, trying to see if there was a testing equivalent of coding katas.

From Agile 2012
  • weekend testing group – choose a target, collaborate on Skype on their findings
  • Wikimedia Foundation – looking at ways crowd source testing to test infrastructure (rather than content) – more on this initiative to be announced in the near future
  • why is it any different to coding katas? Safer and smaller so you get more practice, practice collaboration too
  • organise a community like a book club
  • code roast – put the code up and everybody critiques it, be careful not to attach to a person!
  • get practice at driving different interfaces – Triangle Tester exercise, parking lot calculator
  • hard to practice test automation as it takes a lot of time upfron
  • take time to do charter writing sessions or test different items like cheap toys (how would you test this toy?)
  • demonstrate value of quality using simulations eg. origami games
  • add tests to open source – many of the existing tests are average
From Agile 2012
From Agile 2012

Holes / Editors

Chzy led this discussion to discuss holes in the existing frameworks.

From Agile 2012
  • the HTML report from Cucumber is very average – chzy is releasing a new gem based on discussion from a recent testing conference in Austin
  • editors – people now bundling these in TextMate, Eclipse and Visual Studio
  • JetBrainsRubyMine has gherkin support and refactoring support for Ruby, plus a lot of support for steps in Cucumber
  • big picture view of feature coverage – would be cool to map this to Sonar, suites represent functional areas, tags to represent cross-cutting concerns
  • SpecFlow is trying to map to story maps using SpecLog
  • Relish allows you to create higher level specification of your scenarios
  • there is a plugin for Cucumber that allows github integration
  • Thucydides has a built in feature coverage report
  • Twist has Cucumber support
  • test data management – FactoryGirl gem – build up snapshots but want to be able manipulate values down the stack, Faker, ActiveRecord
From Agile 2012
From Agile 2012

AA-FTT – The Future

Elisabeth Hendrickson led this session as part of her handing the leadership over to Joseph Wilk.

From Agile 2012
  • mission is to advance the state of the art of functional testing tools
  • community building is the best way to spend the money, tool builders and tool users
  • Yahoo group is main repository of knowledge, current wiki probably needs to be moved
  • need people who have time and energy and interest to take this forward
  • biggest issues with wikis is managing all the wiki spam
  • have a leadership issue to curate the content and grow the community
  • the other options are to create static content, like business analysts and leadership
  • important to have a knowledge repository that at least captures outcomes
  • would like have more organised meetings worldwide
  • is our mandate just functional testing? It has really been just about “agile testing”
  • probably need to rewrite the charter
From Agile 2012

Wrap Up

We finished up the open space by writing what action we were taking from the day and giving them to another participant to keep us honest (mine was to write this post!)

From Agile 2012

Another good open space, and good to catch up with many of the leaders in the testing community once again.

Podcast

I recorded a short audio podcast for The Agile Revolution wrapping up AAFTT.

Episode 39: Agile 2012 Day 0

The Agile Revolution Podcast

Agile 2012Craig is in Dallas, Texas, USA at the Agile 2012 conference. Today was day 0 and the annual AA-FTT workshop.

In-depth notes can be found on Craig’s blog.

TheAgileRevolution-39 (8 minutes)

View original post

AAFTT Workshop 2011 (Salt Lake City)

Agile AllianceThe Agile Alliance Functional Testing Tools Workshop (AAFTT), held the day before the Agile 2011 conference in Salt Lake City, was once again one of the highlights of the conference. Organised by Jennita Andrea and Elisabeth Hendrickson, it was as always a wide variety of participants with a passion for testing and testing tools. Here are my notes from the day held on August 7, 2011.

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011


Energy Kickoff & Networking

The session was facilitated by Ainsley Nies, and all of the official session notes are stored on the AAFTT wiki: http://aaftt.agilealliance.org:8080/display/AAFTT/agile2011.

From AAFTT 2011
From AAFTT 2011

We started the day with some networking and sharing some areas of passion. Some of these included:

The theme of the AAFTT is: “Advancing the state of the art and the practice of Acceptance Test Driven Development”.

From AAFTT 2011

Ainsley started walking the circle to explain the day and how open space works, but frankly it make me feel a little dizzy! She went on to explain that Harrison Owen invented the open space idea as he noticed the real content at conferences was the passionate conversations. The rules of open space are:

  • whoever shows up are the right people
  • do not hang on to pre-conceived ideas
  • it starts when it starts
  • discussion does not need to be over until it’s over
  • wherever it happens is the right place
From AAFTT 2011

The law of mobility and responsibility (also known as the law of two feet) is if you are not learning or contributing where you are, go some place where you will. Also, butterflies and bumblebees cross pollinate ideas.

From AAFTT 2011
From AAFTT 2011

Finally, we were warned to be prepared to be surprised.

From AAFTT 2011
From AAFTT 2011

Developers are testers and testers are developers – how do we dissolve and combine the roles 

This was the first session that I attended.

From AAFTT 2011
  • there are two mindsets – offence and defence, testers are defence
  • job is not to find defects but to prevent defects – build quality in
  • define quality and what does it mean to us
  • startups don’t often have the problem – multiple skills required
  • what is the biggest impediment – are we missing the skill
  • there is no team of quality anymore – drive quality through the organization
  • functional testers tend to exploratory test and drive from the UI, technical analysts tend to multiple-skill
  • you need to have a team focus and a product focus
  • don’t start with practices but start with a common vision (eg. zero defects)
  • fear of losing identity if you dissolve roles
  • understanding the historical roles sometimes helps understands why things are the way they are
  • need time – Lisa Crispin mentioned that in her company they were going out of business because the system was not good quality, so management were smart to support the initiative
  • helps if everybody on the team has experienced the entire value chain and needs to understand the value of everybody’s piece of the chain – tendency to optimise the piece of the chain you understand
  • developers often underestimate the precision of data and scenarios and developers underestimate the difficulty of some requests
  • personality issues often get in the way
  • mostly about having the right people – need to let some people go
  • we assign labels to roles which create barriers – break down on teams but need to break down at the HR level
  • payroll is also an issue – need to compensate for people taking on more responsibility
  • need to put queue limits on the testing queue to drive behaviours
  • pairing with developer if they do not understand the scenarios
  • some people have the questioning mindset, some have the practical focus – need both to make sure you ship a quality product
  • mini waterfall problem – long tail feedback loop, change workflow that developer needs to work with tester, avoid lean batching problem
From AAFTT 2011

ATDD Patterns

Jennitta Andrea led this session about the work so far in this space:

Last Mile Tools

Elisabeth Hendricksen led this session on tools that are attempting to solve the problem at the last mile.

From AAFTT 2011
From AAFTT 2011
  • NUnitLiz Keogh – were using Fitnesse but added another level of complication, wrote a DSL that separates tests to make it easier read, WiPFlash is the automation tool, examples are on the website, can call the fixtures from another testing tool like Fitnesse, capture scenarios on a wiki first to get the best out of the automation tool
From AAFTT 2011
From AAFTT 2011
  • SpecFlow – Christian Hassa – similar to Cucumber, scenarios written as steps that are bound to execution, uses Gherkin parser (this is a plus as a number of tools use this)
  • SpecLog – maps of your product backlog, capture results of collaboration with the business (Jeff Patton’s story maps), data stored in a single file, stories are initially mapped to a feature file but ultimately get linked to a feature tree
  • SpecRun is under development currently, not bound to SpecFlow or test runner/execution, currently Windows only
From AAFTT 2011
  • Limited RedJoseph Wilk – uses the probability of failure to run those tests first in Cucumber, can then get failure statistics at a feature level, working on a refactoring tool at the moment
  • Relish – publish Cucumber features to a website
From AAFTT 2011
  • The Smallest Federated WikiWard CunninghamJSON for data scrubbing, thin columns to display well on mobile, refactoring is the number one edit so allow it to drag and drop refactor, fit for any analytic or outcome-oriented endeavor, sponsored by Nike, under very early development, meant to take spreadsheet data to the next level
From AAFTT 2011
From AAFTT 2011

Business Rules

Mary Gorman led this discussion.

From AAFTT 2011
  • business rules – conference website has rules, such as group pack for 5 registrations, what happens to the sixth person, what if someone pulls out
  • need to capture these to describe what our system does
  • business rules manifesto – Mary gives a copy to everyone she work with
  • separation of concerns – a rule is separate from the action which makes the process more brittle and more difficult to test
  • rules are a form of requirements and live beyond the building
  • one process is to extract the rules of a legacy system and then the regression tests – code archaeology
  • the business does not always know the rules of the system or how they got there – rules get added to the system over time or evolve and documentation is unlikely to get updated
  • one insurance company had spent $100 million dollars to bring in a business rule engine, returned investment in two years due to being able to be able to look for conflicting rules
  • put analysis of rules in the hands of developers for way too long
  • simplest part of business rules is having a glossary
  • rules engine enables our rules in productions, and use examples to ensure the engine works correctly
  • testing could look like this – given this data when these rules are applied then I expect this output
  • you need both rules and examples to test them – you need enough examples for now, need to be different paths, decision points, infliction points rather than different values
  • examples are not as expressive as arithmetic, but they are not as understandable
  • lots of rules that we do not think of as business rules because they are baked into the process eg. security access, database schemas
  • “business logic is not” (Martin Fowler)
  • you can’t read English as if it were rules, so we need to use examples
  • the worst systems are the ones that do not have a manual override, humans are usually the best at determining this
  • lots of business rules change due to jurisdiction
  • something will always fall to the bottom – rules need to be valued on risk and value – where is the tipping rule
  • rules are the expression of intent
  • Mars issue – crashed, six week window too costly to fix
  • guts to keep it simple – reporting system (Ward Cunningham) – resisted urge to put in a formula system, wait for requests from users, got 6 requests, sold system based on simplicity of the system
From AAFTT 2011
From AAFTT 2011

Other Sessions

As with any conference, there are always sessions you would have liked to have got along to.

Richard Lawrence led a discussion on Static Analysis for Gherkin which turned into a discussion on design patterns for Cucumber.

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011

George Dinwiddie led a discussion about conversations between roles:

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011

My mate Jason Montague led a session on Building Conditions Conducive for ATDD Adoption.

From AAFTT 2011
From AAFTT 2011

Closing Circle

We shared some takeaways in the closing circle, he were some that stood out at me:

From AAFTT 2011
  • issues with dealing of people was a theme
  • what are good ways to express a large amount of test data
  • challenge to get corporations over the hump to release data, plus have good tests and examples around the rules
  • testing needs to be a nation, not just a community
  • it’s time we got more respect in our organisations, it’s time we show more respect to those we work with
  • teams need to dependent on the production of the build
  • federated wikis could help solve the test ownership problem

As for me, my comment was the day had renewed my energy again. ATDD is hard, and as a community we need to try harder.

Podcast

Finally, I recorded a short audio podcast for The Agile Revolution wrapping up AAFTT.

AAFTT Workshop 2009 (Chicago)

Agile AllianceI had the great pleasure to attend the Agile Alliance Functional Testing Tools (AAFTT) workshop on the Sunday before the Agile 2009 conference in Chicago, and share discussion with some of the best minds in the testing community from around the world.

The location was right across the road from the Willis Tower (better known by its previous name, the Sears Tower). Some of the notable attendees amongst many others included:

There were at least 4 tracks to choose from, these are the notes from the ones I participated in.

Screencasting

Small group discussion led by Jason Huggins about a different way of thinking about test artefacts (basically producing an iPhone commercial)

Photo 3 of 4 from #agile2009 in Chicago at the pre-conference... on Twitpic

  • the Rails screencast sold Rails because it sold the idea and then the product sold itself
  • now, with YouTube, etc, we have the tools available
  • used to be RTFM, not it is WTFV
  • ideal is to produce automated tests like the iPhone commercial, instead of a test report
  • use the “dailies” concept, like in the movies
  • perhaps the movie should be at a feature level, because the video should be interesting
  • best suited for happy path testing, is a way to secure project funding and money, remember that the iPhone commercial does not show the AT&T network being down
  • there is a separation between pre-project and during testing
  • tools currently exist, including the Castanaut DSL
  • part of the offering of Sauce Labs, currently recording Selenium tests
  • from the command line utility vnc2swf, created an API called Castro
  • at the moment you need to clean up the screens that are recorded
  • the advantage, being VNC, is that you can use all sorts of hardware, including the iPhone
  • suggest that you use something like uLimit to stop runaway videos, especially when being run in an automated test, to limit the size of the directory or the length of the video
  • suggest make a rule that no test is longer than five minutes
  • given the current tools are written in Python, DocTest is good for testing

Lightning Talks on Tools

I came in mid-way through this session, but caught some of the tools being discussed at the end

  • some tools are too hard to get passed the basic level, but quick to setup
  • tests are procedural, engineers tend to over-engineer

Robot IDE (RIDE)

  • most tools have a basic vocabulary to overcome
  • IDE is worth looking at
  • Robot has a Selenium plugin, but it is easy to write your own framework

Twist

  • specify tests as requirements, looks like a document, stored as text, write whatever you want
  • refactoring support as a first level concept
  • out of the box support for Selenium and Frankenstein (Swing)
  • write acceptance test – brown shows not implemented, allows developer to know what to implement, turns blue when done
  • refactoring concept “rephrase”
  • supports business rule tables (ie. Fitnesse for data driven tests)
  • support to mark a test as manual and generate the same reports
  • commercial software, licenced in packs
  • plugins to Eclipse, but don’t need to be familiar with this unless you are developing the automation

WebDriver

  • been around for three years

UltiFit

  • Ultimate Software, internal currently, allows to select Fitnesse tests, setup and teardown, close browser windows, nice GUI, etc…
  • uses TestRunner under the covers

SWAT

  • been around for two years, more traction now that Lisa Crispin works for Ultimate Software
  • simple editor for SWAT (& somewhat Fitnesse)
  • has a database access editor
  • uses Fitnesse syntax
  • there is a recorder, only good for teaching, people get lazy and don’t refactor
  • can take screenshots, borrowed from WatiN
  • can’t run SWAT when Fitnesse is running as a server
  • SWAT is a C# library at its core
  • can run macros, tests from other tests
  • run script – write script (eg. JavaScript) to help things that are hard to test

High Performance Browser Testing / Selenium

Jason Huggins led this conversation which was more a roundtable debate than anything else. The group discussed how we can get tests running quicker and reduce feedback times considerably.

This discussion led to a couple of the quotes of the workshop from Jason Huggins:

  • “Selenium IDE is the place to start with Selenium, but it is Selenium on training wheels”
  • “Record/playback testing tools should be clearly labeled as “training wheels”
  • “What to do with the Selenium IDE, no self respecting developer will use it.” Thinking of renaming the IDE to Selenium Trainer.
  • Amazing how many people in the testing community are red, green colour blind”

When Can / Do You Automate Too Much?

This started as a discussion on testing led by Brandon Carlson…

  • get your business people to write the tests – they will understand how hard it is, have seen outcome that amount of scope reduced because they have to do the work

…but ended up as a great discussion on agile approaches and rollout, discussing a number of war stories led by Dana Wells and Jason Montague from Wells Fargo

  • still early in their agile deployment
  • wish to emulate some of the good work done by some of the early agile teams
  • estimate in NUTs (Nebulus Units of Time)

Miscellaneous and Other Links

Some other miscellaenous observations from the workshop:

  • a number of sessions were recorded
  • of those using Windows laptops, a large percentage were running Google Chrome
  • Wikispaces is good to setup a quick wiki

A number of posts about the workshop have been posted since including:

And you can view the photos that I took from the event at: http://www.flickr.com/photos/33840476@N06/sets/72157622521200928/