The Agile Alliance Functional Testing Tools Workshop (AAFTT), was one again held this year the day before the Agile 2012 conference in Dallas. Despite there being only a small group there this year, the discussion was still open and free flowing under the facilitation of Matt Barcomb and the organisation of Joseph Wilk and Elisabeth Hendrickson.

From Agile 2012

We created an agenda for the day:

From Agile 2012

From Agile 2012

Here are my notes from the day:

Enabling Non-Programmers

George Dinwiddie led this session which turned into a lively discussion! I had proposed what I thought was a related session on Specification By Example and had combined them, but the conversation never really had a chance of getting onto that topic!

From Agile 2012

• George expects the business people to be able to read and understand the tests
• non-programmers should not be writing automation, it is the programmers responsibility
• wants to be able to extract working tests into a step definition rather than needing to rewrite in Ruby (George Dinwiddie)
• there is a difference between a specification and testing (Christian Hassa), this is a fundamental shift
• building a DSL – talk about terminology and how we explore our domain – essential step
• you don’t create a DSL, you build it
• not a problem with the toolset but our training in thinking in a procedural way rather than an example way of thinking (Corey Haines
• testers new to automation create large scripts because it’s their only hope in creating some sort of repetition (@chzy), it does not take a lot of effort and most business people are open to working this way
• enable non-programmers by getting them to come work with us every day (Woody Zuill)
• George is helping people make a transition, don’t want people to throw away what they have,
• ideal is not to have step definitions call step definitions, Cucumber community is becoming a community of programmers and are moving away from this
• Robot Framework is more keyword driven, more aligned to non-programmers, you can also make a mess, “it is a double edged sword” (Elisabeth Hendrickson)
• testers like to test the negative cases, should they be expressed at a high level or expressed as a unit test by pairing developers and testers
• if you are testers and you cannot write simple Ruby scripts, then you have no place on my team (Corey Haines), this opinion is probably shared by the Cucumber community (George disagreed…)
• need to use the same design patterns in both Robot and Cucumber (@chzy)
• in an environment that is test centric and BDD, Cucumber is the tool (usually environments with little to no QA),  in a business centric environment where you an get the business involved Robot Framework is your tool
• Corey works in environments where there is very few Cucumber specifications per scenario, backed by lots of unit tests
• Cucumber came out of environments where the team is predominantly developers, hence the desire to drill down to Ruby code sooner
• at a large household name company – theyexpect testers to be more technical, happening more in the industry, eliminated the role of tester due to different pay grades (@chzy)
• moving traditional organizations to a collaborative way of working is hard (@chzy)
• wants simple refactorings that are are a bridge from one place to another (George Dinwiddie)

From Agile 2012

From Agile 2012

Not Testing

Joseph Wilk led this discussion on thoughts that are coming from the Lean Startup movement.

From Agile 2012

• at a startup Joseph was at, tests were taking up to 8 hours to run and costs for distributed architecture was high
• Forward Internet (London) – let developers do what they want – by not testing they could be faster and more interactive than their competitors – did testing in Production, a risk that sometimes things could fail – testing should not block deployment
• in some situations it is just worth hacking it out, particularly in a lean startup
• if it is faster to rewrite rather than maintain it, then don’t write tests (Fred George via Corey Haines)
• a big question of this is the skill level of your developers – do you have the skill level to make the choice to not do it (Corey Haines), primary impact of success is the skill level of your developers
• cost of failure?
• complexity is in the eye of the beholder
• Etsy – check error rates in Production (and decide whether to roll back or not)
• Scribd – were having trouble with test speed and found out the developers were scared of breaking the PDF (which is the heart of the business) – they separated the PDF out to speed up development (so developers weren’t worried about breaking it)
• quick delivery – need the quick feedback cycle to make this work, simulate production
• need effective tests – small suite of tests that are 5-10 minutes long
• test what you are most scared of
• Silicon Valley’s issue is hiring – Facebook is stealing developers from Google because they hire good people and enable them to just hack it out
• 2 software industries – small companies and large corporations, very different worlds
• question everything – can only do this if you have experienced it before and understand it
• need a model to help others adopt this

From Agile 2012

What Are The Better Ways To Specify Tests With Large Test Data

I unfortunately did not get to this session as it was running at the same time as the No Testing session, but here is the output from that session.

From Agile 2012

From Agile 2012

From Agile 2012

Deliberate Test Practice

Brandon Leiran led this session, trying to see if there was a testing equivalent of coding katas.

From Agile 2012

• weekend testing group – choose a target, collaborate on Skype on their findings
• Wikimedia Foundation – looking at ways crowd source testing to test infrastructure (rather than content) – more on this initiative to be announced in the near future
• why is it any different to coding katas? Safer and smaller so you get more practice, practice collaboration too
• organise a community like a book club
• code roast – put the code up and everybody critiques it, be careful not to attach to a person!
• get practice at driving different interfaces – Triangle Tester exercise, parking lot calculator
• hard to practice test automation as it takes a lot of time upfron
• take time to do charter writing sessions or test different items like cheap toys (how would you test this toy?)
• demonstrate value of quality using simulations eg. origami games
• add tests to open source – many of the existing tests are average

From Agile 2012

From Agile 2012

Holes / Editors

Chzy led this discussion to discuss holes in the existing frameworks.

From Agile 2012

• the HTML report from Cucumber is very average – chzy is releasing a new gem based on discussion from a recent testing conference in Austin
• editors – people now bundling these in TextMate, Eclipse and Visual Studio
• JetBrains – RubyMine has gherkin support and refactoring support for Ruby, plus a lot of support for steps in Cucumber
• big picture view of feature coverage – would be cool to map this to Sonar, suites represent functional areas, tags to represent cross-cutting concerns
• SpecFlow is trying to map to story maps using SpecLog
• Relish allows you to create higher level specification of your scenarios
• there is a plugin for Cucumber that allows github integration
• Thucydides has a built in feature coverage report
• Twist has Cucumber support
• test data management – FactoryGirl gem – build up snapshots but want to be able manipulate values down the stack, Faker, ActiveRecord

From Agile 2012

From Agile 2012

AA-FTT – The Future

Elisabeth Hendrickson led this session as part of her handing the leadership over to Joseph Wilk.

From Agile 2012

• mission is to advance the state of the art of functional testing tools
• community building is the best way to spend the money, tool builders and tool users
• Yahoo group is main repository of knowledge, current wiki probably needs to be moved
• need people who have time and energy and interest to take this forward
• biggest issues with wikis is managing all the wiki spam
• have a leadership issue to curate the content and grow the community
• the other options are to create static content, like business analysts and leadership
• important to have a knowledge repository that at least captures outcomes
• would like have more organised meetings worldwide
• is our mandate just functional testing? It has really been just about “agile testing”
• probably need to rewrite the charter

From Agile 2012

Wrap Up

We finished up the open space by writing what action we were taking from the day and giving them to another participant to keep us honest (mine was to write this post!)

From Agile 2012

Another good open space, and good to catch up with many of the leaders in the testing community once again.

Podcast

I recorded a short audio podcast for The Agile Revolution wrapping up AAFTT.