I had the great pleasure to attend the Agile Alliance Functional Testing Tools (AAFTT) workshop on the Sunday before the Agile 2009 conference in Chicago, and share discussion with some of the best minds in the testing community from around the world.

The location was right across the road from the Willis Tower (better known by its previous name, the Sears Tower). Some of the notable attendees amongst many others included:

• Jason Huggins – Selenium & Sauce Labs
• Lisa Crispin / Janet Gregory – authors of Agile Testing
• Aslak Hellesøy – Cucumber & RSpec
• Chad Wathington – Thoughtworks Studios (Twist)
• Mike Longin – SWAT
• Gerard Meszaros – author of xUnit Test Patterns
• Elizabeth Hendrickson – organiser & test obsessed!
• Pekka Klärck – Robot Framework
• Richard Lawrence – WatiR & WatiN
• Paul King – Groovy & Canoo WebTest

There were at least 4 tracks to choose from, these are the notes from the ones I participated in.

Screencasting

Small group discussion led by Jason Huggins about a different way of thinking about test artefacts (basically producing an iPhone commercial)

• the Rails screencast sold Rails because it sold the idea and then the product sold itself
• now, with YouTube, etc, we have the tools available
• used to be RTFM, not it is WTFV
• ideal is to produce automated tests like the iPhone commercial, instead of a test report
• use the “dailies” concept, like in the movies
• perhaps the movie should be at a feature level, because the video should be interesting
• best suited for happy path testing, is a way to secure project funding and money, remember that the iPhone commercial does not show the AT&T network being down
• there is a separation between pre-project and during testing
• tools currently exist, including the Castanaut DSL
• part of the offering of Sauce Labs, currently recording Selenium tests
• from the command line utility vnc2swf, created an API called Castro
• at the moment you need to clean up the screens that are recorded
• the advantage, being VNC, is that you can use all sorts of hardware, including the iPhone
• suggest that you use something like uLimit to stop runaway videos, especially when being run in an automated test, to limit the size of the directory or the length of the video
• suggest make a rule that no test is longer than five minutes
• given the current tools are written in Python, DocTest is good for testing

Lightning Talks on Tools

I came in mid-way through this session, but caught some of the tools being discussed at the end

• some tools are too hard to get passed the basic level, but quick to setup
• tests are procedural, engineers tend to over-engineer

Robot IDE (RIDE)

• most tools have a basic vocabulary to overcome
• IDE is worth looking at
• Robot has a Selenium plugin, but it is easy to write your own framework

Twist

• specify tests as requirements, looks like a document, stored as text, write whatever you want
• refactoring support as a first level concept
• out of the box support for Selenium and Frankenstein (Swing)
• write acceptance test – brown shows not implemented, allows developer to know what to implement, turns blue when done
• refactoring concept “rephrase”
• supports business rule tables (ie. Fitnesse for data driven tests)
• support to mark a test as manual and generate the same reports
• commercial software, licenced in packs
• plugins to Eclipse, but don’t need to be familiar with this unless you are developing the automation

WebDriver

• been around for three years

UltiFit

• Ultimate Software, internal currently, allows to select Fitnesse tests, setup and teardown, close browser windows, nice GUI, etc…
• uses TestRunner under the covers

SWAT

• been around for two years, more traction now that Lisa Crispin works for Ultimate Software
• simple editor for SWAT (& somewhat Fitnesse)
• has a database access editor
• uses Fitnesse syntax
• there is a recorder, only good for teaching, people get lazy and don’t refactor
• can take screenshots, borrowed from WatiN
• can’t run SWAT when Fitnesse is running as a server
• SWAT is a C# library at its core
• can run macros, tests from other tests
• run script – write script (eg. JavaScript) to help things that are hard to test

High Performance Browser Testing / Selenium

Jason Huggins led this conversation which was more a roundtable debate than anything else. The group discussed how we can get tests running quicker and reduce feedback times considerably.

This discussion led to a couple of the quotes of the workshop from Jason Huggins:

• “Selenium IDE is the place to start with Selenium, but it is Selenium on training wheels”
• “Record/playback testing tools should be clearly labeled as “training wheels”
• “What to do with the Selenium IDE, no self respecting developer will use it.” Thinking of renaming the IDE to Selenium Trainer.
• Amazing how many people in the testing community are red, green colour blind”

When Can / Do You Automate Too Much?

This started as a discussion on testing led by Brandon Carlson…

• get your business people to write the tests – they will understand how hard it is, have seen outcome that amount of scope reduced because they have to do the work

…but ended up as a great discussion on agile approaches and rollout, discussing a number of war stories led by Dana Wells and Jason Montague from Wells Fargo

• still early in their agile deployment
• wish to emulate some of the good work done by some of the early agile teams
• estimate in NUTs (Nebulus Units of Time)

Miscellaneous and Other Links

Some other miscellaenous observations from the workshop:

• a number of sessions were recorded
• of those using Windows laptops, a large percentage were running Google Chrome
• Wikispaces is good to setup a quick wiki

A number of posts about the workshop have been posted since including:

• Wrapup from Mark Levison on InfoQ: http://www.infoq.com/news/2009/09/functional-test-tools
• Wrapup from Mike Longin: http://devxero.wordpress.com/2009/08/24/aa-ftt-meetup-in-chicago-an-overview/
• Wrapup from Adam Goucher: http://adam.goucher.ca/?p=1153
• Wrapup from Lisa Crispin, including the deep dish pizza dinner I was fortunate enough to tag along too as well: http://lisacrispin.com/wordpress/2009/08/30/agile-2009-aa-ftt-workshop-more/ (and the photos: http://picasaweb.google.com/lisa.crispin/AAFTTWorkshopActivities#)
• Wrapup from Matt Wynne: http://blog.mattwynne.net/2009/09/10/the-agile-alliance-functional-testing-tools-workshop/

And you can view the photos that I took from the event at: http://www.flickr.com/photos/33840476@N06/sets/72157622521200928/