Craig chats with Paul Rayner, a BDD and DDD expert who helps people bridge the gap of collaborative design between developers and business representatives, at YOW! West in Perth, and two old friends talk about the following:
George Dinwiddie led this session which turned into a lively discussion! I had proposed what I thought was a related session on Specification By Example and had combined them, but the conversation never really had a chance of getting onto that topic!
George expects the business people to be able to read and understand the tests
non-programmers should not be writing automation, it is the programmers responsibility
wants to be able to extract working tests into a step definition rather than needing to rewrite in Ruby (George Dinwiddie)
there is a difference between a specification and testing (Christian Hassa), this is a fundamental shift
building a DSL – talk about terminology and how we explore our domain – essential step
you don’t create a DSL, you build it
not a problem with the toolset but our training in thinking in a procedural way rather than an example way of thinking (Corey Haines
testers new to automation create large scripts because it’s their only hope in creating some sort of repetition (@chzy), it does not take a lot of effort and most business people are open to working this way
enable non-programmers by getting them to come work with us every day (Woody Zuill)
George is helping people make a transition, don’t want people to throw away what they have,
ideal is not to have step definitions call step definitions, Cucumber community is becoming a community of programmers and are moving away from this
Robot Framework is more keyword driven, more aligned to non-programmers, you can also make a mess, “it is a double edged sword” (Elisabeth Hendrickson)
testers like to test the negative cases, should they be expressed at a high level or expressed as a unit test by pairing developers and testers
if you are testers and you cannot write simple Ruby scripts, then you have no place on my team (Corey Haines), this opinion is probably shared by the Cucumber community (George disagreed…)
need to use the same design patterns in both Robot and Cucumber (@chzy)
in an environment that is test centric and BDD, Cucumber is the tool (usually environments with little to no QA), in a business centric environment where you an get the business involved Robot Framework is your tool
Corey works in environments where there is very few Cucumber specifications per scenario, backed by lots of unit tests
Cucumber came out of environments where the team is predominantly developers, hence the desire to drill down to Ruby code sooner
at a large household name company – theyexpect testers to be more technical, happening more in the industry, eliminated the role of tester due to different pay grades (@chzy)
moving traditional organizations to a collaborative way of working is hard (@chzy)
wants simple refactorings that are are a bridge from one place to another (George Dinwiddie)
at a startup Joseph was at, tests were taking up to 8 hours to run and costs for distributed architecture was high
Forward Internet (London) – let developers do what they want – by not testing they could be faster and more interactive than their competitors – did testing in Production, a risk that sometimes things could fail – testing should not block deployment
in some situations it is just worth hacking it out, particularly in a lean startup
if it is faster to rewrite rather than maintain it, then don’t write tests (Fred George via Corey Haines)
a big question of this is the skill level of your developers – do you have the skill level to make the choice to not do it (Corey Haines), primary impact of success is the skill level of your developers
Scribd – were having trouble with test speed and found out the developers were scared of breaking the PDF (which is the heart of the business) – they separated the PDF out to speed up development (so developers weren’t worried about breaking it)
quick delivery – need the quick feedback cycle to make this work, simulate production
need effective tests – small suite of tests that are 5-10 minutes long
test what you are most scared of
Silicon Valley’s issue is hiring – Facebook is stealing developers from Google because they hire good people and enable them to just hack it out
2 software industries – small companies and large corporations, very different worlds
question everything – can only do this if you have experienced it before and understand it
Wikimedia Foundation – looking at ways crowd source testing to test infrastructure (rather than content) – more on this initiative to be announced in the near future
why is it any different to coding katas? Safer and smaller so you get more practice, practice collaboration too
organise a community like a book club
code roast – put the code up and everybody critiques it, be careful not to attach to a person!
We finished up the open space by writing what action we were taking from the day and giving them to another participant to keep us honest (mine was to write this post!)
Ainsley started walking the circle to explain the day and how open space works, but frankly it make me feel a little dizzy! She went on to explain that Harrison Owen invented the open space idea as he noticed the real content at conferences was the passionate conversations. The rules of open space are:
whoever shows up are the right people
do not hang on to pre-conceived ideas
it starts when it starts
discussion does not need to be over until it’s over
The law of mobility and responsibility (also known as the law of two feet) is if you are not learning or contributing where you are, go some place where you will. Also, butterflies and bumblebees cross pollinate ideas.
there are two mindsets – offence and defence, testers are defence
job is not to find defects but to prevent defects – build quality in
define quality and what does it mean to us
startups don’t often have the problem – multiple skills required
what is the biggest impediment – are we missing the skill
there is no team of quality anymore – drive quality through the organization
functional testers tend to exploratory test and drive from the UI, technical analysts tend to multiple-skill
you need to have a team focus and a product focus
don’t start with practices but start with a common vision (eg. zero defects)
fear of losing identity if you dissolve roles
understanding the historical roles sometimes helps understands why things are the way they are
need time – Lisa Crispin mentioned that in her company they were going out of business because the system was not good quality, so management were smart to support the initiative
helps if everybody on the team has experienced the entire value chain and needs to understand the value of everybody’s piece of the chain – tendency to optimise the piece of the chain you understand
developers often underestimate the precision of data and scenarios and developers underestimate the difficulty of some requests
personality issues often get in the way
mostly about having the right people – need to let some people go
we assign labels to roles which create barriers – break down on teams but need to break down at the HR level
payroll is also an issue – need to compensate for people taking on more responsibility
need to put queue limits on the testing queue to drive behaviours
pairing with developer if they do not understand the scenarios
some people have the questioning mindset, some have the practical focus – need both to make sure you ship a quality product
mini waterfall problem – long tail feedback loop, change workflow that developer needs to work with tester, avoid lean batching problem
NUnit – Liz Keogh – were using Fitnesse but added another level of complication, wrote a DSL that separates tests to make it easier read, WiPFlash is the automation tool, examples are on the website, can call the fixtures from another testing tool like Fitnesse, capture scenarios on a wiki first to get the best out of the automation tool
SpecFlow – Christian Hassa – similar to Cucumber, scenarios written as steps that are bound to execution, uses Gherkin parser (this is a plus as a number of tools use this)
SpecLog – maps of your product backlog, capture results of collaboration with the business (Jeff Patton’s story maps), data stored in a single file, stories are initially mapped to a feature file but ultimately get linked to a feature tree
SpecRun is under development currently, not bound to SpecFlow or test runner/execution, currently Windows only
Limited Red – Joseph Wilk – uses the probability of failure to run those tests first in Cucumber, can then get failure statistics at a feature level, working on a refactoring tool at the moment
The Smallest Federated Wiki – Ward Cunningham – JSON for data scrubbing, thin columns to display well on mobile, refactoring is the number one edit so allow it to drag and drop refactor, fit for any analytic or outcome-oriented endeavor, sponsored by Nike, under very early development, meant to take spreadsheet data to the next level
separation of concerns – a rule is separate from the action which makes the process more brittle and more difficult to test
rules are a form of requirements and live beyond the building
one process is to extract the rules of a legacy system and then the regression tests – code archaeology
the business does not always know the rules of the system or how they got there – rules get added to the system over time or evolve and documentation is unlikely to get updated
one insurance company had spent $100 million dollars to bring in a business rule engine, returned investment in two years due to being able to be able to look for conflicting rules
put analysis of rules in the hands of developers for way too long
simplest part of business rules is having a glossary
rules engine enables our rules in productions, and use examples to ensure the engine works correctly
testing could look like this – given this data when these rules are applied then I expect this output
you need both rules and examples to test them – you need enough examples for now, need to be different paths, decision points, infliction points rather than different values
examples are not as expressive as arithmetic, but they are not as understandable
lots of rules that we do not think of as business rules because they are baked into the process eg. security access, database schemas
“business logic is not” (Martin Fowler)
you can’t read English as if it were rules, so we need to use examples
the worst systems are the ones that do not have a manual override, humans are usually the best at determining this
lots of business rules change due to jurisdiction
something will always fall to the bottom – rules need to be valued on risk and value – where is the tipping rule
rules are the expression of intent
Mars issue – crashed, six week window too costly to fix
guts to keep it simple – reporting system (Ward Cunningham) – resisted urge to put in a formula system, wait for requests from users, got 6 requests, sold system based on simplicity of the system
perhaps the movie should be at a feature level, because the video should be interesting
best suited for happy path testing, is a way to secure project funding and money, remember that the iPhone commercial does not show the AT&T network being down
there is a separation between pre-project and during testing
from the command line utility vnc2swf, created an API called Castro
at the moment you need to clean up the screens that are recorded
the advantage, being VNC, is that you can use all sorts of hardware, including the iPhone
suggest that you use something like uLimit to stop runaway videos, especially when being run in an automated test, to limit the size of the directory or the length of the video
suggest make a rule that no test is longer than five minutes
given the current tools are written in Python, DocTest is good for testing
Lightning Talks on Tools
I came in mid-way through this session, but caught some of the tools being discussed at the end
some tools are too hard to get passed the basic level, but quick to setup
tests are procedural, engineers tend to over-engineer
can’t run SWAT when Fitnesse is running as a server
SWAT is a C# library at its core
can run macros, tests from other tests
run script – write script (eg. JavaScript) to help things that are hard to test
High Performance Browser Testing / Selenium
Jason Huggins led this conversation which was more a roundtable debate than anything else. The group discussed how we can get tests running quicker and reduce feedback times considerably.
This discussion led to a couple of the quotes of the workshop from Jason Huggins:
“Selenium IDE is the place to start with Selenium, but it is Selenium on training wheels”
“Record/playback testing tools should be clearly labeled as “training wheels”
“What to do with the Selenium IDE, no self respecting developer will use it.” Thinking of renaming the IDE to Selenium Trainer.
Amazing how many people in the testing community are red, green colour blind”
When Can / Do You Automate Too Much?
This started as a discussion on testing led by Brandon Carlson…
get your business people to write the tests – they will understand how hard it is, have seen outcome that amount of scope reduced because they have to do the work
…but ended up as a great discussion on agile approaches and rollout, discussing a number of war stories led by Dana Wells and Jason Montague from Wells Fargo
still early in their agile deployment
wish to emulate some of the good work done by some of the early agile teams
estimate in NUTs (Nebulus Units of Time)
Miscellaneousand Other Links
Some other miscellaenous observations from the workshop:
a number of sessions were recorded
of those using Windows laptops, a large percentage were running Google Chrome
Craig chats with Paul Rayner, a BDD and DDD expert who helps people bridge the gap of collaborative design between developers and business representatives, at YOW! West in Perth, and two old friends talk about the following:
