Testing

STANZ 2011 Day 2 Review

/ Craig Smith / Leave a comment

STANZDay 2 of STANZ 2011 in Melbourne, here are my notes from the sessions.

The Future of Quality

Goranka Bjedov from Facebook gave this somewhat controversial talk first up, the slides and videos are available here.

From STANZ 2011
  • value of quality is lower than the price of quality – quality is dead
  • you may get a quality product (sometimes) but you always get an expensive product
  • quality matters for human life, security and money
  • systems are designed for redundancy these days so most of the time nothing will go wrong
  • people value free over quality
  • we live in a world where quality doesn’t matter, we are used to things failing
  • restructure what you are doing – don’t focus on catching big bugs, but focus on productivity testing, reduce the cost of development and speed it up
  • it is cheaper to fix a bug when the customer finds it now we are moving to the cloud
  • organisations are now paying people to find security bugs, people will test for free for a free device, some companies will offer jobs if you find a bug
  • need to start communicating value of work in a language people understand – how much did we save?

Test Process Improvement: Testers Get Out Of Your Cave!

Jan Jaap Cannegieter presented this session.

From STANZ 2011
  • TMMi – maturity framework for testing, public domain, find out how mature your test processes are
  • CMMi only has 5 pages on testing
  • TMMi has 5 levels – initial > managed > defined > management and measurement > optimization
  • start at level 2 when assessing
  • results from some TMMi quick scan assessments in 20 organisations – test reporting 30%, test planning 41%, test monitoring 47%, test design 60%, test environment 59%
  • test design is probably high because it can be influenced within the testing team, whereas planning and reporting, etc.. require people outside of your team
  • testing teams are in a cave, need to get out of the cave and use the rest of the organisation
  • stakeholder definition – the most important people at the BBQ – the hold the power, they have mindset and ambition
  • it’s all political – politics is a way of life – you need to get in front of the leaders and stakeholders and have political skills

Why Model-Driven Testing is of Great Relevance to Test Managers and Test Analysts

Thomas Hadorn from Tricentis gave this very vendor driven presentation.

From STANZ 2011
  • Gartner believe model driven testing will become dominant in next 5 years
  • capture/replay is too fragile, develop/replay test frameworks are too costly because they need to be programmatically extended
  • model driven only one type of test – no scripts to maintain

The Future Tester At Suncorp: A Journey of Building Quality In Through Agile

I presented this session with Adrian Smith from Ennova and Dallas Thorneycroft from Suncorp. The slides are available in a separate post.

From STANZ 2011
From STANZ 2011
From STANZ 2011

Testing Skills : How To Find and Develop Skilled Testers

Goranka Bjedov from Facebook led this hands on workshop.

Amongst other exercises, she introduced the game of Set.

  • card has characteristics – colour (purple, green, red), pattern (full, empty, striped), shape (oval, diamond, squiggle)
  • when a characteristic matches or is different on all 3 sets you have a match
  • very hard to get a set
  • deal 12 cards
  • makes testing fun

She also introduced the dice game and auction game using decks of cards.

STANZ 2011 Day 1 Review

/ Craig Smith / Leave a comment

STANZThe STANZ (Software Testing Australia New Zealand) 2011 conference was held in Wellington and Melbourne on the last week of August (into September). I was lucky enough to be invited to speak at the Melbourne event by my good friends at Software Education, who were the promoters of the event.  I rolled up on the back of a flight from Los Angeles to Brisbane (and then Brisbane to Melbourne) a little jet lagged, but got heaps from the event.

From STANZ 2011

Here are my notes from day one of the conference.

Am I Creating Value With My Testing?

Jonathon Kohl presented this session, a copy of his video and his slides is available here.

From STANZ 2011
  • ask questions of the CEO about the vision and what the product is supposed to do, listen to customer support calls, talk to marketing, talk to the developers about what bugs they value
  • what are the top 10 things people love and hate about your software?
  • look for efficiency – use checklists instead of test cases, forget about regression testing and use the computer to be more efficient
  • testing is about creating value for the people who matter most, your customers
  • people need an emotional attachment to your product – the share market is an example of a product driven by emotion
  • we need to create value for our customers, but just as importantly for ourselves
  • we can’t just focus on business value – it’s a big stick that will erode morale
  • talk to your customers – what do they need, what do they like, dislike, what is missing?
  • talk to team – what do they like about your work, how can you be better?
  • self evaluation – what is new in the field, am I enjoying work, what do other team members focus on or find things that I miss?
  • avoid blame – excuses rather than finding and solving real problems – “we wouldn’t have this problem if we we doing agile”, “management don’t get testing”, etc… – feels good to say but is not constructive
  • don’t expect tools or processes to rescue you – look out for your own best interests, know the problem you are solving and use the tools/process to solve it and ensure you have a way to measure it
  • the key to creating value is alignment – people in different jobs or teams often have different goals
  • leaders – clearly articulate vision and goals to the testing team and how does that align to our goals for the product and company, leadership comes from everyone in the team, leaders need to manage the politics (an organisation with more than one person will have politics)
  • need to continually inject change and keep people interested
  • people have skills, they are not resources – find your talents and invest in it
  • understand your context – every team will be different
  • tangible quality can be measured by understanding if the stakeholders needs are met and if you are meeting ROI, intangible quality is important and not often taken seriously – would you be afraid if you mother used this, would you like your name on the splash screen?
  • impress the most important stakeholder – you!
  • most people don’t know what great testing is – you can be shocked and appalled by what most people think is good, strive to be better
  • tangibly getting better – learn about planning and strategy and exploit the opportunities, write good bug reports as developers really value this, be good at communicating what needs to be done and where we are going, take more responsibility and display competence in basic technical skills
  • intangibly getting better – be in demand for your testing service, have good problem solving ability
  • use external communities to develop your testing skills
  • work as though your favourite person in testing was coming to visit
  • need to be able justify your work – is your testing defensible
  • use repeatable or intermittent bugs as a clue to something bigger – don’t ignore the anomalies
  • testing is like journalism – need to do crazy things to get the story, move towards the issues, people need the news today not tomorrow
  • need to have a technical curiosity about what is going on in the community – what is coming down the pipe, what are the people that have the ability to change things doing?

Overall this was a refreshing session to see a passion in testing and improving skill, with some excellent sound bytes along the way.

What Does A CEO Want From Testing

Mark Feldman from IV&V Australia delivered this presentation.

From STANZ 2011
  • the CEO is accountable for delivery, protecting his reputation
  • the CEO is not going to check test cases unless you look like a risk (ie. front page of the newspaper)
  • governance is a CEO buzzword that covers a bunch of things
  • need to provide more than alignment – creativity and innovation
  • looking for thought leaders and competitiveness enhancement not 80/20 maintenance work
  • CEO wants creative disruption along with well run divisions
  • testing needs to be proactive rather than reactive
  • have some answers about the cloud – how it affects the team
  • CEOs like ERP because they believe there is less risk

Working With Remote & Distributed Teams

Karen Johnson delivered this session.

From STANZ 2011
  • we are not alone – through Twitter and Skype you can connect with great people
  • understand time zones and calculate meetings for each persons time zone, put the number in the meeting request
  • rotate inconvenient team calls – when people are in very inconvenient time zones such as India
  • recalculate time differences again when people are travelling
  • important to have a usable workable space – particularly when working from home
  • some people have trust issues, so ask what have you done for them to have doubts
  • you just can’t work from Starbucks, could you invite your boss to your home workspace
  • be aware on calls when people are not in the room – handing out documents or drawing on the whiteboard
  • get to know your remote people and get to meet them in person when you can
  • observe with your ears – look for clues to mood and listen for tone

From Jaded To Jubilant: Invigorating Your Test Team

Anne-Marie Charrett delivered this presentation.

From STANZ 2011
  • wanted a team that had long term motivation, so could not motivate with carrots
  • Outliers by Malcolm Gladwell – mostly people are successful because they are in the right place at the right time
  • before you can motivate a team you need to ask yourself how motivated you are – what gets you up in the morning about testing
  • know your testers – give your testers a testing challenge to understand how they test, also understand what they want to get out of testing
  • important that your test team knows that you believe in them and that they are being listened to, important that they get excited about testing again
  • testers are paid to think – test scenarios often go against that
  • think about for every test, how is it adding value to the company
  • testers need to take responsibility – make and defend decisions
  • you sometimes need to let go of your own goals – the team need to feel empowered
  • exploratory testing – the tester needs to decide when it is good enough, this is the way testing is and it is hard to estimate – session based test management (SBTM) and Rapid Reporter (enter your charter/objective – time stamps and records test sessions)

I really enjoyed this session, although it reminded me how many organisations still have large test separate teams.

Test Planning for Mobile Application Projects

Jonathon Kohl delivered this session, based on some Techwell articles (part 1 and part 2).

From STANZ 2011
  • implications – power, display size, portability, connectivity, radios, large number of devices
  • less power than a PC – multitasking can freeze memory, interactions with O/S can have a big impact, kinetic input (tapping, touching, pinching) can have strange behaviours, needed to test using physical movement to replicate locking
  • connectivity – strange things happen when moving between WiFi, 3G and 4G, driving also causes issues
  • distribution – you do not have control of distribution in app stores, read the guidelines and understand the timelines early
  • mobile project issues – time pressures due to market competition, smaller applications, constant change in environments, handsets, software, very programmer centric environments so planning, testing, etc is viewed as a bat anchor, lots of competition, high risk if your application does not work as expected
  • testers need to prove their worth as rigid approaches will leave you behind
  • key is to focus on test execution rather than planning, because everything is going to change anyway
  • need a strategy on how you are going to test, what devices you are going to buy, how are you going to manage the devices/cables because they go missing easily (had to chain cables to a hubcap!)
  • find out strategies that you are targeting so you can procure equipment
  • emulators are useful for basic testing, better to use real device of target platform, developers would have used the emulator anyway
  • supporting IOS 3 to IOS 4.1 resulted in 104 combinations between multiple devices, etc – classification trees are good to explain permutations and combinations
  • automation is still in initial infancy – not as nice as web applications at this point
  • devices are being exploited to do combined activities so need to exploit this in testing
  • we use these devices in environments where we do not use a PC – they are addictive and are part of our lives
  • testing will involve leaving the office and moving around to mimic what the users are doing – determine high value because everybody will want to do this testing!
  • tricky to get devices that you are targeting – standing in line for the iPhone!
  • may need to target different carriers and plans as technologies can be different
  • think about logistics of storage, charging, etc…
  • ergonomics are an issue whe testing mobile devices – shorter work days, can be painful on fingers, people are 25% less productive on these devices than PCs
  • health is an issue because devices are shared and illness spreads fast – hand sanitizers, wiping devices after use, washing hands frequently
  • need to factor in training as there are lots of way to use devices
  • taking screen shots is a lot more painful than web applications
  • usability testing – no standards unfortunately, look for user emotions, perceived lack of performance, one of the most important things on these devices
  • performance testing – no real tools, can jailbreak IOS, some emlators have rudimentary tools, can affect performance of device, use stopwatches, spoof the headers, emulate on a machine using small memory footprints and look for speed
  • security is often a trade-off with performance
  • can automate using emulator in a browser, tools are rudimentary, vendors are clamouring in the space, Opera has a mobile mode

Planning

  • influenced by James Bach’s Statisfice Test Plan Evaluation Model and Test Planning Guide
  • planning needs to be a parallel activity, do just enough in regulated environments, video can be good to replace test cases, need to meet their intent and needs but rather than giving them what they ask for give them something better
  • research your customers for your scenario tests – how they will use the app, are they locals or visitors, is it easy to understand outside context (eg. train schedules)
  • trick – search ” sucks” to find and exploit common problems
  • allow time to keep up-to-date with platform changes
  • remember to test technology like GPS, graphics, camera, video, sound, messaging, data
  • Smashing Magazine – good resource for usability
  • modeling state allows you to get understanding quickly
  • risk vs reward testing – focus on what is value – if you need to demo to get funding, test that the demo will work and not crash
  • quality attributes – HP’s FURPS
  • may need to set time aside for guidance documentation
  • put structure and timebox around exploratory testing so that everybody knows what mission and goal is – look at application from different perspectives
  • express completeness as how have we done and how much we have to go from different perspectives
  • Session Tester – video is good for brining new testers in, easier to digest than written down test cases
  • estimating – use uncertainty models (Software Estimation by Steve McConnell), Galton Estimation tool – like to use P90 – give a range, use S curve to give confidence matched to dates
  • regulators are worried about repeatability – they like formal session based testing
  • adapted James Bach’s testing dashboard

10th Anniversary Conference Dinner

Anders Sorman-Nilsson, author of Thinque Funky, gave a very entertaining and thought provoking dinner speech.

Agile 2011: The Speed To Cool – Agile Testing & Building Quality In

/ Craig Smith / 1 Comment

My presentation from Agile 2011 that I delivered with Adrian Smith called “The Speed To Cool: Agile Testing and Building Quality In” is available on Slideshare.

Ensuring that the approach to testing and quality is understood and appropriately valued in an agile world can be a struggle for many organisations, especially when resources are limited and our customers are expecting business value in a timely manner. In this session we will define what quality means and share a number of tools for measuring it, discuss approaches to improving the skills, empowerment and role of testing in the organisation and share why testing is the coolest role on the team and why it is everyones responsibility.

Some of the comments on Twitter included:

@BrianGress: We tend to test only what we can see. #agile2011 @adrianlsmith

@tonyrockyhorror: @smithcdau Speed to Cool was best talk I’ve seen all week. It will take a mighty effort to top it. #agile2011

Adrian also posted about the talk on the Ennova blog.

AAFTT Workshop 2011 (Salt Lake City)

/ Craig Smith / 3 Comments

Agile AllianceThe Agile Alliance Functional Testing Tools Workshop (AAFTT), held the day before the Agile 2011 conference in Salt Lake City, was once again one of the highlights of the conference. Organised by Jennita Andrea and Elisabeth Hendrickson, it was as always a wide variety of participants with a passion for testing and testing tools. Here are my notes from the day held on August 7, 2011.

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011


Energy Kickoff & Networking

The session was facilitated by Ainsley Nies, and all of the official session notes are stored on the AAFTT wiki: http://aaftt.agilealliance.org:8080/display/AAFTT/agile2011.

From AAFTT 2011
From AAFTT 2011

We started the day with some networking and sharing some areas of passion. Some of these included:

The theme of the AAFTT is: “Advancing the state of the art and the practice of Acceptance Test Driven Development”.

From AAFTT 2011

Ainsley started walking the circle to explain the day and how open space works, but frankly it make me feel a little dizzy! She went on to explain that Harrison Owen invented the open space idea as he noticed the real content at conferences was the passionate conversations. The rules of open space are:

  • whoever shows up are the right people
  • do not hang on to pre-conceived ideas
  • it starts when it starts
  • discussion does not need to be over until it’s over
  • wherever it happens is the right place
From AAFTT 2011

The law of mobility and responsibility (also known as the law of two feet) is if you are not learning or contributing where you are, go some place where you will. Also, butterflies and bumblebees cross pollinate ideas.

From AAFTT 2011
From AAFTT 2011

Finally, we were warned to be prepared to be surprised.

From AAFTT 2011
From AAFTT 2011

Developers are testers and testers are developers – how do we dissolve and combine the roles 

This was the first session that I attended.

From AAFTT 2011
  • there are two mindsets – offence and defence, testers are defence
  • job is not to find defects but to prevent defects – build quality in
  • define quality and what does it mean to us
  • startups don’t often have the problem – multiple skills required
  • what is the biggest impediment – are we missing the skill
  • there is no team of quality anymore – drive quality through the organization
  • functional testers tend to exploratory test and drive from the UI, technical analysts tend to multiple-skill
  • you need to have a team focus and a product focus
  • don’t start with practices but start with a common vision (eg. zero defects)
  • fear of losing identity if you dissolve roles
  • understanding the historical roles sometimes helps understands why things are the way they are
  • need time – Lisa Crispin mentioned that in her company they were going out of business because the system was not good quality, so management were smart to support the initiative
  • helps if everybody on the team has experienced the entire value chain and needs to understand the value of everybody’s piece of the chain – tendency to optimise the piece of the chain you understand
  • developers often underestimate the precision of data and scenarios and developers underestimate the difficulty of some requests
  • personality issues often get in the way
  • mostly about having the right people – need to let some people go
  • we assign labels to roles which create barriers – break down on teams but need to break down at the HR level
  • payroll is also an issue – need to compensate for people taking on more responsibility
  • need to put queue limits on the testing queue to drive behaviours
  • pairing with developer if they do not understand the scenarios
  • some people have the questioning mindset, some have the practical focus – need both to make sure you ship a quality product
  • mini waterfall problem – long tail feedback loop, change workflow that developer needs to work with tester, avoid lean batching problem
From AAFTT 2011

ATDD Patterns

Jennitta Andrea led this session about the work so far in this space:

Last Mile Tools

Elisabeth Hendricksen led this session on tools that are attempting to solve the problem at the last mile.

From AAFTT 2011
From AAFTT 2011
  • NUnit – Liz Keogh – were using Fitnesse but added another level of complication, wrote a DSL that separates tests to make it easier read, WiPFlash is the automation tool, examples are on the website, can call the fixtures from another testing tool like Fitnesse, capture scenarios on a wiki first to get the best out of the automation tool
From AAFTT 2011
From AAFTT 2011
  • SpecFlow – Christian Hassa – similar to Cucumber, scenarios written as steps that are bound to execution, uses Gherkin parser (this is a plus as a number of tools use this)
  • SpecLog – maps of your product backlog, capture results of collaboration with the business (Jeff Patton’s story maps), data stored in a single file, stories are initially mapped to a feature file but ultimately get linked to a feature tree
  • SpecRun is under development currently, not bound to SpecFlow or test runner/execution, currently Windows only
From AAFTT 2011
  • Limited Red – Joseph Wilk – uses the probability of failure to run those tests first in Cucumber, can then get failure statistics at a feature level, working on a refactoring tool at the moment
  • Relish – publish Cucumber features to a website
From AAFTT 2011
  • The Smallest Federated Wiki – Ward Cunningham – JSON for data scrubbing, thin columns to display well on mobile, refactoring is the number one edit so allow it to drag and drop refactor, fit for any analytic or outcome-oriented endeavor, sponsored by Nike, under very early development, meant to take spreadsheet data to the next level
From AAFTT 2011
From AAFTT 2011

Business Rules

Mary Gorman led this discussion.

From AAFTT 2011
  • business rules – conference website has rules, such as group pack for 5 registrations, what happens to the sixth person, what if someone pulls out
  • need to capture these to describe what our system does
  • business rules manifesto – Mary gives a copy to everyone she work with
  • separation of concerns – a rule is separate from the action which makes the process more brittle and more difficult to test
  • rules are a form of requirements and live beyond the building
  • one process is to extract the rules of a legacy system and then the regression tests – code archaeology
  • the business does not always know the rules of the system or how they got there – rules get added to the system over time or evolve and documentation is unlikely to get updated
  • one insurance company had spent $100 million dollars to bring in a business rule engine, returned investment in two years due to being able to be able to look for conflicting rules
  • put analysis of rules in the hands of developers for way too long
  • simplest part of business rules is having a glossary
  • rules engine enables our rules in productions, and use examples to ensure the engine works correctly
  • testing could look like this – given this data when these rules are applied then I expect this output
  • you need both rules and examples to test them – you need enough examples for now, need to be different paths, decision points, infliction points rather than different values
  • examples are not as expressive as arithmetic, but they are not as understandable
  • lots of rules that we do not think of as business rules because they are baked into the process eg. security access, database schemas
  • “business logic is not” (Martin Fowler)
  • you can’t read English as if it were rules, so we need to use examples
  • the worst systems are the ones that do not have a manual override, humans are usually the best at determining this
  • lots of business rules change due to jurisdiction
  • something will always fall to the bottom – rules need to be valued on risk and value – where is the tipping rule
  • rules are the expression of intent
  • Mars issue – crashed, six week window too costly to fix
  • guts to keep it simple – reporting system (Ward Cunningham) – resisted urge to put in a formula system, wait for requests from users, got 6 requests, sold system based on simplicity of the system
From AAFTT 2011
From AAFTT 2011

Other Sessions

As with any conference, there are always sessions you would have liked to have got along to.

Richard Lawrence led a discussion on Static Analysis for Gherkin which turned into a discussion on design patterns for Cucumber.

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011

George Dinwiddie led a discussion about conversations between roles:

From AAFTT 2011
From AAFTT 2011
From AAFTT 2011

My mate Jason Montague led a session on Building Conditions Conducive for ATDD Adoption.

From AAFTT 2011
From AAFTT 2011

Closing Circle

We shared some takeaways in the closing circle, he were some that stood out at me:

From AAFTT 2011
  • issues with dealing of people was a theme
  • what are good ways to express a large amount of test data
  • challenge to get corporations over the hump to release data, plus have good tests and examples around the rules
  • testing needs to be a nation, not just a community
  • it’s time we got more respect in our organisations, it’s time we show more respect to those we work with
  • teams need to dependent on the production of the build
  • federated wikis could help solve the test ownership problem

As for me, my comment was the day had renewed my energy again. ATDD is hard, and as a community we need to try harder.

Podcast

Finally, I recorded a short audio podcast for The Agile Revolution wrapping up AAFTT.

Specification By Example (Book Review & Summary)

/ Craig Smith / 1 Comment

I was lucky enough to be a reviewer on Specification By Example by Gojko Adzic, and the final version was recently released to print by Manning. And I was stoked to see not only my name in the acknowledgements, but that my quote made it to the cover of the book. The following is my brief review and notes from the book.

Review

“I love this book. This is testing done right.” That is my quote on the back cover of the book, and I meant every word of it. Having been a quality advocate in the agile space for a few years now, this is the first book I have read in a long time which had me nodding my head all of the way through, as it resonated with my ideas on how development teams need to reconsider specifications and testing.

The book starts out by summarising why specification by example is so important and outlines some key patterns for success and then, through examples throughout the book, steps through the patterns pointing out the warning signs along the way. The key steps are to ensure the culture is fit, then approach specification in a collaborative manner, use examples and automate and finally evolving a living document / specification.

I really appreciated the fact that the examples were not just the run of the mill greenfield Java web applications that are used in most books. There is a good sampling of different organisations, most of which are using this technique on existing legacy applications on a variety of different platforms. The book is an easy read for the entire team, which means it can (and should) be required reading for the developer, tester, analyst and project manager. I have encouraged many of my teams to take a look at the book, and a couple of my colleagues have indicated this book helped convince and reinforce why this approach is so valuable.

My only concern when reviewing was the fact that the title of this book may not standout to testers and developers (not perhaps as much as Acceptance Test Driven Development or ATDD might). Currently the community has a number of similar approaches with similar names, although I must acknowledge that the specification by example tag has grown on me over the last few months.

The book does not expend much effort talking about tools in this space, by design, I think this fact makes the book more readable and accessible to a wider audience, but that said it suggests to me that there is still a gap for a good text that matches specification by example to particular tools like Concordion, Fitnesse and the like.

Overall, this book is a definite must read for any teams (particularly agile teams) who are trying to balance or find a decent approach to specifications and testing. It is a good balance of patterns and real case studies on how testing and specifications should be approached in an agile world. It would make my list of Top 5 must read testing books and Top 10 must read agile books. And now I know what the proper name is for the cats eyes that are embedded in the freeway!

Finally, I had some other suggestions for summaries for the book that did not make it to cover, but they are just as relevant of my feelings about the book:

  • “One of the best Agile related books I have ever read. Buy it, read it, recommend it to your colleagues.”
  • “This book sums up the right way to attack requirements and testing while delivering to your customer. A must read for all agile teams.”
  • “I loved this book. I could not stop raving about it to my colleagues. It’s testing done right”

Summary

Here are my key notes from the book:

  • a people problem, not a technical one
  • building the product right and building the right product are two very different things, we need both to be successful
  • living documents – fundamental – a source of information about system functionality that is as reliable as the programming language code but much easier to access and understand
  • allows easier management of product backlogs
  • proceed with specifications only when the team is ready to start implementing an item eg. at the start of an iteration
  • derive scope from goals – business communicate the intent and team suggest a solution
  • verbose descriptions over-constrain the system – how something should be done rather than just what is to be done
  • traditional validation – we risk introducing problems if things get lost in translation between the business specification and technical automation
  • an automated specification with examples, still in a human readable form and easily accessible to all team members, becomes an executable specification
  • tests are specifications, specifications are tests
  • consider living documentation as a separate product with different customers and stakeholders
  • may find that Specification By Example means that UAT is no longer needed
  • changing the process – push Specification By Example as part of a culture change, focus on improving quality, start with functional test automation, introduce a new tool, use TDD as a stepping stone
  • changing the culture – avoid agile terminology, management support, Specification By Example a better way to do UAT, don’t make automation the end goal, don’t focus on a tool, leave one person behind to migrate legacy scripts (batman), track who is/isn’t running automated tests, hire someone who has done it before, bring in a consultant, introduce training
  • dealing with signoff and tracebility – keep specifications in a version control system, get signoff of living documentation, get signoff on scope not specifications, get signoff on slimmed down use cases, introduce use case realisations
  • warning signs – watch out for tests that change frequently, boomerangs, test slippage, just in case code and shotgun surgery
  • F16 – asked to be built for speed but real problem was to escape enemy combat – still very successful 30+ years later
  • scope implies solutions – work out the goals and collaborately work out the scope to meet goals
  • people tell you what they think they need, and by asking them ‘why’ you can identify new implicit goals they have
  • understanding why something is needed, and who needs it, is crucial to evaluating a suggested solution.
  • discuss, prioritise and estimate at goals level for better understanding and reduced effort
  • outside-in design – start with the outputs of the system and investigate why they are needed and how the software can provide them (comes from BDD)
  • one approach is to get developers to write the “I want” part of the storycard
  • when you don’t have control of scope – ask how something is useful, ask for an alternative solution, don’t only look at lowest level, deliver complete features
  • collaboration is valuable – big all team workshops, smaller workshops (three amigos), developers and analysts pairing on tests, developers review tests, informal conversations
  • business analysts are part of the delivery team, not customer representatives
  • right level of detail is picking up a card and saying ‘I’m not quite sure’, it pushes you to have a conversation
  • collaboration – hold introductory meetings, involve stakeholders, work ahead to prepare, developers and testers review stories, prepare only basic examples, overprescribing hinders discussion
  • one of the best ways to check if the requirements are complete is to try to design black-box test cases against them. If we don’t have enough information to design good test cases, we definitely don’t have enough information to build the system.
  • feature examples should be precise (no yes/no answers, use concrete examples), realistic (use real data, get realistic examples from customers), complete (experiment with data combinations, check for alternate ways to test) and easy to understand (don’t explore every combination, look for implied concepts)
  • whenever you see too many examples or very complicated examples in a specification, try to raise the level of abstraction for those descriptions
  • illustrate non-functional requirements – get precice performance requirements, use low-fi prototypes for UI, use the QUPER model, use a checklist for discussions, build a reference example for things that are hard to quantify (such as fun) to compare against
  • good specifications – should be precise and testable, not written as a script, not written as a flow
  • watch out for descriptions of how the system should work, think about what the system should do
  • specifications should not be about software design – not tightly coupled with code, work around technical difficulties, trapped in user interface details
  • specifications should be self explanatory – descriptive title and short paragraph of the goal, understood by others, not over-specified, start basic and then expanded
  • specifications should be focussed – use given-when-then, don’t explicitly detail all the dependencies, put defaults at the technical layer but don’t rely on them
  • define and use an ubiquitous language
  • starting with automation – try a small sample project, plan upfront, don’t postpone or delegate, avoid automating existing manual scripts, gain trust with UI tests
  • managing test automation – don’t treat as second-grade code, describe validation. don’t replicate business logic, automate along system boundaries, don’t check business logic through the UI
  • automating user interfaces – specify interaction at a higher level (logging rather than filling out the login page), check UI functionality with UI specifications, avoid record and playback, setup context in a database
  • test data management – avoid using pre-populated data, use pre-populated reference data, pull prototypes from the database,
  • Bott’s Dott’s are the lane markers on the roads that alert you when you move out of your lane, continuous integration has that function in software, run it with Specification By Example and you have continuous validation
  • reducing unreliability – find most annoying thing and fix it, identify unstable tests, setup dedicated validation environment, automated deployment, test doubles for external systems, multi-stage validation, execute tests in transactions, run quick checks for reference data, wait for events not elapsed time, make asynchronous processing optional, don’t use specification as an end to end validation
  • faster feedback – introduce business time, break long tests into smaller modules, avoid in-memory databases for testing, separate quick and slow tests, keep overnight tests stable, create a current iteration pack, parallelise test runs
  • managing failing tests – sometimes you can’t fix tests – create a known regression failures pack, automatically check disabled tests
  • easy to understand documentation – avoid long specifications, avoid lots of small specifications for a single feature, look for higher level concepts, avoid technical automation concepts
  • consistent documentation – evolve an ubiquitous language, use personas, collaborate on defining language, document building blocks
  • organize for easy access – by stories, functional areas, UI navigation routes, business processes, use tags instead of URLs

Agile Australia 2011: The Speed to Cool – Valuing Testing & Quality in Agile Teams

/ Craig Smith / 5 Comments

Agile Australia 2011My presentation from Agile Australia 2011 called “The Speed to Cool – Valuing Testing & Quality in Agile Teams” is available on SlideShare.

Ensuring that the approach to testing and quality is understood and appropriately valued in an Agile world can be a struggle for many organisations, especially when resources are limited and our customers are expecting business value in a timely manner. In this session Craig Smith will define what quality means, share a number of tools for measuring it as well as discussing approaches to improving the skills, empowerment and role of testing in the organisation and share why testing is the coolest role on the team and why it is everyone’s responsibility.

Some of the comments on Twitter included:

@AgileAcademy: Good luck today with your presentation at 11.30am on The Speed to Cool: Valuing testing and quality in Agile teams @smithcdau #agileaus

@vivierose: Waiting for room to fill at @smithcdau was standing room only last year! #agileaus

@adrianlsmith: #agileaus @smithcdau Software is a crime – Testers are detectives

@vivierose: Testers think ‘Everything is guilty until proven innocent’ @smithcdau #agileaus

@adrianlsmith: #agileaus @smithcdau discusses increasing technical skills of testers http://t.co/WPhbDsV

@mrembach: seeing an excellent talk by @smithcdau at the qualityInn #agileaus

@vivierose: Everyone likes to be seen as valuing quality, just like they love kittens, but it’s the 1st thing to be dumped #agileaus @smithcdau

@stephlouisesays: #agileaus loving @smithcdau challenging how well we apply the manifesto to testing. Card wall stages look remarkably like a waterfall… Hmm

@adrianlsmith: #agileaus @smithcdau applies agile manifesto to testing practices – great analogies

@AgileRenee: @smithcdau use the quality assessment tool now avail on the @AgileAcademywebsite http://lockerz.com/s/111080989

@stephlouisesays: #agileaus acceptance test development = specification by example… Beats heavy documentation any day @smithcdau

@timechanter: Fabulous presentation by Craig smith on agile testing. Liking the specification by example stuff. #agileaus

@stephlouisesays: #agileaus awesome session by @smithcdau quality and testing – relevant content, interesting slides (love the pics!) and fab speaker #newfave

@SMRobson: #agileaus @craigsmith finally!! Well done!

@AgileAcademy: 150+ watching great talk by @smithcdau on Valuing testing & quality in Agile teams. Terrific energy & passion. #agileaus #yam

@AgileAcademy: A tester is like Robocop – part man/woman; part machine but all tester! @smithcdau #agileaus #testing #quality #yam

@AgileAcademy: Thanks for the mention about the Agile Quality Practices sheet on our website. @smithcdau agileacademy.com.au #agileaus #yam

@seat_paul: #agileaus very good talk by Craig smith. As he says testers can be very cool!!

@mrembach: @smithcdau great talk Craig. Lots of take-aways

@smamol: Really cool #agileauspresso – beautiful slides: The Speed to Cool – Valuing Testing in Agile Teams http://t.co/1m1jwBL #in

ANZTB SIGIST Brisbane: Agile Testing & How We Need To Change

/ Craig Smith / Leave a comment

ANZTBA couple of weeks ago I had the opportunity to present at the ANZTB SIGIST Brisbane September meeting with my colleague Rene Maslen. Our talk was “Agile Testing and How We Need To Change” and the slides are available on Slideshare.

Some of my other colleagues also presented on the night including Ben Sullivan and Brent Acworth who spoke on BDD and some work they are doing on an open source framework for JBehave and Craig Aspinall who spoke on Automated Black Blob Testing.

Alister Scott had some nice words to say on my presentation on his blog and was also nice enough to take some pictures, which I have embedded below:

AAFTT Workshop 2010 (Orlando)

/ Craig Smith / 1 Comment

Agile AllianceThis year, I again had the great opportunity to attend the Agile Alliance Functional Testing Tools Workshop (AAFTT), the day before the Agile 2010 conference in Orlando. Organised by Jennita Andrea, Elisabeth Hendrickson, Aimme Keener and Patrick Wilson-Welsh, it was once again a wide variety of participants with a passion for testing and testing tools.

IMG_5958

It was also awesome to have Rachel Davies (co-author of Agile Coaching) facilitate the session.

First of all, the the agenda was set:

IMG_5959

We then did a round circle stating our name, role, claim to fame and hopes from the workshop. Due to the high percentage of Canadians in the workshop, our locations quickly turned in Canada-location jokes (OK, you had to be there…)

IMG_5956

Rachel then asked us to re-check the goal of the workshop. She suggested that we should check this with the group, even though it is usually set before the workshop. She asked the organisers to clarify the goals before re-checking what is in and out of scope for the workshop

The goal was: To reflect on the state of the practices and to identify ways to move forward.

IMG_5975

In scope for the workshop was ATDD, marketing, pre-requisites, how we teach, effective ways we talk about terms and use of tools to support practice. Out of scope was TDD, terminology / ontology and test after.

Pleasure and Plains of ATDD

We then broke into a number of groups to draw pictures of our pleasures and pains in relation to ATDD.

IMG_5967
IMG_5974
IMG_5973
IMG_5972
IMG_5971
IMG_5970
IMG_5969
IMG_5968
IMG_5960

This resulted in the following:

IMG_5979
IMG_5978

ATDD Retrospective

We then did an ATDD retrospective, which revealed a rich history!

IMG_5976

Open Space

After lunch, we conducted 2 open space sessions.

IMG_5986

Sales and Adoption Strategies

In the first session I attended was on Sales and Adoption Strategies which was hosted by Mark Levision:

  • Elisabeth Hendrickson suggested that ATDD does not get traction because we need to bring everyone to the table, however it can give power to a struggling agile adoption, we need to understand the fears and concerns of people and we need to have skills to talk to all of the audiences
  • Jim Cornelius said that managers think they want BDD for everything, yet we do not want to express everything in Given When Then format
  • I told the story of using ATDD on non-software development teams, as well as some of our challenges in getting test team buy-in from our testing teams to tools such as Concordion
  • Patrick Wilson-Welsh told a story about him introducing story testing to a team with no skills in testing, yet they were a rock star development team, they used JBehave and a thin TeamCity wrapper too convince them of the value
  • it was suggested that continuous integration needs to be in place to give ATDD adoption teeth
  • Andreas Ebbert-Karroum told a story of success when his team lost track of what they wanted to show at the end of the sprint, the team then wanted to do it, the barrier has been a steep technical learning curve, they are using Robot Framework but are always finding pieces missing (and wonder why in this day and age they seem to be the first person finding these issues)
  • we then heard the example of simple DSL Lookup screen (the one where you type a phone number to see if you can get DSL coverage) at BellSouth which was an example of an extremely simple user interface and lots of business logic, they used Fitnesse, as they found it was hard to test manually
  • Bob Galen suggested the need for effective grooming sessions to start collaboration, create a shared vision at the organisation level not just at the developer level, and the fact you need grass roots and top level support
  • Mark  Levison told all of his executives to read Agile Testing and now he is getting buy in and getting questions about how it might work
  • Jason Montague suggested his success has been through just in time communication, need to teams to get it working and sell the results for you

IMG_5987

Pleasure and Pain with Tools

The second session I attended was on Pleasure and Pain with Tools hosted by Elisabeth Hendrickson:

  • Express expectations in English and relate to underlying specifications
  • unit testing successful because it is in my language, my idea, nobody else involved – all you need to do (sounds easy) is to change the way of thinking, and it works for both test first and after
  • Robot Framework allows a test to be written in text, you can link from the wiki to the version control system and read directly from the trunk
  • test from a photo of the whiteboard using ApprovalTests – put a whiteboard photo in the test directory, get the system under test to create a bitmap, first manually match the photo and once the customer is happy then get the system to diff automatically moving forward
  • Sikuli uses images for tests – push a button that looks most like the bitmap in the test
  • ATDD frameworks facilitate collaboration, like Cucumber, then you need a driver
  • there was some discussion of Fitnesse versus Slim and the fact that the GPL2 code prevents adoption
  • Jason Huggins suggested that Concordion / Robot Framework / Fitnesse are too much bondage, as a developer all I need is assert
  • the question of testing Android was raised, with Jason Huggins suggesting Robotium
  • White is a .Net test framework
  • Given When Then loses detail if business people afe solely thinking that way
  • it is awesome that most of the tools now support tables, given when then and other more recent test tool approaches
  • too many tools are solely web focussed, Robot Framework is one exception although there are others
  • wikis are not comfortable for non-technical users (Elisabeth Hendrickson called this the last mile problem), many tools do not have command completion or refactoring support and output from the tools is not always management friendly (who usually want to know number of tests passed / failed)
  • to get ATDD tool buy in you need to get from scratch to test infected in 20 minutes
  • need to discover you are done sooner than you expect
  • a good analogy is why we have brakes on a car, not so you can stop but so you can go fast, the same stands for tests

Lightning Talks

We then kicked off a round of lightning talks:

IMG_5985

Every Acceptance Test Should Look Like An iPhone Commercial (Jason Huggins)

I had seen this last year at AAFTT, but good to see Hugs was still passionate about it and has some updated examples this year

How Does ATDD Work with Kanban (Matt Philip)

Principle of TD (Llewelyn Falco)

  • need to do something and then verify it (do verify)
  • benefits of a test – specifications, feedback, regression, granularity

Structure of Scalable & Maintainable Test Suite (With Robot Framework) (Andreas Ebbert-Karroum)

IMG_5997

A copy of the model is available in the blog post: How to Structure a Scalable and Maintainable Acceptance Test Suite

  • if I have stable platform and libraries, tests should be stable
  • the resource layer is the layer that should be variable (in the middle)\
  • Given When Then (GWT) tests grouped into themes, import file shield tests from business object changes

Testing Circle (Llewellyn Falco)

IMG_5999

  • discuss story on whiteboard
  • then becomes a story in written firm
  • then becomes code
  • result then looks like whiteboard

WWII & ATDD (Brian Marick)

IMG_6001

  • places with good infrastructure had their infrastructure pounded to rubble in World War II but the USA did not, therefore we have no fast trains, no fast Internet, bridges that fall down, etc.,.
  • all of XP is just what Ward Cunningham does naturally
  • spend too much time figuring out how to survive with legacy code, we should go back to the small!

Open Space Summaries

    • Why Have We Not Yet Discussed ATDD (J. B. Rainsberger) – came to conclusion that it is the same as BDD, How Test Driven Development Works (And More!)
    • Adoption and Sales Strategies (Mark Levison) – small pilots under the covers, ping pong developer tester ATDD, automation directors and managers and developers have different perspectives, talk at an enterprise level, semantic patterns of needing to slow down versus switching everyone all at once, how do we maintain momentum, always feel we are the first person tacking the issue
    • Facilitating Conversations (George Dinwiddie) – describing the problem rather than the solution, getting the language right, personas, myth of single product owner
    • Coding Dojo (Llewellyn Falco) – wrote tests for Yahtzee
    • Start a Business Example to Be Used for Many Tools (Mark Levison) – looked at example that Brian Marick wrote in Ruby, but still needs work
    • Fixtureless Testing (Declan Whelan) – fixtures break cadence in flow, especially when a tester needs to talk to a developer, wiki storage format not optimal as not easy to refactor, should be stored as code so we can refactor and reuse, want language to be close between system under test and acceptance test (use the domain language)
    • Pleasure and Pain with Tools (Elisabeth Hendrickson) – see above for notes
    • Workflow – Activities and Deliverables – built up a workflow

IMG_5988

Wrap Up

We then wrapped up with the future of AAFTT Program, in a discussion led by Jenitta Andrea:

IMG_5984

All in all it was a great day. Lisa Crispin was taking some video so if the quality was good I will help her get it uploaded. I also have more pictures up on Flickr for those that are interested.

Agile Alliance Brisbane: Changing Role of a Tester

/ Craig Smith / 2 Comments

The inaugural Agile Alliance Meetup in Brisbane kicked off on May 13 at the Hilton, Brisbane with a talk by Kristan Vingrys (Global Test Lead for ThoughtWorks) talking about the changing role of a tester. I have had the good fortune to work with Kristan a couple of times this year, so was looking forward to catching up again as well as seeing this talk.

Among other announcements, Robin Mack is the Brisbane liason for Agile Alliance Australia and they are currently looking for volunteers, sponsors and speakers.

The following are my notes from the session, the slides are also available online:

Agile Alliance Brisbane Kristan Vingrys

  • testing – find out what system is and what it does
  • testing is not about putting a quality assurance stamp on software before it goes out the door, its about understanding the quality of the application
  • traditional testing – V model – independent, based on a feature complete system, exit criteria from each phase
  • collaboration – more related to agile – single velocity for entire team (not just developers), co-located and interspersed team, loose boundary of roles, work closely with developers (not throwing information over the wall)
  • should not be iterative testing (testing 2-3 iterations behind), independent or separate velocity
  • stop finger pointing, no longer a gatekeeper
  • tester should be able to check out code, try it out whenever they want to
  • tester should have direct contact with business to break down direct animosity with the developer
  • measured by value added to the team not by defects found or number of tests written
  • test early – make sure the story is testable, do all testing earlier (including non-functional testing such as UAT)
  • Acceptance Test Driven Development – building quality in, not testing it in, this is not just about automation
  • handover – developer should call tester over to ensure that the requirement has been met, give the tester the keyboard for 2 minutes to see if they can break it before check-in
  • exploratory testing – difference between exploratory and ad-hoc, it should have a plan, allows to execute test cases while learning the system, a good approach to test the tests, looking for different ways to learn about and break the system
  • automation – a good way to continually do regression testing, it is not a test strategy, is not just tests (can also be used for automated assisted testing, trawling through log files, etc), automation is code so treat it like code (not using good behaviours will end up as brittle code)
  • unit testing – benefits the tester, know what comes from the developers has been tested to a certain level
  • continuous integration – push the regression suite so that it becomes the responsibility of the team
  • pairing – way to transfer knowledge, to and from the tester, focus on higher value testing
  • co-location – can talk directly to the team about issues or further understanding
  • standups – understanding what the team is doing, what they are working on, know areas of system to spend time on or stay away from
  • automated functional testing – not running the same test again and again and again
  • ratio of testers to developers – hard to gauge, depends on the skills, not everybody on the testing team needs to be a tester
  • test analysis – what are we going to test, understand requirements, architecture, test execution (by a human or a machine), environment management (testing the right thing, a stable environment)
  • “you can’t find a defect if it doesn’t exist”
  • troubleshooting – being able to track down a problem, find the artefacts and hand it off to the correct team to fix
  • every role on the team should contribute to testing in some way
  • metrics that you collect change when you move from waterfall to agile testing
  • generating a test report should not take very long – should be an indication as to whether someone needs to have a conversation with me, answers questions about the state of the test team
  • people make assumptions on vanilla metrics

Metrics:

  • points in test over time – number of stories that are in the testing column on the storywall, indicate if the team is keeping up with testing activities
  • outstanding defects over time  – how defects are tracking over time
  • number of bounces per story point – bounces back and forward between the development and the testing columns
  • not all defects need to be logged – only needs to be captured if it is not going to be fixed straight away, better to spend time on automated test to check defect will not come back rather than logging it
  • be aware of fake defect debt – when a story gets pushed through and defect/s get raised to be fixed later
  • add value by getting team focused on quality

Then there were some questions from the crowd:

  • environments – want enough environments to avoid test scheduling (where possible), happy to do UAT in an earlier environment if it means you are getting early feedback that is useful
  • tester velocity – should be focused on the velocity of the team and the definition of done, investigating using Kanban techniques to set a limit on a column (perhaps for testing to enable team flow)
  • automated tests – depends on where business logic, for AJAX you need to do more UI tests, main issue is getting tests running as quickly as possible and getting fast feedback, prefer not to test via the UI because it is very fragile
  • skillsets – pulling a team together will look at troubleshooting and may need to hire somebody just for this task, a team with developers that strong minded makes it hard for a tester to get their opinions across
  • test plans – does not necessarily need to be a document, the important thing is that there is a shared understanding, should never be signed off because it should be constantly revisited
  • skilled team of testers and developers – who should write the automated tests – look at skills, should not be written by those whp do not understand programming or test analyst – answer what I want to test and how do I go about developing it
  • regression test definition of done – business need to articulate the level of risk they are willing to accept, understand where the business value is in the system, finding a defect may encourage you to spend more time