“BDD In Action” is a book that aims to cover the full spectrum of BDD practices from requirements through to the development of production code backed by executable specifications and automated tests.
Gojko Adzic “does computers” which means he helps people deliver software and he caught up with Craig on a recent YOW! DepthFirst tour of Australia. Gojko is the author of numerous books including “Bridging The Communication Gap“, “Specification by Example“, “Impact Mapping” and “50 Quick Ideas to Improve Your User Stories“.
- XP – started with “Extreme Programming Explained” which was really about developers ruling the world – XP is not dead, it won!
- TDD has crossed the chasm to mainstream
- Sturgeons Law – 90% of anything is going to be crap
- Continuous integration and automation has opened up a world of possibilities
- “Bridging the Communication Gap” – about finding ways to break dysfunctional processes in organisations
- “Agile Testing” by Lisa Crispin and Janet Gregory
- The most valuable companies in the world are software companies
- It’s more about the right people…
View original post 170 more words
I was priviliged to be an early reviewer for Ilan Goldstein’s book “Scrum Shortcuts Without Cutting Corners” and to also count him as a colleague and a friend. The Agile community in Australia is relatively small, so it is always exciting to see a book come out the community, particularly one that is part of one of the leading Agile Series by Mike Cohn. I was also honoured to be asked to provide a quote for the book. The following is my brief review and notes from the book.
The following is my quote from the front of the book:
“If Scrum and Agile were easy, everybody would be doing it! Now that so many are, this book is the virtual Agile coach I wish I had when I was on the early steps of my Scrum journey. Ilan is a world-class coach, and he has packed this book full of ideas and approaches to all of the common questions and issues that are bound to come up as you transform your world of work to Scrum.”
–Craig Smith, Agile coach and editor at InfoQ
The book is broken into 10 chapters, each with 3 shortcuts each. It is written in a very easy to read and laid back style and feels like you are having a conversation with Ilan. It’s organisation means you can read the book from cover to cover of ad-hoc as a reference. It is clearly written with the Scrum Master / Iteration Manager as the primary audience, although it is suitable for anyone on the Agile team.
The key concept I really liked in the book was the notion of a Chief Scrum Master. As organisations begin to scale, it is a necessary role that is not always taken seriously. The book also covers the notion of One Scrum Master = One Team and deals with the ramifications and alternatives to this setup in a reasonable amount of pragmatic detail.
There were a couple of areas I disagreed with, mainly from an hybrid Agile approach as opposed to Scrum
- protected sprint – Scrum talks about the idea of a protected sprint so that the team can focus. I can see some merit for new teams, but in this day and age of constant change I much prefer working on the next highest priority and the kanban approach to focussing on one-piece flow
- sprint lengths – Ilan mentions that “1 week is too short, 4 weeks is too long, leaving me sitting on the fence between 2 and 3 weeks”. I have had a lot of success with 1 week sprints, particularly in business teams. Also, with teams adopting kanban, I have found 1 week sprints a good starting point. I have had some mainframe teams argue that they need 3 week sprints, but I personally find them too long.
- creating and estimating tasks – Ilan writes “break the PBIs into more granular technical tasks and to estimate each task to the nearest hour”. I find this a complete overkill and much prefer to have nicely sized cards of no more than 3 days. That said. Ilan has some good strategies for splitting stories in the book that I completely agree with.
- tracking time – further in the book, it is written “before going home each day, everyone on the development team should adjust the remaining time for any tasks they had been working on that day to ensure that up-to-date data is being fed into the sprint burndown chart.” As per the above point, I find this to be in most cases a useless metric, particularly if you cannot get the estimates from everyone in the team. In this age of #noestimates, I personally see this as an administrative overkill.
Overall, this is a book that belongs on any Agile pratitioners bookshelf as a trusted advisor or useful reference. As you become more experienced, this is the book you will definitely hand-on and recommend to your more novice colleagues as they take on the Scrum Master role.
Here are my key notes from the book:
Shortcut 1: Scrum On The Pitch
- protected sprint allows the developers to completely fix their focus on what they committed to during the sprint planning session
- the idea of personal achievement is overshadowed by team achievement
- ScrumMaster protects the team from disruptive outside influences and removes issues that may be impeding development progress.
- Scrum promotes transparency as a core tenet
Shortcut 2: Fragile Agile
- Scrum is a framework not a method – Scrum is a framework of practices tied together by a small set of clearly defined rules
- a requirement should not be considered done unless it has met the quality requirements stipulated in the definition of done
- sprint zero – artificial term often used to describe the preliminary work that a team might undertake before it is ready to commence an actual sprint. The issue is inappropriate work is bundled into it that delays the starting of real sprints
- poor ScrumMaster selection – unilateral product or technical decisions, micromanaging task assignments, driving the team to put in regular overtime, or generally acting like a tyrant
Shortcut 3: Creative Comfort
- important to ensure that everyone is feeling energized and excited about coming to work
- teams are made up of individuals, and individuals still maintain a sense of self-worth and appreciate having their hard work recognized.
- when our value feels at risk, as it so often does, that worry becomes preoccupying, which drains and diverts our energy from creating value
- basics for the environment: plentiful whiteboard and wall space, plenty of light, open desk space for each team, ample chair space, small round discussion table, readily available large meeting rooms, do-not-disturb zones, private areas, buffering from the noisier elements of the organisation
- developers who are offered their choice of the latest and greatest equipment see it as the most wonderful benefit – keeping developers happy but also improving overall productivity!
- jelled teams are usually marked by a strong sense of identity . . . team members feel they’re part of something unique. They feel they’re better than the run of the mill.
- the more passionate people are about their projects, the more likely they are to fully engage in them each day
- include the developers in some of the early user story workshops so that they not only feel involved in the conception of the product but also get an early idea of what they will be expected to develop and why.
- a warm greeting in the morning, a sincere pat on the back for a task well done, and the feeling that you are part of a unique team can often be all that is required to maintain smiling faces.
Shortcut 4: Masterful ScrumMaster
- begins with the natural feeling that one wants to serve, to serve first. Then conscious choice brings one to aspire to lead
- 7 abilities: leading without authority, bring about change without fear, be diplomatic without being political, behave selflessly without downplaying the role, protect without being overprotective, maintain technical knowledge without being an expert, be comfortable never finishing, next generation leadership
- ScrumMasters form part of a new generation of enlightened professionals. The role of the ScrumMaster is deep and complex and should never be seen simply as a laundry list of operational functions
- although not everyone can be a ScrumMaster, a ScrumMaster can be anyone
Shortcut 5: Rock Stars or Studio Musicians
- we prefer attitude over aptitude
- teammates are expected to step up to the plate and, if necessary, temporarily help carry any additional load in the same way that a fellow soldier will help stretcher a wounded comrade off the battlefield
- constant feeling of safety should be generated from the knowledge that teammates will be respectful even if they aren’t particularly enamored with an idea or opinion
- the effort you’re willing to contribute goes down the more times you hear “You’re wrong!”
Shortcut 6: Picking Your Team Line-Up
- consider many factors when running your own Scrum “draft pick,” including attitude, compatibility, skill sets, team size, ratio of functional specialties, shared resources, and workplace logistics
- everyone is a developer – Scrum views all development team members as equals and labels them accordingly with the collective title “developer.” – it is like being a medical specialist; irrespective of specialty, all specialists are still doctors
- Scrum team size – 5-7
- development team ratios – 3 programmers : 1 tester : 0.5 “deep specialist(s)”
- preferable to limit the work in progress and instead encourage as many developers as practicable to focus on completing a smaller number of PBIs
- in your next sprint planning meeting, agree that one specialist on the team will not work in that specialty for the duration of the sprint. The specialist can advise others who do the specialty work but cannot do the work personally
- have often found it unnecessary (from a requirement perspective) and unrealistic (from a budgeting perspective) to have deep specialists, such as database administrators, dedicated full time to a development team – treat the as consultants and trainers for the rest of the team
- one team = one ScrumMaster
Shortcut 7: Setting the Scrum Stage
- keep the team together for the duration of the project
- a reasonable assessment of startup cost (for a new team member) is therefore approximately three lost work-months per new hire
- work conducted in ad hoc space has got more energy and a higher success rate. People suffer less from noise and interruption and frustration
- sitting together is the most effective way I know to build empathy
- overtime should be the exception, not the rule – symptom of a serious problem on the project,” not simply business as usual
- big believer in initially running a pilot project. I recommend this approach even if the business is champing at the bit to roll Scrum out en masse
- as an industry we have enough evidence that Scrum works; what individual organizations need to learn is how to make Scrum work inside their organizations
Shortcut 8: Plan the Sprint, Sprint the Plan
- by collectively resetting the goals for the upcoming sprint every few weeks, the team can start afresh rather than remain stuck on a seemingly endless treadmill of ongoing work
- ensure that the product owner (with relevant assistance) not only has determined the next priority requirements for the upcoming sprint but also has fleshed them out in just enough detail to allow the developers to get started
- 1 week is too short, 4 weeks is too long, leaving me sitting on the fence between 2 and 3 weeks
- don’t fall into the trap of believing that those who are dedicated full time to the sprint will be able to spend their entire working day on sprint-related tasks
- 7 Ps – proper planning and preparation prevents piss-poor performance
Shortcut 9: Incriminating Impediments
- anything impeding your team’s progress becomes the number-one priority for the ScrumMaster to tackle
- impediments – meetings of large magnitude, illness, broken builds, issues with the tools of the trade, unreliable supplier, unrefined product backlog, absent or unempowered Product Owner, incentive schemes focussed on the individual
- impediment ConTROL – Confirm, Triage, Remove, Outline, Learn
- an obstruction that has stopped progress on a particular task but hasn’t necessarily slowed down overall progress (a block) versus an obstruction that is slowing down the team’s sprint progress (an impediment)
- you want clear visibility of all blocked tasks, irrespective of how temporary the block may be – spin the corresponding sticky-note 45 degrees
Shortcut 10: Structuring Stories
- an epic story is a user story that will take more than one or two sprints to develop and test
- not necessary to have detailed user stories for every requirement only the top-priority items that are going to be worked on in the next one or two sprints
- key is that stories should be written so that the customer can value them
- if you can split a story into smaller ones and it is possible to independently prioritize them, it makes sense to treat them as actual stories rather than tasks
- technical stories are things that need to get done but that are uninteresting to the customer, such as upgrading a database, cleaning out unused code, refactoring a messy design, or catching up on test automation for old features.
Shortcut 11: Developing the Definition of Done
- DoD becomes the governing agreement that guides all developmental activities, clearly stating what is required for a piece of work to be categorically classified as “done”
- we should be regularly inspecting and adapting the DoD
- acceptance criteria or DoD – comes down to whether the requirement is applicable to every user story or to a smaller subset of stories
- be realistic and get the ball rolling with a minimally acceptable DoD that everyone can live with
Shortcut 12: Progressive Revelations
- inspect and adapt the user stories under development on a day-to-day basis throughout the sprint
- typically a hands-on demonstration of the work requiring verification/validation, it usually occurs at the desk of the applicable developer
Shortcut 13: Relating to Estimating
- estimate to make trade-off decisions and help set goals
- relative estimation applies the principle that comparing is much quicker and more accurate than deconstructing
- determine the effort required to complete a PBI using three factors: complexity, repetition, and risk
Shortcut 14: Planning Poker at Pace
- the issue when using Fibonacci numbers is that people can get into the bad habit of equating 13 points to 13 hours, for example
- some teams adopt more abstract classifiers, such as T-shirt sizes – requires the extra step of mapping to a numeric value to enable forecasting during release planning
- ScrumMaster acts as the facilitator throughout and is not involved in the actual estimation
- Reestimation should be required only when a whole class of PBIs suddenly becomes smaller or larger
- important to circulate a small number of reference PBIs that correspond to the point levels in the card deck
- typically advocate removing the big cards (20, 40, 100, infinity) as well as the ½ card from the Planning Poker deck – fewer options equals less analysis paralysis
- group of PBIs will inevitably rely on some of the same important research or technical plumbing – underlying work should be incorporated into the estimation of only one of the PBIs, not into all of them
Shortcut 15: Transitioning Relatively
- identify the smallest user story in the product backlog and designate it to be the initial ½-point story – this approach can certainly work, and it seems straightforward on the surface, but the reality is that it can end up taking considerably more time than you might expect
- map historical work to the new point scale system – identify, sort and stack, sizing up, subclassify, final filter
Shortcut 16: Bah! Scrum Bug!
- an issue is a problem that occurs during the sprint and is tightly coupled to a user story – not a product backlog item
- bug is a bug only if it is identified after a user story has been completed and accepted by the product owner – type of product backlog item
- unless a user story meets the definition of done, it might as well not exist as far as the customer is concerned
Shortcut 17: We Still Love The Testers!
- loss of identity fear – lose QA identity, lack of skills, won’t get support they need
- testers think in alternative problem-solving patterns that are, generally speaking, mutually exclusive to the way programmers think
- pair testing (when a tester pairs up with a programmer) is potentially even more powerful because there is additional scope to encourage functional skills transfer
- there will always be the need for manual exploratory testing that no level of automation is able to replicate
Shortcut 18: Automation Nation
- if your programmers are using TDD as a mechanism to write their tests, then they are not only creating a great regression suite, but they are using them to design high-quality, robust code
- functional testing is also often called acceptance testing. Perhaps one day we will call it user story testing, as the idea is to be able to test and automate the full end-to-end functionality of a particular user story
- integration testing is all about ensuring that new functionality is able to play nicely with the broader ecosystem and not just in isolation
- performance testing is to measure the operation of the product when under pressure, typically brought about by increasing the number of users and/or increasing the amount of data processing required
- recommend running a secondary build in the development environment (traditionally known as the nightly build) that is triggered manually and less frequently. The difference between the CI build and the secondary build is that the latter should be given the luxury of time, and therefore, it can include the full set of tests (including all of the heavier functional and UI tests that take significantly longer to run
- should configure registry settings, initialize database schemas, set up web servers, launch processes—everything you need to build and test your software from scratch without human intervention
- while continuous delivery makes every build releasable to users, continuous deployment actually releases every change to users (sometimes multiple times a day)
- Scrum does not say that you must release at the end of every sprint, but it does say that you should do everything possible to have something releasable at the end of a sprint
- just start somewhere – recommend that if you are new to automation, you allocate a percentage of your sprint capacity to chipping away at it
Shortcut 19: Metrics That Matter
- use them only for good, not for evil
- good metric – used as a signal to help the team identify roughly where things are at and, more importantly, as a guide to help the team inspect and adapt its processes to improve over time
- evil metric – used as an inflexible indicator for micromanaging an individual’s performance over time and, more importantly, for beating people up and killing morale
- meaningful metrics – sprint burndown, enhanced release burndown, sprint intereference, remedial focus
Shortcut 20: Outstanding Stand-Ups
- GIFTS – good start, improvement, focus, team, status
- stand-up ambassadors – act as observers in the other groups to pick up on any potential contention and/or lessons
- if you notice a team member addressing the ScrumMaster without making eye contact with anyone else, a tip is to start slowly turning around or looking up at the ceiling
- daily scrum is a simple concept, but without care it can quickly turn into a daily mess
Shortcut 21: Taming the Task Board
- colorful, sticky-note-adorned board has almost become the symbol of Scrum
- visceral “ceremony” of this movement really appeals to our natural sense of achievement because of the visual recognition of work completed
Shortcut 22: To-Dos for your Sprint Reviews
- sprint review is rarely simple, and in fact, I consider it to be the most delicate session to facilitate
- nothing is more embarrassing than so-called technology experts unable to get the projector to work before the session even begins
- demo of the sprint’s completed work should act as a prompt to encourage a two-way conversation between the business and the Scrum team
- team should demonstrate only stories that meet the definition of done
- acknowledge any suggestions made (no matter how outlandish they might sound) by writing them down on the whiteboard or index cards
Shortcut 23: Retrospective Irrespective
- irony is that this session is most valuable when the pressure is on and/or when things aren’t running as expected
- without an atmosphere of openness, you will never get to the heart of issues; without courage, the team won’t be willing to confront problems head on; without respect, the team won’t present criticism in a constructive fashion; and without focus and commitment, the team won’t care enough to resolve the issues
- recommend a relaxed setting such as a coffee shop, a break room (if you have one), or even the kitchen area
- focus areas – communication, processes, scope, quality, environment, skills
- ensure that all improvement suggestions are documented, but aim to tackle no more than a few issues
- circles technique employs an affinity mapping variation, bubbles technique requires each person to document on paper the three most urgent issues that he or she feels should be focused on pair and keep bubbling up the top 3 issues
Shortcut 24: Risk Takers and Mistake Makers
- to successfully implement Scrum, we must overcome a range of fears
- spend your time and effort helping those who are already enthusiastic
- the few bad eggs are often the ones who are most afraid of exposure
- Scrum turns software development into an open book where mistakes are clearly visible
- when openly discussing mistakes and vulnerabilities is the forging of closer bonds between team members
Shoprtcut 25: Perception is Reality
- you have a role as the ScrumMaster to keep the sponsor from blowing a gasket
- reinforce the positive changes that are occurring while also gauging the sponsor’s current perception of how the project is going
- sponsors become less upset when confronted with problems if they are involved in the resolution
- take the sponsor on a “Tour of the Task Boards” now and again
- never say no is a lesson that I have learned over time
Shortcut 26: Our Lords and Masters
- Chief ScrumMaster – facilitator of the ScrumMaster Community of Practice – training and coaching, challenge existing behaviours, provide and maintain tools, define and refine metrics, help establish communities of practice, ensure consistency, coordinate teams, ongoing Scrum promotion, developing the approach, company-level education, aligning the teams DoD, continual process improvement via Collective Retrospectives, impediment escalation, HR for the ScrumMasters, creating a physical environment conducive to Scrum
- ScrumMaster – process improvement, impediment management, diplomacy, coaching, managing change, maintaining the DoD, maintainin effective communication, updating artifacts, faciliating workshops, faciliating Scrum activities
Shortcut 27: Morphing Managers in the Matrix
- requires visionary leadership that is genuinely interested in encouraging continuous improvement as opposed to playing politics
- intra-organizational coordination, logistical planning, scheduling, and tracking are massive roles and ones that are nicely suited to the traditional project manager
- the manager in Scrum is less of a ‘nanny’ to the Team and more of a mentor or ‘guru,’ helping them learn, grow and perform
Shortcut 28: Scrum Rollout Reckoning
- you are either doing Scrum or not – it is binary
- Scrum is not a mechanical process… it is so reliant on people and culture that even with fantastic quantitative results, the introduction of Scrum may have caused such upheaval that too many people are unhappy and that is not good for Scrum’s long-term survival
- Benefield and Deemer used a simple survey based on the following six criteria: How much the team got done in 30 days; Clarity of goals—what the team was supposed to deliver; Collaboration and cooperation within the team; Business value of what the team produced in 30 days; Amount of time wasted, work thrown out, cycles misused; Overall quality and “rightness” of what the team produced
- Comparative Agility – teamwork, requirements, planning, technical practices, quality, culture, knowledge creation
- 3 questions – are your clients happier, are your team members happier, are your stakeholders happier
Shortcut 29: Eyes on the Prize
- development team has no top-down command-and-control authority that tells the team how to do its work. Instead, a cross-functionally diverse team of people organize themselves in the most appropriate way to get work done
- self-organization needs to be grown and nurtured in a particular environment with distinct boundaries, or it risks spreading wildly out of control
- ScrumMaster should assist management in the selection of team members to ensure that the team has the natural predisposition to self-organize and work together as one
- ScrumMasters should not become obsolete – impediments are never predictable, perfection is impossible
Shortcut 30: Shortcut to the Final Level
- if you remain disciplined, follow the Scrum rules, and adhere to the key practices, then, at the very worst, Scrum will act like a mirror, helping you to uncover the dysfunctions that caused the project to fizzle
- three words to remember from the book: transparency, inspection, adaption
- A Scrum adoption should be seen as a big collection of small experiments, wrapped up in a big experiment
- convince every single person in your organization that “agility needs to be seen as a business strategy and not just something the IT guys do
My presentation from the Agile 2013 conference called “ 7 Deadly Sins of Agile Software Test Automation” is available on Slideshare.
Adrian Smith was unfortunately unable to join me to present this extended version of the talk that he has presented previously at Agile Australia and Fusion.
Automated software testing is a key enabler for teams wanting to build high quality software that can be progressively enhanced and continuously released. To ensure development practices are sustainable, automated testing must be treated as a first-class citizen and not all approaches are created equal. Some approaches can accumulate technical debt, cause duplication of effort and even team dysfunctions.
The seven deadly sins of automated software testing are a set of common anti-patterns that have been found to erode the value of automated testing resulting in long term maintenance issues and ultimately affecting the ability of development teams to respond to change and continuously deliver.
Taking the classic seven sins (Gluttony, Sloth, Lust, Envy, Rage, Pride, Greed) as they might be applied to test automation we will discuss how to identify each automated sin and more importantly provide guidance on recommended solutions and how to avoid them in the first place.
A full house for the talk, some really positive feedback and heaps of questions following the talk, so thanks to everyone who attended!
And here are the comments from the feedback cards that were handed in and nothing negative!:
- Great speaker, am so glad I came here
- Excellent slides, pictures
- Very humorous – kept me awake!
- Super content, gave me some great ideas to take back to my workplace. THANKYOU!
- Great analogy, good tips / info
- By far, the absolute best QA session I have attended this week. I wish my entire company could have heard this presentation. It was engaging, meaningful and practical information that I can take directly back to my colleagues. Well done!
- Very good session, got a lot out of it – got some good direction, fun presentation
- Best session I have attended! Great speaker delivering the content in a very entertaining manner
- Excellent session! Craig is a great speaker, content was SO good! Nice I can go get preso and link to templates and materials
- Pragmatic testing!! 🙂
- Enjoyed the session, this will make me look for other opportunities (tools) for automation testing
- Great speaker! Although new to testing sessions, I gained good insight from this session to put into use back at the business! 🙂
- This was the most insightful and best talk I’ve attended thus far
- Excellent session
The Agile Alliance Functional Testing Tools Workshop (AAFTT), was one again held this year the day before the Agile 2012 conference in Dallas. Despite there being only a small group there this year, the discussion was still open and free flowing under the facilitation of Matt Barcomb and the organisation of Joseph Wilk and Elisabeth Hendrickson.
|From Agile 2012|
We created an agenda for the day:
|From Agile 2012|
|From Agile 2012|
Here are my notes from the day:
George Dinwiddie led this session which turned into a lively discussion! I had proposed what I thought was a related session on Specification By Example and had combined them, but the conversation never really had a chance of getting onto that topic!
|From Agile 2012|
- George expects the business people to be able to read and understand the tests
- non-programmers should not be writing automation, it is the programmers responsibility
- wants to be able to extract working tests into a step definition rather than needing to rewrite in Ruby (George Dinwiddie)
- there is a difference between a specification and testing (Christian Hassa), this is a fundamental shift
- building a DSL – talk about terminology and how we explore our domain – essential step
- you don’t create a DSL, you build it
- not a problem with the toolset but our training in thinking in a procedural way rather than an example way of thinking (Corey Haines
- testers new to automation create large scripts because it’s their only hope in creating some sort of repetition (@chzy), it does not take a lot of effort and most business people are open to working this way
- enable non-programmers by getting them to come work with us every day (Woody Zuill)
- George is helping people make a transition, don’t want people to throw away what they have,
- ideal is not to have step definitions call step definitions, Cucumber community is becoming a community of programmers and are moving away from this
- Robot Framework is more keyword driven, more aligned to non-programmers, you can also make a mess, “it is a double edged sword” (Elisabeth Hendrickson)
- testers like to test the negative cases, should they be expressed at a high level or expressed as a unit test by pairing developers and testers
- if you are testers and you cannot write simple Ruby scripts, then you have no place on my team (Corey Haines), this opinion is probably shared by the Cucumber community (George disagreed…)
- need to use the same design patterns in both Robot and Cucumber (@chzy)
- in an environment that is test centric and BDD, Cucumber is the tool (usually environments with little to no QA), in a business centric environment where you an get the business involved Robot Framework is your tool
- Corey works in environments where there is very few Cucumber specifications per scenario, backed by lots of unit tests
- Cucumber came out of environments where the team is predominantly developers, hence the desire to drill down to Ruby code sooner
- at a large household name company – theyexpect testers to be more technical, happening more in the industry, eliminated the role of tester due to different pay grades (@chzy)
- moving traditional organizations to a collaborative way of working is hard (@chzy)
- wants simple refactorings that are are a bridge from one place to another (George Dinwiddie)
|From Agile 2012|
|From Agile 2012|
Joseph Wilk led this discussion on thoughts that are coming from the Lean Startup movement.
|From Agile 2012|
- at a startup Joseph was at, tests were taking up to 8 hours to run and costs for distributed architecture was high
- Forward Internet (London) – let developers do what they want – by not testing they could be faster and more interactive than their competitors – did testing in Production, a risk that sometimes things could fail – testing should not block deployment
- in some situations it is just worth hacking it out, particularly in a lean startup
- if it is faster to rewrite rather than maintain it, then don’t write tests (Fred George via Corey Haines)
- a big question of this is the skill level of your developers – do you have the skill level to make the choice to not do it (Corey Haines), primary impact of success is the skill level of your developers
- cost of failure?
- complexity is in the eye of the beholder
- Etsy – check error rates in Production (and decide whether to roll back or not)
- Scribd – were having trouble with test speed and found out the developers were scared of breaking the PDF (which is the heart of the business) – they separated the PDF out to speed up development (so developers weren’t worried about breaking it)
- quick delivery – need the quick feedback cycle to make this work, simulate production
- need effective tests – small suite of tests that are 5-10 minutes long
- test what you are most scared of
- Silicon Valley’s issue is hiring – Facebook is stealing developers from Google because they hire good people and enable them to just hack it out
- 2 software industries – small companies and large corporations, very different worlds
- question everything – can only do this if you have experienced it before and understand it
- need a model to help others adopt this
|From Agile 2012|
What Are The Better Ways To Specify Tests With Large Test Data
I unfortunately did not get to this session as it was running at the same time as the No Testing session, but here is the output from that session.
|From Agile 2012|
|From Agile 2012|
|From Agile 2012|
Deliberate Test Practice
|From Agile 2012|
- weekend testing group – choose a target, collaborate on Skype on their findings
- Wikimedia Foundation – looking at ways crowd source testing to test infrastructure (rather than content) – more on this initiative to be announced in the near future
- why is it any different to coding katas? Safer and smaller so you get more practice, practice collaboration too
- organise a community like a book club
- code roast – put the code up and everybody critiques it, be careful not to attach to a person!
- get practice at driving different interfaces – Triangle Tester exercise, parking lot calculator
- hard to practice test automation as it takes a lot of time upfron
- take time to do charter writing sessions or test different items like cheap toys (how would you test this toy?)
- demonstrate value of quality using simulations eg. origami games
- add tests to open source – many of the existing tests are average
|From Agile 2012|
|From Agile 2012|
Holes / Editors
Chzy led this discussion to discuss holes in the existing frameworks.
|From Agile 2012|
- the HTML report from Cucumber is very average – chzy is releasing a new gem based on discussion from a recent testing conference in Austin
- editors – people now bundling these in TextMate, Eclipse and Visual Studio
- JetBrains – RubyMine has gherkin support and refactoring support for Ruby, plus a lot of support for steps in Cucumber
- big picture view of feature coverage – would be cool to map this to Sonar, suites represent functional areas, tags to represent cross-cutting concerns
- SpecFlow is trying to map to story maps using SpecLog
- Relish allows you to create higher level specification of your scenarios
- there is a plugin for Cucumber that allows github integration
- Thucydides has a built in feature coverage report
- Twist has Cucumber support
- test data management – FactoryGirl gem – build up snapshots but want to be able manipulate values down the stack, Faker, ActiveRecord
|From Agile 2012|
|From Agile 2012|
AA-FTT – The Future
Elisabeth Hendrickson led this session as part of her handing the leadership over to Joseph Wilk.
|From Agile 2012|
- mission is to advance the state of the art of functional testing tools
- community building is the best way to spend the money, tool builders and tool users
- Yahoo group is main repository of knowledge, current wiki probably needs to be moved
- need people who have time and energy and interest to take this forward
- biggest issues with wikis is managing all the wiki spam
- have a leadership issue to curate the content and grow the community
- the other options are to create static content, like business analysts and leadership
- important to have a knowledge repository that at least captures outcomes
- would like have more organised meetings worldwide
- is our mandate just functional testing? It has really been just about “agile testing”
- probably need to rewrite the charter
|From Agile 2012|
We finished up the open space by writing what action we were taking from the day and giving them to another participant to keep us honest (mine was to write this post!)
|From Agile 2012|
Another good open space, and good to catch up with many of the leaders in the testing community once again.
A couple of years ago I received an awesome opportunity to attend James Bach deliver his Rapid Software Testing course in Adelaide. At the time I was working with Sharon Robson from Software Education to help re-develop the Agile Testing course for the Agile Academy, and she thought it might be good for us to sit in the back. The two day course was awesome (one of the best courses I have ever attended), although the animated debate between James and Sharon over breakfast in relation to ISTQB is one I will never forget either.
One of the great things about the course is that the notes are freely available from the Statisfice site (slides and appendices). Although it is the insight and passion from James that makes the course extremely worthwhile. Unfortunately I did not earn my “testing stars” from James from this course, but I did learn a lot. I recently dug out my notes from the course and here they are below.
- the secret – “watch people test” – then follow the patterns
- traditionally testers muddled through, as you got more experienced you just muddled better
- there is lots of practices yet to be written about
- James is “walking through an orchard rip with apples”
- “nobody expects a tester to be right about anything” – we are in the evidence and inference business
- tester tip – did you do “booja booja” testing? Your answer should be “not by that name”
- method of concommonant testing – vary x for y (eg. dimmer switches) (John Stuart Mill – A System of Logic)
- you test under uncertainity and time pressure – if not you are about to be laid off!, organisations keep testers at minimum number
- heuristics – essential to rapid testing, eg. walking into a foreign building – “I’ll know it when I see it”
- “creep and leap” – leap is the most outrageous test you can do, creep is to gently shatter the pattern in your mind – creep and leap may fail because you don’t leap far enough or you don’t creep enough
- minimum number of cases has no meaning – infinite – no light flashes when you have finished testing / understand the pattern
- pattern in the test cases is just the pattern in the test cases, not the program
- need to leap beyond imagination
- rapid testing is not about techniques – a way of thinking, a set of skills
- what do testers do? – they are the “headlights of a project”, don’t need testers in the daylight (no risks)
- testers don’t ensure quality of a product, they report the quality of the product
- key definitions: quality is value to some person (who matters), a bug is anything about the product that threatens its value
- testers represent the people whos opinion matters
- defect is a bad word legally; not sure it is a defect when you find it, assumes more than you know (emotional word: bug, issue, incident)
- testing and questioning are the same thing
- there is a motivating question behind each test (if not, a zombie walk)
- first principle – know your mission – allows you to test what matters, gets you more focussed
- we are chasing risk
- quality criteria – what is important, who are users
- curse of expertise – people who know a lot, don’t always see a lot (why you need developers and testers)
- need an oracle / result – otherwise you are just touring (an oracle is a principle or mechanism by which you find a problem)
- rapid test teams should be a team of superheroes – what is your super power? Seek test teams that have variety
- critical thinking – “huh”, “really”, “so” – say these words and you are on the road to critical thinking, you have to make assumptions to get work done
- “huh” = what exactly does that mean?
- “really” = what are the facts, how do we know it is true?
- “so” = does any of this really matter, who cares?
- safety language – this desk “appears” brown, have “not yet seen” a number 127 work, when you see this language your brain keeps thinking about the problem (interim conclusion only)
- if you have stopped questioning you have stopped testing (and turned yourself into a test tool)
- video tape your tests – take notes at timestamps, good for audit when you need that
- The Amazing Colour Changing Card Trick – look from a different angle, view things more than once
- Steve McQueen was a great tester in The Towering Inferno, know the questions to ask to get to an issue quickly
- ask a question without asking a question – make a statement / fact and wait for a reaction
- model it differently – look at it in a different way
- need to have the ability to slow down your thinking and go step-by-step and explain/examine your steps and inferences
- exploratory testing is about trying to de-focus – seeing things in a different way
- there is no instruction you can write down that won’t require some judgement from a human
- irresponsible to answer a question without knowing some context – allows you to establish a risk landscape
- James remembers his testing approach as a heuristic – CIDTESTDSFDPDTCRUSSPICSTMPLFDSFSCURA (his notes go on to explain this one!)
- when you hear “high level”, substitute “not really”
- HICCUPS(F) heuristic, a set of patterns all testers seem can be an answer to justify why something might be: History (something has changed), Image (OK, but something makes us looks stupid), Comparable products (like another system), Claims (said in a meeting, hallway), User’s expectations (do you understand users), Product (consistency), Purpose (why and what is it trying to accomplish), Statutes (something legal), Familiarity (a familiar feeling)
- Oracles – calculator (ON 2 + 2 =4; not heuristic, answer won’t be 5, burst into flames, number won’t disappear), Word saving files (came up with 37 alternatives), Notepad (this application can break, Microsoft suggested it was not a bug)
- Ask for testability – give me controllability (command line version and visibility, text version of display), when developers say no send email so you have documented evidence on why didn’t or it takes so long to test
- ask “is there a reason I have been brought into test this?”
- ad-hoc / exploratory does not equal sloppy
- testing is not the mechanical act but the questioning process, only people who have a goal of 100% automated testing are people who hate to test, don’t hear about automated programming (what about compiling?)
- everybody does exploratory testing – creating scripts, when a script breaks, learning after a script runs, doing a script in a different way
- exploratory testing acts on itself
- “HP Mercury is in the business of avoiding blame”
- script – to get the most out of an extremely expensive test cycle, for interactive calculations, auditable processes
- mix scripting and exploration – what can we do in advance and what can we do as we go, James always starts at exploratory and moves back towards scripting
- use a testing dashboard – break down by key components in the system, all management cares about is a schedule threat so get to the point, count the number of test sessions (uninterrupted block of testing time – 90 minutes) as management understand this (session test management), the key is simplicity, what does management usually ask for / need (usually a different measure), counts give the wrong impression, numbers out of context, number of test cases is useless, use coverage (0 = nothing, 1 = assessed, 2 = minimum only, 3 = level we are happy to ship) and status (green = no suspected problems, yellow = testers suspect problem, red = everybody nervous)
- equivalence partitioning – you treat differences as if they are the same, models of technology allow us to understand risk (eg. dead pixels on a button), critical tester skill to slow your thinking down (is that a button?)
- galumphing – doing something in an intential, over exuberant way (eg. skipping down the street), some inexpensive galumphing can be be beneficial, takes advantages of accidents to help you test better
- An Introduction to General Systems Thinking (Gerry Weinberg, 1974) – basic text of software testing
- many people are hired to fake testing – not to find bugs but to point fingers (“we hired testers”)
- good testers build credibility
- testers question beliefs (we are not in the belief business) – cannot believe anything that the developers tell you
- lots of people can test – like surgery in the 14th century
- reality steamroller method – maximise expenses from the value that they are going to have – record decisions, do your best to help out, let go of the result, write emails to get your hands clean (helpful, timestamp documented)
- get all of the documentation and create a testing playbook – diagrams, tables, test strategy
- The Art of Software Testing (Glenford Myers) – introduced the triangle exercise
- calendar exercise – visualise your test coverage whenever you can, plot times on a grid, bar chart, wheel
- choose a number between 1 and 20 – 17, 7, 3 – 20 is the least popular – what about pi, floating points – choose because they look less random
- anti-random testing – heuristic – every molecule trying to get away from the other molecule – as every test is trying to do something different
- Crazy Ivan Testing Manoeuvre – defocussing approach, looking for approaches you weren’t looking for (The Hunt for Red October)
- finding bugs – testing exhaustively, focus on the right risk, indulge curiosity, use a defocussing strategy
- curiosity – urge to learn something you don’t need to know
- good usability checklist (medical devices) – ISO 60601-1-4
- base testing on activities (what a user does) rather than on test cases
- playbook – table – goal, key, idea, motivation, coverage, etc… – is just a list of ideas
- you can’t check an always – but you can test aggressively for confidence
- stopping heuristic – piñata heuristic (when you have enough candy), cost vs value (when cost exceeds value), convention (what is expected of you), loss of mission, ship
- basic boundary is testing is not one over / one under –> fairy tale boundary testing
- Meaningful work whilst investing for the future on the RallyDev blog
- Coaching Agile Teams on the Scrum Alliance site
- Failure IS an option
- Passionate software testing by TestJutsu and Craig’s Atlassian article
“Management does not know what a system is” – Deming
“The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small, manageable tasks, and then starting on the first one” – Mark Twain
View original post 36 more words
My presentation from STANZ 2011 that I delivered with Adrian Smith and Dallas Thorneycroft called “The Future Tester At Suncorp: A Journey of Building Quality In Through Agile” is available on Slideshare.
When Suncorp started down the path of rolling out its agile program over four years ago, it was viewed by many internally and the industry with much scepticism and angst, yet now it is approaching mainstream adoption in the industry.
One of the key challenges of becoming agile was improving our approach to testing and quality.
In this talk we will talk about why we had to change, why we had to improve the “speed to cool” in relation to testing, our challenges and approach and our blueprint for the “future tester” at Suncorp.
Like our agile journey, our vision for testing has been regarded as ambitious, so join us to hear why we believe raising the profile, empowerment and skillset of testing is critical to our (and your) future success.
Day 2 of STANZ 2011 in Melbourne, here are my notes from the sessions.
The Future of Quality
|From STANZ 2011|
- value of quality is lower than the price of quality – quality is dead
- you may get a quality product (sometimes) but you always get an expensive product
- quality matters for human life, security and money
- systems are designed for redundancy these days so most of the time nothing will go wrong
- people value free over quality
- we live in a world where quality doesn’t matter, we are used to things failing
- restructure what you are doing – don’t focus on catching big bugs, but focus on productivity testing, reduce the cost of development and speed it up
- it is cheaper to fix a bug when the customer finds it now we are moving to the cloud
- organisations are now paying people to find security bugs, people will test for free for a free device, some companies will offer jobs if you find a bug
- need to start communicating value of work in a language people understand – how much did we save?
Test Process Improvement: Testers Get Out Of Your Cave!
Jan Jaap Cannegieter presented this session.
|From STANZ 2011|
- TMMi – maturity framework for testing, public domain, find out how mature your test processes are
- CMMi only has 5 pages on testing
- TMMi has 5 levels – initial > managed > defined > management and measurement > optimization
- start at level 2 when assessing
- results from some TMMi quick scan assessments in 20 organisations – test reporting 30%, test planning 41%, test monitoring 47%, test design 60%, test environment 59%
- test design is probably high because it can be influenced within the testing team, whereas planning and reporting, etc.. require people outside of your team
- testing teams are in a cave, need to get out of the cave and use the rest of the organisation
- stakeholder definition – the most important people at the BBQ – the hold the power, they have mindset and ambition
- it’s all political – politics is a way of life – you need to get in front of the leaders and stakeholders and have political skills
Why Model-Driven Testing is of Great Relevance to Test Managers and Test Analysts
Thomas Hadorn from Tricentis gave this very vendor driven presentation.
|From STANZ 2011|
- Gartner believe model driven testing will become dominant in next 5 years
- capture/replay is too fragile, develop/replay test frameworks are too costly because they need to be programmatically extended
- model driven only one type of test – no scripts to maintain
The Future Tester At Suncorp: A Journey of Building Quality In Through Agile
|From STANZ 2011|
|From STANZ 2011|
|From STANZ 2011|
Testing Skills : How To Find and Develop Skilled Testers
Goranka Bjedov from Facebook led this hands on workshop.
Amongst other exercises, she introduced the game of Set.
- card has characteristics – colour (purple, green, red), pattern (full, empty, striped), shape (oval, diamond, squiggle)
- when a characteristic matches or is different on all 3 sets you have a match
- very hard to get a set
- deal 12 cards
- makes testing fun
She also introduced the dice game and auction game using decks of cards.