Dave Thomas on Maximum Software Productivity – Breaking The Rules!

Dave Thomas paid a visit to Brisbane to present his talk Maximum Software Productivity – Breaking The Rules! at the Microsoft office at Waterfront Place. He set the scene by suggesting the last time he gave this talk he was innundated with hate mail from agilista and objectas! Dave promised the slides would be posted, but I have not been able to locate them if they have. Here are my notes from the session:

  • tired of need to get agile and need to get objects – both good, but got nothing from it, where is the value?
  • objects good, agile good, but are we any better off?
  • most software late, bloated, poor to maintain
  • business does not define requirements well, don’t engage or talk to the customers (NEHITO visits – “Nothing Ever Happens In The Office”), insist on the industry norms because that is what everybody else is using
  • storypoints – training wheels for people who don’t know how to estimate
  • IT does not estimate well, do not build continuously or automatically test (need to be prepared to write a big cheque), also fixed in technology
  • if not ready to do TDD, just stop! Scrum will only make you feel good
  • need to change the rules to be competitive
  • living in legacy due to OO mud ball – legacy is code where there is no tests (see Brian Foote – Big Ball Of Mud)
  • objects too hard for normal people, first thing to be voted off the island should be Hibernate, objects don’t work for lots of things (queries, rules, transformations, etc…)
  • 80% of objects are CRUD – no objects except for the junk in the middle – just data, no simulations or data model – taking a solution and making it complicated and slowing it down
  • frameworks – so many to choose from, new versions, latest things, lots of dependencies
  • dependency injection is proof of how much we have screwed up objects
  • object libraries are unstable and languages are complex (attributes, generics, concurrency)
  • very few people use interfaces properly and keep them stable release to release
  • little reuse – promise not a reality
  • serialization carries all of the baggage of the objects contained within
  • objects suck up performance and memory – bulky and computationally expensive
  • objects cumbersome and slow for multi-core processors to run efficiently
  • objects are sequential and parallel
  • Java application will be 4 to 5 times slower than PHP, because objects are slower
  • can we shorten software value chain – shorter, faster, cheaper – Facebook built in PHP but the money rolls in
  • agile shows predictability and productivity but zip about quality
  • agility is good, but in Java it is very difficult to change the code quickly
  • scrum increases morale (everybody feels better), but makes no difference to quality unless doing TDD
  • lean thinking – software waste – if not prepared to spend $2 million to do continuous integration and TDD then they are not serious about their software output and quality
  • don’t go to meetings unless it is increasing the bottom line – will it help ship code?
  • how to make zero defect code? Don’t ship anything!
  • lean – simple – why am I doing this? – do we need a new framework? – NO!
  • staff a team with people who have shipped software (have a track record)
  • fix price your consultants and enforce that they make delivery with acceptance tests – pay when they pass
  • reward people on delivery, not how long they work
  • tangible requirements – story on the front, acceptance criteria on the back, start with acceptance tests, they are more valuable
  • envisioning – what the developer sees is not what the customer wants
  • agile great for prototyping – building small requirements on the fly
  • a backlog filled with lumpy requirements will burn out designers and product owners – original Scrum paper said sprint 0 should be 3 months – envisioning
  • architects – not a job, a role – should be able to code
  • some companies pay senior developers same as VP
  • extreme design – four hours to design software and hardware and cost it – after a few, you can get close and find out quickly what you don’t understand
  • serious engineering needs design
  • API first – design the architecture, should always be able to get architecture from the code (push a button)
  • need API’s versioned in the code
  • want to close gap between needs and solution
  • a picture is 1,000 words, a table 200 and a diagram 50
  • table driven programming – easily understood, easy to refactor, easy to consistency check (look for missing data), easy to version and diff (just data), data driven can be changed live in a running instance
  • integration – talk to old systems use RSS/ATOM feed (almost all old systems will give a feed for each transation) then you can talk it without custom api’s, REST/JSON your services or use ODBC as a simple interface (its not just for databases), use mashup tools to deliver an integrated application view
  • use scripts to save time & dollars – most software gets thrown way even before turned on, C# and Java too heavyweight, Ruby, Python, PHP, Groovy, Clojure, can easily leverage cloud services and existing services
  • productive languages – LINQ and Reactive LINQ (Haskell underneath), Erlang (good for swicthing and moving traffic, highly efficient), F#, Ruby, Scala (a better Java), Kleisli (bioinformatics), Clojure
  • Everybody should read “The Wizard Book”, Structure and Interpretation of Computer Programs by Abelson, Sussman, and Sussman
  • hardware is cheap, cloud is cheaper – all interesting data is in memory, databases are just journals and don’t really exist, Google does everything by brute force search (eg. translation), speed now means complicated stuff can be done easier and cheaper
  • data driven – massive storage means we can store it all very cheaply, run smart algorithms to determine best value customers because we have all of the data, recommendation engines (NetFlix), Net Promoter, complex event processing (open intelligence), real time financials (no end of month financials, know state of company all of the time)
  • query oriented programming (QOP), Greenplum, Q, Aleri (basically extended SQL dialects with functions) and other functional languages
  • array VM’s – better than object VM’s – always boxed, simpler garbage collection, support all data types (array of stuff is an array of stuff), VM can be small (just an interpreter), arrays are column stores already and trivially serialised, take less space and programs concise and compact
  • best practices for functional programming – agile , refactoring tools, FindBugs / Lint
  • challenges for functional programming – get out of math phobia, different way of thinking, need to write literate and understandable code, think in functions
  • yesterday and tomorrow is always wrong – if you get more bang for buck and competitive advantage, why use existing technologies, if you believe IT is strategic you need to do something strategic with it and dare to be different
  • lost of old things work, lots of new things work well, get out of agile and OO box

Barcamp Brisbane IV Wrapup

Barcamp BrisbaneLast weekend I got along to Barcamp Brisbane IV (held at the East Brisbane Bowls Club), and once again it was a worthwhile meetup of locals willing to share their skills with others.

From the lightning talks that I attended:

Speed Networking

One minute to introduce yourself to someone you don’t know. Worked well, although I knew more people this time around (after last BarCamp and other meetups).

Search Engine Optimisation (SEO)

Michael Smale led this discussion on SEO (unfortunately it started a little late and lost a bunch of attendees, including myself, at the end due to a Google Wave presentation following it!). My notes from the session:

  • SEO is optimising for Google (& Yahoo!)
  • 9 out of 10 people search for content, very few click the sponsored search
  • keywords – on page (to help Google index) and off page
  • stem analysis – trunk and branches (eg. golf and balls, clubs, shoes) then leaves (buy golf shoes and Brisbane) – before SEO, find out what target audience is looking for
  • tools to analyse keywords – Google Adwords Keyword Tool (slightly out of date, monthly), worldwide but narrowed down to regions
  • to know backlinks, etc – Traffic Travis, Market Samurai (free and paid version)
  • not your trunk and branches, but for your leaves you may want to buy keywords, you can optimise different landing pages (separate URL but not a duplicate of pages as Google will drop prioritisation)
  • car rental very competitive for SEO
  • Google Trends for search – can see if things are trending up and down or compare
  • on page optimisation – Firebug for Firefox – drill down and inspect code, JavaScript debugger
  • YSlow – tell you how page is loaded and report on how to optimise page loading
  • each page needs to be optimised with its own title – what’s in the title is what the link on Google says
  • meta description after link is the blurb on Google – not visible to users on site, Firebug will help you see competitors meta tags are, but will not get you up in the ranking
  • meta keywords – does not mean anything anymore
  • care about content on site using LSI (Latent Semantic Index)
  • link text important, add href no follow so Google will ignore

Google Wave

Paul O’Keeffe and Steve Dalton led a live demonstration of Google Wave.

  • collaborative tool, still in preview, crashes, interface still weak
  • proliferated from developers in Google sandpit, only give 8 invites to each user
  • a wave is a single collaboration / conversation
  • has Gmail feel, add and save searches, folders, etc…
  • have a wave inbox
  • with:public – see any waves that are public
  • search with:public gardening
  • new wave by default is not public, add public@a.gwave.com
  • to start, drag contact in, give wave a name
  • drag and drop seems to depend on Google Gears, works out of box with Chrome
  • bots and plugins eg. pirate speak or add a Google Map / Twitter in
  • open source version of Chrome – Chromium
  • Sweepy bot – remove the empty conversations
  • can mute conversation and replay, has version control so you can see how it was and then fork it off

Business Structures

Malcolm Burrows from Rostron Caryle gave this presentation. I hope the slides are made available, as this was a large topic for a 20 minute slot. These are my notes but should not be relied upon an advice or for accuracy!

  • sole trader – liable for own debts, etc, house on the line, no protection freom risks, okay if you have little risk
  • partnership – not sure why anybody would do this now, agreement and governed by those terms, in Queensland partners are liable for acts of the other, everything has to be tailored, risks
  • company structure Pty Ltd – level of risk reduction such as corporate veil, shareholders only liable for the capital put in as long as you don’t do stupid stuff like trading insolvent, as directors do not profit from position of power, need to disclose, 12/20 rule can’t make more than 20 offers in 12 month period, no more than 50 shareholders, replaceable rules (eg. regulate by ASIC or regulate yourself in your constitution)
  • company structures – Limited – Public – all of baggage of public company without the good stuff, horrible!
  • trust – discretionary and unit
  • joint ventures – used a lot in mining, in IT where people agree to do stuff, like a trust is a feature of contract, rights of joint ventures can get very long
  • income distribution structure and IP protection structures
  • options for IP – spin out trading company, spin out company owned by trusts, spin out company licences another

Smile! Say Cheese!

DJ Paine from Studio Promise dropped by, and offered attendees a free portrait, which I certainly took advantage of. Just wished I had of known, and I would have had a shave and worn a nicer shirt!

All of the shots from the day are here and if you need professional photography, support those that support BarCamp!

Symphony – Open Source Content Management

Allen Chang and Alisair Kearney led this session on Symphony:

  • originally called TypeWorks
  • 2.0.6 out now, 2.1 on the way
  • uses XML as data format, output format standards compliant
  • Drupal and Joomla! cores are huge, they wanted a small footprint and control over data structure
  • use XSLT to transform XML to any format you like (eg. HTML, CSV, JSON, etc..)
  • native intergration REST API for Twitter, RSS, etc…)
  • uses open standard templating language, as per all CMS systems
  • a number of data sources for which you can apply rules
  • around 8,000 members, 10% of these contribute
  • users include Australian Museum of Democracy, Heineken and City of Westminster (London) amongst many others
  • ensemble – fully functional website package, Symphony itself is an ensemble

Agile Overview – The Three T’s

It occurred to me in the speed networking session that a number of attendees did not know what this agile hype was about, so I decided to on short notice to propose the talk I gave at Agile Australia 2009 to try and give that overview. Not sure if I succeeded, but got some questions afterwards nonetheless.

Had to laugh at one of the tweets from @funkygorilla (Simon Griffiths): “Agile web development in a 10 min presentation. That’s agile!”

Overview of Agile 2009 / Agile Australia 2009 / AAFTT Workshop

A couple of people decided they wanted to chat about some of the learnings and trends from the conferences I had recently, so a couple of us sat around and chatted about agile testing mainly.

New Hotness

Greg Luck led this discussion as he mentioned to me he came to Barcamp to hear about the new hotness. He has written the notes, but here were the notes I was taking at the discussion:

Wrapup

Paul and Steve reminded everybody about the Queensland Legion of Tech and Greg Luck announced the inaugural Brisbane Jelly (adhoc working together at a location)

Barcamp Brisbane III: The Search For Flock Wrapup

Barcamp BrisbaneHere is my better late than never wrapup of Barcamp Brisbane III (held last weekend at the East Brisbane Bowls Club), a worthwhile meetup of locals willing to share their skills with others.

Discussion at Barcamp Brisbane III

Discussion at Barcamp Brisbane III

From the lightning talks that I attended:

Speed Intro

A new concept was the speed introductions, one minute to introduce yourself to someone you don’t know. Worked quite well, met a bunch of new people and got a good vibe of the different passions of the attendees.

Favourite Cloud Applications

Session presented by Michael Rees.

None of the applications were particularly new to me, but these were my takeaways for more of a look:

  • Gliffy – online diagram and flowchart editor
  • Slideshare – for uploading of presentations
  • Prezi – requires Flash, but is a big space to show text and is a new and interesting way to show presentations
  • Delicious – not new, but Michael uses it as a home page
  • Archive.org – from the guys that brought us the wayback machine, they have a service that allows you to upload audio and video maintaining the original resolution (although the upload is quite slow apparently)
  • Evernote – I have used OneNote for convenience of late, but a place to store and search for notes on multiple platforms and phones, this is a service that Michael pays for as well
  • Online Storage – LiveMesh is a favourite, gives you 5GB that syncs anywhere. He also mentioned Live Sync P2P, SkyDrive and Amazon S3

Introduction to Git

Attended two sessions on Git, a discussion and then an online overview.

  • A local repository, distributed
  • Competitors are Mercurial and Bazaar
  • Git is not as good on Windows environments right now
  • Github uses for public hosting
  • Gitgui is a an interface to Git, amongst many others

This is certainly the next generation of version control, but I have concerns on how to get this working in the enterprise especially since I have enough trouble convincing people to commit let alone to commit often. Can see its potential for open source and independent or small developers however.

Groovy, Griffon & Grails

Paul King and Bob Brown gave a good introduction to the G3 technologies. I was especially interested in Griffon, since I hadn’t spent any time looking at it previously.

Government 2.0

Was interested to listen into the discussion about PublicSphere / Government 2.0 by Des Walsh, and the opportunities it may present. Des has posted a more in-depth post here: http://deswalsh.com/2009/07/19/government-2-0-at-barcamp-brisbane/

Drupal Hosting

Discussion about Drupal Hosting:

  • Open Atrium – intranet website incorporating wiki, forum, internal-twitter – theme around existing modules
  • Aegir – Drupal Hosting System
  • Suspect that much like Linux, we will see many distributions in future
  • Acquia are the Red Hat of the Drupal world
  • Drupal 7 has gone full TDD with 80% coverage

Fish Shell

An online demonstration of the Fish Shell, which can be best described as a shell that adds a bunch of added functionality to bash, such as better history and visualisations.

Conclusion

A good way to get a launch into some new and interesting technologies and meet some new people. There is apparently another planned before the end of the year.

How to set JAVA_HOME in Ubuntu

JavaI can never remember this, so for mainly for my reference and anybody else who may Google this.

  • Get Java via the Synaptic Update Manager
  • sudo vi /etc/bash.bashrc and add the following to the end of the file:
    JAVA_HOME=/usr/lib/jvm/java-6-sun
    export JAVA_HOME
  • Close your command window and reopen
  • echo $JAVA_HOME to confirm it is set

Wrap up from CITCON Brisbane

CITCONI attended the CITCON (Continuous Integration and Testing Conference) in Brisbane last weekend and had an awesome time discussing a range of topics with the most passionate in this field.

Deep in CITCON discussion

Deep in CITCON discussion

I have added my notes to the conference wiki,but my takeaways from the sessions I attended are:

Elements of Enterprise Continuous Integration

Jeff Frederick led a discussion based around the Elements of Continuous Integration maturity model:

  • for teams that are already doing continuous intgration, it gives you a target to obtain
  • is obnoxious after insane (where to for teams that are already at the top level)?
  • tooling makes continuous integration trivial now (when Cruise Control was released many people thought it crazy that you might build on every release, not its a given)
  • the model was developed because people assume what is possible is based around their personal experiences
  • the model shows the industry norms and targets, and if your team is not at these levels you are behind the curve

The discussion branched out around the following ideas:

  • scrum does not prescribe continuous integration, but continuous integration is a good development technique
  • that it should be acknowledged that there is a difference between project builds and full product builds (which can take days)
  • I raised the idea that perhaps there should be an element around team principles, and that things like performance (and more importantly, the team realisation that performance should be monitored and improved) should be an indicator to maturity (there was much debate about this!)
  • a number of industries potentially have their continuous integration processes audited, such as defence, gaming  and financial organisations that have Sarbanes-Oxley requirements
  • it was acknowledged that most large organisations have teams at different levels on the maturity scale (this is certainly my experience)
  • dynamic languages and don’t really build or deploy. This then raised discussion that dynamic languages are not compiling as opposed to not building, and that in many cases one man consultants can manage their deployment process in a much more lightweight manner
  • parallel to CMMI, is there a payoff to getting to insane?
  • maturity is often determined when we move from dropping code to testers versus testing the development build (where testers are writing the code while the code is being developed)
  • where is the line that determines that the build is complete? It should be the entire team, not just the developers or the QA team
  • the QA team is traditionally where much of the auditing happens, therefore many testers are reluctant to change as they have built up processes to deal with audits over a number of years

For the record, the cutting-edge agile teams I have worked with over the last few years were at the following levels:

  • Building (Intermediate)
  • Deploying (Intermediate)
  • Testing (Insane)
  • Reporting (Intermediate)

We still have work to do!

Virtualisation & CI

I used the “law of two feet” during this session, but was interested to hear that many people are using virtualisation very effectively in their test labs, and that it makes getting environments and data ready for testing much easier.

Long Build Times

The discussion was well-established by the time I got to this session, but some of the key points for me from the discussion were:

  • question as to when static analysis checks should be run in the build – the consensus that running them first means you get the quickest feedback
  • longer builds should be run nightly so as not to hold up developers
  • prioritising build queues or using different machines sounds like a good idea, but nobody is doing it
  • you can reuse functional tests for performance tests, but targeting specific tests seems to work better
  • Atlassian use JMeter for performance tests and have a variety of Maven and Ant builds, but use Maven for managing repositories
  • Ant is still well regarded, Idea support is awesome, many people do not understand the power of custom ant tasks or the ant idoms
  • the build should be regarded as part of your code
  • discussion about using a Java build tool, such as Hammer and why we can’t articulate why it seems wrong
  • not enough people understand Maven, usually there is “one guy” on the team
  • Vizant is a good tool to graph the build
  • EasyAnt combines all of the Ant idioms plus Ivy

Is Scrum Evil?

Jeff Frederick led a discussion that he is led at previous CITCON’s around the world

The team first debated why Scrum is Evil. During this discussion I really thought the whole agile movement was done for. Jeff asked the group to finish the sentence Scrum Is Evil because…:

  • it becomes an excuse
  • that’s not Scrum
  • tested as a silver bullet
  • hides poor personal estimation
  • master as dictator, project manager
  • two days to agile master certification
  • daily standup equals agile
  • agile by the numbers
  • is dessert first
  • you lose the baby with the bathwater
  • Scrum teams don’y play well with others including customers
  • it has certification
  • is the new RUP

Jeff then proposed a way to think about Scrum adoption as outlined in Geoffrey Moore’s “Crossing The Chasm”. The early adopters had success while the early majority are putting their faith in training everybody as Certified Scrum Masters (a problem that appears to be a far greater issue in Europe than Australia).

Then, just as though all hope had gone, Jeff asked the group to finish the sentence Scrum is Good beacuse…:

  • people can get it
  • an easy introduction
  • a good starting point
  • it is better than a cowboy shop
  • people can actually follow it
  • improves visibility
  • blockers are highlighted
  • testers can start work early
  • provides a forum for communication
  • can engage customers in a much richer way
  • states there should be a facilitator
  • results focussed
  • makes everybody responsible for end result
  • better communication from end result

The key outcome by the group was “Scrum is not evil… people are evil”

This was a great way of trying to tease out the issues and advantages to using an agile process and one that we may be able to use in the enterprise with teams who have been on training but appear to be resistant to change.

Seeding Test Data

A good discussion about ways to seed test data

  • Erik Petersen introduced the group to GenerateData.com, a free site that generates real adddresses and data based on factors a random amount of times that you can then inject into SQL – the site looks awesome!
  • others in the group mentioned LiquiBase that can be used to version the database, is designed for database management but can be used to seed data
  • Unitils is used to setup data scripts
  • one suggestion was to build a reset database function into the system
  • HSQL (Hypersonic) is a good way to create databases from Hibernate, in memory

The discussion got a little more generic and talked about:

  • Roo, a java Grails-like platform
  • WebObjects by Apple is a lot better than it used to be

Extending CI Past Traditional Dev & Release Process

I led this discussion, and whilst it focussed mainly on different usages that I have been involved with (with assistance from Paul O’Keeffe and Paul King), we also had a good discussion about Tableaux and the build process at Atlassian.

Conclusions

A great open conference attended by people passionate enough to give up their Saturday to talk about continuous integration and testing.

ASWEC 2009: Experiences from Agile Projects Great & Small

My presentation from Australian Software Engineering Conference (ASWEC) 2009  that I delivered with Paul King called “Experiences from Agile Projects Great and Small” is available on Slideshare.