Google Cloud Data Pipeline Patterns

At the recent YOW! Night in Brisbane (as well and Sydney and Melbourne), Lynn Langit, and independent software consultant and cloud expert presented “Google Cloud Data Pipeline Patterns”.  It was great to meet her, here are my notes from the event:

Lynn Langit

  • Storage is so simple you don’t need to think about it – data lake architecture
  • Simply select how much memory and how many cores and it computes the cost
  • Google is quick because they are laying their own fibre and using their own infrastructure – less than a minute to spin up a VM, with no local data centre
  • Chrome has a RDP client – who knew
  • Google philosophy is you shouldn’t have to hire someone to manage the pricing
  • Bioinformatics is new to the cloud and they are in great need of engineers, Google is having a stab at a genomics API
  • BigQuery is NoOps, has been evolved to be a data warehouse, with no servers,
  • Big Relational if you just want SQL, Amazon Aurora is the fastest growing cloud database, Google has released Cloud Spanner, first database to truly meet CAP theory
  • BigQuery is column store, Cloud Spanner is OLTP
  • MQTT seems to be coming the defacto standard for IoT
  • IoT uses BigTable first and BigQuery second in Google reference architectures, because it is cheaper
  • Python is the emerging ML language for some reason
  • Teaching Kids Programming

Jenkins Gets a Facelift with Release of Blue Ocean 1.0

Jenkins, the popular open source automation server that is used by development teams worldwide for continuous integration and continuous delivery pipelines, has recently announced the general availability of Blue Ocean 1.0.

Source: Jenkins Gets a Facelift with Release of Blue Ocean 1.0

AgilityHealth: Quality

ah-logo-transAt Agile 2016, Sally Elatta from AgilityHealth asked me to do a short video on quality.

You can watch it on Vimeo.

Craig Smith talks to us about quality, technical debt and rework.

csq

Breaking the Cylinders of Excellence (in the Australian Government)

YOW-Nights_Logo_stackedAt the recent YOW! Night in Brisbane (as well and Sydney and Melbourne), Lindsay Holmwood (the Head of Technology at the DTA) presented “Breaking the Cylinders of Excellence”. It was a rare experience to hear the story of how the DTA is using cutting edge development practices to help the government catch up with, and even exceed, the public sector. 

 

  • DTA – aid transformation in government, small agency
  • Delivery hubs in Sydney and Canberra – help identify and plug capability gaps in teams
  • Prototype of how government services could work  gov.au/alpha
  • Digital Service Standard – 13 characteristics on what good looks like in government, useful in organisations as well
  • Cloud.gov.au  – government cloud service, usage growing, continuous delivery pipeline (which is a major change for government who are used to 2 changes per year)
  • The unit of delivery is the team – not about individuals, but the team – borrowed from GDS
  • Government is slow, but government is designed to be stable, they cannot fail, they have characteristics that are resistant to change
  • Myth that organisations must choose between speed and reliability, high performing organisations deploy more frequently, have shorter lead times, fewer failures and recover faster, but they also have a greater profit
  • Want to deliver like a startup but be stable like a government
  • Not a lot of cross pollination between departments currently
  • Read the policy! – quite often the process is not mandated
  • Document what works and doesn’t so it becomes a repeatable pattern – ie. running a meetup inhouse, don’t tell me I can’t do it, tell me how I can run it without being thrown in jail!
  • Stick with technologies the government is comfortable with if you are changing the delivery engine
  • Security matters – prevention is a battle you will always lose, detection is your best defence – aggregate and log in one place, identify threat signatures, etc
  • Embed security people on big services so it is part of the architecture
  • Proactive testing between different governments around the world on similar platforms
  • Simplest security breaches make the most mess – infected excel macros, leaving free USB keys in the foyer that are malware infected
  • Need to put user needs first – alpha mockup using tools like Jeckyll, then beta then live
  • Lots of people strictly interpret the design and delivery guides – they are guides not rules!
  • Create a longer runway by pulling tech forward – turn down the volume of design, turn up the volume of tech
  • If it hurts, do it more often!
  • Fixed cost delivery with agile is a thing, agile is a way to de-risk in the government
  • Don’t put manual testing on the critical deployment path – have special skills on hand for accessibility, performance and security

Atlassian Bamboo 5.11 Delivers Continuous Integration At Scale

InfoQAtlassian, makers of development tools such as JIRA and Confluence, have just released version 5.11 of their continuous delivery tool Bamboo with a host of new features to help teams scale and collaborate. The key feature in this new release is the ability to scale from 100 to 250 elastic build agents.

Source: Atlassian Bamboo 5.11 Delivers Continuous Integration At Scale

Martin Fowler on Microservices, Event Sourcing and Infrastructure as Code

YOW-Nights_Logo_stackedIt was great to have Martin Fowler back in Australia and host him in conjunction with ThoughtWorks for a YOW! Night in Sydney (he was also in Melbourne but I was unfortunately not able to attend).

Martin followed his usual approach of breaking his talk into three mini talks on Microservices, Event Sourcing and Infrastructure as Code. Here are the videos I shot from the session for YOW!

Episode 98: The Heckle Revolution YOW! 2015 Edition

The Agile Revolution Podcast

YOW_2015_Conference_-stacked-PNGCraig joins Darren Rogan, Ben Morgan and Leigh Appel in another special cross over episode with the Hack && Heckle Podcast to talk software development, Agile and preview the upcoming YOW! conference that will be covered by both podcasts.

This episode has been released simulatenously as H&&H 131 – Hack and Heckle / Agile Revolution Discuss the YOW! Conference.

Discussion points included:

TheAgileRevolution-98 (50 minutes)

View original post

Learn Me A Haskell Development Tool Setup

HaskellI attended the Brisbane Functional Programing Group meetup this week, and one of the talks was by Chris McKay on “The Whirlwind tour of Haskell Development Tools”. I always enjoy these talks because often the books don’t adequately explain these things and the experts often assume that these things are known.

Here are my notes:

  • getting started books – Real World Haskell, Learn You A Haskell For Great Good, Thiinking Functionally with Haskell
  • ghc – core compiler, supported pretty much everywhere
  • Cabal – package management system for Haskell, general purpose build tool
  • Haskell Platform – bundle of the previous 2 bundled for your platform, plus commonly used libraries, safe place to start
  • Windows – pressure for the committers to kept his up-to-date, Haskell platform takes care of the hard parts
  • OS X – ghcformacosx for an installer or via brew / macports
  • sandbox – supported by Cabal – will check here before going to the system libraries
  • src/Main.hs is the starting point
  • licences – choosealicence.com or tldrlegal.com
  • default-language – Haskell2010
  • GHCi – REPL console, also use for inspections at the command line, used by editor
  • GHCi-NG – extension with better debugging and extra features, required for some plugins
  • Hlint – like lint on other languages, looks for code smells (patterns of functional rules), good for beginners
  • Hoogle is Google for Haskell – idea of what you want to do and want to know a function to do it
  • Hackage – repository of all Haskell knowledge
  • GhcMod – used by plugins, may need to install it, unlikely to use it directly
  • Emacs – haskell-mode (syntax highlighting, indentation, REPL integration), ghc-mod (access to a whole bunch of tools, nice place to start), flycheck-haskell (syntax checking, can’t use with ghc), company-ghc (auto completion), shm (structured Haskell mode, write code in a tree and it edits for you)
  • https://github.com/chrisdone/emacs-haskell-config – good place to start and works out of the box
  • vim – vim2hs (syntax highlighting), syntastic (syntax checking and lint), neco-ghc (auto completion), ghcmod-vim (compliation errors and warnings, not as good as syntastic)
  • other editors that work OK – Sublime, EclipseFP, Atom (from Github, still immature), intelliJ-haskell, FP Haskell Center

Yehuda Katz on Ember.js (Ninefold Event Brisbane)

ninefoldYehuda Katz is a well respected figure in the Ruby and JS communities, so it was no surprise that it was packed house that turned up to see him last night in Brisbane. And whilst the room was hot, there was a lack of chairs and Yehuda himself was getting frustrated with the feedback and cutoff in the house audio system, it was a great night organised by Nigel Rausch on behalf of the guys at Ninefold.

Here were a couple of items I picked up on his talk on Ember.js:

  • like Ruby and Rails, Ember.js is not optimised for computer science but for developer happiness
  • ambitious app – optimised for building entire applications, long lived applications (like Gmail, lasts on the screen for a very long time) need a sophisticated architecture
  • MVC is not quite enough – model (all sessions), controller (just this session), view (what is currently visible)
  • use HTML and CSS to layout application, is the lingua franca of the web
  • HTML needs more dynamic features – {{ for data binding, # block form in Ember
  • URLs are important – good applications have good URL support – like Twitter or Rdio – primary way that people share information in Twitter and email for example – is the UI of the web, we should use them more universally and use them everywhere

Yehuda then gave a demo. Having not used Ember.js before I was impressed by how easy it was to build URLs and interactivity. I certainly will give it a closer look at some stage.

  • future of the web – take a look at the google project – Polymer
  • evergreen browsers – Chrome started this, once you download the browser that is permission enough to download new versions of the browsers ; Safari has not adopted this and will probably become the next IE6!
  • some good resources include embercasts and Ember 101

There was also an interesting kick off presentation by Toby Hede about some of the cool stuff Ninefold are doing, if for nothing else the animations were amusing!