Google Cloud Data Pipeline Patterns

At the recent YOW! Night in Brisbane (as well and Sydney and Melbourne), Lynn Langit, and independent software consultant and cloud expert presented “Google Cloud Data Pipeline Patterns”.  It was great to meet her, here are my notes from the event:

Lynn Langit

  • Storage is so simple you don’t need to think about it – data lake architecture
  • Simply select how much memory and how many cores and it computes the cost
  • Google is quick because they are laying their own fibre and using their own infrastructure – less than a minute to spin up a VM, with no local data centre
  • Chrome has a RDP client – who knew
  • Google philosophy is you shouldn’t have to hire someone to manage the pricing
  • Bioinformatics is new to the cloud and they are in great need of engineers, Google is having a stab at a genomics API
  • BigQuery is NoOps, has been evolved to be a data warehouse, with no servers,
  • Big Relational if you just want SQL, Amazon Aurora is the fastest growing cloud database, Google has released Cloud Spanner, first database to truly meet CAP theory
  • BigQuery is column store, Cloud Spanner is OLTP
  • MQTT seems to be coming the defacto standard for IoT
  • IoT uses BigTable first and BigQuery second in Google reference architectures, because it is cheaper
  • Python is the emerging ML language for some reason
  • Teaching Kids Programming
Advertisements

Breaking the Cylinders of Excellence (in the Australian Government)

YOW-Nights_Logo_stackedAt the recent YOW! Night in Brisbane (as well and Sydney and Melbourne), Lindsay Holmwood (the Head of Technology at the DTA) presented “Breaking the Cylinders of Excellence”. It was a rare experience to hear the story of how the DTA is using cutting edge development practices to help the government catch up with, and even exceed, the public sector. 

 

  • DTA – aid transformation in government, small agency
  • Delivery hubs in Sydney and Canberra – help identify and plug capability gaps in teams
  • Prototype of how government services could work  gov.au/alpha
  • Digital Service Standard – 13 characteristics on what good looks like in government, useful in organisations as well
  • Cloud.gov.au  – government cloud service, usage growing, continuous delivery pipeline (which is a major change for government who are used to 2 changes per year)
  • The unit of delivery is the team – not about individuals, but the team – borrowed from GDS
  • Government is slow, but government is designed to be stable, they cannot fail, they have characteristics that are resistant to change
  • Myth that organisations must choose between speed and reliability, high performing organisations deploy more frequently, have shorter lead times, fewer failures and recover faster, but they also have a greater profit
  • Want to deliver like a startup but be stable like a government
  • Not a lot of cross pollination between departments currently
  • Read the policy! – quite often the process is not mandated
  • Document what works and doesn’t so it becomes a repeatable pattern – ie. running a meetup inhouse, don’t tell me I can’t do it, tell me how I can run it without being thrown in jail!
  • Stick with technologies the government is comfortable with if you are changing the delivery engine
  • Security matters – prevention is a battle you will always lose, detection is your best defence – aggregate and log in one place, identify threat signatures, etc
  • Embed security people on big services so it is part of the architecture
  • Proactive testing between different governments around the world on similar platforms
  • Simplest security breaches make the most mess – infected excel macros, leaving free USB keys in the foyer that are malware infected
  • Need to put user needs first – alpha mockup using tools like Jeckyll, then beta then live
  • Lots of people strictly interpret the design and delivery guides – they are guides not rules!
  • Create a longer runway by pulling tech forward – turn down the volume of design, turn up the volume of tech
  • If it hurts, do it more often!
  • Fixed cost delivery with agile is a thing, agile is a way to de-risk in the government
  • Don’t put manual testing on the critical deployment path – have special skills on hand for accessibility, performance and security

Social Good and Flying Robots

YOW-Nights_Logo_stackedAt the recent YOW! Night in Melbourne (as well and Sydney and Brisbane), Daryl Wilding-McBride (the CTO of DiUS) presented “What I Learned while Teaching Kids at Flying Robot School”. It was an interesting story on the importance of social good for those of us in the technical space.

  • Waking hours capacity – families, hobbies, paid work, unpaid work
  • Not all work has equal social impact – pays the bills > interesting > impactful > worthwhile
  • Worthwhile work creates a legacy and passes the BBQ test (something you are proud to convey and recognised as value by the other person)
  • 80000hours.org – the average hours you have from university to retirement, help you decide how to spend that time and be effective
  • William MacAskill “Doing Good Better” – how do you know your social impact is not being wasted – doing good, lean
  • Dimensions for measuring social impact – scale, neglectedness, tractability, personal fit
  • A lot of untapped potential in rural areas
  • Interest in science and maths drops considerably between year 6 and year 9
  • Number of girls continuing with maths after year 10 – 21% drop out, and for boys and girls the percentage has tripled over the last 10 years
  • Flying Robot School – started 2014, overcome barriers for rural schools, free program to lower barrier of entry, blend of technologists and teachers
  • Drones are not only fun but are a self contained package that cuts across science, maths and technology
  • Had lofty goals on topics to teach, but had to prioritise to mix theory and practice
  • Other social outlets – Random Hacks of Kindness (RHoK), Code Club, FIRST Robotics, NodeBots, Robogals
  • We have an obligation as technologists to make things better

Martin Fowler on Microservices, Event Sourcing and Infrastructure as Code

YOW-Nights_Logo_stackedIt was great to have Martin Fowler back in Australia and host him in conjunction with ThoughtWorks for a YOW! Night in Sydney (he was also in Melbourne but I was unfortunately not able to attend).

Martin followed his usual approach of breaking his talk into three mini talks on Microservices, Event Sourcing and Infrastructure as Code. Here are the videos I shot from the session for YOW!

Instantly Better Presentations

IMG_0796At a recent YOW! Night, Damian Conway gave an excellent presentation on “Instantly Better Presentations”. His notes are online and the video of the presentation is below.

My notes from the session:

  • if you need to give the audience bad news, give it first
  • instantly does not mean effortlessly

1. Talk about your passion

  • to feel more confident, you need competence – talk about subjects you genuinely understand
  • seeing someone who is excited… is exciting
  • energy, enthusiasm and passion through your actions and speech will translate to your audience
  • find something in the required topic that gives you passion – even if you loathe the topic or have been forced by your boss to present it

2. Tell them a story

  • our memory is very volatile – stays for 8-10 seconds unless we do something with it
  • 7+/- 2 is horribly optimistic and not backed by research, real number is 4 +/- 1
  • stories are our oldest information processing tool
  • stories have a flow to assist acquisition and memorisation (all our memories are reconstructed from a storyline), have a hierarchy to assist comprehension and recollection
  • tell the historical story or the story of what happened, process or funny anecdotes
  • story is for your benefit to get the sequence and content right – audience don’t necessarily need to know
  • stories make complexity comprehensible, structure recognisable, information easy to remember, make audiences feel more comfortable

3. Don’t search for content, select it

  • what should I say is the wrong question, question you should start with is what could I say
  • humans are good at recognising important stuff rather than recalling important stuff
  • start with a blank sheet and write down everything you know about the topic that you might want to say – stream of consciousness
  • whittle down to 3-5 most relevant and important topics to talk about
  • these 5 points becomes the chapters, so go looking for the narrative that connects them – they may not connect so look for a couple of lesser topics that better connect the 5 important things
  • competency – think about the questions you were asking when you were learning

4. Simplify your slides

  • tools encourage a bad job
  • content matters but not as much as style
  • content is your payload to explode the audiences brains
  • style – the stuff the audience doesn’t see that prevents them seeing what they should see
  • bad style – anything that prevents the audience seeing what they should see
  • a wall of text – technical audience will read everything, regardless of whether it is relevant or not
  • Apple is good at presentations – simple but effective
  • big words – people at the back can still read them
  • slide numbers turn your presentation into a death march – get rid of background, name and title on every slide, get rid of the logos (audience sees salesperson)
  • slide deck is to focus audience on the presentation – if they need context give them a separate PDF or notes
  • each message is a different slide
  • cluttered is overwhelming and as a result they switch off the attention channel as they are trying to read everything
  • show less on more slides

5. Manage the questions

  • a presentation should always be for the benefit of the audience – give them what they need
  • have an explicit questions policy – hold to the end of each topic, end of the talk, or interactive through the talk (can however affect the flow)
  • always be keen to take questions – shows you care
  • make the questions fit in with your question – “that’s a really good question” makes others more comfortable to ask question

6. Animate code simulations

  • explain code temporally, not spatially
  • use animations to reveal information one thing at a time
  • walk through code as an animation and highlighting
  • low tech animations – use the same slide over and over – cell animation
  • don’t export your slides – notes
  • live coding – synthesise, automate or have a partner – need to keep contact with the audience

7. Deliver your message fearlessly

  • use your nervousness – turn fear into energy
  • never give a presentation for the first time – practice it live at least 3 times
  • use an audience image on a big screen