Gavin Terrill's Blog

RESTful API and Deploying Updates Safely

I've been thinking about this for a while, but haven't really come up with a great solution so I'm looking for ideas on how others have approached it.

Say you have a web service with a RESTful API. The service provides a centralized place to capture straight through processing exceptions and handle their remediation. There are loads of other loosely coupled applications in the environment, all with the need to be able to post exception notifications on a more or less real time basis.

Now, the exception service needs an update - there are some bug fixes and a few new shiny features we want to push out. It is early days but you have 2 of those applications using the service already. The update only takes 30 minutes or so, but during that time the clients won't be able to post any new exceptions.

So, the conundrum...

How to deploy the new service without breaking client applications?

Being based on HTTP, the client will get a 503. Are they expected to store their payload and try again later? Seems like the client is taking on a lot of burden responsibility here, and what if it is time critical?

Or, do you forgo the synchronous nature of a REST call and put in a message queue? Less hassle for the client app but they do need to install a messaging driver now. At least Ops can look at the meesage queue and see what is backing up. But hey, isn't this still a single point of failure though?

Perhaps the exception service needs to be clustered so you get fail-over during updates. Trouble is the update needs a database schema change. How will it work if the new instances are expecting the new schema but the old instances are still serving out requests?

Am I missing something?

Getting Back to Basics

I've been hobnobbing it a bit with the locals lately.

First, I checked out the Ruby on Rails Sydney meetup a couple of weeks ago. An ex-colleague was doing a lightning talk on his experiences attending startup school in Chi-town. I wanted to come along and lend support. The talks were on emerging tech including Ansible which looks pretty cool from a devops point of view, and another on "DCI: Data, Context, Interaction". DCI is kinda like the command pattern from the Java days if you squint at it sideways. Awe, Ruby is getting all growed up. Met a lot of smart, passionate and interesting people that night. Seems to be characteristic of the Ruby community.

Tonight I went along to Silicon Beach Drinks after partaking of a couple of celebatory drinks at the Thoughtworks Christmas party (thanks for the invite Kelly). We were a bit late, but I met some cool folks and would definitely like to come back. While pretty laid back, I got the sense that there are some serious folks just getting on and building out product they believe in. Would have loved to have chatted more.

Quite a few people were interested in talking about building Apps on the Shopify platform. Might tee up a session at an upcoming RoRo meeting to talk to that if they'll have me, but in the mean time I guess it is fair to say that I didn't realize how much I missed getting out there at a grass roots level and meeting people who with a passion. Sign me up for more.

 

Agile Findings from the GAO

According to the GAO, US Federal Agencies spend $76 Billion bucks annually on IT so to reduce the risk of budget overruns and delivery delays Agile is now recommened by the OMB as the preferred development methodology. They've just released a freely available report on what practices have been effective, and what the challenges have been. These challenges tend to not get talked about so much so it's great to see this commentary.

Some of the biggest challenges identified were:

• "Teams having difficulty collaborating closely": Getting the mix right with pair programming, as well as personality clashes when everyone is working so closely together, spring to mind as examples of where this comes up
• "Self directed work" and "difficulty managing iterative requirements": Particularly when transitioning from a traditional project manager led approach. The Scrum Master is critical in fostering and shepherding this cultural change. Guidance around roles and responsibilities was also called out in this area.
• Setting up technical environments.
• Status and reporting tracking.
• Bringing along the rest of the organization - e.g. procurement and compliance checking.

Very consistent with what I've experienced in both small and large organizations.

Phusion Passenger RewindableInput Error Parsing with REXML

Ran into this funny little bug and didn't find too much about it on the interwebs, so for prosperities' sake here are the gory details...

The error is "RuntimeError: PhusionPassenger::Utils::RewindableInput is not a valid input stream. It must walk like either a String, an IO, or a Source."

The offending line of code occures when using REXML to parse an XML document as part of a webhook controller:

doc = REXML::Document.new request.body()

Under the covers, it is this line of code that generates the error:

/opt/ruby-enterprise-1.8.7-2011.03/lib/ruby/1.8/rexml/source.rb:21:in `create_from'

This is a rails 3 app, running under Phusion Passenger 3.0.10 and REE 1.8.7.

The solution, courtesy of jsierles, is to fixup Passenger with an ImprovedRewindableInput: https://gist.github.com/281126 Basically, create a /config/initializers/phusion-rexml-phixup.rb file and paste the code from that gist.

 

 

Remembering Carl Adam Petri

Just found out that Carl Adam Petri passed away July 2, 2010. Petri invented the "Petri Net" - a tool for modeling process flow that is used in a wide variety of applications, from circuit design to traffic light coordination to business process management.

I was introduced to Petri Nets by a grizzled veteran software developer back when I knew everything. I looked into it and was struck by the simplicity of the model, plus the mathematics. You see, Petri Nets can be formally verified so you can prove that your flow is "correct", meaning it will complete and doesn't get stuck in an endless loop for example. You can even do simulations with it to help predict capacity requirements.

The basic idea is you have a diagram with boxes that represent "doing something", called a Transition. Transitions are connected to other transitions via a circle, called a Place. The connecting lines show the direction of the flow and are called Arcs. A network has start and end places and can not have cycles (i.e a Petri Net is a DAG).

Now, to model the process flow you place a Token in the starting place. The token represents a unit of work and moves through the network of transitions and places until it gets to the end place. As each transition fires the the token moves to one of the output places.

 

What is interesting is that a place can hold more than one token, so you can add another token to the starting place before the first one has reached the end place. This is why a Petri Net is good for modeling concurrent processes.

Wil Van Der Aalst has written some great papers on applying Petri Net theory to business process management. A few years ago I was inspired by one of his great books and created a Swing based Petri Net editor. The code has long since been lost unfortunately, but I remember feeling quite chuffed at being able to simulate and optimize an insurance claims process by playing with the capacity of certain transitions in order to improve overall system throughput. It was very Kanban like.

Petri was a true pioneer of our industry, and the amazing thing to me is that he came up with the idea when he was just 13! 

37 Signals Interviews Shopify

There is a nice Q & A with Shopify's CEO Tobias Lütke over at 37 Signals talking about the history, culture, product and philosophy of the company.

It's a great, motivating success story, and I'm proud to have played a small part of that by making the introduction to John Phillips after I had experienced the product and was so impressed with the company and technology choices:

We ended up taking an investment from an angel investor, who just called us up one day because he heard about our company. This is certainly an outlier, but taking this investment from John H. Phillips was probably the best decision we ever made. Apart from allowing us to meet payroll for the first year, he has spent countless hours teaching me all the things I needed to know about running a company. Without him, we simply could not have survived the years and would not be around today.

There are some interesting comments on Hacker News, including Tobi talking about a momentous occasion in their lifetime when the new business model was put in place (which I blogged about at the time):

The problem was that our old system was setup so that Shopify was very cheap for people who weren't successful and became quickly expensive for people who actually sold a lot of product. Everyone who expected to setup a 1M revenue store ( e.g. everyone ) picked up a calculator and realized that.

I really admire how Shopify stuck to their guns on the new pricing structure despite huge pushback from the community at the time. Now with the benefit of hindsight, it was clearly a very smart move.

Target non-mobile Devices using JS

Some regex to use when you don't want something to happen on a mobile device:

if (!/android|iphone|ipod|series60|symbian|windows ce|blackberry/i.test(navigator.userAgent)) {
 // regular browser stuff here...
}

I'm All Over It Too

A little slip by one of the bigger banks down here, Westpac, with a wayward tweet (or maybe it was on purpose?!). It's been removed now, but not before I managed to get a screen shot:

Hopefully whoever screwed it up isn't reprimanded too strongly. I think it is nice to see the humans behind the curtain come out every now and again.

 

10 Critical Thinking Traps & Tips

There is a great list of critical thinking traps & tips over at litemind.com. I'm summarizing them here so I remember them!

1. The Anchoring Trap: Over-Relying on First Thoughts.

initial impressions, ideas, estimates or data “anchor” subsequent thoughts

• Always view a problem from different perspectives
• Think on your own before consulting others
• Seek information from a wide variety of sources

2. The Status Quo Trap: Keeping on Keeping On

 status quo automatically has an advantage over every other alternative

• Consider the status quo as just another alternative.
• Know your objectives.
• Avoid exaggerating switching costs.

3. The Sunk Cost Trap: Protecting Earlier Choices

it's a sunk cost, and it shouldn't influence your decision

• Be OK with making mistakes.
• Listen to people who were not involved in the earlier decisions.
• Focus on your goals.

4. The Confirmation Trap: Seeing What You Want to See

looking for information that will most likely support your initial point of view

• Expose yourself to conflicting information.
• Get a devil's advocate.
• Don't ask leading questions.

5. The Incomplete Information Trap: Review Your Assumptions

(my favorite, but hard to execute!)

overlooking a simple data element can make our intuitions go completely astray

• Make your assumptions explicit.
• Always favor hard data over mental simplifications.

6. The Conformity Trap: Everybody Else Is Doing It

other people's actions do heavily influence ours

• Discount the influence of others.
• Beware "social proof".
• Be courageous.

7. The Illusion of Control Trap: Shooting in the Dark

we still tend to irrationally believe that we can somehow influence results. We just love to feel in control

• Understand that randomness is part and parcel of life.
• Beware of superstitions.

8. The Coincidence Trap: We Suck at Probabilities

Probabilities can be unintuitive.

• Don't over-rely on gut estimates.
• Beware of "after the fact" probabilities.

9. The Recall Trap: Not All Memories Are Created Equal

we analyze information based on experience, on what we can remember from it

• Get hard data. (there's that hard data thing again - see # 5)
• Be aware of your emotions.

10. The Superiority Trap: The Average is Above Average

people have much inflated views of themselves

• Be humble.
• Don't go overboard. (wrt applying rigor and rational thinking)

I encourage you to read the full article as it contains several good examples and great explanations of the tips.

Giving the Benefit of the Doubt

There was a lively thread on proggit the other day about how difficult programmers can be to work with:

• "always trying to one up each other and prove how smart they are"
• "petty infighting and ruthless backstabbing"
• "constantly seek refuge in argument by authority, blindly quoting their favorite gurus to support their positions"
• "desperate need to believe in absolutes"

I have to admit, I am certainly guilty of pretty much all of the traits listed in the post. Some more than others, much to my chagrin.

It occurred to me that as I have gained more experience (read: got older) these tendencies have tended to diminish. I have realized the more aware I have become with the business side of the equation, ideals I was once passionate about now seem rather insignificant in the grand scheme of things (and of course with the benefit of 20/20 hindsight).

My interaction style has also changed a lot. Whereas before I would quickly jump to conclusions when I saw things that were "wrong", I now try to come up with possible explanations for why it is the way it is before flying off the handle. What I have found is that I often learn something new - if not with the code or process then certainly about the way the individual works.

So today I came across a post in The Fishbowl about a thing called the "Principle of Charity". It's a good read. Fundamentally it says be charitable to whom it is you are dealing with and assume the best of them:

1. They are not idiots, brain dead or morons.
2. They are honest and not trying to scam, manipulate or deceive you.
3. They did their homework.

I like it! It is nice to find out about things you have a sense of but can't quite put a finger on. Getting back to that proggit thread, I wish someone had told me about this 20 years ago. I dare say my technical discussions would of been much more efficient and less aggravating had I afforded my poor colleagues the benefit of the doubt occasionally.

Charles says his challenge is to remember the principles. I think my challenge (now that I am enlightened) is how to respond to individuals who violate those principles when dealing with me. Let me clarify that. I mean to say how to respond "politely".