Gavin Terrill's Blog

RESTful API and Deploying Updates Safely

I've been thinking about this for a while, but haven't really come up with a great solution so I'm looking for ideas on how others have approached it.

Say you have a web service with a RESTful API. The service provides a centralized place to capture straight through processing exceptions and handle their remediation. There are loads of other loosely coupled applications in the environment, all with the need to be able to post exception notifications on a more or less real time basis.

Now, the exception service needs an update - there are some bug fixes and a few new shiny features we want to push out. It is early days but you have 2 of those applications using the service already. The update only takes 30 minutes or so, but during that time the clients won't be able to post any new exceptions.

So, the conundrum...

How to deploy the new service without breaking client applications?

Being based on HTTP, the client will get a 503. Are they expected to store their payload and try again later? Seems like the client is taking on a lot of burden responsibility here, and what if it is time critical?

Or, do you forgo the synchronous nature of a REST call and put in a message queue? Less hassle for the client app but they do need to install a messaging driver now. At least Ops can look at the meesage queue and see what is backing up. But hey, isn't this still a single point of failure though?

Perhaps the exception service needs to be clustered so you get fail-over during updates. Trouble is the update needs a database schema change. How will it work if the new instances are expecting the new schema but the old instances are still serving out requests?

Am I missing something?

I'll see your Groovy and raise you a Python (and Ruby, PHP for good measure)

Inspired by what is clearly an interesting trend in want ads mentioning Groovy, I ran the same query against Python, Ruby & PHP. Groovy is yet to be where Ruby was five years ago but has at least managed to break out from the x axis. Python and PHP keep on gaining traction.

ruby, python, php, groovy Job Trends

ruby jobs - python jobs - php jobs - groovy jobs

ThingDB - you haven't slayed SQL just yet

I came across ThingDB the other day (via Reg, who pretends he isn't blogging, but one wonders how long until the siren call beckons?!). It is part of the Pharos project from the reddit guys. ThingDB is basically an abstraction layer that sits on an entity table that has a one-to-many relationship to a properties table, with versioning thrown in for good measure. Anyone who has built a system requiring extensible user defined attributes or configurable metadata (e.g. a document management system) would instantly recognize this. It partitions data (i.e. shards) to make it scale beyond a single machine, and they talk of using bloom filters, so it seems pretty cool.

However... what got me to post about this was not so much the technology, but the statements made about SQL. Consider:

But SQL has some problems. For one thing, it's inflexible and poorly-integrated into most programming languages.

Reality check guys: SQL is one of the most successful technical things we have done in our field. I credit that to its flexibility. In fact, I can't think of a situation in my 25 year programming career where I have found SQL to be so inflexible that I need to use something else. Maybe it is because I've predominantly done mainstream CRUD-style enterprise apps, but guess what - that is what the majority of the work is.

I don't get the claim that ThingDB is superior from a language integration point of view. Take a look at the top 9 programming languages (constituting 80% of the space) as at November '08 (according to TIOBE):

1. Java (20.30%)
2. C  (15.28%)
3. C++  (10.36%)
4. (Visual) Basic (9.27%)
5. PHP (8.94%)
6. Python (5.14%)
7. C# (4.03%)
8. Delphi (4.01%)
9. Perl (3.88%)

Everyone one of those languages has excellent support and integration with SQL. In some cases, you can even embed SQL directly into the code (the merits of doing this nowadays are a bit dodgy, but I will say we built a very clean, fast, client/server app in the eighties using this approach). I would go so far as to say that LAMP would not have happened were it not for the integration between the M and P in LAMP.

Side note: check out this cool feature in Netbeans.next - SQL code completion in PHP.

Anyways, a final note on ThingDB. The discussion about how it can be scaled up is very interesting, and I don't want to detract from that as it is very leading edge and well... cool. But, and there is always a 'but', for the average corporate application this is going to be massive overkill. Your typical RDBMS is a very capable technology and isn't going to run out of steam anytime soon. For example, we have a deployment of our product that is tracking issues for 50,000 users worldwide. They are happily running it using Sybase on a 4-way box, and with millions of rows performance is still great.

Bottom line: In spite of claims to the contrary, SQL is still a very useful technology that is entirely appropriate for a large class of applications, and will be be for the foresable future.

15 Minutes of REST

No, this post isn't about workplace ergonomics. Well, maybe it is... If you are interested in getting up to speed on REST quickly, do your self a favour and take 15 minutes to check out Google's "Introduction to REST" video.

I like the way it covers the basics quickly, and also gets into some of the subtleties that are very easy to overlook at first glance (e.g. the importance of intermediaries).

Passion and Architecture

Seth Godin is a marketing guy, but I think he would make a pretty dab hand at developing software as well:

If you want to do something worth doing, you'll need two things: passion and architecture. The tools will take care of themselves.

Want to build great software?

Don't get hung up on Maven versus Ant, Hudson versus Bamboo, Flex versus ExtJS, TDD versus DBC...

Do get hung up on REST versus RPC, Push versus Pull versus Push and Pull, Relational versus Column Stores versus Distributed Hash Tables versus Shards...

Lindaspaces eBook Now Available

I believe the hardcopy is now out of print, so this is pretty sweet - one of the seminal books on the ideas behind Tuplespaces by Nicholas Carriero and David Gelernter is now available online: "How to write parallel programs - a first course". The last edition was written in 1992, but this is still very relevant today.

Reference Architecture circa 2008

Archetype: High availability, scalable enterprise application supporting '000s of users; running on multi-core b0xen.

• Ajaxy ExtJS front-end.
• Client talks REST.
• Request handling via Lift REST - dispatch to business logic via Scala Event Based Actors.
• Lift JPA integration for persistence.
• Comet for push.

Does documenting your architecture make it high quality?

I just posted a piece to InfoQ on the SEI's new 87 page "Evaluating and improving architectural competence" paper. Towards the end of the report there is a section that discusses principles embodied in the models, expressing that goals should be clearly articulated. One of the examples they used made me chuckle:

Documenting the architecture is likely to lead to high-quality architectures because documentation is essential to effective communication, which is essential to effective understanding and use by the architecture’s stakeholders, which is in turn essential to providing timely and useful feedback.

Don't get me wrong - I love most every aspect of my job, but "documenting the architecture" is not one of my favorite activities by a long shot. Its right up there with filling in timesheets and doing SR&ED claims.

I think part of the problem is figuring out what to document. Most of the stuff I've read suggests spending weeks working through the various architectural views. Obvious guy (aka Heraclitus) says that since change is the only constant, you are taking on a significant maintenance burden in synchronizing the document with the evolving system. Meanwhile, the developers and other architects have filed your treatise in the tl;dr folder.

I agree that "effective communication" is "essential to effective understanding and use by the architecture’s stakeholders", but I'll take providing working code anyday.

BAUF - Big Architecture Up Front

Bob Warfield has just written how the lead architect of Twitter has bolted, leaving behind speculation that there are deep scalability issues that need to be overcome before Twitter can achieve world domination.

I don't think you can take shortcuts when it comes to doing due diligence on your architecture choices before building a solution. I'm all for the YAGNI mindset and avoiding BDUF when building out feature, but there are some things that are simply not possible to change after the architectural foundations have been set without incurring significant renovation cost.

To make matters worse, it is extraordinarily difficult to get this right as you typically have very little information at your disposal when you are making these key decisions. Other factors such as the talent pool available to work on the project further muddy the waters. You end up making trade-offs based on educated guesses, and relying heavily on your intuition.

Some of the key decisions include:

• Framework choice - you get time to market, but the trade off is losing control of architectural decisions. In a Rails app, your CRUD operations are synchronous, so if one day you read about the CAP theorem and think to yourself this eventual consistency thing might be the way to go you are going to have a tough time.  Language choice implicitly follows your framework choice, and it is almost impossible to change that later. Imagine trying to get an MS Access based product to work in a clustered environment for example.
• Caching & Security. Two things that are expensive and extremely difficult to retrofit.
• Client platform. Will you need rich client or offline functionality? How will it communicate with your web based service? If you build it all loosely coupled and cohesive you may have a shot, but these qualities rarely fall out of a system built using an ad-hoc approach.
• Distributed API. In an SOA world, changing the protocol of an established application is pretty much out of the question. So, you are forced to think about future requirements ahead of time. For example, you can pretty much guarantee you have to deal with clients on different versions of the API. Tip: having an extensible data format helps, as well as  making sure your application protocol utilizes caching extensively.

I've also covered the "Premature Scalaculation" idea on InfoQ - some good comments there.

Introductory REST Article

Stefan Tilkov has written an excellent article over at InfoQ covering the ideas behind RESTful architectures: "A Brief Introduction to REST".

The thing I really liked about this article was the low-barrier-to-entry approach Stefan took. For example, instead of starting out with why things are called "resources", he simply says "give every thing an id". I can see this resonating with developers new to REST because words like resource are quite overloaded. Another thing that takes some getting used to is the fact you can also model processes as a resource:

Taking the idea of identifying everything that is worth being identified further leads to the creation of resources that you usually don’t see in a typical application design: A process or process step, a sale, a negotiation, a request for a quote — these are all examples of “things” that merit identification.

Stefan also covers several other fundamental REST ideas:

• Using links to traverse the web of relationships between things.
• The fact that there is a standard set of operations (GET, PUT, POST, DELETE, HEAD, OPTIONS), each with formal semantics, that are sufficient to express your (web) services and can be used by ubiquitous clients such as browsers or curl.
• Using content negotiation to ask for different representations of things (e.g. JSON v's XML).
• Communicating statelessly so that the system is more robust and easier to scale.

There is also a nice summary of the theory behind REST. I started getting into REST a few years back when Paul Prescod was educating the great unwashed on xml-dev. This is the best introductory article on REST I've seen so far, so I highly recommend taking a look if you are at all interested in this emerging architectural style.