RESTful API and Deploying Updates Safely

I've been thinking about this for a while, but haven't really come up with a great solution so I'm looking for ideas on how others have approached it.

Say you have a web service with a RESTful API. The service provides a centralized place to capture straight through processing exceptions and handle their remediation. There are loads of other loosely coupled applications in the environment, all with the need to be able to post exception notifications on a more or less real time basis.

Now, the exception service needs an update - there are some bug fixes and a few new shiny features we want to push out. It is early days but you have 2 of those applications using the service already. The update only takes 30 minutes or so, but during that time the clients won't be able to post any new exceptions.

So, the conundrum...

How to deploy the new service without breaking client applications?

Being based on HTTP, the client will get a 503. Are they expected to store their payload and try again later? Seems like the client is taking on a lot of burden responsibility here, and what if it is time critical?

Or, do you forgo the synchronous nature of a REST call and put in a message queue? Less hassle for the client app but they do need to install a messaging driver now. At least Ops can look at the meesage queue and see what is backing up. But hey, isn't this still a single point of failure though?

Perhaps the exception service needs to be clustered so you get fail-over during updates. Trouble is the update needs a database schema change. How will it work if the new instances are expecting the new schema but the old instances are still serving out requests?

Am I missing something?

Comments

[Thanh Tran]

You mentioned service clustering which deals with the 503 issue. For time critical and high availability apps, wouldn't database clustering (and replication) be able to deal with the issue of schema change, assuming the change is backward compatible?