BAUF - Big Architecture Up Front

Bob Warfield has just written how the lead architect of Twitter has bolted, leaving behind speculation that there are deep scalability issues that need to be overcome before Twitter can achieve world domination.

I don't think you can take shortcuts when it comes to doing due diligence on your architecture choices before building a solution. I'm all for the YAGNI mindset and avoiding BDUF when building out feature, but there are some things that are simply not possible to change after the architectural foundations have been set without incurring significant renovation cost.

To make matters worse, it is extraordinarily difficult to get this right as you typically have very little information at your disposal when you are making these key decisions. Other factors such as the talent pool available to work on the project further muddy the waters. You end up making trade-offs based on educated guesses, and relying heavily on your intuition.

Some of the key decisions include:

• Framework choice - you get time to market, but the trade off is losing control of architectural decisions. In a Rails app, your CRUD operations are synchronous, so if one day you read about the CAP theorem and think to yourself this eventual consistency thing might be the way to go you are going to have a tough time.  Language choice implicitly follows your framework choice, and it is almost impossible to change that later. Imagine trying to get an MS Access based product to work in a clustered environment for example.
• Caching & Security. Two things that are expensive and extremely difficult to retrofit.
• Client platform. Will you need rich client or offline functionality? How will it communicate with your web based service? If you build it all loosely coupled and cohesive you may have a shot, but these qualities rarely fall out of a system built using an ad-hoc approach.
• Distributed API. In an SOA world, changing the protocol of an established application is pretty much out of the question. So, you are forced to think about future requirements ahead of time. For example, you can pretty much guarantee you have to deal with clients on different versions of the API. Tip: having an extensible data format helps, as well as  making sure your application protocol utilizes caching extensively.

I've also covered the "Premature Scalaculation" idea on InfoQ - some good comments there.