Gavin Terrill's Blog

Vent: RubyMine/Gems/Bundler/Backward Compatibility

My preferred framework of choice for a lot of the web apps I'm playing with these days is Rails. But I've got to tell you there is some stuff that really ticks me off....

First is backward compatibility. 2 years is like a whole new eon in Ruby land. Going from Rails 2 to 3 breaks all sorts of things outright, and other stuff in subtle ways. The community's answer to this is to ensure you've got tests up the wazoo (which is a good thing, but not everything can be automated especially the edge cases). Ruby itself from 1.8 to 1.9 breaks things. Net result for me is locking in gems at specific versions and deferring updates so ultimately this is driving up Tech debt.

As an example, I'm trying to upgrade one of my Shopify Apps to use OAuth because the "old" way is going to be turned off. Of course, the OAuth Gem is not compatible with Rails 2 so upgrade here we come. 2 weeks later and I'm still working through it. Partly because of...

Next pain point is the whole ruby gem nightmare. Maybe it's because of my preferred IDE (RubyMine - which on the whole is awesome by the way) in conjunction with rbenv, but yesterday was blown because after doing a reboot it couldn't find Rails in the SDK. I haven't rebooted for a while so it is likely something I did weeks ago. I tried all sorts of stuff to get it working again - reinstalling rbenv, gems, setting environment variables etc. In the end I closed all projects, rebooted again and restarted the IDE and guess what it starting working again (sort of, now it says I've got the wrong version of rack installed, sheesh!).

It all just feels kind of random and fragile. I don't know what the answer is, but I think this really turns a lot of people off Ruby. In my day job we are planning on moving to a Micro-Service style architecture and it would be great to see some of those implemented in Ruby but the folks who have tried it at some point got turned away after not being able to get their heads around the environmental stuff.

I do see some positive signs. So far ruby 2 seems to play nicely with my existing stuff.

People talk about the drag maintaining backward compatibility would involve (looking at you Java) and how it stiffles innovation. In this case I think the pendulum may have swung a little too far. It may be better to cut down the surface area - E.g. the Asset Pipeline stuff introduced in Rails 3 probably should have been an optional extra, particularly now that our thinking about the performance benefits of it are changing.

Ok, vent over. Thanks for listening!

Update 3 days later: Ok, so to get Rubymine to understand the gems, go to Rubymine > Preferences and delete your SDK. After that a little reset button will appear in the top right. Click that and it sorts itself out.

 

 

 

Remembering Carl Adam Petri

Just found out that Carl Adam Petri passed away July 2, 2010. Petri invented the "Petri Net" - a tool for modeling process flow that is used in a wide variety of applications, from circuit design to traffic light coordination to business process management.

I was introduced to Petri Nets by a grizzled veteran software developer back when I knew everything. I looked into it and was struck by the simplicity of the model, plus the mathematics. You see, Petri Nets can be formally verified so you can prove that your flow is "correct", meaning it will complete and doesn't get stuck in an endless loop for example. You can even do simulations with it to help predict capacity requirements.

The basic idea is you have a diagram with boxes that represent "doing something", called a Transition. Transitions are connected to other transitions via a circle, called a Place. The connecting lines show the direction of the flow and are called Arcs. A network has start and end places and can not have cycles (i.e a Petri Net is a DAG).

Now, to model the process flow you place a Token in the starting place. The token represents a unit of work and moves through the network of transitions and places until it gets to the end place. As each transition fires the the token moves to one of the output places.

 

What is interesting is that a place can hold more than one token, so you can add another token to the starting place before the first one has reached the end place. This is why a Petri Net is good for modeling concurrent processes.

Wil Van Der Aalst has written some great papers on applying Petri Net theory to business process management. A few years ago I was inspired by one of his great books and created a Swing based Petri Net editor. The code has long since been lost unfortunately, but I remember feeling quite chuffed at being able to simulate and optimize an insurance claims process by playing with the capacity of certain transitions in order to improve overall system throughput. It was very Kanban like.

Petri was a true pioneer of our industry, and the amazing thing to me is that he came up with the idea when he was just 13! 

Probability Distribution for Aircraft Safety?

A couple of weeks ago a Qantas flight experienced a topsy-turvy ride enroute to Perth. 70 people were hurt, some seriously. At the time it was put down to an air pocket, but today I read that it was actually due to something called an ADIRU (Air Data Inertial Reference Unit):

Investigators said last night the false information from an air data inertial reference unit (ADIRU) fed "very high, random and incorrect values". The ADIRU supplies information such as air speed, altitude and position.

If you are constantly logging air speed, altitude and position for every flight then you have a very large dataset to work with, making it extremely useful for seeing if the readings you seeing are what you expect.  Similarly, you can record the rate of change between readings and my bet is that is shows a highly normalized distribution. I am wondering if an Error Function is utilized to check for dodgy readings? Anyone know of the inner workings of these things from a software point of view?

G'day Mate

I see it is already April 1st back home: Search tomorrow’s web, today!

Learning Scala

One of my new year revelations is to learn a new language (I'm investing in my knowledge portfolio) and I've decided to go with Scala. I was also looking at Erlang, but in the end Scala made the most sense to me because:

• It runs on the JVM. This is important to enterprise customers, and if Scala works out I'd like to be able to use it for our existing products.
• I am more comfortable with the language syntax, coming from a Java background.
• I can mix and match parts of our existing Spring based Java application with Scala bits. I'm actually interested in how far into component reuse utilizing Spring Dynamic Modules for OSGi I can get - swapping out Java backed components with ones written in Scala.
• I am interested in exploring Scala Closure and Actor idioms for revalidate (ETag helper for Spring & Hibernate applications), inspired by this post on using Scala for HTTP Precondition processing in Jersey.
• I'd like to explore the benefits in terms of code quality using Scala that Dave Pollack reports in building lift using Scala.

Ted Neward's introductory article on developerworks "The busy Java developer's guide to Scala: Functional programming for the object oriented" is easy to digest, and timely. After reading that, I followed the tutorials on the Scala site.

So far, the learning curve has been gentle. Type inference is cool. I find myself thinking about how I can apply the Scala way of doing things to problems areas in our product. I'm currently reading about "mixins", which allow you to treat classes as if they implemented an interface, even when they don't.

The next step is to install the Scala Eclipse plugin and start getting into more meaty examples (Updated a few hours later: see here for instructions on getting around the "org.eclipse.pde.runtime is missing" problem).

A real RESTful API for SimpleDB

Amazon is catching a lot of flak over their implementation of a RESTful API for SimpleDB.  Stefan calls it "the shittiest REST API in a long time". Bill says that "Ascribing REST to the SimpleDb API might be down to lack of knowledge or marketecture". 

So what is the fuss about? Take a look at the following sample code from the Amazon SimpleDB developer guide:

https://sdb.amazonaws.com/?Action=PutAttributes
    &DomainName=MyDomain
    &ItemName=Item123
    &Attribute.1.Name=Color&Attribute.1.Value=Blue
    &Attribute.2.Name=Size&Attribute.2.Value=Med
    &Attribute.3.Name=Price&Attribute.3.Value=14.99
    &AWSAccessKeyId=
    &Version=2007-11-07
    &Signature=Dqlp3Sd6ljTUA9Uf6SGtEExwUQE=
    &SignatureVersion=1
    &Timestamp=2007-06-25T15%3A01%3A28-07%3A00

Can you see what the problem is? It's kind of subtle, so I'll give you a hint - what is  up with that "Action=PutAttributes" bit in the URL?

The problem is that in REST you use the HTTP methods GET, PUT, POST, DELETE, HEAD and OPTIONS. In the above example, GET is being used. Now, the interesting thing about the GET method is that according to the HTTP spec (RFC 2616), it is "safe":

GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval

In other words, the GET method is what you use for reading things (e.g. queries or looking up items). GET should not be used to update things. Lots of web infrastructure rely on these semantics to do their job. Breaking the contract hurts the web.

So how would you make it RESTful? First, I'd recommend taking a look at Joe Gregorio's excellent advice from 2004 on designing a REST API. Joe suggests asking four questions:

1. What are the URIs?
2. What's the format?
3. What methods are supported?
4. What status codes should be returned?

So lets map out a strawman SimpleDB REST API using these questions as a guideline.

The URI

In this case, we are updating something in MyDomain which is identified by ItemName.

Resource	URI
MyDomain List	/mydomains/
MyDomain	/mydomains/{item-name}

 

The Format

These days, JSON is a pretty decent format to use as a generic starting place. XML is still fine & dandy, but is a little harder to work with in browser clients. You might want to take a look at Atom as well. In any case, clients can use Content Negotiation to ask for their preferred representation.

A JSON format for a SimpleDB item could look like this:

  {
      "DomainName": "MyDomain",
      "ItemName": "Item123",
      "Timestamp": "2007-06-25T15%3A01%3A28-07%3A00",
      "Attributes": {
		"Color": "Blue",
		"Size": "Med",
		"Price": "14.99”
	}
  }

The list of items would look like this:

  [
    {"ItemName": "Item123", "href": "https://sdb.amazonaws.com/mydomains/123"},
    {"ItemName": "Item321", "href": "https://sdb.amazonaws.com/mydomains/321”},
    ...
  ]

Notice that in the list we have indicated the href of each specific item.

The Methods

Mapping the available operations to the MyDomain items/item we get the following:

Resource	URI	Method	Description
MyDomain List	/mydomains/	GET	Retrieve the list of MyDomain items
		POST	Create a new MyDomain item
MyDomain	/mydomains/{item-name}	GET	Retrieve a specific MyDomain iem
		PUT	Update a MyDomain item
		DELETE	Remove a MyDomain item

 

The Status Codes

Joe recommends that you document the status codes that you expect to return for each operation. Some of the more common ones are:

• 200 - OK.
• 201 - Created. When an item is successfully created return a 201 and the response will include a Location: header with the URI of the newly created item resource.
• 204 - No Content. Used for a successful DELETE.
• 301 - Moved Permanently.
• 400 - Bad Request.
• 410 - Gone.

Note that in HTTP there is the idea of status code fallback. The 2xx series means success, 3xx means a redirection and the 4xx series means a client error. If a client doesn't understand 201 for example, it can still safely assume that the operation was a success.

So there you have it. My version of a RESTful API for SimpleDB. SimpleDB seems like a very cool idea to me. Let's hope they sort this mess out soon.

Cool Image Resizing Technique

"Seam Carving" is a technique for resizing images based on "seams". Seams are runs of similar pixels, and are identified by using gradient magnitude. When you cut out less significant pixels in a seam, you end up being able to resize an image in a non-uniform manner, but retain important features. You can even use it to remove items, like say an ex from that last vacation...

It is a lot easier to see it in action. Take a look:

Larry O'Brien talks about the algorithms behind this here. I had never heard of A* before this. Interesting stuff.

(via hackzine.com)

BFGs on a plane

It was Socrates who said "It's the journey, not the destination." I believe he may have been referring to Virgin America, who apparently have a mp3 playlist you can set up before the flight, chat and Doom (amongst other things including internet access at some point) on their inflight entertainment system. Oh, I am so flying with these guys. Hope they come to Toronto.

No backup, no problem - thanks to the wayback machine

I did the original web site for Karora Technologies just after we got the company incorporated. It was nothing fancy, and I mostly used Microsoft Publisher because it came bundled with the first laptop we purchased. Later on, we hired a graphic design firm to come up with a clean, simple design that incorporated Aboriginal imagery. The designers delivered and we ended up with an awesome website in my opinion.

Later on as the company got bigger, management of the website was handed over to the marketing/sales area who had someone on the team who had "done websites in the past" using FrontPage. The decision was made to "revamp" the site, and along the way we lost the simplicity and elegant of the "Karora Dreaming" site. In fairness, the requirement was to be able to easily update the site, and this was difficult at the time because it was using bare HTML.

Anyway, five years down the track and the new site badly needs to be updated. Karora has since downsized, and no longer has FrontPage people on staff, so I offered to help out in my copious amounts of spare time (yeah, right!).

The first thing I wanted to do was go back to the look and feel the design firm had come up with so I asked for the backups. Backups? You guessed it - there were no backups. Sure, they were done at the time the site was swapped over, but the machine they were on has since disappeared into the ether. The design firm didn't have anything either.

Then I remembered reading about the "Wayback Machine". This is an initiative of the Internet Archive organization. Basically, they take snapshots of the web so you can see what a site looked like at a particular point in time. I wondered if the Karora site was on it, and sure enough there were entries going way back to my initial dodgy version from 1999.

So, I was able to recover our old website design (with a little help from HTTrack). Thanks WayBack Machine!

Boggles the mind

These guys are messing with the theory of relativity: Scientists break speed of light. I guess that's why it is still a theory, and not a law.

Practical applications? Two words: Fibre Optic.

File that under "how would you measure it?".