Monday 6 August 2012

Two phase commit (XA Transactions)

Some time ago I had the "opportunity" to lose some sleep worrying about a single transaction across two resource managers in a C/C++ application.  To the uninitiated this sounds simple, trivial even.  Since then, I've heard estimates anywhere from 20 days to 200 days to implement two phase commit.  Usually I can only hope they realise their mistake before the project is committed and certainly before it goes live.



If someone produces a significant estimate, say 200 days, for two phase commit it is probably because they imagine implementing their own version of the XA specification.  However, it is also likely they haven't actually read the spec, and will only implement a partial solution.

It seems questionable to me that any team could even partially implement two phase commit for 20 days effort.  Even linking with a proper transaction manager Tuxedo, Encina, etc would take more than 20 days.  There are other alternatives, but none seem that (20 days) easy:  http://linuxfinances.info/info/tpmonitor.html

If you introduce the constraint of just 2 resource managers you have a smaller problem, but it still seems unlikely that this system could function semi properly for just 20 days effort.  A typical approach (shortcut) might be to use a last resource commit 'optimisation' - where one resource manager (say an application server) supports XA and one other resource manager only supports a local transaction.  The thought goes, we can simply commit the local transaction last and thereby safely modify both resource managers in one transaction.  The Weblogic documentation provides a good description of how it accomplishes that, but it is also clear that this is an optimisation.


One of my good friends provided the following description of what a two phase commit really needs to accomplish:



The reason for two phase commit is to simplify the programming model (no matter how many resources the programmer has ‘touched’ all she needs to do is check consistency and then commit) and to simplify recovery. Using a short cut like last committer breaks the programming model (so you should only use it in middleware like integration, not in a programming model) and does not enable recovery.

Here is the recovery scenario. The program commits, we prepare the resource manager that supports two phase and then we commit the one that doesn’t. We go down before getting a response from the single phase resource manager. We are now ‘in doubt’. When we come back up, what true two phase commit does is first exchange log identifiers between the transaction manager and the two resource managers. This establishes the context for recovery. The transaction manager discovers the in doubt transaction and asks the two resource managers if they had all prepared and if at least one had committed. If so they are all committed. If not the transaction is rolled back.

Now try that with last committer. When we come up, there is a transaction prepared and no way, in the logs, to know if the single phase resource committed or not. So either we heuristically commit (guess) or we ask the operator to decide. Or worse, the transaction manager automatically rolls back the prepared resource even though the last committer committed. Either way we end up with a delay or an inconsistent database. We lose the automatic recovery of two phase, which is its real bonus for a customer. 

Two phase commit is not "easy" in C++; don't believe anyone who tells you it is.  One of the hidden gems (to a C++ programmer) of the Java Application Server is this programming model for distributed transactions (two phase commit).  In JEE is really is pretty trivial to update a queue and a database in one transaction - sadly the same cannot be said for unmanaged environments like the typical C++ application.

Wednesday 25 July 2012

Go forth and create something RESTful


Every time I speak to someone starting out with REST there seems to be a familiar pattern.
1.  Very likely someone has given you a directive to create something RESTful - with little more reasoning than "because".  (sadly is it uncommon, almost unusual to have a problem and then discover REST as the solution)
2.  You begin to investigate REST and are overwhelmed with libraries, implementations, advice, hearsay, and very little information about why your situation requires REST.  I think everyone should strive to find out "why" their problem needs REST first, but lets summarise by saying reduced client/server coupling (service evolve-ability), horizontal scalability (transparent caching, stateless) as two major benefits.
3.  You start to implement (or are "lucky" enough to have inherited) something that is meant to be RESTful, but clearly lacks some of the advertised RESTful benefits you figured out in step 2.  Go back and challenge step 1 or step 2 as appropriate. 

Here's what happened to me in step 1.
Our group was given a directive of sorts "to provide an interaction architecture with loose client/server coupling".  At this point someone suggested we would benefit from REST.  Being a natural skeptic meant step 2 was quite involved.  We concluded that despite all the libraries, books, and implementations - it appears you have to implement everything yourself or at least put it all together yourself.  Some will say that this is natural because REST is an architectural style.

There are libraries and various standards that will help, but at the moment it is down to you to build your application and ensure that you meet all the RESTful constraints - unfortunately there are just so many ways to go wrong.  Let me try to explain some of the constraints that helped us avoid too many Step 3, Step 2 iterations.

With REST there are 4 key constraints (arguable 5 as Fielding identified 
another after his initial dissertation)

Identification of Resources
Most REST libraries do this pretty well.  For example, JAX-RS uses the @Path annotation to make it easy to bind a method/class to a resource.  Here is a cool list of libraries:  
http://code.google.com/p/implementing-rest/wiki/RESTFrameworks  Your main task is to avoid creating URIs that are Verbs.

Manipulation of resources through representations
Here you may have chosen HAL.  Other good choices might be AtomPub or HTML - depends.  It is quite a lot of effort to standardise and describe a media type to User Agent implementers.  This leads us to choose a media type that is not specific to individual things in your domain, nor something generic such as application/xml as other user agents won't have any idea what to do with your bespoke media type.  Essentially you want to ensure that the client / server can negotiate a good media type, one that suits the purpose of the interaction.  HAL could be good for interacting with application entities, AtomPub could be good for collections of things such as blogs or other structured data. Some other examples could be:  I'd like to add up my list of things - spreadsheet might be ideal (application/vnd.ms-excel) or I'd like to print my list of things - a PDF document might be ideal (application/pdf)

Self descriptive messages
This is one of the nice things that an uncomplicated media type like HAL could give you.  If you describe your resource separately from the view of that resource you open up all sorts of problems.  For example, by having a /metadata resource and a /someentity - you cannot ever really guarantee they are of the same version.  Client or intermediaries could cache your resource or metadata for quite some time.

Hypermedia as the engine of application state
Perhaps you identified HAL links or AtomPub links as a good way to achieve this.  Not the only way I might add, there are good alternatives.  In my opinion, REST is ideally suited to user interaction as you can move your user agent from one application state to another with the user agent responding to each resource to the best of its abilities.  In our earlier example we could use Excel or Acrobat in response to application/vnd.ms-excel or application/pdf media types.  More over, by controlling the user agent from the server, you benefit from some of the key features of RESTs dynamic behavior / upgrades in place (evolve-ability).  (as an aside - Believe it or not Users are pretty smart and know what they want to achieve, systems on the other hand are not smart and don't respond well to interface changes.  I personally clearly separate Interaction from Integration for this reason)  A lot of libraries seem to fall down in this space.  JAX-RS1.1 does not have support for links in the annotations, some people have proposed extensions (http://code.google.com/p/jax-rs-hateoas/)  RestEasy has some support for links http://docs.jboss.org/resteasy/2.0.0.GA/userguide/html/LinkHeader.html  Even as of JAX-RS2.0, the JAX-RS standard and the RestEasy library are still basically "roll your own" as far as links go.

No. 5 Resource generic interaction semantics
Here you've probably chosen HTTP.  Many have said this is the only resource interaction you ever need.  Jury is probably still out on that - see WEBDAV, or even WAKA as other examples.  It is a very well accepted standard and deviating from this will require you to build quite a bit of tooling on the client side and potentially intermediaries.


Take heart.  It's tough job building an architecture according to RESTful constraints.  I'm pretty sure Fielding said "REST is simple, but difficult to get right".

If you found this little blog interesting, perhaps you'll like one of my other blogs on REST in an Enterprise application.  http://aphethean.blogspot.co.uk/2012/06/giving-enterprise-applications-rest.html




http://aphethean.blogspot.co.uk/2012/06/giving-enterprise-applications-rest.html

Friday 20 July 2012

MTB C2C, now what?

A fortnight ago I was about to start the Tim Woodcock Coast to Coast.  Those in the know will agree this is one challenging little MTB ride.  The forecast couldn't have been much worse - 76 flood warnings in place across England - and I'd just read an article recommending 6-7 days to complete it - we'd planned to do it in 4!  Despite all evidence to the contrary, we managed it; with just one shower of rain each of the first 3 days, and a memorable 4th day of nearly constant rain and mud.  It's funny how I've already started to elevate all the really difficult and challenging parts of the ride in my memory and pretty well forgotten about the huge distances, long hours, and other easy bits.  

Our main mistake was nearly missing dinner and a pint each day.  This was supposed to be a holiday after all.  On reflection I think we made this mistake months before we even started the ride by booking the train up for the morning of Day 1 - pushing everything back.  
- Day 1 we left Watford Junction at 6:20am, meaning we'd be at St. Bees and riding by 12pm at the earliest.  Despite trying to keep up a good pace all day we arrived in Coniston at 9pm and were literally the last meal served at the pub.  With no real rest or down time we were late setting off the next day - about 9:30am.  
- Day 2 was a big one and setting of that late was always going to make it tough to get in for the all important dinner and a pint. We had to stop at a pub a few miles short of our nights accommodation and eventually checked in at 10pm. 
- Day 3 saw a similarly late departure after some much needed bike maintenance - 10am ish.  Desperate for a relaxed dinner and pint, the longest day of mileage ahead of us pretty much forced us to take in a few more road miles than I had in mind.  We did have a good dinner that night, and treated ourselves to 2 pints.  
- The easy miles on Day 3 were more than made up for on Day 4.  Wet, muddy, cold, very difficult, and to add to our misery the North York Moors destroyed our brake pads.  My biked limped the last 30km with no rear brake and a front brake that looked to fail in the same way at any moment.  


Here's my kit list and notes from the train home:

Long distance biking (C2C - 4 days)
3 fine days with showers, 1 really wet day.
Flats - good, but did knock my legs a lot
Brake pads!  Sandy conditions complete wore out brake pads on last day
Shoes - need light weight shoes.  Managed without, but was a faf going to the pub
Seat - such a sore arse day 3 & 4.  Adjustment?  New seat?  Seat is nothing special, not that old.  OEM specialised seat?
Soap- small soap if staying at hostels, did without, but not ideal.
Dry bag - Revelate bags don't seem waterproof (Fantastic bags and would never consider panniers after using these bad boys with 12L dry bag up front, and Viscacha seat bag
 http://www.revelatedesigns.com/ )

Stuff:
4 - cycling shirts
1 - cycling shorts
4 - jocks (one day just in shorts, was good)
5 - socks (dumped each day)
1 - wool ice breaker jumper
1 - long sleeve Tshirt
1 - Swiss army knife (did not use)
1 - toothbrush, floss, toothpaste
1 - small bottle oil (essential)
1 - multi tool
1 - tube
1 - pump
1 - repair kit
1 - commuter light front & back
1 - emergency cag
1 – Sunglasses / sunscreen

1 - Tshirt
1 - cycling trousers (not much use)
1 - light weight trousers.  Not white next time, could probably reuse cycle trousers as did not cycle in the trousers
1 - cycle jacket
2 - light shorts
1 – MTB head lamp, 2 batteries
1 - phone charging battery pack
1 - phone charger
1 - phone / camera
1 – Maps, GPS (Darren), Compass


Food:
2/3 tablespoons of flax seed at breakfast
SIS powder drink mix
Mint cake each day
1/2 bag 150g? Nut/raisin mix each day
Biscuits or muesli bar each day
Fig rolls / pillow biscuits

Things my mate carried that I consider essential:
Cable ties


Now for the main reason I started this blog - now what?  I love riding, I love adventures, I love my family, and I'm so lucky that my main problem is choosing what to do next.  So here's a thought - MTB 3 peaks:

1.  Must be bridleways (obvious)
2.  Must be highest in England, Scotland, and Wales
3.  Must be done in 24hr's
4.  Must ride all of it (is that possible?)

Helvellyn (England, 950m):
http://www.youtube.com/watch?v=fXFYZUd7UQs

Snowdon (Wales, 1085m):
http://www.youtube.com/watch?v=wqUFUQuygvs

Ben Nevis (Scotland, 1344m):
http://www.youtube.com/watch?v=EMFkHGLcK-Q 


Not a totally unique idea perhaps:
http://www.mbr.co.uk/news/trail_news/mountain-bike-route-%E2%80%93-helvellyn/


Saturday 16 June 2012

Giving an Enterprise Application a REST


I work in a product development group with a software product in use at over 600 companies, which they depend on for their mission critical record keeping.  One day, not so long ago, we woke up and the world had changed.  No, this was not the financial crisis.  It was far more significant for our development team.  We discovered that our users’ perception of what was a good application had changed. The ones they liked could browse, mash-up, poke, and prod any number of resources. But our Enterprise Application couldn’t.  We were practically selling legacy software.


As we found out, to satisfy a modern Enterprise Application user it is no longer sufficient to transact with a single application based on queries from a single data source.  We were spending an increasing amount of time and effort masking the deficiencies of our architecture by creating non-reusable mash-ups within our various user agents, or worse still, some of our clients and development teams were unwittingly limiting scalability by introducing call outs within the resource manager  - see Figure 1.  Furthermore, because we had multiple user agents each managing their own resource manager connectivity, a single functional change often had to be repeated for each user agent.  We needed a change, a change that would allow us to draw information from, and transact with, many sources at the back end, while at the same time be able to rapidly alter the functionality across many channels at the front end.

Figure 1 Enterprise Application mash-ups and callouts


Interaction Layer

After some deep thought, we decided on a change to our architecture that seemed to satisfy all our requirements. We changed our application from being monolithic, all the code in one place, to one with code in two places. This journey actually started many releases ago and is shown in the diagrams below.

Figure 2 Original Architecture

In a monolithic application, see Figure 2 Original Architecture, the user agent is ‘dumb’ and the session state is held at the resource manager. This is usually done using some kind of scratch pad. One interaction from the user may be multiple calls into the business and data logic to manage the state of the resource.


Figure 3 An early evolution of our architecture

In Figure 3 you can observe an early evolution of our application architecture; some user agents began to separate the user interface logic from the business and data logic.  The user interface logic can be defined as the part that depends on the session state.  To keep the user’s application up to date, we interact with the resource manager as many times as we need.  This change enables features like AJAX.


 Figure 4 Current application architecture

By adding the ability to route requests and transform them to our models – as seen in Figure 4, we could now do mash-ups.  However, we want to have a uniform model of our application to stop repeating ourselves; where with multiple user agents (one for the desktop, one for mobile, one for touchscreen and so on) can all share a similar interaction.


Figure 5 Interaction architecture

Our latest change, as seen in Figure 5, proposed a new interaction layer that enables us to route requests to resource managers for all the user agents.  In MVC (Model View Controller) terms we have put a gateway between the view and controller in the user agent (providing rich interaction) and a model mapped to the resource managers.

Our Flex based Rich Internet Application (RIA), servlet based web applications, mobile Apps, and Interactive Voice Response (IVR) user agents can now be completely decoupled from our various resource managers - in much the same way that a web browser is decoupled from the various websites throughout the world.  To accomplish this we use an XML media type over HTTP as the only means of communicating from the user agents to the resource managers and a HATEOAS (Hypertext As The Engine Of Application State) constraint to control the behaviour of all the user agents from the interaction framework – see Figure 6.  The interaction frameworks’ other responsibilities include a mash-up and synchronous call-out capability, and the ability to rapidly define new resources to be managed by our user agents.


 Figure 6 Interaction layer


REST Interaction

Introducing this level of change into a mature Enterprise Application was never going to be an easy task.  We quickly decided that the REST (REpresentational State Transfer) architectural style[1] was exactly what we needed; as a way of interacting between the user agents and our interaction framework. The requirement on us, if we are to provide a RESTful resource manager, is to comply with the constraints of the REST architecture style.  These are:

  • Client Server – we get this by binding to HTTP
  • Stateless – we get this by binding to HTTP
  • Cache – we get this by binding to HTTP
  • Uniform Interface – this is the main subject of this article
  • Layered System – we get this by binding to HTTP
  • Code on Demand (optional) – this I not discussed in this article
  • Representations – this is discussed in the uniform interface section below
  • Resources – we get this by binding to HTTP
Our analysis of the RESTful constraints indicated that a crucial constraint for us was the uniform interface.  An application binding to HTTP has to comply with the uniform interface on top of its HTTP infrastructure.  Our challenge is to make our mature Enterprise Application RESTful and conform to these same constraints.


Our Approach

Our first priority was to identify what ‘things’ could be managed by our application and organise them into resources user agents can navigate and render.  Once we had identified what resources could be managed, we needed to establish the links between them and how they would transition state.  With more than three and a half thousand individual, or non collection, resources managed by our application, and a staggering mass of relationships, we were going to need something a bit more substantial than a pen and paper.

Resources

UML class diagrams proved to be an excellent way to describe our resources and the relationships between them.  We used the attribute property to represent the elements of our resources, and the association property to represent the links between the resources – see Figure 7.  At runtime these two pieces of information are used by the user agents to render a screen and navigate from one screen to another – in the same fashion that a web browser renders html into a page and links to other pages.  With UML diagram in hand we can analyse our resources connections to one another and visualise how we might interact with them; as you can imagine this was incredibly useful during our development phase.



Figure 7 – Simplified resource example

Application State and Resource State

A few recent and frankly excellent pieces of research are emerging that seek to establish techniques for describing application state using State Diagrams and certainly could have helped us during our analysis phase – notably REST: From Research to Practice, Chapter 5, Beyond CRUD (Rauf and Porres 2011) [2] and Formal Modeling of RESTful Systems Using Finite-State Machines (Zuzak, Budiselic and Delac 2011) [3].  We also found State Diagrams useful for modelling our system, but we’re primarily interested in modelling our resource state transitions rather than application state transitions.  Our users manage resources according to easily understandable interaction patterns and we have limited need for wizards, or other controlled user interactions, therefore we seemed to have little need for sophisticated HATEOAS modelling in the early stages.  However, modelling resource state transition surely saved us many months of pain.

Once we had modelled our resource state transitions and attempted to map these transitions to a uniform interface based on HTTP methods and URIs, some interesting findings started to unfold.  As noted earlier, we really only needed to do this for one resource type as all our resources have a uniform lifecycle.  In our first attempts to map the transitions to URLs with GET, PUT, POST and DELETE we struggled to produce any sensible RESTful looking URIs, and our attempts to map the transitions to POSTs seemed to be getting away from RESTful principles.  In the end we were starting to acknowledge that something didn’t fit – a gap in our knowledge perhaps?

 Figure 8 - One of our early Resource State models


If we take a step back for a moment and look at the most popular REST architecture in the world – the World Wide Web – we can note that most interactions are read-only, safe, GET interactions.  This might go some way to explaining why we could not find the same depth of research for update based REST interactions as were available for read-only interactions.  By way of example, the reference books available at the time often dedicated just a handful of pages to updates, but hundreds of pages to other REST concepts.  Furthermore, some of the concrete examples within these books seemed to contradict one another.  Could we therefore conclude that no one was using REST to build Enterprise Applications which are predominantly update based or could we conclude that REST is not suitable for use in such a scenario?


Uniform Interface

The uniform interface constraints are described as follows by (R. Fielding, Representational State Transfer (REST) 2000):
“REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state.”
This is not Fielding’s last word on the subject however.  In (R. Fielding, Designing a New Protocol for the Web 2002) he adds the following constraint to the list of four above: “Resource-generic interaction semantics”.  This is in the context of creating a new Web protocol called Waka, where Fielding creates new methods for manipulating Waka resources which include RENDER and MONITOR.
Let us deal with each of these five constraints in turn.

Identification of Resources

We use URIs to identify all our resources; being defined by Fielding as any information that can be named, the number of resources in our system far out-weighs the number of domain objects.  Excluding a few well known service roots our resources are accessible by links from at least one other resource.  Although the use of URI templates [5] is still within the RESTful constraints, no meaning should be derived from our URI structure.  To convey meaning about any given resource we rely on the information within the media type just as Atom does with its link relations [6].


Manipulation of resources through representations

We are in the process of defining our representations, but it is likely we will take an approach similar to OData.  The media type to carry our representation is quite fixed, but the representation of our resources is continuously changing as the application evolves through both our enhancements and our customer’s specialisations.  To support this flexible payload requirement we think an Atom based media type with an Entry that supports a set of property values, as OData Property has done, will be the way forward.
HTTP defines a set of methods for manipulating resources, used by AtomPub and also OData for instance, for manipulating resources.  However, we found our individual resources could be uniformly manipulated using a different set of methods, just as Fielding found in Waka.  We discuss this in more detail below and this is the main subject of this article.

Self descriptive messages

We intend to define self-descriptive messages within our XML or JSON based media type with inline metadata for our representations.  The important point here is not to rely on ‘out of band’ information for understanding messages.  Fielding has criticised some other projects which define custom http methods, such as WEBDAV [8], for not being RESTful as they require out of band knowledge [9].  There has also being criticism [10] of OData for requiring user agents to construct URIs to the metadata and its failure to define a media type for the Atom Entry.  Both of these criticisms seem justified for large scale deployment scenarios due to the client-server coupling this introduces.

Hypermedia as the engine of application state

Usually abbreviated as HATEOAS, this is absolutely crucial for us.  The whole point of our interaction framework is to move session state out of the resource manager and in so doing separate session state from resource state.  The reason we chose the REST architectural style is because it uses hypermedia to change the state of the user agent (what REST calls application state) and thereby decouples the user agent from the resource manager. 

Resource-generic interaction semantics

Although a huge number of systems derive significant value from a RESTful architecture using HTTP GETs alone, many of the same benefits are available for applications that need to manage resources (that is, change their state).  In our case we manage a large number of resources and the main benefit is user agent decoupling.  However, we can also benefit from other infrastructure optimisations provided we establish very clear interaction definitions or reuse an existing interaction definition such as HTTP’s (Fielding, Hypertext Transfer Protocol -- HTTP/1.1 (9 Method Definitions) [11]).  Therein is the key, by establishing a very clear definition of the Uniform Interface any suitably informed user with a RESTful user agent can manage our resources without the user agent having any explicit knowledge of what it is managing. 

To determine the right way to transition resource state within our RESTful Enterprise Application we established principles to keep us on track: resources must have a uniform interface; we must not accidentally implement RPC; and we must not rely on out of band information for the user agent to interact with our resources.  Following our application analysis and modelling exercise, which revealed that all of our three and a half thousand resources support the same six interactions, we had high hopes of being able to define a suitable uniform interface without too much difficulty.  Three main options seemed to present themselves: one that reused HTTP methods GET, PUT, POST, DELETE; one that “overloaded” POST; and a third based on defining custom methods.


The Options for Resource-generic interaction semantics

As discussed in various REST research papers, the uniform interface constraints give rise to many of the strengths of a RESTful architecture.  One aspect of defining, or understanding, our Enterprise Application’s resource-generic interactions was to establish the safe and idempotent characteristics of each operation.  This knowledge not only affords many optimisations in the underlying infrastructure, but also provides a user agent implementer with a definition of any explicit guarantees for a given interaction.  Our application heavily utilised idempotent operations to ensure consistency of the underlying resource state, a single duplicate transaction could result in a significant loss for our customers, so we were keen to define a uniform interface that elegantly supported this characteristic.  As already discussed, HTTP defines a set of methods with well-defined safe and idempotent properties, but the question remained how to best utilise these for our interactions.  The options are summarised in Table 1 Uniform Interface Options.

Table 1 Uniform Interface Options
Interaction Description
Our Requirement
HTTP Method
Overloaded POST
Custom Method
Retrieve a resource
Safe, Idempotent
GET
GET
SEE (GET)
Move a resource from live to history
Unsafe, Idempotent
PUT
POST
HISTORY
Add an authorisation to a resource
Unsafe, Idempotent
PUT
POST
AUTHORISE
Remove authorisation from a resource
Unsafe, Idempotent
PUT
POST
REVERSE
Delete a resource
Unsafe, Idempotent
DELETE
POST
DELETE
Update a resource
Unsafe, Idempotent
PUT
POST
INPUT (PUT)
Create a resource
Unsafe
POST
POST
POST

Our Enterprise Application has two kinds of resources, as do most: collections and individuals. A collection might be /customers and an individual might be /customers/123. It is the individuals that could accept our custom SHARDI methods. Collections can be treated using HTTP methods, perhaps in future we could use AtomPub as an approach.  To create a new individual we POST to the collection, as is normal in REST.  However, in our case we POST to the collection to get back a new URI with a unique identifier.  This is a resource we can, if necessary, throw away.  This makes the inputting of a new resource idempotent and is similar to the approach in "Reliable Delivery inHTTP" [13].

HTTP’s use throughout the World Wide Web is clearly a great example of the success of a uniform interface and if we look more carefully it is also a great example of how infrastructure can support interaction by understanding the characteristics of the HTTP methods.  The ability for infrastructure at all levels to cache GET operations is one of the obvious ways we transparently benefit from its well defined safe characteristics.    By establishing a standard that makes explicit guarantees about a given interaction, as HTTP has done, many optimisations in the underlying infrastructure and user agents become possible.  On the whole these standards have been hugely beneficial to the web, but can of course be ignored and cause havoc as was the case for the Google Web Accelerator [12].


Our Enterprise Application certainly could benefit from an HTTP infrastructure optimised for GET operations, but perhaps less obviously, by understanding which operations are idempotent our interactions can be enhanced by also caching writes.  Imagine for a moment that you’ve built an email server with a RESTful interface.  An idempotent operation to mark an email as “read” is entirely cacheable.  This operation can be retried upon failure, at any time, and the result will still be the same.  If we mark a large number of emails as “read” there is also no need to wait for these operations to complete; our users can carry on with other tasks.

You could make a strong case for an interface based on HTTP methods as any RESTful user agent available today would be able use it (although most Web browsers can actually only use POST and GET today).  Any agent writer can take an HTTP connection class and start transacting with the resources.  Although our primary goal is not to open up our system to any user agent, this is certainly a very attractive feature.  Furthermore, an HTTP interface allows us to leverage community knowledge of the HTTP methods, readily reuse available test clients, and of course benefit from any existing HTTP infrastructure. 

As already stated, our product already makes extensive use of idempotent operations for managing resources and, when put to use in a REST architecture, this knowledge allows us to safely retry operations without risking unintended consequences.  For an interaction based on HTTP, the idempotent operations PUT and DELETE were certainly going to feature quite heavily in our design.  The challenge then seemed to be how to create a sensible URI scheme for our resources.  In keeping with RESTful principles, recall that a resource is any information that can be named.  Put another way, REST is based on nouns and the few methods in our uniform interface are the verbs.

Option 1: HTTP GET, PUT, POST and DELETE

By far the greatest advantage of using HTTP’s resource-generic interface is that any RESTful user agent available today could use it.  Any user agent developer can take an HTTP connection class and start transacting with your resources.  Furthermore, there is a wealth of community knowledge for the HTTP methods, readily available test clients, and the existing HTTP infrastructure has been optimised for them.  However, we found that creating suitable resource state transitions within the RESTful constraints can be difficult to understand and difficult to implement correctly.


If all resources need to be named, we can easily check the logic of our URI scheme by simply asking ourselves “Is this URI a noun?”, or alternatively “Have we created a URI that is a verb by mistake?”.  We are going to create a noun based URI scheme and transition the state of a fictitious Customer resource.
This fictitious example of transitioning a Customer resources’ state is going to require a little imagination.  To avoid getting bogged down in our HATEOAS interactions we are only going to show the requests that are needed to transition the resource state and the accompanying response headers.  In the real world we are of course going to navigate our way through this resource transition as guided by the server through our chosen media type.  To begin, imagine that we have navigated our way to a GET /customers HTTP/1.1”.
The first representation we see offers us several choices for managing customer resources; the /customers/new resource provides a way to create new customers. 

Request
POST /customers/new HTTP/1.1
Response
HTTP/1.1 200 OK
...
[body includes a form that can filled out for subsequent PUT /customers/123/unauth]

POST seems to be a good choice here as the operation is neither safe nor idempotent.  A POST to the collection resource returns a throw-away-able resource in the form of the next available customer ID (ID 123 in this case).  We return an HTTP 200 OK response as the next available ID has been incremented, but nothing has been created at this stage.  Once we’ve filled out the form our next interaction with the server is to create the customer content.

Request
PUT /customers/123/unauth HTTP/1.1
Response
HTTP/1.1 201 Created
Location: /customer/123/unauth
...

At this point we have now created a Customer.  However, you might have noticed that the server instructed us to PUT our Customer to an unauthorised version of the customer.  In our application, every resource creation or modification should be authorised by at least one other individual.  To perform this authorisation we provide an authorised user the ability to add their authorisation to a resource.

Request
PUT /customers/123/unauth/authorisers/Bob HTTP/1.1
Response
HTTP/1.1 201 Created

Bob has now authorised this new Customer.  PUT is a good choice here as Bob could PUT his authorisation on this resource any number of times, but logically he can only be added as an authoriser once.  This particular resource actually requires two authorisers; it will need a second approval from Bob’s manager Louisa.  Louisa has a reputation for being on the ball and has spotted a problem - this Customer already exists.  At this point Louisa decides it is best to remove the duplicate Customer and inform our unwitting data entry clerk of their mistake.

Request
DELETE /customers/123/unauth HTTP/1.1
Response
HTTP/1.1 202 Accepted

Louisa’s action has marked the Customer for deletion in this evening’s overnight run as indicated by the 202 Accepted response.  Again, DELETE can be performed any number of times, but it is still only possible to be flagged for deletion once.  Another manager who wasn’t so on the ball and wasn’t aware of the duplicate Customer might have added their authorisation with the following request:

Request
PUT /customers/123/unauth/authorisers/Nick HTTP/1.1
Response
HTTP/1.1 303 See Other
Location: /customers/123

This example shows that we can perform our SHARDI interactions using standard HTTP methods as can also be seen in Figure 9 State Machine Diagram SHARDI-HTTP.  By using a combination the URIs we return in the representation of our resources and the allowed methods for those resources. The complete mapping of SHARDI to HTTP is shown in Table 1 Uniform Interface Options on page 7 above.


 Figure 9 State Machine Diagram SHARDI-HTTP


Option 2: Overloaded POST

An alternative to mapping our methods to HTTP methods is to use POST for all the operations that are not safe, indicating somewhere in the URI what the operation is, either with query parameters in your URI, by defining the URI in a verb like way or using some other element which breaks our principle of not using remote procedure calls in our URIs.  In addition to breaking that constraint, we are now unable to tell the infrastructure if our operation is idempotent and so lose the advantages of being able to retry these operations.

Many resource interfaces that claim to be RESTful break this constraint, as the difference between a PUT to a noun and an POST to a URI with the verb is quite subtle and we’re happy to admit to that we have created interfaces in the past that required user agents to POST to URIs along the lines of /myresource?action=doStuff.  Someone has even given this technique a name -“overloaded” POST [14], but this is not how we want to build our uniform interface as it breaks the principle of avoiding remote procedure calls in our resource interface.


Option 3: Custom Methods

Another alternative to mapping our SHARDI methods to HTTP methods is to provide a new resource-generic interface for our resources which explicitly uses the methods SEE, HISTORY, AUTHORISE, REVERSE, DELETE and INPUT (SHARDI).  Three of these can actually be replaced directly with the equivalent HTTP method as SEE has the same meaning as GET, INPUT as PUT and DELETE as DELETE.  That leaves just three methods for our new generic interface HISTORY, AUTHORISE and REVERSE.  In each case these create a new resource.  In the case of AUTHORISE, a new live resource is created. In the case of HISTORY, an archived version is created.  In the case of REVERSE a new unauthorised resource is created.  However, just as in the case of INPUT, which is idempotent because we already know the unique identifier of the resource, these are idempotent methods.

There is a precedent for adding methods to HTTP interactions.  This is what WEBDAV has done in its use of HTTP and what Fielding intends to do with Waka.  WEBDAV added methods for manipulating folders and files which includes PROPFIND, MOVE and COPY.  Fielding has criticised WEBDAV, and specifically PROPFIND, for not being RESTful by breaking the constraint that every resource should be addressable by a URI.  However, we think it is possible and sometimes desirable to introduce new methods in an HTTP exchange provided we don’t break the RESTful constraints.

In particular we need to ensure that a SHARDI user agent can discover which resources allow the SHARDI methods without requiring out of band information.  This is crucial as we want to design for the long term, so that a user agent written today can work with resource managers delivered tomorrow and vice versa.  We intend to achieve this by using the HTTP Allow header to communicate the valid next state transitions for a resource and furthermore we aim to allow the client to request a resource state transition, but we’ll make no guarantees this transition will occur as per the HTTP specification for Allow.  We will however advise what transitions are allowed when asked to do so via OPTIONS, and return 405 Method Not Allowed like a good citizen when a transition is not valid.

This is how the same interaction from option 1 would look with our SHARDI Customer resource-generic interface:

The first representation that we see will still offer us several choices for managing customer resources and one choice will allow us to create new customers. This is the /customers/new resource.

Request
POST /customers/new HTTP/1.1
Response
HTTP/1.1 200 OK
...
[body includes a form that can filled out for subsequent PUT /customers/123/unauth]

No change here, POST still seems like the best choice as the operation is neither safe nor idempotent.  Once we’ve filled out the form our next interaction with the server is to create the customer.

Request
INPUT /customers/123 HTTP/1.1
Response
HTTP/1.1 201 Created
Allow: GET, HEAD, INPUT, AUTHORISE, DELETE
...

At this point we have now created a Customer.  However, you might have noticed that the server has advised us that we can now attempt to AUTHORISE this resource.  To perform this authorisation we need to be an authorised user, but this request may result in 405 Method Not Allowed if someone else has already authorised this resource.

Request
AUTHORISE /customers/123 HTTP/1.1
Response
HTTP/1.1 200 OK
Allow: GET, HEAD, INPUT, AUTHORISE, DELETE

Bob has now authorised this new Customer.  AUTHORISE is an idempotent operation and as such Bob could AUTHORISE this resource any number of times, but assured that this authorisation will only occur once.  From the Allow header we can tell that this resource needs another authoriser; it will need a second approval from Bob’s manager Louisa.  There is no change to the DELETE interaction and Louisa still needs to inform our data entry clerk of their mistake.

Request
DELETE /customers/123/unauth HTTP/1.1
Response
HTTP/1.1 202 Accepted

We now know Nick is not as on the ball as Louisa and he simply authorises in the same fashion as Bob.

Request
AUTHORISE /customers/123 HTTP/1.1
Response
HTTP/1.1 200 OK
Allow: GET, HEAD, INPUT, HISTORY

This example shows that we can perform our SHARDI interactions using custom HTTP methods as can also be seen in Figure 10 State Machine Diagram SHARDI-Custom.  Using a combination of custom http methods and the Allow header we have established a resource-generic interface semantic that is much easier to negotiate than our HTTP equivalent, plus we have not strayed from the RESTful constraints at any point.  The complete mapping of SHARDI to Custom methods is shown in Table 1 Uniform Interface Options on page 7 above.

Figure 10 State Machine Diagram SHARDI-Custom



I've made an attempt to discuss three options to transition resource state, but there are certainly plenty more e.g. the use of custom link relations.  When searching for alternatives my main aim would be to simplify the communication of resource state change options and a mechanism that provides a self-descriptive nature that could inform the infrastructure of our interaction semantics.   


Conclusion

Enterprise Applications like ours have a legacy of user interaction issues caused by tight coupling of the client to the server.  We think REST is our vehicle to achieve openness, innovation at user interaction, and web scale.  In a structured, constrained, and regulated environment of predominately record keeping activities a resource-generic interface such as SHARDI could allow us to greatly simplify resource state transition.  However, the HTTP resource-generic interface is more suitable because of the user agents, tooling, and infrastructure already built on this standard.  Creating self-describing resource-generic interfaces could simplify resource state transition and provide opportunities to optimize unsafe, idempotent operations within external apps and infrastructure.



Bibliography



[1] R. Fielding, "Representational State Transfer (REST)," 2000. [Online]. Available: http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm.
[2] I. Rauf and I. Porres, "Beyond CRUD," in REST: From Research to Practice, 2011.
[3] I. Zuzak, I. Budiselic and G. Delac, "Formal Modeling of RESTful Systems Using Finite-State Machines," 2011. [Online]. Available: http://www.springerlink.com/content/j233175h2q3h7631/.
[4] R. Fielding, "Designing a New Protocol for the Web," 2002. [Online]. Available: http://www.gbiv.com/protocols/waka/200211_fielding_apachecon.ppt. [Accessed 2011].
[5] J. Gregorio, "IETF URI Templates," 2007. [Online]. Available: http://tools.ietf.org/html/draft-gregorio-uritemplate-08.
[6] M. Nottingham, "The Atom Syndication Format," 2005. [Online]. Available: http://www.ietf.org/rfc/rfc4287.
[7] OData, "OData," circa 2008. [Online]. Available: http://www.odata.org/.
[8] E. L. Dusseault, "WEBDAV," [Online]. Available: http://www.ietf.org/rfc/rfc4918.txt.
[9] R. Fielding, "No REST in CMIS," 29 09 2008. [Online]. Available: http://roy.gbiv.com/untangled/2008/no-rest-in-cmis.
[10] D. Miller, "Oh Data," 2009. [Online]. Available: http://www.bizcoder.com/index.php/2009/11/30/oh-data/.
[11] R. Fielding, "Hypertext Transfer Protocol -- HTTP/1.1 (9 Method Definitions)," June 1999. [Online]. Available: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html.
[12] "Google Web Accelerator," [Online]. Available: http://37signals.com/svn/archives2/google_web_accelerator_hey_not_so_fast_an_alert_for_web_app_designers.php.
[13] P. Prescod, "Reliable Delivery in HTTP," [Online]. Available: http://www.prescod.net/reliable_http.html. [Accessed 11 October 2011].
[14] "Overloaded Post," [Online]. Available: http://blog.robustsoftware.co.uk/2009/07/post-overload-implementation.html.