Hypermedia APIs: Document resources through Vocabularies

In my previous post, I discussed the need to adhere closely to the HTTP specifications.  In a similar way, in this post I would like to discuss the need to standardize your resources and actions through vocabulary definitions.

Much of the work done while designing CRUD APIs is spent building the URI resource hierarchies and determining how to reveal behavioral capabilities through non-standard designs to consumers.  There are many well-known and supported frameworks and technologies to support the CRUD API design pattern including OAS formerly Swagger, RAML, Apiary.io and more.  However, these techniques all result in interfaces which are tightly coupled to their clients, are very brittle, and become increasingly difficult to maintain over time.  Hypermedia APIs can alleviate many of the symptoms, but simply adding hypermedia to a brittle foundation (like Spring-HATEOAS) will not resolve these issues.  The problem with all of these approaches is the static nature of the API definition, the contracts they publish or describe are not expected to change over time.  If the contract is to change in any way, some external mechanism for handling the change is to be employed, and coordination or intervention is expected to resolve any disparities resulting from the change.  In other words, the API provider makes changes and the all the consumers clients break.

The way to get around the fragility and brittle nature of the static binding is simple, don’t guarantee a static contract.  Instead your design should aim to publish or subscribe to dynamic contracts which are malleable, robust, and capable of supporting constant interrogation.  It is much less effort overall to define domain vocabularies which encapsulate the resource and message representations and behaviors which obviates the need for strict static contracts.  There are a variety of ways to produce these vocabularies, including Schema.org and Application Level Profile Semantics (ALPS) among others.  The suggested approach will vary based on the use case, however Schema.org definitions are tightly coupled to the HTTP transport making ALPS the better generic decision for a transport agnostic vocabulary definitions.

There are a few often cited perceived drawbacks to this approach, and they fall hand in hand with the standard arguments against hypermedia APIs.  First, is the enhanced complexity of creating vocabularies makes the design process more difficult overall, not less.  The second is the use of these published and discoverable semantics makes the consumption of the service a more complex and costly process.

The simple answer for the first perception is to understand the vocabulary definitions are not static.  While it is good practice to attempt to account for the future the dynamic nature of the vocabulary definitions provides the opportunity to fix design mistakes and other changes.  All too often voices from the semantic web community clamoring for the creation of perfect vocabularies are confused for proponents of hypermedia driven API design who simply advocate for designs to promote lower coupling.  More importantly, the barrier to entry for contributing value in vocabulary design is not limited to the highly technical members of your team.  Members of your team with domain expertise can play a much more direct role in the design of vocabularies, allowing more technical members to focus their effort on more technical tasks.  Quite simply this perception is a fallacy, the design process should be no more complicated to define a vocabulary than to barter away on the URI structure of a CRUD API.

Discoverable semantics, late-binding definitions, or follow your nose APIs; regardless of how you want to refer to the concept there is truth to the notion that a hypermedia API is more complex to consume.  Despite this admission the sky is not actually falling on the hypermedia concept.  In fact this small complexity is one of the tradeoffs which allows the API and clients to be more loosely coupled.  A vocabulary defined hypermedia API enhances the API design flexibility while increasing the clients’ longevity by strongly urging the client design to discover the API’s resources where the consumer can go next.  Our first guideline has come back into play to provide us a solution to another difficult problem.  The service does not need to be discovered on every single request to the API.  In fact, all that needs to be done is to build a client which responds appropriately to correctly formatted cache control headers from the HTTP specification and we will greatly reduce the overhead of discovery.  Additionally, by leveraging different HTTP status codes we can broadcast to consumers a temporary or permanent move of a resource.  With these statuses the server is able to gently broadcast deprecation of resources providing a buffer period to allow clients and consumers time to adjust to the evolved API.  This greatly reduces the stress and pressure on a designer to create a perfect API on the initial version by providing non-breaking ways to make changes over time.

Both of these common misconceptions about hypermedia APIs result from a fundamental misunderstanding of the benefits and difficulties of hypermedia.  The burdens are overblown, the perceived risk is amplified, however a well-designed API should see a net reduction in complexity through its initial release and beyond.

By leveraging the documentation of your resources through vocabulary definition your design greatly reduces its fragility in the face of change, and your consumers can expect a much longer lifetime from their investments in clients.

Hypermedia APIs: Embrace the HTTP standards

In my previous post I established some guideline constraints to follow when designing a hypermedia API.  Heavy emphasis on constraints, as the complexity we intentionally introduce now will greatly reduce the complexity of larger scaling and performance issues in time.  In this post I will to discuss the first constraint, ‘The interface MUST embrace the underlying HTTP protocol’.

If you haven’t spent time on IANA or IEFT websites perusing the nearly endless supply of standards papers filled with normative and dry text, then you probably would consider yourself lucky.  There is just an absolute mountain of current and proposed standards which fall under the HTTP umbrella, so you would be forgiven for not immediately rushing there to read everything related to HTTP.  However, as an API designer you are responsible for the behavior of your design and you really should be familiar with the entirety of the basic specification of HTTP, as well as some of the more popular or useful additions.

I’ll step back and ask the obvious question, why is HTTP so important?  A restful architecture is not a goal to be sought in its own right, but for the benefits the constraints offer to the overall performance and behavior of the application.  By ensuring adherence to standards, you can expect a large range of clients to support your service immediately and for an extended service life without the burden of further technical intervention.  Perhaps most importantly, adhering to standard functional definitions within the protocol will allow a service to fully leverage the benefits of a restful architecture.  Presumably you are designing a service to send messages through HTTP, in that case as an example it would be beneficial to know when the protocol itself has a solution to a particularly difficult asynchronous application problem, like resource locking.

There are many reasons why asynchronous systems require the ability to lock a particular resource, be it exclusive access to physical or hardware resources, simultaneous editing, or even reserving items for a specific user’s purchase.  There are two main approaches to locking ‘pessimistic’ and ‘optimistic’.  Pessimistic locking requires the service to maintain the state of a resource as owned by a particular entity, but within a restful architecture we really really don’t want to add statefulness.  This is not an ivory tower argument at all, the primary desire for statelessness in this case is the driven by the desire to allow caching.  If a design were to introduce statefulness on any resource, it would be impossible for any caches to ever retain the resource as there would be no way to determine its current state.  This leaves us the optimistic approach as the only viable option for a restful hypermedia API.  Fortunately, the HTTP specification has a standard way to support optimistic locking.

To utilize optimistic locking, an HTTP service will provide metadata about a resource representation when responding to a request through the inclusion of an ETag header with a unique value for the resources state at the time of the response.  When a consumer would like to modify the resource they will simply conduct the appropriate request along with supplying the previously returned ETag header value within the ‘If-Match’ header field and ‘return=minimal’ within the ‘Prefer’ header field.  If the resource is unmodified, the service can perform the update and return a 204 status code, with a ‘Preference-Applied’ header value of ‘return’ and an empty body.  However, if the resource has changed, the service can return a 409 conflict response while ignoring the request preference and returning the current representation, or with a ‘Location’ header with the resource URI and the ‘Preference-Applied’ header of ‘return’.  It is important to note there is flexibility for implementation details within the protocol specifications for optimistic locking and most use cases should be covered.  For applications intended to scale and effectively utilize a restful architecture caching is crucial to the performance of the system, and in the scenario presented above the resource is always cacheable.

This entire scenario is certainly more engaged than a simple request to a lock sub-URI on a resource, however a very large portion of a very difficult problem has already been handled through a standard, well known, and supported process.

Embracing the use of HTTP constructs to solve application concerns for your API, will allow you as the designer to focus on tasks which build value in your product and avoid extraneous boilerplate design.

Development Guidelines for Hypermedia Web APIs

I have a confession, I have written the same application many times, probably just like you.  I’m sick of it.  I just don’t get the feeling of accomplishment when I implement core API functionality for the 10th time.  It just isn’t fun anymore.

The same discussions, the same concerns, and the same technology always results in the same application.  I do work as an API designer and developer, so I can see how there is so much overlap between projects.  Yet, in retrospect I’m forced to admit I’ve spent a lot of time doing work which wasn’t very novel and could have been better spent adding value to my employers and clients.

The tools and techniques to focus on domain value didn’t exist, and we all work with the tools we are provided, but now there is a way to look past much of the previous work and focus on getting products done.

Hypermedia Web APIs.

Just to be clear, hypermedia APIs are not a silver bullet swiss army knife to do everything you need, but they come pretty close.  This post is going to be the start of a series of posts where I discuss hypermedia web APIs and how to leverage their power and minimize the perceived or real pain created by the added complexity.

Guidelines for Designing Hypermedia Web API

The following design constraints will provide a strong foundation for your API design.

  1. The interface MUST embrace the underlying HTTP protocol.
  2. The interface MUST document resources and resource capabilities through vocabulary definitions.
  3. The interface MUST present a home document to publish resources and documentation.
  4. The interface MUST define all resources atomically, and MUST flatten resource representations.
  5. The interface MUST NOT couple or document tightly to specific URI paths and patterns.
  6. The interface MUST NOT include any versioning in its representations.
  7. The interface MUST expose applicable resource capabilities through hypermedia controls.
  8. The interface MUST respond to consumer declared goals if the goal is understood.
  9. The interface MUST be decoupled from hypermedia format, the format MUST be negotiable.
  10. The interface MUST promote flexible design, it MUST NOT present breaking changes.
  11. The interface MUST extensively leverage content negotiation.

Got that, no? Well, like everything else these days that is just the tip of the ice berg.  The key is to leverage existing standards and functionality to get this done over HTTP.

Over the next few posts I will delve a little deeper into each point, explain why we are bucking some industry trends, and begin a high level review of how to use this guide to design a hypermedia web API.

Much thanks goes out to Mike Amundsen, Steve Klabnik, Ruben Verborgh, and Kin Lane for providing extremely valuable blogs, talks, links, and dissertations to understand this stuff.