I’ve recently discussed the problems with nearsightedness when designing APIs through the comparison of a OAS (Swagger) API to a hypermedia API.  These discussions have been very technical, targeted at an audience of API designers largely without addressing business and economic ramifications.  In this post I’d like to take a step back to remove my technical hat, and talk about the business, economic, and human benefits of supporting and maturing the proliferation of hypermedia APIs.  I’ll go into some differences between these two options from a business perspective to demonstrate the massive value of hypermedia APIs.  I will end with a little on a related topic, the undue influence on the direction of technology from the venture capital backed world of hyper growth.

Your API is your product.

Let’s talk about the elephant in the room; I would like to address the extremely common misconception that an API is nothing more than your UI app’s gateway to data.  The term REST as the industry understands and uses it, reduces the value proposition of your API development to little more than a gateway between your product, and how you store your product’s data.  It may provide some functionality, enhance performance, and shape the data in a way which is beneficial to the UI app development team, but it provides no value in itself.  Not only does this go directly against the path of progress towards the API economy, but it wastes the opportunity to save time and money on redundant parallel effort.  You have driven initial costs up with duplicated development effort, and maintenance costs also go up due to similar bugs in many places.  Perhaps the most critical effect of this mistake is you have almost certainly increased the time to market for your entire solution.

Your customer is everyone outside your API.

It may be difficult to look at your API as anything more than a means to provide your real products with the data they need to create value, but this thinking is guaranteed to hit your bottom line in a big way.  The API is your product, and anyone interacting with it is your customer.  The internal team developing your new mobile app?  Your customer.  The group responsible to maintain your web application?  Your customer.  The outside parties looking to utilize your service, without having to use your mobile and web apps?  Your customer.  Each of these groups share the exact same goal, they want to utilize the functionality your API provides as quickly and easily as possible.  If your own employees, and your customers all share the same goal, you are missing a tremendous opportunity to capture efficiency gains by making your API easier to use.  Anyone using your API regardless of their affiliation with your company wants to learn as little as possible about your API to meet their goals.

Solutions like OAS offer short term benefits to your product.  Developers can quickly get up and running, there is ample documentation explaining the ins and outs of your system, and users can utilize this to leverage your products very quickly.  The catch is these solutions offer extremely poor performance in the long term.  Over time your customers will need to constantly maintain their code, continue to need to read and understand your business models to meet their goals, and their quick solutions turn into a nightmare of legacy code to fix.  The ample documentation you thought was such a victory becomes a high barrier to entry as your service matures.  The result is an extremely dissatisfied customer, one who won’t refer your product to a friend or colleague, who is only looking for a better opportunity to present itself before they leave your product in the past.  When they are gone, they aren’t likely to come back, they already know how bad it is to use your product.

When you decide to use a product like OAS to form the foundation of your API, you prioritize your needs over the needs of your customers.  The short term benefits of OAS disappear quickly, but the long term negative effects to your business and brand will be extremely costly to remedy.  A product which puts the priorities and goals of the customer front and center will drive referential sales, creating buzz and goodwill in the marketplace surrounding your brand.  If you want to create long term goodwill and revenue security, you need to prioritize how people feel about using your product.

You can’t sell a mega product.

Technology is forcing businesses to change how they sell their products to the market.  The concept of selling an entire package of solutions is quickly yielding to selling smaller incremental sets of solutions which can be independently acquired and used when the customer needs them.  Digital products are quickly becoming commoditized to the logic and value they add to your customers’ business processes.  If your business model is not changing, you will likely soon find your target market has dried up.  Inferior, but modularized products will take the place of your products as customers learn to carve out the functionality they need without unnecessary costs or complexity of larger bundled solutions.  Every organization is in a race to the bottom on cost to provide products and services, if you force a customer to buy products they don’t need, you are continuously inviting them to seek alternative products.  When you promote goodwill and engagement with your customers, the easiest sales channel of current customers gets even easier to sell enhanced services.  Your customers are more likely to buy additional products from you when they are already satisfied with their current solutions.

You can’t sell a mega product, but you might be able to sell a customer all the parts of one.  Fixed API designs like OAS make segmenting your products difficult and unintuitive, while requiring a lot of management overhead which cuts deeply into the margin for the product.  The dynamic design of hypermedia APIs allows your product to be segmented naturally.  This enables marketing initiatives to directly target specific functionality and customer pain points, while requiring very little additional overhead to reduce the profits.  If your segmentation isn’t intuitive and it isn’t easy to determine where the functional and license boundaries lie, your customer experience suffers dragging your future sales potential down with it.

Your business probably isn’t hyper growth.

It is difficult to look at the success of companies like Facebook, Netflix, Amazon, and Uber and resist the temptation to copy the way they operate, however its very likely the needs of this niche market do not match your organization or your industries’ needs.  The move fast and break things mentality of Silicon Valley and other venture capital funded startup hubs pairs extremely well to the short term benefits of API designs like OAS.  In the venture world a problem two weeks away can feel like two or three lifetimes.  Companies who intend to hyper scale for acquisition aren’t concerned with their customer’s success in 2 and a half years, because after they sell in 2 years it will be someone else’s problem.  Google and Amazon build, try, and sunset so many products that it’s folly to waste time concerning themselves with long term benefits to themselves let alone customers.  If you are reading this then there is a very good chance your needs don’t align to such short term goals.  Trying to operate using the same tools and methods as these hyper scale companies will do a disservice to your customers and your brand.  Your business model likely is concerned with your customer satisfaction in two, five, even 10 years.  Hyper scale companies have, and continue to develop, good tools for their needs, my advice is to look carefully to see if those tools fit your needs, because it’s likely they don’t.

Hypermedia APIs are simultaneously an extremely proven design, and an unexplored frontier.  The internet itself runs upon the very same principles of a well-designed hypermedia API.  Developer tools for this space do currently lag behind the alternatives like OAS.  Fortunately, they have the same potential for speed to market, prototyping, and integration tools.  Investments in developing hypermedia APIs and the tooling around them are investments for the future on the scale of decades.  Hyper scale companies have created tooling which prioritizes their goals of short term gains, if your company is not primarily interested in short term gains then it is up to you to create the tools which prioritize the long term benefits to match your goals.  The long term benefits to your company and your customers of developing a hypermedia API have no equal, there isn’t even a good alternative to compare.  If your business is concerned about long term market sustainability, revenue, and customer retention you should be looking into hypermedia APIs.

In my last post I discussed the start of the next group of guidelines with the use of vocabulary provided goals to curate hypermedia interactions with the service.  This exciting idea will allow truly domain driven interaction with your service, while remaining stateless and easy to consume.  The next guideline is more of a cautionary tale, and that is the design should be decoupled from the hypermedia format of choice.

When hypermedia is discussed today, I imagine the conversation ventures around the room discussing the different available formats.  You’ll hear mention of HAL, JSON API, JSON-LD, Hydra, Siren, and Collection+JSON among others.  The pros and cons of each decision are weighed, and eventually a consensus is reached and the team decides to use ‘X’ format.  The particular format picked is irrelevant to this discussion, however there is a chance the format picked will include something it shouldn’t; it will include specifications for URL patterns.  The problem with this is as the hard won victory of building the hypermedia API and client allows near effortless consumption of the service, formats which specify URL patterns, greatly increase the odds a consumer will cheat and bind to a URL besides the root.

However, this isn’t the only concern for supporting a hypermedia format at specific URL patterns.  Suppose you had a requirement to support another format as well?  Not a problem, that format can just use these URLs, as hypermedia makes the URLs irrelevant.  What happens when the requirement for a 3rd format comes in, and this one also has a specific format it requires?  Well things start to break down here, and the service needs to start managing the context between multiple endpoints which are synonymous with each other.  This creates a variety of problems you really don’t want to have to deal with, like reduced caching and cache inconsistency as different URLs aren’t cached as the same resource.

The easy, short, and best answer is to simply avoid utilizing the formats which prescribe a URL pattern, if possible leave out those portions of the format which you can, and if that doesn’t work hopefully one of the other fantastic alternatives will provide the right set of attributes to fit your initial use case.

In my last post I addressed the worst acronym ever, HATEOAS, and how to truly have hypermedia drive the stateful interaction of your application.  The discussion rounded out the more standard guidelines for creating hypermedia APIs, creating a nice foundation for understanding for the next four guidelines which are part of the forthcoming hAPI specification to drive adoption of hypermedia APIs through reduced complexity and better tools.  In this first post in the series addressing the next stage of hypermedia APIs, I would like to address goals.  Specifically, I would like to address the goals of the consumers of your API, and put some serious effort behind helping them achieve their goals.

If you take a step back and look at the interaction of a CRUD, you should be able to see a usage pattern which relies heavily on understanding the model of the service provider’s implementation to understand what interactions are required, and in which order to accomplish a larger goal.  Due to my history in the financial sector and common familiarity with banking, I tend to use the creation of a checking account as a good example to demonstrate the issue.

Suppose you had the following CRUD APIs:

/account
/address
/person

If you wanted to create a new checking account, considering documentation like OAS doesn’t show larger interaction arcs, what would you do?  The most likely implementation scenario requires you to create a person with an address, and then use this person and address to create an account.  The problem is I had to reason my way through to this conclusion as the service implementer.  As a consumer I don’t, and shouldn’t be forced to, care about the concerns of implementing a service in order to consume it.  This was an easy enough example, so you might be inclined to shrug this off as manageable.  If you have not, well then I’m sorry for your trouble because you have been through the same pain I have, it wasn’t fun.  If however, you have yet been spared from the joy of integrating a service designed by someone else’s undocumented and internal data model of confusion, then I have concocted just the elixir for you to prove just how real this problem is.

Assuming at this point, we’re all on the same page regarding the previously mentioned pain we need to look at the hypermedia solution, which is without a doubt a much friendlier interface to a very similar albeit muted frustration.  You see, despite the ease of engagement with the service being significantly better, I am still required to understand the domain model and internal implementation and composition structure enough to make value judgements to determine how to crawl the service intelligently.  As a consumer correctly discovering the service, I have two choices, try everything, or try everything while guessing at relationships.  Neither of which sound particularly appealing, but at least it’s better than using CRUD.

I propose a third option, as part of the vocabulary for the service define domain relevant goals which can be provided by a consumer to express their domain intent for consuming the service.  In the banking example above, it would be much easier if profile linked by the home document contained a goal of “new-customer-create-account” which I could provide to the server in order to tailor the hypermedia of the responses in order to steer my client towards the new account goal.  Hypermedia APIs are a great leap forward in usability, by using goals with hypermedia we can greatly reduce the interaction difficulty and enhance the speed at which we can integrate and release new APIs.

The hypermedia API designer should not only look to create the appropriate vocabulary for the service, but also look to encapsulate the larger goals of the domain to provide stateless hypermedia curation for their consumers!  Together this will allow you to reduce the amount of knowledge a consumer needs to have about a particular domain or implementation in order to successfully consume it.

In my previous post in this guidelines series I discussed many reasons why versioning should not be introduced into your API, despite the existence of convenient tricks to hide some of the side effects for a time.  Many leading tech organizations argue the opposite, which might be reasonable when the likelihood of any particular API aging long enough to get past v1 is extremely low.  However, these organizations aren’t in the business of creating reliable, flexible, and enduring APIs for consumers in the long term.  In this post I’ll discuss perhaps the most mispronounced and ill-conceived acronym in an industry obsessed with acronyms: HATEOAS.

The terrible and often thrown about acronym stands for “Hypermedia as the engine of application state” and is possibly the most frustrating part of Roy Fielding’s entire dissertation.  Once understood the concept is simple, the problem is for better or more likely worse this short sentence represents the entire discussion of hypermedia in the paper.  As one of the primary fundamental tenets to a RESTful application, Roy spent precious little time in his dissertation to expand upon this rather arcane phrase.  To be fair to Dr. Fielding, he is quoted at saying he did want to actually graduate, his dissertation is foundational to many movements within technology, and an in depth discussion of HATEOAS in his paper would have been a large additional undertaking.  With that in mind, I’ll go through it in a quick manner.

The concept boils down to a very simple principle, the state transitions in the application should be invoked by hypermedia driven links.  You can throw the endless discussions on ‘nounifying’ verbs, and shoehorning data models into a CRUD paradigm to match the 4 commonly used HTTP methods.  Your resources are your resources, and their representations are whatever is required by the domain models, and that’s good because as previously discussed you should spend a lot of attention on your resource representations.  By clearly separating stateful transitions from your representations you maintain a stateless interaction and greatly reduce the complexity of your representation design.  This still leaves us with the need to present the current actions available to any particular resource to a consumer of the API.

The good news is, we’ve already set the stage with all the requirements to utilize hypermedia to control the actions of the resources through the vocabulary definition in the profile.  Regardless of the hypermedia format you are going to use, there are two components to the hypermedia you need to present for resource actions, these are the link, the rel name.  The link is URL for the consumer to follow to submit the next request.  This link may be templated in the certain cases but should generally be provided to the consumer fully composed.  This URL is provided by the service and is not constructed by a consumer, however it can be augmented with query parameters like filtering, sorting, sparse field sets, and more.  The rel name is a word in the vocabulary which corresponds to an action a particular resource can take.

The final piece of the puzzle is how do we take the vocabulary of resource and action representations and turn it into an engine of application state.  Up until this point the actions described in the vocabulary have consisted solely of a name, the rest of the definition is the type of transition and the messages or templated messages to send.  In general terms, you have safe, unsafe, and idempotent actions which can be performed on resources.  Through the use of a protocol binding profile we can get a good mapping of the profile semantics to HTTP request types.

With all of these pieces in place, we have all of the components necessary to start our engines!  The hypermedia in a representation is included dynamically in the message as the state of the resource requires to give the user choices in interacting with that resource.  Suppose you had a collection of person resources, your home document guides your hypermedia client to the root of the person resources, which is a collection of all person resources.  In this case it would be helpful to include such links as self, profile, and next to provide both documentation context to the client if necessary through profile, and an easy way for the client to know how to interact with the collection immediately.  Perhaps the first person in the collection was of interest, and the client navigates to the link for the individual person resource.  Suppose the person had a ‘doing’ property which could be sitting, standing, walking or running, and the current value of this person resource is standing.  The server would be able to include hypermedia controls with names ‘sit’, ‘walk’, and ‘run’.  The exciting part is that the client can navigate entirely based on the vocabulary presented to it, as the message and protocol bindings are provided, the client is merely responsible for composing the message as described from the parts available to it in links provided by the service itself.  The client composes a run message, and submits it using the appropriately bound http method to the URL provided in the link and the resource state transition is handled without the client ever having to consider anything about the protocol itself.

With the power of these orchestrated interactions using the right clients you can quickly interact with a hypermedia API with very little previous knowledge of the service or it’s domain.  With this type of interaction model, you don’t try to squeeze behavior into a 4 verb vocabulary, you write the vocabulary and messages which makes sense for your task, and use a protocol binding to directly map them to interactions.  Now we have a more in depth understanding of the mechanism which makes the URL pattern irrelevant to the consumer.

As a developer designing and implementing APIs for the past five years, integrating external services has always been a key component of developing the products.

As you sit in your design meeting, scrum, or stand-up the moment a new integration is mentioned, you can see the pause spread throughout the room and trepidation go through each colleague’s mind.  It is painfully obvious everyone is sharing variations on the same thoughts and questions.  How good is the documentation for this service? How long will it take to wade through the idiosyncrasies and bugs to a stable implementation?  Without knowing the specifics, everyone in the room is instantly aware of the landmines waiting for them.

These common concerns are entirely with cause, the quality range for services you may have to integrate provides a near limitless combination of difficulties.  The service being entirely undocumented isn’t even the worst case, as untrustworthy but thorough documentation can be much worse than discovery by trial and error.

Unfortunately, when implementing our own services, we often overlook or deprioritize the ease of use of our designs for the end user.  It’s an easy trap to fall into with deadlines and deliverables, that is precisely why it is so important we use designs and tools which make this simple.  Through the specification wars of the last 5 years, the CRUD-REST industry has settled on the Swagger specification (Open API Specification – OAS) as the standard for application design. While this represents real improvement over snowflake services, the use of a vocabulary driven hypermedia approach gives us all the beneficial properties of OAS as well as the long term benefits of flexibility, adaptability, and easing the burden on initial design perfection.

There are two primary problems with the solution provided by OAS namely, it tightly couples clients to the service through URLs, and requires orchestrating client changes in step with the service changes.

The first problem is easier to understand, by hardcoding the resource heirarchy to a URL and specific representation you now require tight and explicit versioning for clients to safely consume the service.  Any developer familiar with SOAP web services should be able to notice the similarities to OAS as the WSDL for a SOAP-like service without an envelope, using curly braces, and 3 extra HTTP methods.  The same arguments against the tight binding of the interface in SOAP services are becoming increasingly relevant when discussion the cons of OAS services.  The ramifications for this are immediately felt, but similar tooling has silenced detractors enough to satisfy the majority into adoption of this specification.

The second problem is much more nuanced, but far more frustrating to contend with as it is not immediately felt.  The design of SOAP and OAS lend themselves well to situations where the same group or company has control over both the service and the client.  If you distribute an SDK to wrap your service calls, or you distribute your own mobile applications, or support web applications under your control then the negative effects of the style aren’t felt until you need to perform the first major upgrade to these clients.  In this situation you can manage the negatives to a degree.  This difficulty is entirely unnecessary, but resisting the temptation to wait and deal with that problem when it comes up is hard to do.  You certainly are aware the process will be difficult while consuming resources and time, but the time and resources you are committing to the change management are in the future and your current deadlines are fast approaching.

The worst effects of this ill-advised tradeoff is felt when you are not in control of any portion of your APIs consumers.  This will be felt in cases as small as an internal microservices architecture or as large as your companies external APIs, and it will hit your bottom line directly.  If you deploy microservices which are tightly coupled to URLs and representations, you will need to manage the service dependency trees to fully deploy changes.  Assuming no changes made in one service results in a break in another, you have invited the complexity of a massive organization like Netflix to solve relatively small problem.  If this does result in a breaking change, you have lost a large portion of the benefits of a microservices architecture in tightly coupling two or more services which should be independent.  The benefits of the architectural style to the development team are obvious, but you may lose more time and resources managing the DevOps than you gain from development.  If your public facing APIs are forced to change, frequently requiring your consumers to modify their clients to meet your needs then you shouldn’t be surprised to see some of those clients explore or exit to your competitors.  When breaking changes are introduced this forces a slow release pattern, and requires your consumers as well as your team internally to manage multiple versions of your API.  As the difficulty of maintaining an integration with your service increases, the likelihood of your clients looking for alternative providers goes up from a real chance to a near certainty.

The obvious question you are probably asking is how is hypermedia any different?  If my clients bind to a domain vocabulary hasn’t this just moved the binding point with the same result?

The answer is no.  When transitioning from a CRUD API to a hypermedia API, you have moved from the realm of statically binding consumers to services to dynamic binding.  Hypermedia APIs by their nature should to be discovered at each use.  Hypermedia consumer clients should only ever have the root URL of the service statically bound.  The vocabularies can and should change over time to support changes to the understanding of the domain, or actual changes to the domain itself.  However, it is now possible to gracefully support clients as they migrate themselves at their own pace to newer portions of the vocabulary.  The client is no longer responsible for managing which version or effective version of your service they are interacting with on per call basis, the service handles this for the consumer.  Architecturally it may be necessary or easier for deployment to include multiple effective versions of a service to support this graceful transition, the key takeaway is the consumer is completely unaware of these URL changes.  The consumer is simply discovering, caching, and composing resource representations with metadata through links by their interaction with the service.  Any changes made would propagate to all clients by the end of the maximum caching period delay set by the service.  Any interactions with the service with now malformed or expired resource representations or moved resources can be managed by the ETag headers and HTTP 3xx response codes.  Clients are bound to the vocabulary, which means they are simply looking for resources and link rel-names they know, while caching information to reduce extra calls to the service for resource and service metadata.

This is a slightly more complex integration model, but the development of libraries to manage the increased complexity can release consumers from even more of their burdens, allowing them to focus on their true goal whether it is creating a UI or consuming the service for some useful purpose.

As I have been ramping up my evangelizing of hypermedia APIs through various channels, I have noticed a common argument being thrown out against hypermedia.  It’s taken a few forms, but the crux of the argument has been something like ‘hypermedia APIs just move the hard coded binding from URLs to link names but they don’t actually solve any problems’.  In most of these discussions, the flexibility and maintainability benefits of hypermedia APIs have already been brushed aside as unimportant and irrelevant.  Now I clearly have some things to say about those points, but as the people I’ve had this recurring conversation with had little interest in those properties I’ve decided to instead address the direct and immediate benefits of a hypermedia over CRUD APIs.  Namely, these are the greatly enhanced usability and proper hiding of service implementation details.  To accomplish this, I have put together a portion of an API description in the crud pattern which is intentionally not optimized to exaggerate the point I’m making, namely that crud APIs are less usable and require the consumer to know too much about the internal implementation details of a service to consume it.  I will also list and describe the usability and proper hiding of implementation details of a hypermedia API using my hypermedia API design guidelines.

Ridiculous requirements with a CRUD API

A movie theater has created a CRUD API to managage their lighting system in order to provide optimal viewing experience for all patrons at the lowest cost for the theater.  The following API is used to manage the lighting on an individual seat basis, to provide the best lighting conditions for each screen at the lowest costs.

Documentation

Screens have views where you can see them from, the view will have a type, and those types will have seats.  Those seats will have a source of light, which is natural or artificial.  In order to best optimize the viewing experience for the members of the audience, over the course of the day the lighting requirements will change seat to seat as the cost of electricity and the availability of sunlight fluctuate.  It is also extremely important to keep track of the natural lighting conditions in order to balance the cost of providing artificial light with power provided by utilities with power supplied from the on-site solar installation.

During normal operating conditions at no point should the cost savings of providing natural light reduce the optimality of viewing by more than 5%.  Any optimality below 80% should be immediately disregarded, unless the cost savings exceeds 90%.  Views from a balcony will increase the optimality by 10% for natural light, as it requires fewer redirections on mirrors.  The orientation of the screen will decrease the optimality of natural light by 50% at 0 degrees from north, and at 180 degrees from north there will be no reduction.  There is an exponential growth curve of the decreased optimality from true south to true north.  However, the position of the sun relative to the screen will offset some or all of this decreased optimality in logarithmic fashion as the relative position of the sun approaches 180 degrees.  This factor will be then used in conjunction with sun elevation which will completely offset the orientation degradation at greater than 70% of maximum annual elevation and reduce the factor linearly as the offset approaches 50%, at which point the relative orientation factor is entirely eliminated.

The service will internally validate any request supplied and reject any status changes which do not adhere to these constraints.  The switching mechanism has a finite life and it is critical to reduce the number of attempted switches.  The service is metered to protect the switching mechanism to 1 attempted switch per hour per seat.  Additionally, only 10 switches may be switched within a 60 second rolling window.  If a seat is in the wrong state it will cause extra wear on the switching mechanism at the damage rate of 10 switches per hour that it is in the wrong switch state.

Resource Representations

weather/current
+++haze-rate – a measure of the transparency of the air.
+++overcast-percentage – a measure of the overcast percentage.
weather/current/sun
+++elevation – elevation of the angle of the top of the sun above the horizon.
+++orientation – the degrees from true north of 0 to the center of the sun.
light-concentration-index
+++value – the index value from peak of the intensity of the sun on this day for the given logitude and latitude.
electricity/sources
+++source_id – id.
+++name – name.
cost
+++source_id – source_id for this type of electricity.
+++value – cost of the electricity per unit.
+++unit – the unit of count.
current_distribution
+++sources – source name percentage value pair for current electric use. e.g solar:50%, grid 50%.
…
many other objects
…
In order to facilitate the optimization of these resources we have supplied a system for you to create calculations to simplify the process.  The format for the calculation parameter is as follows: (? parameter_name : operation_id : parameter_name )?+ .  Additionally, any open parenthesis is required to be closed, or it will fail validation.  Names of all parameters must be unique.

usersuppliedcalculation
+++type – the numeric type to be used in this calculation, ex integer, float32, float64… etc.
+++parameters – the name of the parameters used in this calculation.
+++calculation – the calculation formula to be used in this calculation.
calculate
+++parameter_values – the name value pairs for the parameters to be used to create this calculation result.  A calculation result_id can be used as a value in the format result_id={result_id}.

Most likely use case:

The consumer will obtain this document from some out of band process, and will begin to read it.  First creating objects to contain the represented data within the service as described in the documentation, which is both shown and not shown in this example.  Afterword the user will begin to exercise their client by retrieving data.  After noting the physical ramifications of an incorrectly switched light source, they will go about creating local logic in order to perform the prescribed data calculations to accurately manage the lighting.

Depending upon how thoroughly the reader had gone through the entire document to fully understand the system before writing any code, the user will most likely stumble upon the helper endpoint which allows them to declare calculations to be defined and run by the service.  This may or may not result in a rewrite of some of the functionality to leverage the service provided functionality to reduce traffic.

The clients are very tightly coupled to the servers’ representation of the resources in the current hierarchy.  Any effort to simplify the service by changing this URI pattern will result in a broken client which needs to be completely rechecked, against the new version of this documentation which may have changed many of the hierarchies and relationships between resources.  Additionally, the use of any versioning, especially in the case of a breaking change in the above hierarchies will require a complete regression testing of the consumer code in order to verify the service changes do not cause undue wear on the switching mechanisms.

This service requires the consumer to be extremely well versed in the internal workings of the switching mechanisms in order to avoid extremely undesirable outcomes.  It also requires the consumer to duplicate logic from the service creator in order to prevent bad things from happening.  The consumer is forced to take on the responsibility the service designer has decided to ignore.

Ridiculous requirements with Hypermedia

A movie theater has created a semantically driven hypermedia API to manage their lighting system in order to provide optimal viewing experience for all patrons at the lowest cost for the theater.  The following profile is used to orchestrate the service to manage the lighting on an individual seat basis, to provide the best lighting conditions for each screen at the lowest costs.

screen
+++properties …
+++affordances …
+++relationships …
+++goals …
++++++optimize-screen-lighting
view
+++properties …
+++affordances …
+++relationships …
+++goals …
seat
+++properties …
+++affordances …
+++optimize-lighting – this will calculate the optimal lighting source and power source for a given time optimize-at – date time for the service to optimize the seating
+++relationships …
+++goals …
lightsource
+++properties …
++++++type – natural or artificial
+++affordances …
+++relationships …
+++goals …
weather
+++properties …
+++affordances …
+++relationships …
+++goals …
calendar
+++properties …
+++affordances …
+++relationships …
+++goals …
electricity
+++properties …
+++affordances …
+++relationships …
+++goals …
usersuppliedcalculation
+++properties …
+++affordances …
+++relationships …
+++goals …

Documentation

There is none.  The service’s semantic profile contains the human readable descriptions of the resources, their properties, and their affordances which are freely discovered by requesting the root resource “/” of the API.  These documents serve as both the services bounded domain and human readable documentation.  The profile is defined by domain semantics and not by technical terminology, the profile semantics are then separately bound to protocol specific definitions to facilitate the implementation, but these bindings are opaque to the human consumer.

Most likely use case:

The consumer will be given the root URL of the service; with a hypermedia aware client the user will browse the available resources from root “/” where the home document will be served.  Among the contents of this document will be links to the profile which will contextualize the resources returned in the document, as well as begin to populate the local cache-controlled copy of the profile to reduce redundant and unnecessary calls for meta-data.  The user can see all of the resources available to them, where all resources are likely root resources as the hypermedia service has been appropriately flattened and complex representations are composed by link relations.

The user notes the goal ‘optimize-screen-lighting’, and that this sounds like a requirement of current effort, and navigates to the link provided by the home document for the root of the ‘screen’ resources.

The collection of ‘screens’ is returned, the service provides hypermedia metadata about the current affordances and relationships of each ‘screen’, which are rendered as links and forms with helpful descriptions to contextualize the information.  The user notices the screen has the goal noted earlier, navigates to the link within the goal and the service then curates the experience guiding the user through all interaction necessary to accomplish the goal.  However, the interaction necessary is very little, as the goal will supply the user with a link to the seats resource collection, already filtered for seats related to this screen.  Each seat in this collection will have a related link named optimize-lighting.

The client will have already cached the description and documentation which cautions the caller from following this link more than 1 time an hour, and also provides the appropriate message structure to send to perform the optimize-lighting action.

The user will then be able to write this simple procedure within their more advanced client, which would simply follow the semantic links of interest at the root through optimizing every seat.  This client would be able to skip any of the discovery steps which it has a valid cache value for, and if there is a question of validity or the cache period has expired the client can validate the ETag of the response through a HEAD call. In this way the client is utilizing HTTP caching tiers between itself and the service to enhance the apparent performance of the service, and preventing excess load on the application and better response time to the client.  The client can then simply follow the final steps of the hypermedia discovery process in order to achieve its goal, in a much more streamlined, efficient, and dynamic manner.

At no point in this process is the user required to know anything about the technical requirements or implementation details of the service.  By leveraging semantic hypermedia, the use of the service has become as intuitive as possible within the bounded domain of the profile definition, which should be written to maximize human readability and interoperability.  Once understood by a human, a machine can easily follow the same steps.  Ideally this process would be bound to user interface constructs to create generic clients with interface bindings dynamically responding to stateful hypermedia messages.

Disclaimers, conclusions, and more words

As hinted at in the very beginning of this novella of a blog entry, I do understand this example is almost over the top exaggerating the negatives of the CRUD pattern.  I am not writing this with the intent of removing the CRUD pattern from any particular toolbox.  I do hope to demonstrate to more API designers, developers, and most importantly consumers there really is a better way to do this API thing.  The tooling for the CRUD pattern is fantastic, so good in fact that even consumers are happy to take on many burdens of the service provider in order to gain the ability for rapid prototyping and code generation.

I’ll pose some questions though, what if we could remove the entire requirements for rapid code generation to stand up a prototype?  What if all we needed to do was to create the domain profile, and we could have a mock service running immediately?  Would that be enough to start the trend of demanding more usable APIs?

I hope so, because I believe it is time we stop writing snowflake services on the internet, justifying them claiming our use-case is somehow special.  Somehow we have convinced ourselves we don’t need to make our services easy to use.  I can’t see how in the hysteria of speed to market rush, we all seem to have forgotten we need to build good products first before anything can go to market.

Perhaps I’ve been spending so much time looking at the oasis of the future that this example felt so ridiculous to me.  I showed this example to a colleague today, prefacing this as a ridiculous example of a CRUD API, his first reaction “I’m never using this service, but why do you say it’s ridiculous?”.  I think we can do better; I think we should all do better.

Throughout most of my career as a Software Engineer, one constant trend I have seen has been the increasing need to provide context to technical solutions and how to adapt a business requirement to technical limitations.  Sometimes the tool was not mature enough to accomplish what was wanted, and a bargain was struck on what was feasible in the short term using the tools on hand.  Other times the tools might have existed, but the solution was prohibitively costly in time, money or both.  There were countless other causes as well, but the crux of the issue was business was bending towards the will of technology, because computers don’t work the way humans do.  It was becoming my job to explain, convince, cajole, and bargain an adequate solution to a requirement instead of working towards the right solution.

History has shown tools which have provided even a small portion of the right solution, have enjoyed vast success by allowing the technical problem to be circumvented to deal with the business requirement.  WordPress, the very software this blog is running on is testament to the wild success a human tool can achieve when released into the wild.  Salesforce.com has enjoyed massive growth and penetration, despite being largely disliked by the technical community because it satisfies real business needs in a more human way.  As a developer who held a salesforce certification as a requirement of employment for a consulting company, I know how painful the experience can be for developers, but I was also able to see how well loved the platform was by users who leveraged its power with great success.

It is absolutely vital to take the next step in APIs that we don’t lose sight of the human perspective.  We must internalize the lessons of past success and push towards a human centric API space.  We are poised to fundamentally alter the course of human history.  Soon we will be able to build distributed systems upon other distributed systems, laying the groundwork functional composability on a unfathomable scale.  All of our success hinges on how many people can participate in building these new systems, and involve the most people we need to make our APIs as human as possible.

Every realized dream started with a crazy vision, beyond the realm of possibility and slowly materialized.  My vision is to humanize the API space, open the floodgates to the population, and allow everyone to participate.  This is where I am pushing, and with my guidelines for hypermedia web APIs, this is just the beginning.  The story I am going to tell is a story of inclusion, and I intend to include as many as possible.

In my last post I, perhaps controversially, set the constraint that an API should not couple itself or document tightly to URI paths and patterns.  This comes as a stark contrast to many of the popular trends within the API space, but the long term benefits to the service over its life far outweigh the complexity and cost of implementation.  In this post I would like to discuss an additional constraint which is in part corollary to the last guideline: do not version anything at all in the service.

I will first start out by addressing the elephant in the room, this guideline also comes at a stark contrast to popular trends in the API space, as well as the established trend from some of the largest Silicon Valley technology companies.  These two guidelines add a great deal of complexity to initial API design and architecture for public APIs.  Most of these leading organizations are driven by concerns with speed to market, and therefor do not allow themselves the appropriate design time before beginning to build applications, or are far more concerned with a larger audience’s familiarity with a particular design strategy than creating a better API design.  These concerns are certainly important from a business perspective, however it is often incorrectly presented as a technical limitation and guideline, when in fact it is driven almost entirely by business motivations.

If the rush for faster minimum viable products is driven by a business concern, what are the technical benefits associated with a hypermedia web API style?  The often cited axiom in the hypermedia web API space is WWBD, or what would browser do?  This is particularly apt given HTML is probably the most familiar hypermedia format to any user of the internet regardless of their awareness of this fact.  Netflix and other continuous delivery champions are famous for deploying code to production hundreds of times per day, however as a user of their services you are never aware of any change in the service platform.  The only way this can be done is if you are never aware or tightly coupled to any type of versioning within the service interface.  Your browser never knows, or is concerned in any way with, the version of the Netflix software it is querying.  This point perfectly addresses the style of CRUD api which includes a version number within its URI pattern, but does not cover all ways to version.

The other shortcut often taken during API design is to apply a version to the MIME type itself, and has been demonstrated to be beneficial in the very near term.  This solution does manage the version disconnect at the service layer, but leaves un-patched clients unable to fully consume the service until new client versions can be distributed.  Even though this strategy is one we are very familiar with as API service integrators, it is extremely consumer hostile as all the service version management responsibility have been dumped on the consumer.  Worse still this type of versioning is unique to each integration, greatly increasing the difficulty of a service which aggregates functionality from multiple APIs to create some or all of its responses.  This strategy will solve your concerns for versioning, but it comes at a hefty cost to the API’s consumers, and if there is completion in the space, this poor experience could result in the loss of a client or consumer.

Both of these reasons taken in isolation or together should be enough to convince a reasonable designer of the importance of removing all versioning from their API.  However, there is a more fundamental reason to exclude versioning entirely and it goes back to the very first guideline.  Versioning is a solved problem within the HTTP application protocol.  I previously discussed the ETag strategy to perform cache control, but this is nothing more than a more specific form of representation versioning.  If part of the structure or field of a message representation changes between versions, normal validation processing should handle this change, and the client can update its local representation and model cache for the service from the service itself.  If a historical representation of a resource is required for audit or some other need, the memento header exists to handle requesting a resource and representation as it existed at a certain point in time.

Clearly this is not be the simplest solution, however it is a far more useful and standard way to version the resources and messages of an API.  By adhering to the standard way to perform this action, a sophisticated http consumer can always know the status of any data it holds locally and remain unconcerned about the version of the service running.  Furthermore, this sophisticated client can be used to consume other APIs which follow these guidelines, with little or no additional integration effort.

In my last post I discussed the need to carefully construct your resource and message representations in order to increase the flexibility of your design over time.  Great care needs to be taken to avoid many of the pitfalls which can befall your design with insufficient attention.  A considerable portion of your time as a designer of a hypermedia web api should be spent designing your representations.  This post may then come as a small relief as I would like to talk about removing an entire concern from your API designer mind to free up time for the previous discussion; the URI pattern.

For the most part the guidelines previous to this one could largely be followed in a crud or hypermedia web api and you could expect improved results regardless.  This trend ends now, and depending on your attachment to the tools or strategies I’m about to reference, this might upset you.  Immediately stop spending any design energy on your URI pattern philosophy, rotation, choice, or strategy as you simply won’t need it.

The documentation and standardization around a hierarchical URI pattern generally wastes an inordinate amount of designer and developer time before some semblance of a workable design reaches the hands of a capable developer.  In order to help speed up this process of wasting time, many solutions like OAS (Swagger), Apiary.io, RAML, etc.. have been created to cut out chunks of effort to disprove a design.  This statement isn’t to be taken out of the appropriate context, as previously stated hypermedia web api’s are not right for every situation.  There are many viable situations where the solutions they offer are the correct tool for the job.  However, scaling those solutions to a large domain size with limited or no control over consumer code, and desiring high utilization endurance over time, it becomes an exceptionally difficult task to accomplish.  I would argue the longer you desire the API to run unmodified, the closer to effectively impossible the task becomes, and the point where this occurred wouldn’t be anywhere near as far in the future as you might believe.

Overloading the URI structure to contain semantic information places a tremendous amount of pressure on the designer to allow enough gap for future growth and needs within the hierarchy.  It also places a huge burden on the consumer of the service to know a lot of information about your service before being able to fully utilize it.  Kin Lane (“the api evangelist”) on the recent Api Academy podcast described this as human targeted documentation and tooling.  Meaning that while tooling can help set up some of the boilerplate code, the responsibility for tying things together is left for the human developer.

When utilizing a hypermedia web api which matches the preceding four guidelines, the URI pattern for resources is entirely irrelevant as the actual URIs are completely opaque to a consumer of the API.  With the root resource representation name of ‘user’, and assuming complete home document, and vocabulary definitions, the actual URI for the resource is meaningless.

With the same conditions stated above, the user resource being under ‘/uusdfskd231232/’ and the hypermedia client would have no difficulty whatsoever discovering the resource, nor surfacing the context to a human consumer.  The same is obviously not true of the human integration developer, whom must consult some external documentation to identify the change.  As a consequence, moving the resource to another URI would simply be a matter of making some small modifications to the profile, and this would be entirely unrecognized by the clients.  It is important to not, I do not advocate obscuring the URLs of the resource unless your design absolutely calls for it, where possible an intelligible and human friendly format should be used but it should not become an obstacle of any note.

By utilizing the representations painstakingly created following the previous guideline, your service doesn’t need to worry about creating the perfect hierarchy structure for your API, it just doesn’t.

In my previous post, I discussed the need to present a Home document.  Through its presentation your API design is able to be flexible and adaptive over time, while still being easy for clients to consume and discover functionality.  In this post, I want to discuss the design and handling of the resource representations themselves.

In traditional Object Oriented design, the principles of encapsulation encourage a class designer to hierarchically wrap atomic representations in increasingly more complex representations to compose a complete model which is self-sufficient.  In the OO world, this will enhance the flexibility of the implementation as it adheres to the information hiding principle.  A class implementer is free to make any changes to the encapsulated code, and any execution of the code is bound to the interface provided by the object, and is not impacted by internal design changes.  The following json serialization might represent a good OO model for a User class, it has been truncated for brevity.

{"user":{
  "address":{
    "street":"742 Evergreen Terrace",
    "city":"Springfield",
    "state":"?",
    "zip":"doh"
  },
  "userProfile":{
    "name":{
      "first":"Homer",
      "last":"Simpson"
    },
    "accountCredentials":{
      "userName":"homers",
      "email":"homers@example.com"
    }
  }
}}

When designing the representations for resources and messages, a designer needs to abandon this long standing practice and focus on creating the least complex but semantically complete representation possible.  The value of the atomic resource and message design is the enhanced flexibility gained by the API designer to bind related resources via relationships which allows mutability of the interface over time.  This guideline can clearly be taken to an unreasonable extreme, therefor it is important to always keep an understanding of the caveat ‘semantically complete’ in mind as there will be occasions when a representation is not entirely flat.  When successfully completed, a resource model should look to the OO designer eye to be remarkably flat and in desperate need of factoring.  For example, the representation of the above OO model could be 3 separate resources with the following representations.

{
  "user": {
    "userName": "homers",
    "email": "homers@example.com"
  }
}
{
  "userProfile": {
    "name": {
      "first": "Homer",
      "last": "Simpson"
    }
  }
}
{
  "address": {
    "street": "742 Evergreen Terrace",
    "city": "Springfield",
    "state": "?",
    "zip": "doh"
  }
}

If this was the end of the process, the resulting implementation of our API design could be a disastrous and unworkable torrent of traffic for even the most minute of tasks at scale.  In order to aggregate the same collection information in a resource representation, the design must compose the related resources through embedding, transcluding, or link relation.  Through these means, and fine grained use of cache-control, a hypermedia web api can have surprisingly low overhead while still retaining the flexibility and robust nature we seek.

At this point in the discussion of these 11 guidelines for designing a hypermedia web api the reader might have noticed the heavily referential nature towards previous guidelines.  Each guideline addresses a particular constraint or sets of constraints which a designer might be inclined bypass.  However, each step is a crucial building block on the path towards a fully functional hypermedia web api.  In order to achieve the benefits of a flexible, robust, and enduring API we must focus on designing the API right the first time, and allowing it to grow as resource and affordance understanding changes.