Don’t iterate the interaction design of your API.

Recently I have been encountering an increase in how a misunderstood best practice is misapplied to justify a bad decision. I’ve seen a general uptick as I’ve gained experience due to my increased knowledge, but also the increasing diversity of knowledge and experience of developers in the field. One practice in particular has stood out as particularly damaging to the API ecosystem namely the agile tenet of simplicity. This tenet advises the practitioner to add only the functionality and complexity required for the current known requirements. If one were to step back to think about it this would seem like it should be both obvious and harmless practice to follow. How could creating a design with the lowest possible complexity, or cyclomatic complexity for the computer scientist, ever be a bad decision?

We will cross that bridge when we get to it!

I’ve been consistently hearing this argument in regards to adding functional or design complexity to new API development. The practice of simplicity until necessary is generally sound, but fails utterly when applied to API design. The reason is quite simple there is only one chance to design an API, ever. But wait you cry, we can version the API! I’ve previously addressed the poor choice of versioning, nevertheless if you pursue this option the ill-advised use of versioning is a tacit admission of this fact. If there is only one opportunity to define the design of an API, you simply cannot make it any less complex than it will need to be to satisfy the eventual end goals of the API as it evolves.

When best practices go wrong!

The problem comes from the fundamental misunderstanding of the definition of best practices as rules of thumb, not hard and fast rules. Advocates and evangelists loudly tout around the benefits of their process, but often fail to acknowledge the existence of any scenario where their best practice simply isn’t. The argument consistently boils down to, this solution is too complex for now, we will go back and fix it later when we have time. But there is a few subtle built in fallacies which become this approaches Achilles heel.

The first is the belief that with the introduction of this technical debt the price to repay will not grow over time, or at worst will grow linearly. There are certainly situations where this might be the case but it would be the exception not the rule. The term technical debt was coined because of the tendency for the debt to grow like compounding interest or worse. Worse still it is very common that the weight of the legacy system once released would actually prevent you from ever returning to address the problem at all.

The second is the naive assumption that the future will be less busy, the team will maintain a desire fix the flaws, and their fortitude to expend capital to meet the requirements will grow. Case study after case study has proven this is overly optimistic and simply not true. As the cost to fix an implementation or design flaw escalates, the cost benefit tradeoffs with leaving the code in place become ever more biased in the favor of not touching ‘what isn’t broken’.

At the end of the day this is simply the lies told by designers, developers, and stakeholders to themselves and others to justify an increasingly more expensive sub-optimal deliverable.

Assuming your team is stellar and defies the odds by prioritizing the rework process, it following through is still completely dependent upon having the opportunity and control of all dependencies to seamlessly perform the work. If there is even a single client outside of your teams’ immediate control, your ability to complete this work quickly is severely degraded.

Agile: The buzzwordy catalyst and amplifier

There is nothing earth shattering here, but I haven’t even touched on the whole story. In the same paper as ‘cyclomatic complexity’ Arthur McCabe also introduces the concept of essential complexity, or the complexity innately required for the program to do what it intends to accomplish. Under the guise of the tenet of simplicity, the essential complexity is often left unsatisfied because the agile methodology places a burden of proof on additional complexity which is unforgiving and ultimately unsatisfiable. In order to reach the known essential complexity of a program, you first have to prove adding the complexity is actually essential. It’s a classic ‘chicken or an egg’ problem with no answer. Ultimately this will most often result in the process directing your actions to failing to meet essential requirements through a failure to define, justify, or evaluate essentiality of the added complexity.

The business decision, and business imperative to do only the required work for now is deaf to technical concerns outside of the short term, regardless of the costs or savings. This isn’t to say developers should always be in control of these decisions, but it is very important to be aware of the increased importance of communicating technical pitfalls and their costs outside of the technical audience as the process is heavily biased against technical concerns. The adoption of agile practices has actually increased the importance of a highly knowledgeable technical liaison who can push back when shortsighted goals will provide a quick positive payout saddled with a negative longer term value. This is where it all comes back to the misunderstanding of best practices.

These teams are more often being led by practitioners without truly understanding the best practices business purpose. Rigid adherence to, and often weaponization of, ‘best practice’ in these design discussions has only served to hide the inevitable costs associated with poor design until a later date with the debt relentlessly compounding unimpeded.

You can’t put design off, so don’t!

I started this off by saying you can’t iterate away the interaction design, so I want to be very clear what parts of the API design can and cannot be iterated. The design of an API is actually composed of two relatively straightforward and separate concerns, what I will call the interaction design and the semantic design. The interaction design is the complete package of the way a client will interact with your service. It includes security, protocol concerns, message responses, and required handling behavior which cuts across multiple resources among many others. The semantic design encompasses everything else and this can and should be created and enhanced over time as domain requirements change.

Knowing the interaction design of the API is permanent once completed, it’s important to not only get it right, but to ensure the design defines the capability for expansion of specific functionality which will need to change over time, for example the use of a new authentication scheme, or filtering strategy.

It is impossible to list the requirements which will fall under the interaction design of your API, but I provide some questions I’ve used which will help you go through the initial design period of your API to exclude the design and implementation of features which can wait.

Does this feature change the way a consumer interacts with the API?
Does this feature change the flow of an interaction with the API?
Could later introduction of this feature break consumer clients?
Could later introduction of this feature break cached resource resolution?

With a rigorous initial design session, utilizing these questions you should be able to determine the essential complexity of your API interaction design with much higher accuracy, and prevent cost increases and consumer adoption pain from adding new value to your services in the future.

Posted on June 30, 2017 by Mike Hibay 0

Unleashing generic hypermedia API clients

A true restful API has been called many things, hypermedia web APIs, ‘the rest of REST’, HATEOAS – the world’s worst acronym, or perhaps the newest hAPIs. Regardless of what you call it, this concept has long been proclaimed to solve nearly all of your most difficult design problems when building a web service interface. There is plenty of evidence to support the claims made by hypermedia evangelists over the years, however one glaring omission is likely the cause for the slow adoption of hypermedia on restful services. How do you consume this service, and what do all of these link relations mean? Building an effective hypermedia client is more complex a task than consuming a CRUD API, an extremely difficult question to answer has been when do the benefits outweigh the cost of complexity? Once past this hurdle, how does a consumer know how to interact with the service?

It is no wonder adoption of a superior design is so slow when a more complex design leads to more complex clients. The primary selling points for this style are longevity, scalability, and flexibility, however the benefit from these traits is seen over a long period of time making the complexity a difficult tradeoff to evaluate at the start.

We are all very familiar with good, seemingly simple hypermedia clients. In fact, you are likely using your favorite one right now to read this. If we know so much about building good hypermedia clients, why are hypermedia APIs still not the de facto standard?

The key to enabling adoption of hypermedia APIs is very simple, make them easier to consume. The Open API Initiative through the swagger specification has demonstrated the power and appeal of standard formats to enable rapid adoption of best practices in accelerated development cycles. I often will call out the shortcomings of the specifications, but it is critical to understand the cause of the successful proliferation to the web at large. The trick is to apply the lessons learned from this success to driving the adoption of semantic hypermedia. To make a hypermedia API easier to consume you create generic clients to encapsulate the complexity by establishing and adhering to a strict http behavior profile. Then you subscribe to or publish a semantic profile of the application adding domain boundaries to the messages and actions. Finally, allowing clients to tailor their hypermedia through requested goals of supported interaction chains.

Often hypermedia is used to augment CRUD services using binding formats like OAS. In this scenario it simply can’t be relied on to drive the interaction with the service as it has no guaranteed, or an unbounded, range of responses. Establishing a range for the hypermedia domain semantics is critical to transition the role of hypermedia from augmentation to the vehicle for application state and resource capabilities.

The takeaway here is simple, if you want to have the robust flexibility offered by hypermedia APIs then your focus should be on enabling strong generic hypermedia clients. To build strong generic hypermedia clients, you need to adhere to strict service behavioral profiles to isolate the domain from the underlying protocol behavior.

Posted on June 14, 2017 by Mike Hibay 0

Hypermedia APIs: Use extensive content negotiation

In my last post I touched on how important it was to insulate consumers from the immediacy of a breaking change. Nothing you can do as a designer will allow you to create the perfect API which will never require change on the first try. What you can and should do is reduce the likelihood of the occurrence of a breaking change as much as is feasible, and then allow consumers to gradually adopt to the changes on their own schedule. In this post I’ll discuss the need for extensive content negotiation.

It has been stated, in the comments on these very guidelines no less that there is a striking similarity between the 9th and this the 11th guidelines, as both rely on or discuss content negotiation. Much like the first guideline to embrace the http protocol, the benefits, constraints, and reasons for content negotiation are sufficiently board to merit multiple discussions to be properly addressed. It is imperative a designer avoids hypermedia formats which prescribe URL patterning because this could lessen the proper attention being given to resource representation and affordance design. The goal of this discussion is to address the rest of the content negotiation constraints to prepare your designs for interaction with real traffic volume and diverse consumer demands.

As the API designer, your job is to provide the simplest service you possibly can to your consumers. CRUD APIs like OAS (swagger) often struggle with complex designs when domain functionality doesn’t map to 4 methods very well. Other solutions like GraphQL provide excellent solutions to captive audiences and internal services, but for external consumers often result in the same poor consumer experience. Quite simply the act of consuming the service correctly requires too much knowledge about how the service is built. So how do you avoid making these same mistakes with hypermedia APIs? You allow your consumers to interact with your service just about any way they want. The fact is you will never be able to guess all the particular ways a consumer would want to interact with your service or tailor their requests, so don’t try. The solution is to build your service as generic as possible and allow the consumers to choose the interaction mediums will be used.

What all should be negotiated? The short answer is everything you can reasonably support which adds to the consumer experience. A longer non exhaustive list of potential negotiated points:

Hypermedia Format (Content-Type)
Filter Strategy
Query Strategy
Pagination Strategy
Cache Control Strategy
Goals
Vocabulary
Sparse Fieldsets
Representation or Document Shaping

It’s a long list, does your service really need to support all of those negotiation points? It should aim to support all of these and more if they are reasonable and feasible to your service domain. Yes, this adds a lot of complexity but it’s crucial to focus on the consumer experience, and the long term payoff of creating a service which will happily satisfy the consumer needs for years to come.

These negotiation points are all critical to supporting a wide breadth of consumers, but they are also central to providing service flexibility over time. A service designed from the beginning to be generic, and support a wide range of many different properties already has the capability to support one more option in any particular property. When a new hypermedia format comes out, or a new standard filter strategy, your service already provides multiple options for this properties and supporting the change is nothing more than plugging in the appropriate functionality. You can’t know what formats will be wanted in 5 years, but your service has been designed to account for changes over time, and the required upkeep is vastly lower than any alternative presented to date.

Design your API to negotiate with your consumers as much as possible, and you will have an enduring service your consumers will love to use for years.

Posted on June 1, 2017 by Mike Hibay 0

Hypermedia APIs: Use flexible non-breaking design

In my last post in the series of hypermedia API guidelines, I discussed the need to decouple the design and implementation details of your API from the constraints of any particular format. You likely aren’t designing your own format, but it is a good decision to avoid formats which require URL patterns, as they can provide confusion and increase the odds a consumer will make calls directly to URLs. In this post I’d like to go through the follow up guideline to don’t version anything, which will fill in the remaining gaps in dealing with resource and representation change. To support long term API flexibility, your design should leverage a strict non-breaking change policy, with a managed long lived deprecation process.

As time passes, an APIs design can lose relevance to the piece of reality it is built to model. Processes change, properties change, and priorities change so it is crucial to maximize flexibility for change over time. When using hypermedia APIs, it is important to understand the three types of changes you can make to your profile, and the appropriate way to manage each kind. Optional changes will make modifications to the representations and their actions without any effect on current consumers and their bindings. Required changes will make additions to the profile which can be gracefully handled by the generic client. Breaking changes, or removing items from the profile, will require a client update to maintain compatibility.

In traditional statically bound API styles the handling of the optional changes would likely lead directly to consumer client changes as the representations of resources are strongly coupled to the consumer. However, a generic hypermedia client is intentionally dumb when it comes to the properties of resources, so the addition of any unknown resource simply behaves in the default manner.

The story of required changes is much the same as the optional changes. The highly coupled service and consumer relationship requires constant maintenance and attention to continue to function. A hypermedia API consumer client will manage the required changes by standard approaches, generic fields which are required can be flagged to the consumer as invalid without requiring any strong bindings to the consumer client.

In this way, the two changes which represent any difficulty for hypermedia APIs are the required and breaking changes. In the case of the required changes, a previously valid representation is no longer valid because a new property has been added. Alternatively, there is a new action has been added to a representation which was not previously expected without a client binding to the action. The breaking change is a representation or action being removed from the profile which is has previously been required or bound by consumers. With these definitions, it’s clear the real difficulty is in addressing the breaking changes. The solution to breaking changes again can be found in the very first guideline I discussed, use the HTTP protocol to advertise change.

Previously in these discussions I have noted how the hypermedia API will manage the range of bounded contexts available to consumers. Diving into this concept a little further, the primary benefits in supporting a range of bounded contexts is to allow transparent incremental versioning and consumer preference in the resource representations to be utilized. Many leading tech organizations and methodologies stress the importance of versioning the API, unaware or uncaring of the fact that doing so has sown the seeds of future breaking changes. By tracking the changes of your representations in the supported vocabularies, your service is able to leverage the HTTP 3xx response code family to inform consumers that change is imminent while still respecting their interaction in the vocabulary they know. This allows consumers to upgrade gracefully on their own schedule, and greatly reduces the occurrence of high stress deadlines caused by your services’ evolution. Through nuanced activity tracking and API orchestration, you will have an accurate view on exactly when particular representations or portions of the API are no longer in use. Allowing you to confidently sunset old functionality knowing it will not likely result in a rude awakening to one of your extremely valuable customers.

By leveraging the protocol in the standard way, we can avoid breaking changes from immediately impacting consumers and requiring their full attention. As I’ve mentioned elsewhere, creating the good consumer experience is critical to the success of your API, and a great way to keep consumers happy with your service is to not break their clients at 3am on Saturday night.