The Definitive Guide to 
api integration

For nearly a decade, Cloud Elements has focused on how to take the pain out of integrations for developers. Even so, we’re consistently impressed by how the demand for API integration has grown.

83% of all internet traffic is API-driven according to Akamai, and growing. The key trends driving integration needs include software proliferation (i.e., thousands and thousands of SaaS companies), which in turn means users need to aggregate and contextualize more data to make data-driven decisions and automate processes. Gartner estimates that API integration work accounts for 50% of the time and cost of building a digital platform. More and more, the cool things we want to create rely on integration; the winners will be those who do integration well.

Yet, every API and data model is unique, like a snowflake. Many APIs implement standards, like OAuth for authentication, differently and many do not offer functionality like webhooks or bulk data operations. API nuance forces developers to sort out the same complexities repeatedly, remap and transform data, and rewrite programs for error handling, polling, and other critical functionality. The result: brittle, point-to-point networks that are poorly documented and hard for colleagues to maintain.

That’s why Cloud Elements exists and that’s where we focus all of our time and effort—not just our developers’ time, but also keeping pace with trends and best practices. But as the demands for integration grow, that means more and more people get involved. So... 

Our goal here is to share the key lessons we’ve learned helping hundreds of customers, from startups to massive enterprises like SAP, implement thousands of integrations — some that just connect A-to-B, some that are product features serving hundreds of thousands of end users every month.

A note on users

API integrations serve a wide variety of needs. If you’re developing with microservices, they’re a simple fact of life and your teammates are your users—they just need the documentation. But often, integrations either support an internal or ‘business user’ (i.e. a salesperson at your company syncing CRM updates or the HR leader requesting account provisioning for a new employee) or a customer interacting with your product (i.e. someone who uses the application you’re building). In what follows there are differences between the former (internal integration) and the latter (productized integration or professional services delivery) but, for the most part, the process and recommendations are the same.

You Say Tomato, I Say...

Before we get started we have to acknowledge an uncomfortable fact: much of the challenge in API integration, like so much else, is not a technical challenge as much as a communication challenge.

To put it bluntly: software engineers and developers speak a slightly different language than product designers and a very different language than non-technical end users (whether that end user is in the finance or HR department of your company or the finance/HR department of your customer’s company).

Communication Tips For Developers

As you’re researching an integration and building requirements (yep, we have a whole section on building requirements) and assuming you’re talking to end users (vs. simply being handed a requirements doc), the main challenge is getting non-technical end-users to recognize unstated assumptions and think about corner cases.

They might say, “I need the order info from the closed won opportunity in the CRM to sync with the items for fulfillment in the ERP.” Sounds simple enough.

But when a sales order is closed won, they probably assume that contracts have been signed, shipping information is validated, and tax has been calculated for invoicing & payment. But if you don’t think to ask about those unstated assumptions, you could miss important validation criteria and data requirements to create a functional integration.

What about corner cases? What if the closed-won opportunity includes items that are listed
as ‘sold out’ in the ERP? What happens then? The business user should be able to tell you the process (“Oh, then we have to create a backorder ticket for inventory…”) and that will give you clues about the data and orchestration logic you need to consider. So we recommend to:

1. Ask business users to draw out the steps in their process. Use a whiteboard or chart maker or even a slide (they love slides, don't they) so everyone has a visual to come back to, add on to, and to anchor requirements around.
2. Explore corner cases. Ask things like "Is there ever a time where you can't move to [step B]?" Or "If you were doing it all manually, what would need to move from [step b] to [step C]?"
3. Get at unstated assumptions. Ask things like "What has to happen before [step A] so that the process can move to [step B]?" Or "What information is the user at [step A] required to have to move to [step B]?"

Let’s also face one other fact: non-technical colleagues may not (initially) understand why solid test data and sandboxes can be blockers. And since they likely don’t have a handle on how complex a solid integration truly is (say, one that covers off the corner cases) they might hassle you on timeline.

More practically, try to explain up front all the steps you’ll go through (turns out you have a
handy guide right here), what the blockers can be (ahem, access to sandboxes that someone else controls and doesn’t like to provision for others), and where you’ll need their future input (like UAT?).

Communication Tips For Product Leaders and Execs Who Have Requests in IT’s Backlog:

Integration work is complex and takes time and effort. Start by saying thanks to the developers across the table from you. Read below to get a taste of that complexity (even if you need to look up a few words).

Second, remember that software programs can’t make judgement calls like people can.
Integration work takes a process where people make decisions based on many inputs and turns them into a set of logical steps and actions. Often 80% of what you need is straightforward (like when you get formal signoff for a PO on an important initiative and can start contracting) - it’s the remaining 20% that is hard (like when you call Kelley in procurement because the vendor needs non-standard payment terms in exchange for the discount you negotiated).

If instead you’re talking about creating an integration experience so that users of a digital product you make and sell can connect your app with others they use, that adds on yet another layer of complexity. You need a UI for users to securely enter their login credentials to other applications (some of which contain sensitive data like a CRM or ERP), you need to think about multi-tenanted integrations and self-serve data mapping so that your PS or CS teams don’t have to step in each time a user wants to leverage the integration, and you need to think about how updates and events flow back and forth between apps.

In any case, what seems like a simple request or a simple feature to add is probably much harder than you think. So be patient. Give grace. And overcommunicate what the intended user experience (UX) should be (pictures help a lot here).

The first thing we’ve learned over the years: An integration is never ‘done.’ There’s always maintenance, endpoint changes, things breaking, or necessary upgrades because the integration is wildly successful with users but was never designed to work like that….

We find it more useful to think about integrations as living things within ecosystems that adapt and evolve. Things that have a lifecycle, like the following:

The second thing we’d recommend: don’t be dogmatic about this process. If you’re working on integrating microservices or even some internal integration projects, a lot of the design—and implied requirements—might be handed to you (if they’re not, we don’t recommend skipping those steps).

More, let complexity be your guide: if you’re trying to automate transactions between a highly customized ERP and a highly-customized CRM, more work up front will save time in development and QA. The same goes if you’re not building an integration but rather creating reusable components (i.e. API connectors, standardized mapping/transformation modules, callable process-based APIs others build on, etc…). Better research and design usually equals less maintenance. With a quick data dip into a martech app, on the other hand, you can probably outline the key requirements in an hour or less.

And finally, putting the CTO hat on, sometimes you might be asked to do more work than strictly necessary at first glance to meet requirements. There are times when the ask to do extra work requires an honest conversation with management; other times the ask is meant to keep data secure, govern it centrally, or use common patterns and tools that, frankly, might make one integration take a little longer but save time across a thousand others. All we’ll say about this: one integration rarely solves the whole puzzle. It’s a big world out there. Let’s dive in.

[Insert overused Abe Lincoln quote about ‘sharpening the ax’ here]

To build the requirements for your integration, start with the user, then the non-functional requirements of the integration itself, then translate those needs into the test cases and more importantly the test data and sandbox(es) you’ll need in order to begin design & development.

Start With the End in Mind by Creating User Stories

• Identify the different user personas that will be interacting with your integrations: admin, end user, onboarding team member, etc… There are loads of good resources on writing user stories out there from product development teams and we go deeper in this blog series, but a few quick examples:
  • “As a Marketing Admin at Company A, I need to be able to log into [new martech
    application], and from there quickly and easily enable a connection to my instances of Tableau and ZoomInfo.”
  • Continuing above: “After connecting to Tableau and ZoomInfo, I then need to integrate my Hubspot instance and need to customize the field mappings to work with my custom objects.”
  • “As an junior specialist working for Connected-Consultants, I need a low-code environment to build integrations between the applications my customers use in the Hire-to-Retire Process”

• With an initial set of user stories, think through the broader business processes your integration will support: do you have all the user stories you need? Focus on where your integration fits along the ‘data journey’ through that business process.
  • Consider user stories for admin users vs non-privileged users. Your target application endpoint may allow authenticated admin users to manage objects and user access in addition to CRUD operations (see below) on the data itself.
  • User stories can also be defined as a system persona but beware becoming too focused on technical aspects to the detriment or the people who need to accomplish some job. Ex: The Webinar Management Application will automatically create leads and associate them with accounts in the selected Marketing Automation System so that the two systems will stay in-sync.
• Think about triggers for workflows: how and when does data need to be moved?
• Think about dependencies of data objects: does the data in underlying objects need to be regularly kept in sync with another system?
  • For example, integrations touching Invoices often assume that Customer records and/or Product line items are kept in sync because an Invoice is tied to a Customer and includes various Products as line items.
  • In this example, if the bi-directional integrations for Customers and Products don’t already exist, the scope of your integration just grew. Welcome to yak shaving...
• Think about any parameters the user might need to configure or change: particularly important when building self-serve integrations for non-technical end users

Identify Functional and Non-Functional Requirements

• Functional requirements are the requirements around the product’s functionality, capabilities, features, and usability. These are typically driven by your user stories.
• Non-functional requirements include all other requirements not tied to the user experience—things like performance, security, and other technical specifications. These are typically driven by infrastructure limitations, company policies, or product design principles (i.e. latency for the user). 

Think Through Sandbox Needs (and Make Requests Sooner Than Later)

• A sandbox is necessary to create a successful integration—especially when you’re productizing a self-serve integration for your customers (i.e. to test scenarios) or when you don’t want test data triggering real-life actions (i.e. to keep production environments clean).
• Obtain sandboxes as soon as you know what applications you want to integrate with. When building productized integrations, remember that some integration targets require paid partnerships or formal application approvals that take weeks or longer to finish.
  • If the lack of sandbox access becomes a blocker, some software vendors allow licensed access to third parties. This may mean your customers' sandboxes can be utilized for each customer’s respective integration. You will need your own sandbox for continuous testing and development or if the customer revokes access.
  • Additionally, testing with your customer’s sandbox makes your tests subject to the customizations that the customer has made. This means edge cases for other customers may go untested but can also help make sure your self-serve integration can handle user-specific parameters in the wild.

Outline Test Cases and Define Test Data

• Identifying test cases during the requirements creation phase helps to ensure you’re capturing all edge cases and hidden requirements before development and testing begins.
  • Let’s be honest, there’s nothing worse than thinking you’ve completed QA on a new integration only to have your business user share additional needs they forgot to mention/assumed you’d know.
• Create business and technical test cases so you can ensure your end-user gets the functionality they need and that your solution meets your non-functional requirements.
• Best Practice: Throughout development of your integration, save test payloads and responses. This makes automated testing, expected result validation, and long-term maintenance easier (for you or the coworker who will have to pick this up in the future when it needs to expand).

Think of requirements as the inputs or ingredients for your integration. But before diving in to start cooking, errr... developing, there’s more work to prepare.

From our point of view, developers’ knowledge and experience really shines at the Design phase (and yes, we’re biased). The non-technical users you’re building for or working with are likely to assume that your skills are best served writing code—because they can’t write code.

But we know it’s not about who can churn out code the fastest. It’s about solving problems elegantly. That comes from simulating the problem in your head ahead of time and getting smart before you start writing code.

The key elements of this simulation and research: the endpoints themselves, the authentication mechanism and authorization, data mapping/transformation, and data operations (i.e. event-driven, bulk operations, historical data, etc…).

Research and Select the Appropriate Endpoints

• To say modern applications (think Salesforce) “offer an API” is a misnomer. They offer hundreds of programmatic endpoints that all offer different data resources, require input parameters, and come with different functionality/limitations. Even legacy applications with SOAP/XML APIs offer multiple endpoints. Based on your requirements (above), analyze the data objects you need to access for your product or user story.
• Research which target APIs provide the data objects you need. It’s not uncommon that you’ll need to orchestrate several sequential and/or contingent API calls to get the data you need from multiple endpoints.
  • As this comes up, we recommend drawing out the orchestration flow and logical operators in a simple flowchart to help you keep track of what data you need to input and retrieve from which target endpoints.
• Every data model is different - another snowflake. Make sure you understand the data configuration you’ll get back from the endpoints you’re researching.
  • You need to determine how you will handle these different structures and how the data needs to map from one application to another or from third-party applications to your own product.
• Note: particularly if you’re working with ecommerce, publishing, marketing technology, or software development tooling, you may encounter GraphQL APIs.
  • For many use cases, GraphQL has the potential to leapfrog hypermedia APIs. Hypermedia APIs require good metadata and well-structured endpoint linking to discover data and endpoint capabilities dynamically but can return bloated responses that slow down other applications’ performance.
  • One of the key benefits of GraphQL—the ability to create a data abstraction layer that can combine multiple sources through a single gateway and endpoint—will garner growing interest from industry sectors that have complex data supply chains. Data science, healthcare, and city services will likely be drawn to GraphQL in 2020 in the same way that publishing, social media, and e-commerce have been to date.

A Note on Authentication

Authentication itself is complex, there are books on the topic, and as a developer you have to work within the authentication framework of the APIs you’re working with. We’ve outlined the primary auth types here, though this is an area where we find a unified auth framework like what Cloud Elements has built really pays dividends for developers. [Stepping off the soapbox...]

Determine the Authentication Mechanism(s) Used and the Workflow Required

• A quick primer on the primary auth types you’re likely to encounter:
  • Basic Auth - A widely used protocol for simple username/password authentication. This type of mechanism provides no confidentiality protection for the transmitted credentials. Example: 
  • OAuth (1) - An Open Data Protocol that provides a process for end users to authorize third-party access to their server resources without sharing their credentials using user- agent redirections. Credential tokens are long lived, typically a year. Example flow: 
  • OAuth2 - Delegates security to the HTTPS protocol. OAuth (1) does not require this and uses alternative methods to remain secure. OAuth2 also introduced the use of refresh tokens that allow authentications to expire, unless “refreshed” on a periodic basis. Example flow: 
  • OAuth2 Password Grant - The password grant is one of the simplest OAuth grants and involves only one step: the application presents a traditional username and password login form to collect the user's credentials and makes a POST request to the server to exchange the password for an access token.
  • OpenID - OpenID Connect is an open standard and decentralized authentication protocol based on the OAuth 2.0 family of specifications. Promoted by the non-profit OpenID Foundation, it allows users to authenticate to websites and applications (known as relying parties, or RPs) using the third-party service UI (OpenID) so that developers do not have to manage password files.
  • SAML -An XML-based open standard data format for exchanging authentication and authorization data between parties, in particular, between an identity provider and a service provider. SAML is popular with large single sign on (SSO) organizations, corporate applications, and many older applications. Example of an SAML authentication request: 
  • TLS - A TLS handshake is the process that kicks off a communication session that uses TLS encryption. During a TLS handshake, the two communicating sides exchange messages to acknowledge each other, verify each other, establish the encryption algorithms they will use, and agree on session keys.
  • JSON Web Token (JWT) authentication - The claims in a JWT are encoded as a JSON object that is digitally signed using JSON Web Signature (JWS) and/or encrypted using JSON Web Encryption (JWE). In simple terms, it is just another way of encoding a JSON object and using that encoded object as access tokens for authentication from the server. Example flow: 
• For token-based auth, determine how to manage refresh tokens and make sure they are securely stored.
  • For example: if you want your user to authenticate once and then not have to re-authenticate again as they interact with the endpoint, your application will need to manage the refresh token from the endpoint (when available) in addition to the initial access token. Some access tokens expire in an hour, while others may last as long as a year or never expire.
  • Pro tip: OAuth 2 is the only mechanism that has refresh tokens. This allows you to keep the token alive so that users do not have to log in again. SAML does not use refresh tokens. Services like Xero, Quickbooks, etc., do not yet support OAuth 2.0.
  • Pro tip: Refresh tokens have similar variations in how long they are authorized for access. You will need to develop a strategy for managing access and refresh tokens for each endpoint that you connect to in order to provide a seamless user experience for your clients.
  • It’s your responsibility to protect tokens. If your application stores tokens, it is highly recommended that you encrypt with 256-bit encryption at rest within your data storage system with the key owned by the end user. Encrypted tokens stored with 256-bit encryptions are really tough to break, protecting your application and your customers' usernames and passwords.
• Within the authentication workflow, your user stories may require you to call an identity provider (IDP) service instead of, or in addition to, building the auth workflow and managing user tokens yourself.
  • This type of federated identity and authentication management (SAML or OIDC) may require i) registering and retrieving a management token from the IDP for your application and ii) calling the IDP API and retrieving an access token for the individual user.
  • The contents of third-party access tokens will vary depending on the issuing identity provider. Because tokens are created and managed by a third-party (such as Facebook, GitHub, etc.), the validity period for third-party tokens will vary by the issuing IDP. If you believe these tokens have been compromised, you will need to revoke or reset them with the third-party that issued them.

More thoughts on auth flows for productized integrations

To get started, ask yourself these questions: Which persona will be authenticating to the service?
A system administrator or the individual end users? What are the types of authentication mechanisms supported by each service I’m connecting my app with? Are there specific security concerns based on the type of data that I'm enabling my user to access? Where should the data for logins, passwords, keys, and refresh keys be securely stored?

Then, check with your product designer to make sure that you’re aligned—the typical workflow looks something like: i) a user is presented with the UI to select the endpoint they wish to connect to within your product, ii) your application will open an authentication screen (that you design or leverage open source), iii) the user enters their own login credentials to access the service and grant access to connect the apps.

Architect your users’ data access via permissions/role-based access controls

• Some 3rd party applications allow the administrator to configure user permissions, but not all. From an integration standpoint, this is typically limited; OAuth2 provides better role-based access control than OAuth or Basic Auth, for example.
• These permissions are different per application and authenticated user.
  You need to take into account that executing certain API calls might be restricted for one user versus another user and build verification logic in your integration workflow(s).
• Consider whether or not your client will be authenticating an administrator account or an end user account: how will you handle and pass permissions from the endpoint? An admin account at the endpoint may allow you to manage objects and data access — is this included in your user stories?

A note for those building products serving enterprise customers

An applicant tracking system (ATS) likely needs to integrate with the system of record (HRIS), just as an AR/AP automation solution likely needs to integrate with the ERP, but unrestrained access to these systems can create data integrity vulnerabilities for your customer if there’s not a way for your pre-built integration to set permissions (i.e. read from the general ledger but not write; update the pending/paid status of invoices only, etc...) Simply putting this concern back on user behavior, as so many have done for years, no longer meets CISO’s demands for improved controls.

Outline data mapping and transformation needs

• Consider whether your integration use case is static or dynamic:
  • Static integrations provide a fixed mapping between data fields and do not discover custom data and do not provide the end user with a facility to change mappings. Static integrations are suitable for services with fixed payloads such as cloud storage, payments, and messaging services.
  • Dynamic integrations are designed to accommodate custom data. These integrations require services to discover custom data, the ability to map data and custom fields for each instance of an integration, and often the ability to transform data values between endpoints. They’re “dynamic” in the sense that they enable each integration to be uniquely designed by the user based on the data discovered at the endpoint.
• In a nutshell, static integrations typically rely on the developer for the initial mapping as well as any subsequent changes. Dynamic integrations typically require more work upfront, but also empower more people (customers, business users, etc...) to configure the integrations they need and may require search or browse functionality.
  • If you’re providing a Dynamic integration experience for your users, will you need to create a UI for data mapping and transformation, or is there another UI your integration can lean on? The data mapping UI should feel as seamless as the underlying integration you’re creating.
• Here are some key considerations for data transformations:
  • Evaluate each endpoint and determine if values/types are consistent with the values/types the other application(s) expect. For example, many help desk ticketing systems use different priority schemes (‘High, Medium, Low...’ vs. ‘1, 2, 3...’) - consider if you will transform these values into a consistent data set or map bi-directionally.
  • Determine each endpoint’s payload structure and prepare to recreate the necessary payloads. Legacy applications, databases, and cloud services encompass a variety of payload formats, including XML, SOAP, JSON, etc... For more complex enterprise integrations, there may be a variety of payload formats coming from different data objects.
• For integrations serving multiple users and/or highly-customized enterprise applications, it is unlikely you can anticipate all of the transformation needs for each endpoint and each user during the Design phase — consider providing a programmatic facility to incorporate custom logic into your transformations.

A note on creating self-serve integrations within your application vs offering an API customers can consume for integration needs

Why consider this in the first place? Usually because competing apps offer it and/or customers have come to expect it. Still, productized self-serve integrations can be a lot of work — they can help your users use your product, keep them in your UI, and drive sales/reduce churn. Beyond building an integration UI, or “marketplace” as it’s commonly referred to, you often must also provide users a data mapping UI to configure their necessary mappings (data transformations are implicit in your self-serve mapping UI) or pre-defined mappings.

If this seems too complicated either because of target system customizations or limited dev resources, most shift the burden to Support teams or Professional Services during the onboarding process where file-based mapping templates may mitigate some of the customer pain and cost of these approaches.

Consider a range of integration operations that may be included or implied by your user stories

• Consider limitations to moving large volumes of data via HTTP (APIs): in the past, connecting to databases (ETL) made it easy to grab large data sets; today, many APIs present new challenges and limits. Some applications offer purpose-built bulk APIs (see endpoint selection above), while others restrict array size or limit the total daily calls, making bulk operations difficult.
• Do your user stories account for data sitting in other applications the user will want to migrate to the new application? If so, how will your integration handle this migration?
  • File-based migrations are common (download everything from system A, transform and map, then upload everything to system B) but can create several data integrity issues.
  • Application vendors have an incentive to build migration integrations to make it easier to lure customers away from the competition, but maintaining these integrations can be difficult if competitors revoke sandbox access (see sandboxes, above).
• If a target application does not provide bulk endpoints, multiply the time it takes to make a /GET call for one page in one application by the number of pages of data you will need to loop through. This product will help you determine if you want to handle this job in the background in an asynchronous fashion or not.
• Consider timing bulk operations for opportune times of day or night when system loads are lower.
• For integrations that require near real-time response, determine if the target endpoint offers webhooks or similar that provide real-time push notifications whenever certain actions have occurred. If the target API does not offer webhooks (or similar), you will have to build (or reuse) a polling framework to request status updates that ‘listen’ for target actions.
  • Polling typically includes a simple status call to the target endpoint at regular intervals (sometimes very frequent intervals); when the status changes, integration logic makes another call to retrieve updated data. When setting polling frequency, consider any API limits and system load.

Prepare to track and log activity for support, monitoring, and security purposes

• Users almost always need support for integrations. Tag API calls with an identifier for each user and/or customer to make it easier to manage and support/debug integration usage over time.
• Pro Tip: Pitfalls to watch out for include insufficient log data to produce useful insights, lack of health checks to ensure target APIs are up, missing latency data, missing tags
  (ex. user, environment—prod vs sandbox, etc...), and not bothering to normalize error codes.

Dealing with internal users (your company) and external users

At some point the integration team or Support team will need access to API call activity logs, error reports, and alerts. For internal integrations, you will need to build this logging and monitoring or your API management (APIM) and/or integration platform (iPaaS) may provide it for you. For productized integrations for users, you may be able to leverage existing logging and notification tools, or an iPaaS, instead of building this infrastructure, as well.

This part you know how to do. API integrations have existed since we moved from direct-to-database ETL, though the tooling that supports integration development sometimes feels like it was designed for that era.

Why does this matter? If you’re new to integration work, your company may ask you to learn a purpose-built integration IDE or a point-and-click tool. Like any new programming language, integration tooling offers benefits only after you’ve invested the time to learn the basics.

Beyond the tooling, it’s worth pointing out that “your integration” is really a mini-application or microservice in and of itself: upon some event (trigger), it will take an action (CRUD + data mapping/transformation) using defined methods (we define for REST APIs below) that will depend on logical operations (integration orchestration). All this will need to be created, tested, debugged, and deployed to a production environment to meet a user’s needs.

Why the pedantic rundown? Because ultimately we can’t cover all of how to develop an application here. Instead, we’ll try to hit on key points that are specific to integration and point you to helpful resources where we can.

A (very) quick primer on REST API methods

• RESTful methods initiate Create, Retrieve, Update, and Delete actions to resources at endpoints using GET, POST, PUT, PATCH, and DELETE methods.
• Since the goal is (typically) to synchronize data when an event occurs, your integration will need to consume events that Create, Update, and Delete data.
• Consuming events will enable your integration to execute corresponding methods to Create, Update, or Delete when notified that an event has occurred at the endpoint, thereby synchronizing actions that change data.

These methods are straightforward (and not dissimilar from methods for working with SOAP APIs, for example - just simpler). The meat of integration development lies in the logical orchestration of various actions and data mapping and transformation.

• Think through the flow of your integration just as you would for any other application — in the end you need to perform an action on acceptable or usable data (i.e. update records in a target application) - so, work backwards.
  • Lay out your integration workflow: Complex integration workflows usually require multiple API calls to multiple endpoints to aggregate and transform data.
    • For example, the vendor name, address, and contact information may be contained in one single data object in one CRM, but spread across several data objects in another CRM.
  • Add any necessary pre- and post-hooks to API calls: In many cases you’ll need to add global or resource-specific hooks to manage endpoint nuances.
    • Pre-request hook: A script that executes prior to sending API requests. For example, use a pre-request hook to manipulate or add query parameters, headers, the path, or the message body.
    • Post-response hook: A script that executes after receiving a response from an API provider. For example, use a post-response hook to manipulate response headers or the message body on response.
  • When updating multiple records, create loop functions that cycle through all of the relevant records along with the query parameters that will help you ensure some records have not been missed.
    • Pro tip: watch out for paged responses from an API if you’re not the one setting the page size so you don’t miss data records.
  • During development, consider timeout requirements when using 3rd party or cloud services (i.e. AWS lambdas) and create graceful failure conditions to avoid situations that leave the user wondering what happened.

Fancy words to know - idempotency

From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result. In other words, making multiple identical requests has the same effect as making a single request. Note that while idempotent operations produce the same result on the server (no side effects), the response itself may not be the same (e.g. a resource's state may change between requests).

The PUT and DELETE methods are defined to be idempotent. However, there is a caveat on DELETE. The problem with DELETE, which if successful would normally return a 200 (OK) or 204 (No Content), will often return a 404 (Not Found) on subsequent calls, unless the service is configured to "mark" resources for deletion without actually deleting them. However, when the service actually deletes the resource, the next call will not find the resource to delete it and return a 404. However, the state on the server is the same after each DELETE call, but the response is different.

GET, HEAD, OPTIONS, and TRACE methods are defined as safe, meaning they are only intended for retrieving data. This makes them idempotent as well since multiple identical requests will behave the same.

Note on integration logic

When you create integration logic, is it simply wrapped up in your code or in the iPaaS tooling you’re using, or can others across your company build on that workflow via API call? At Cloud Elements, we believe that intelligent reuse is the only way to meet the growing demand for purpose-built APIs and user-ready integrations. In our platform, every workflow your team creates and deploys automatically gets its own REST API, or “Formula-as-a-Resource” as we call it.

This API-accessible resource allows anyone in the organization to call that process and build on top of it: a fit-for-purpose, process-based API. Enterprises and large banks find this functionality particularly useful when it comes to reuse and deploying an integration strategy that scales. Moreover, when Cloud Elements surveyed hundreds of integration professionals for the 4th annual State of API Integration report, we found that respondents overwhelmingly said that “Customized APIs that fit a specific need” are most in need in order to get value and innovate (55% vs the next closest answer of 20% for “no-code integration templates.”).

Data Mapping & Transformation

Data mapping & transformation can quickly become repetitive and maddeningly detail-oriented, especially if you’re designing and building for a self-serve mapping and transformation UI serving a range of users vs. a single-purpose, bi-directional integration. If it’s the former, here’s what we recommend:

• Define a default mapping:
  • A default mapping is a template that will define how standard data structures from an endpoint will map into your application's data model. This mapping establishes the association of standard fields within a standard object into the fields within your application.
  • The primary goal here is to save your users’ time by pre-mapping standard data from the endpoint, reducing the effort they need to undertake when connecting to your app.
• Present discovered data:
  • Your integration/application must discover the data structure of the user’s target application and will need to present the discovered data structure, custom objects, and custom fields for your users to evaluate and map based on their understanding of the data.
• Provide the UI for the user to map custom data:
  • Custom data objects and custom fields within standard objects are common. The challenge here is that you’re going to need to support unique data maps at the “Instance” level. The instance level data maps will be specific to that authenticated account of an endpoint.
  • There is no way to fully anticipate the type of data coming in, as it only exists specific to that “instance” of a service. The most dynamic user experience provides the user with full, intuitive, self-service control over these mappings.
  • There is also no way to always anticipate how your users may use standard fields. Occasionally, users will use standard fields for a custom purpose and they will likely want to map this data to a field other than your default mapping. Just as you handle custom data on a dynamic, self-service basis, it’s best to enable your users to override your default mappings.
  • Finally, within these many data mapping permutations, create the value/type transformations that will convert the user’s data from one format, such as XML or SOAP, or even currencies and time formats, into another.

Events and Polling

The bottom line is hope that the applications/ APIs you’re working with offer some type of event notification, like webhooks. If they don’t, you’ll have to build a polling framework.

• The AsyncAPI Initiative is one group aiming to publish industry standards and reduce the protocol fragmentation we see today.
• Event-driven APIs typically remove complexity from the consumer—there’s no question of maintaining state or considering race conditions. But that puts a higher burden on the API publisher, a burden some teams push back onto the API consumer via API design (see more from our friends at Ably Realtime in section 6 of the 2020 SOAI report). 
• Bulk data operations: what to do when your design requirements include batch operations but the target API doesn’t support bulk:
  • Building a bulk facade is simply no fun and getting one out-of-the-box for any Element in our platform is one of the key benefits Cloud Elements’ users comment on (to toot our own horn a bit).
  • If bulk operations are required by users, consider the frequency and self-sufficiency requirements they have — many applications offer a .csv or flat file upload for data migrations that you can take advantage of (formatting pains notwithstanding).
  • If users need regular batch operations, or if you’re working on regular sync integrations (i.e. nightly backups or data syncs), we suggest you work in steps: i) downloading the bulk data, ii) transforming the data, then iii) uploading your records to the target system. This is so you can work around API limits for the different systems separately (i.e. page size, API call limits, status and record failures, etc...) That said, be cautious where you store the data in flight and how/when you purge it — you don’t want your integration inadvertently compromising security protocols and exposing sensitive data.

A few notes on integration security, API design, and management:

From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result. In other words, making multiple identical requests has the same effect as making a single request. Note that while idempotent operations produce the same result on the server (no side effects), the response itself may not be the same (e.g. a resource's state may change between requests).

As we now know, "security is everyone’s job." Developers building API integrations, especially self-serve productized integrations for users, need to make sure their applications are resistant to code injections, for example, and securely manage users’ credentials (for non- OAuth 2.0 applications). That said, modern APIs are for a “zero-trust environment” in which the endpoint dictates the security mechanisms for the integration developer. So a few notes on API Management and Security — there’s lots of good content out there if you’d like to learn more (like Apigee’s web API design guide):

• APIs should be treated as products with a deliberate lifecycle
• All APIs should “enforce consistent security and governance”
• A key to making APIs usable is clear, concise documentation
• For enterprises moving from monolithic to microservices architectures, an API facade can help relieve the pressure on back-end systems and help limit disruption
• Don’t rely only on web application firewalls (WAFs) for security
• Ensure you mitigate OWASP threats, prevent volumetric attacks, and protect against adaptive threats

While just one statistic, in 2019, researchers estimated that three-quarters of mobile apps had vulnerabilities and 38%-43% had “high-risk” vulnerabilities, according to Positive Technologies.

The approach to testing and QA usually depends on the type of integration: on the spectrum from back-end data sync to multi-tenanted self-serve integration for non-technical users, simple smoke tests and test cases will usually drive the former while user story-focused QA testing typical of any new feature drives the latter. Depending on the situation, consider load testing as well.

In any case, you need a complete set of integration test cases and quality test data to work with as well as the right sandbox environment(s) to validate formatting and field dependencies (see Design section above). Finally, we have a few other recommendations before your final deployment of your integration:

• Create a real-time report of integration usage data and feed the data into your application team's performance monitoring tools.
• Automate health checks for your app’s integration to stay ahead of API maintenance (see log reviews in the next section).
• Work with your DevOps team to monitor response time from individual endpoints — isolate your application’s performance vs. the performance of the endpoint you’re connecting to.
• Tag your usage data to enable rapid filtering of your logs and isolate/debug issues.
• Standardize error response because your operations team will not have time to research every vendor's error code.
• When building multi-tenanted integrations (i.e. productized within your application for user self-service), tag and log activity for each user’s instance for support, monitoring, and security purposes. 

When done right, successful development, testing, and deployment of your integration (or your integration microservice if you prefer) is a lot like enjoying a special dinner you’ve spent all day preparing. Building and deploying successfully is extremely satisfying, and if you’ve done the research and design well, there’s not much that ruins the build. But at the risk of holding onto an analogy too long — if enjoying the meal is build + test + deploy, maintenance is like doing the dishes.

That said, someone has to do them. Period. In this world, if it’s not you at any given point in time, it’s a friend or colleague. And it’ll probably be you at some point in the future (ask our CTO, Vineet, if you ever fully get to leave this work behind...) Since sooner or later we all have to do certain chores, we might as well do them well and efficiently, if nothing else, to move on to other work ASAP.

Maintenance falls into two categories—i) API changes that require a response from you, and ii) updates to the user experience (i.e. new edge case or enhanced functionality). For the former, here’s what you need to know:

• Most important: keep an eye on API deprecation & migration requirements.
  • As new API versions become available, API providers may deprecate or reduce support for existing APIs. Nearly every API provider will let you know this is coming, so make sure you’re not entirely filtering out provider announcements and emails. Moreover, most providers will share ways for users to leverage the new or other API endpoints for similar needs, though occasionally providers deprecate or restrict functionality entirely (which requires a redesign of the integration).
  • Once you see an API deprecation announcement for an API supporting an active integration, schedule time to research the change in an upcoming sprint - we recommend scheduling that work sooner than later so there’s ample time to prioritize changes to the integration. When the deprecation deadline drives prioritization (i.e. ‘it has to be this sprint because the API goes offline in two weeks...’), you leave yourself little room for added complexity or mistakes, as well as risk downtime that affects users.
• Short of a full deprecation, API providers can make material changes to how the API works. These changes are usually (but not always) included as a new version of the API.
  • Again, nearly all API providers will give you advanced notification. It’s not a bad idea to designate a role on your team to monitor notifications from API providers regularly.
  • In our experience, most changes include added functionality that you as the consumer can choose to take advantage of or not (i.e. back to the Requirements & Design phases). Sometimes changes to vendor-supported query parameter syntax or versions that are not backwards-compatible can break an existing integration. Oftentimes, with query parameters and versions related to API design and load the provider has to manage
    (i.e. you’re not the only consumer of that particular API...), it can be tough to remember the details of the query logic you use, so we recommend solid documentation as well as scheduling a research spike when you see these types of announcements come through.
• Usually in line with added security and/or added functionality, API providers may also make authentication changes. Like the changes above, we recommend prioritizing some research time early-on before the changes will go live; however, these changes usually require relatively less time, except in cases where the auth type changes wholesale (i.e. from basic to OAuth2.0). Keep an eye out for:
  • Changes to OAuth apps such as which scopes are required or optional
  • Changes to authentication header requirements
  • Changes to authentication for specific endpoints (e.g. a provider may accept OAuth2 tokens for one endpoint, but only basic authentication for others)

The other category of integration maintenance tasks relate to how the integration functions for users (reactive or proactive) rather than reacting to changes to the APIs you consume. Logging is foundational (see the end of the Design section) to give you data on failures that might represent new edge cases or user-focused enhancements.

• Presumably you have automated notifications for application downtime; beyond that a regular review of logs helps guide error troubleshooting and to debug/pinpoint areas of failure. In particular, experience has taught us to focus on:
  • Failed API Calls
  • Looking for failed authentication and re-authentication tracking
• Further, even the best research and design cannot cover every edge case and integrations will need maintenance to cover newly-discovered edge cases. The error logs above often translate into new user-focused edge cases:
  • Custom Data including custom objects, custom fields, and custom formats in objects like Vendor Accounts
  • Special character support both in payloads and in queries
  • Field or data type changes in connected apps often require redesign of the integration’s mapping and transformation logic
• Finally, the pinnacle of happy/sad state for most integration developers we work with relates to enhancements: happy because the integration is serving user needs well, sad because there are now opportunities to do more, which just means more work. But beware piecemeal enhancements made on the fly without going back to a redesign:
  • Maintaining enhancements that multiple users rely on and keeping them current with the latest provider functionality can be time-consuming. Depending on where you store the logic to normalize the experience from a user-facing perspective, it can be challenging to scale the applicability of an enhancement to multiple use cases.

First, we understand there’s a lot here. We get it. There’s a lot of detail in this guide because there’s a lot of complexity in the world of API integration: every API and data model is unique, like a snowflake, forcing integration developers to deal with loads of complexity and nuance.

Second, that’s why Cloud Elements and other integration platforms (iPaaS) exist. For developers, integrations quickly become tedious, undifferentiated work — especially when you build from scratch and the team has to document and maintain that custom code. That’s not to say engineering teams should never write and maintain their own integration services. But before you do, ask yourself and your team if integration is a strategic/ differentiating part of what you offer or a supporting feature/service? If you’re an integration center of excellence (COE) or a workflow platform, the answer is likely ‘yes’ - if you’re a SaaS payments app, maybe not.

Integration platforms (iPaaS) come in two varieties:

1. Point-and-click tools: allow non-technical users or business analysts (what Gartner calls Citizen and Ad Hoc integrators, respectively) to quickly automate relatively simple data flows.
2. Developer platforms: allow Application Developers and Integration Specialists (again, Gartner terms) to create new solutions and to incorporate complex, highly- customized applications like ERP, HRIS, and core banking systems into seamless processes.

Congrats on making it to the end! Now that you know everything
you need to build an API integration, check out Cloud Elements.

We can't wait to see what you'll create.