This part you know how to do. API integrations have existed since we moved from direct-to-database ETL, though the tooling that supports integration development sometimes feels like it was designed for that era.
Why does this matter? If you’re new to integration work, your company may ask you to learn a purpose-built integration IDE or a point-and-click tool. Like any new programming language, integration tooling offers benefits only after you’ve invested the time to learn the basics.
Beyond the tooling, it’s worth pointing out that “your integration” is really a mini-application or microservice in and of itself: upon some event (trigger), it will take an action (CRUD + data mapping/transformation) using defined methods (we define for REST APIs below) that will depend on logical operations (integration orchestration). All this will need to be created, tested, debugged, and deployed to a production environment to meet a user’s needs.
Why the pedantic rundown? Because ultimately we can’t cover all of how to develop an application here. Instead, we’ll try to hit on key points that are specific to integration and point you to helpful resources where we can.
A (very) quick primer on REST API methods
- RESTful methods initiate Create, Retrieve, Update, and Delete actions to resources at endpoints using GET, POST, PUT, PATCH, and DELETE methods.
- Since the goal is (typically) to synchronize data when an event occurs, your integration will need to consume events that Create, Update, and Delete data.
- Consuming events will enable your integration to execute corresponding methods to Create, Update, or Delete when notified that an event has occurred at the endpoint, thereby synchronizing actions that change data.
These methods are straightforward (and not dissimilar from methods for working with SOAP APIs, for example - just simpler). The meat of integration development lies in the logical orchestration of various actions and data mapping and transformation.
- Think through the flow of your integration just as you would for any other application — in the end you need to perform an action on acceptable or usable data (i.e. update records in a target application) - so, work backwards.
- Lay out your integration workflow: Complex integration workflows usually require multiple API calls to multiple endpoints to aggregate and transform data.
- For example, the vendor name, address, and contact information may be contained in one single data object in one CRM, but spread across several data objects in another CRM.
- Add any necessary pre- and post-hooks to API calls: In many cases you’ll need to add global or resource-specific hooks to manage endpoint nuances.
- Pre-request hook: A script that executes prior to sending API requests. For example, use a pre-request hook to manipulate or add query parameters, headers, the path, or the message body.
- Post-response hook: A script that executes after receiving a response from an API provider. For example, use a post-response hook to manipulate response headers or the message body on response.
- When updating multiple records, create loop functions that cycle through all of the relevant records along with the query parameters that will help you ensure some records have not been missed.
- Pro tip: watch out for paged responses from an API if you’re not the one setting the page size so you don’t miss data records.
- During development, consider timeout requirements when using 3rd party or cloud services (i.e. AWS lambdas) and create graceful failure conditions to avoid situations that leave the user wondering what happened.
Fancy words to know - idempotency
From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result. In other words, making multiple identical requests has the same effect as making a single request. Note that while idempotent operations produce the same result on the server (no side effects), the response itself may not be the same (e.g. a resource's state may change between requests).
The PUT and DELETE methods are defined to be idempotent. However, there is a caveat on DELETE. The problem with DELETE, which if successful would normally return a 200 (OK) or 204 (No Content), will often return a 404 (Not Found) on subsequent calls, unless the service is configured to "mark" resources for deletion without actually deleting them. However, when the service actually deletes the resource, the next call will not find the resource to delete it and return a 404. However, the state on the server is the same after each DELETE call, but the response is different.
GET, HEAD, OPTIONS, and TRACE methods are defined as safe, meaning they are only intended for retrieving data. This makes them idempotent as well since multiple identical requests will behave the same.
Note on integration logic
When you create integration logic, is it simply wrapped up in your code or in the iPaaS tooling you’re using, or can others across your company build on that workflow via API call? At Cloud Elements, we believe that intelligent reuse is the only way to meet the growing demand for purpose-built APIs and user-ready integrations. In our platform, every workflow your team creates and deploys automatically gets its own REST API, or “Formula-as-a-Resource” as we call it.
This API-accessible resource allows anyone in the organization to call that process and build on top of it: a fit-for-purpose, process-based API. Enterprises and large banks find this functionality particularly useful when it comes to reuse and deploying an integration strategy that scales. Moreover, when Cloud Elements surveyed hundreds of integration professionals for the 4th annual State of API Integration report, we found that respondents overwhelmingly said that “Customized APIs that fit a specific need” are most in need in order to get value and innovate (55% vs the next closest answer of 20% for “no-code integration templates.”).
Data Mapping & Transformation
Data mapping & transformation can quickly become repetitive and maddeningly detail-oriented, especially if you’re designing and building for a self-serve mapping and transformation UI serving a range of users vs. a single-purpose, bi-directional integration. If it’s the former, here’s what we recommend:
- Define a default mapping:
- A default mapping is a template that will define how standard data structures from an endpoint will map into your application's data model. This mapping establishes the association of standard fields within a standard object into the fields within your application.
- The primary goal here is to save your users’ time by pre-mapping standard data from the endpoint, reducing the effort they need to undertake when connecting to your app.
- Present discovered data:
- Provide the UI for the user to map custom data:
- Custom data objects and custom fields within standard objects are common. The challenge here is that you’re going to need to support unique data maps at the “Instance” level. The instance level data maps will be specific to that authenticated account of an endpoint.
- There is no way to fully anticipate the type of data coming in, as it only exists specific to that “instance” of a service. The most dynamic user experience provides the user with full, intuitive, self-service control over these mappings.
- There is also no way to always anticipate how your users may use standard fields. Occasionally, users will use standard fields for a custom purpose and they will likely want to map this data to a field other than your default mapping. Just as you handle custom data on a dynamic, self-service basis, it’s best to enable your users to override your default mappings.
- Finally, within these many data mapping permutations, create the value/type transformations that will convert the user’s data from one format, such as XML or SOAP, or even currencies and time formats, into another.

Events and Polling
The bottom line is hope that the applications/ APIs you’re working with offer some type of event notification, like webhooks. If they don’t, you’ll have to build a polling framework.
- The AsyncAPI Initiative is one group aiming to publish industry standards and reduce the protocol fragmentation we see today.
- Event-driven APIs typically remove complexity from the consumer—there’s no question of maintaining state or considering race conditions. But that puts a higher burden on the API publisher, a burden some teams push back onto the API consumer via API design (see more from our friends at Ably Realtime in section 6 of the 2020 SOAI report).

- Bulk data operations: what to do when your design requirements include batch operations but the target API doesn’t support bulk:
- Building a bulk facade is simply no fun and getting one out-of-the-box for any Element in our platform is one of the key benefits Cloud Elements’ users comment on (to toot our own horn a bit).
- If bulk operations are required by users, consider the frequency and self-sufficiency requirements they have — many applications offer a .csv or flat file upload for data migrations that you can take advantage of (formatting pains notwithstanding).
- If users need regular batch operations, or if you’re working on regular sync integrations (i.e. nightly backups or data syncs), we suggest you work in steps: i) downloading the bulk data, ii) transforming the data, then iii) uploading your records to the target system. This is so you can work around API limits for the different systems separately (i.e. page size, API call limits, status and record failures, etc...) That said, be cautious where you store the data in flight and how/when you purge it — you don’t want your integration inadvertently compromising security protocols and exposing sensitive data.
A few notes on integration security, API design, and management:
From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result. In other words, making multiple identical requests has the same effect as making a single request. Note that while idempotent operations produce the same result on the server (no side effects), the response itself may not be the same (e.g. a resource's state may change between requests).
As we now know, "security is everyone’s job." Developers building API integrations, especially self-serve productized integrations for users, need to make sure their applications are resistant to code injections, for example, and securely manage users’ credentials (for non- OAuth 2.0 applications). That said, modern APIs are for a “zero-trust environment” in which the endpoint dictates the security mechanisms for the integration developer. So a few notes on API Management and Security — there’s lots of good content out there if you’d like to learn more (like Apigee’s web API design guide):
- APIs should be treated as products with a deliberate lifecycle
- All APIs should “enforce consistent security and governance”
- A key to making APIs usable is clear, concise documentation
- For enterprises moving from monolithic to microservices architectures, an API facade can help relieve the pressure on back-end systems and help limit disruption
- Don’t rely only on web application firewalls (WAFs) for security
- Ensure you mitigate OWASP threats, prevent volumetric attacks, and protect against adaptive threats
While just one statistic, in 2019, researchers estimated that three-quarters of mobile apps had vulnerabilities and 38%-43% had “high-risk” vulnerabilities, according to Positive Technologies.
