The easy part is the part everyone plans for
Ask a team how an integration is going and they will tell you about authentication, rate limits and payload formats. Those are real problems and they are the small ones. The three expensive problems, in the order they will cost you money, are identity, meaning and timing. Which record over here is the same thing as that record over there. What each field actually means to the people who fill it in. And when a fact became true, given that one system updates instantly and the other updates at two in the morning.
Transport is a week. Identity, meaning and timing are the project. A proposal that spends four pages on the API and one paragraph on field mapping is describing the wrong job.
Identity: nothing has a shared key
Two systems built independently have no common identifier. The order system knows customer 40219. The accounting package knows ACME-004. Nothing anywhere states that these are the same company, and both of them have three records for it because someone re-entered the address in 2021.
You will be tempted to match on the fly — compare names, normalise the address, take the best score, move on. Do not. Fuzzy matching at run time means the same pair can match today and not tomorrow, the logic lives inside a sync script nobody reads, and nobody can answer why a payment landed on the wrong account.
Build a crosswalk instead: a table you own, with one row per matched pair, holding both source identifiers, how the match was made, the date, and who approved it if a human did. It is unglamorous and it is the single most valuable artifact the project produces, because it outlives the integration. When you replace one of the two systems in four years, the crosswalk is still true.
Realistic numbers: on business names with addresses, expect eighty-five to ninety-five percent to match confidently by rule. The remainder is where the cost is. Budget human review for it — a person clearing a queue of ambiguous pairs at perhaps a hundred an hour — and accept that a few will stay unmatched forever because they are genuinely ambiguous. Then keep the queue open, because new unmatched records arrive every week and a crosswalk with no maintenance quietly stops covering the business.
Two rules make this durable. First, never delete a crosswalk row; supersede it, so history stays explainable. Second, never let the crosswalk be a spreadsheet on someone's laptop, which is where about half of them start and all of them should not stay.
Meaning: the same word, two definitions
This is where the money goes, and it is invisible in every architecture diagram.
Status words. “Closed” in the operations system means the work is finished. “Closed” in the finance system means it has been paid. Sync one to the other and you have just told the accounting package that unpaid work is settled.
Enumerations that nearly overlap. One system has six order statuses, the other has nine, and four of them map cleanly. The other five are the argument. There is no technical answer to this; somebody with authority has to decide, and the decision has consequences for reports both sides already rely on.
Dates without time zones. A date field with no zone is a fact with a twenty-four hour error bar, and month-end is precisely where that error bar lands on the wrong side. Establish, in writing, whether a date is local to the site, local to headquarters, or absolute.
Amounts. Tax included or excluded. Currency stated or assumed. Discounts applied at line level or invoice level. Rounding at two decimals or six. Two systems that each add up correctly will still disagree by a few cents per line, and a few cents per line at volume is a reconciliation meeting every month.
Free-text fields carrying structure. Somewhere in every older system is a notes field where the team encoded something the software would not let them record properly — a purchase order reference, a delivery instruction, a flag. It is load-bearing, and it is not in the schema.
The fix is a field-by-field mapping document, written before code, that states for each field: the source, the target, the transformation, what happens when the value is missing, and what happens when it is a value nobody anticipated. Then it gets read and approved by the person who owns each system — not their technical contact, the person accountable for what the numbers mean. That approval is the cheapest insurance in the project. Every integration disaster I have seen up close was visible in a mapping table that nobody senior read.
Timing: when is a fact true
One system is live. The other posts in a nightly batch. At four in the afternoon the two reports disagree, and by nine the next morning they agree. Everyone concludes the integration is broken. It is not broken; it is a design that nobody stated out loud.
Decide and publish a “truth time.” If the reconciled view is accurate as of the close of the previous business day, say so on the report, and stop trying to make an intraday number agree with a nightly one. Half the credibility problems in this work are solved by labelling the clock.
Then handle three specific cases explicitly. Late-arriving records, which show up dated for last week and must not silently change a closed period. Corrections and reversals, which are not updates and should not overwrite the original. And back-dated edits made by a user who fixed a typo in a record from March, which will change a number somebody already reported unless you decided in advance whether it can.
One direction or two
One-way sync costs roughly a tenth of two-way, and most requirements described as two-way are actually two independent one-way flows on different fields.
Two-way brings problems that have no clean answers. Which side wins when both changed since the last sync? How do you stop an echo, where your write to system B triggers a webhook that writes back to system A that triggers a webhook? What happens to a record deleted on one side and edited on the other?
The honest resolution is ownership at the field level rather than the system level. The customer's billing address is owned by finance and flows outward. The delivery window is owned by operations and flows outward. Nothing is owned by both. Write that ownership map down; it is a one-page document and it eliminates most of the conflict logic before it is ever written.
| Transport | When it is right | What it costs you later |
|---|---|---|
| Vendor API, polled | Default. Predictable, testable, easy to reason about | Latency equal to your poll interval; rate limits at volume |
| Webhooks | Low latency matters and volume is moderate | Delivery is not guaranteed; you still need a nightly sweep to find what was missed |
| Scheduled file exchange | Underrated. Daily is genuinely fine for most finance flows | Almost nothing; it is auditable, replayable and boring, which is a compliment |
| Database replica or change capture | You control both databases and volumes are high | You are coupled to someone's internal schema, which will change without notice |
| Integration platform (drag-and-drop) | A handful of simple, low-volume flows | Cost climbs with volume; logic hides inside a tool nobody can review or test properly |
| Screen automation | The vendor has no interface and will not build one | Breaks on any interface change; price it as a subscription to breakage, not a build |
You are probably here because
- Two reports disagree and nobody can say which one is right
- Someone is re-keying data between systems for hours a week
- An integration was built, works most days, and nobody knows when it does not
- You just acquired a company and now own two of everything
The identity and meaning sections are the work. The reconciliation section is the thing to insist on if you insist on only one thing.
The one job to build first
If you take one item from this article into a vendor conversation, take this: build the reconciliation before you build the sync.
A reconciliation job runs on a schedule and answers a short list of questions. How many records exist on each side for yesterday. What do they sum to. Which identifiers are present on one side and absent on the other. Which pairs exist on both sides but disagree on a field that matters. Then it delivers that to a person who can do something about it, and it says something even when everything is fine, so silence is never ambiguous.
Without it, you learn that the connector stopped on the fourteenth when a customer calls on the twenty-second. With it, integration failures become a five-minute morning check. It is perhaps five percent of the build cost and it is the difference between a system you trust and a system you hope about.
Where the effort actually goes on a typical integration — our read
Our judgment of a typical split, not a measured average. If a proposal inverts this, it was written by someone who has not done the second one.
What the vendor's API will do to you
Assume none of these and check all of them in week one, because each has stopped a project cold.
Deletes may be invisible. If records vanish without a marker, you cannot detect deletions incrementally. Your only option is a periodic full comparison, and you need to know that before you promise anything.
The modified timestamp may lie. Bulk operations and administrative edits frequently do not update it, so an incremental pull based on it will silently skip records. Test by changing something in an unusual way and seeing whether the timestamp moves.
Pagination has a floor and a ceiling. Plenty of interfaces work beautifully to ten thousand records and behave strangely past that. Test at the volume you will actually run, not the volume in the sample.
The sandbox is not production. Different data shapes, different limits, sometimes a different version. Reserve time for surprises after the first live run.
Rate limits are the schedule. If you can pull two hundred records a minute and you have four hundred thousand, your initial load is a day and a half. That is fine if it is planned and a crisis if it is discovered on cutover weekend.
Retries, order and the message that will not process
Every integration needs four unglamorous properties. Every operation must be safe to run twice, because it will be. Failures must retry with increasing delay rather than hammering a system that is already unwell. Anything that fails repeatedly must land in a holding area with the original payload preserved, so a person can look at it. And you must be able to replay a day's traffic after a fix, without duplicating anything.
Order matters more than people expect. An update arriving before the record it updates is a common and confusing failure. Either sequence the flows deliberately or make the receiving side tolerate out-of-order arrival by creating a placeholder. Decide which; do not discover it.
What it costs
A first one-way integration between two systems with decent interfaces, one object type, and a mapping that somebody has actually approved: two to five weeks. Add a second object type and its relationships, and it grows less than the first — the plumbing is reusable, the meaning is not. Two-way with genuine conflict handling: six to twelve weeks, and it stays more expensive forever.
Ongoing, budget ten to twenty percent of the build cost per year per integration. Interfaces change, a vendor deprecates a version, someone adds a required field. And watch the count: point-to-point connections between six systems can be fifteen relationships, and each one has an owner, a failure mode and a maintenance bill. At three systems, point-to-point is right. Past five or six, a canonical model in the middle starts paying — but building the hub first, before you have three working integrations to generalise from, produces an expensive abstraction of a business you did not yet understand.
When you do not need an integration
Be honest about volume. If forty records a week move between the two systems, a person copying them takes twenty minutes a week. An integration costs weeks to build and carries a permanent maintenance obligation. The integration wins on error rate and on not depending on one person, and it loses badly on cost. Say the arithmetic out loud before starting.
If either vendor offers a supported native connector to the other, use it, even if it is imperfect. Their maintenance obligation is worth more than your elegance.
And if the two systems overlap enough that one could replace the other within a year, integrating them is paying to preserve a situation you are about to end.
How integrations fail
- Matching records at run time instead of recording match decisions in a crosswalk you own
- No approved field mapping, so “closed” means two things and nobody noticed until month-end
- Two-way sync by default when two one-way flows on different fields were the requirement
- No reconciliation, so the first indication of failure is a customer
- Operations that are not safe to repeat, producing duplicates on the first retry
- Trusting a modified timestamp that bulk updates do not touch
- Building the hub first and generalising a business nobody had mapped yet
- Screen automation sold as an integration, priced as a build and paid for as breakage
Before you start
- A field-level mapping exists and the owner of each system has read and approved it
- Every field has a single owning system; nothing is owned by both
- A crosswalk table is designed, owned, versioned, and never overwritten in place
- The unmatched queue has a person and a working rhythm
- Time zones, tax treatment and rounding are written down, not assumed
- The truth time is stated on every report the integration feeds
- Deletes, corrections and back-dated edits each have a defined behaviour
- Every operation is safe to repeat, and a replay procedure has been tested
- A daily reconciliation reports counts, sums and differences to a named person
- Someone has confirmed the API's real limits at your real volume
Bottom line
Spend the first two weeks on paper, not on code. Agree what the fields mean with the people accountable for them, build a crosswalk you own rather than matching on the fly, name a single owner for every field, and state the truth time so nobody argues about a four-o'clock discrepancy that is working as designed. Then build the reconciliation before the sync, so the system tells you when it is wrong instead of leaving you to find out from a customer. The transport will take a week and nobody will ever thank you for it.
Frequently asked questions
For a few simple, low-volume flows, genuinely yes, and it is often the right call. It stops being the right call when the logic gets conditional, when volumes make per-operation pricing painful, or when you need to review, test and version what the integration actually does. Business rules living inside a drag-and-drop canvas are difficult to audit and difficult to move, which matters most on the day you want to leave.
On business names with addresses, typically eighty-five to ninety-five percent by rule. The rest needs a person, and a stubborn few percent stay genuinely ambiguous. Plan for a review queue rather than a one-off cleanup, because new unmatched records arrive continuously. Never let a low-confidence match flow through silently — a wrongly matched customer is worse than an unmatched one.
Ask what decision changes if the data is a day old. For most finance and reporting flows, nothing does, and a nightly file exchange is simpler, cheaper, auditable and easy to re-run. Real time earns its cost when a person is waiting on the answer or when a customer sees the result. Choosing real time for its own sake buys latency you cannot use and complexity you keep.
In order: ask for a scheduled export, which surprisingly often exists; ask for read access to a reporting database or replica; then screen automation as a last resort. Automation of a user interface works and breaks on any layout change, so budget it as an ongoing subscription to maintenance rather than a build. It is also worth asking what the vendor's roadmap says, and what leaving them would cost, because no interface is itself a fact about the relationship.
Not first. With three systems, direct connections are simpler and cheaper. Past five or six, the number of relationships grows faster than the number of systems and a canonical model starts to pay. Build two or three point-to-point integrations, learn what the shared definitions actually are, and generalise from real examples — a hub designed before that is an abstraction of a guess.