Adaptive data integration keeps a connection useful as source data changes. It combines agreed data meaning, controlled flexibility, validation, and recovery. The goal is to detect a change, decide how it should be handled, and preserve enough evidence to explain the resulting records.
Write down what the receiving process needs
Start with the business use of the data: paying an invoice, updating an order, or reporting sales. For each required field, define its name, type, units, permitted values, meaning, and treatment of missing values. Establish the record's identity and the rule for updates and deletions.
A field named “amount” is incomplete without currency and a definition such as unit price before tax. A timestamp needs a time-zone convention. A customer ID needs a source system and a rule for duplicate or merged records. Assign a source owner who can explain changes and a receiving owner who can approve their treatment.
Preserve an access-controlled copy of received data with source, batch ID, receipt time, and mapping version. This permits later diagnosis without confusing a transformed record with its original input.
Choose flexibility at the field level
Microsoft's Azure Data Factory schema-drift documentation describes accepting fields absent from the defined projection, with a tradeoff in early knowledge of names and types. In that product, newly detected fields arrive as strings by default unless type inference is enabled. Those are platform-specific behaviors to verify in a deployment.
For your own pipeline, classify changes by their consequences. Accepting an optional descriptive field can be reasonable while a change in currency or identity requires review. Automatic type conversion deserves particular attention when empty strings, decimal separators, leading zeros, or units carry meaning.
On a small screen, scroll the table sideways to read all columns.
| Change | Possible handling | Evidence needed |
|---|---|---|
| New optional delivery note | Retain in raw input; map only if approved | Confirm its meaning, access needs, and destination. |
| Quantity changes from number to free text | Quarantine the record | Get an unambiguous corrected value from the source. |
| Required customer identifier disappears | Stop or quarantine the affected scope | Establish identity before writing destination records. |
| Price changes from cents to dollars | Hold the affected feed | Approve a versioned unit conversion and reconcile totals. |
Walk through a small validation exercise
This fictional supplier feed uses positive integer quantities, nonnegative integer unit prices in U.S. cents, currency USD, and unique order IDs. It represents order lines with one line per order for this exercise. Taxes, shipping, discounts, and multiple-line orders are outside its model.
On a small screen, scroll the table sideways to read all columns.
| Order ID | Quantity | Unit price, cents | Initial result |
|---|---|---|---|
| A101 | 2 | 1,250 | Accept: 2,500 cents |
| A102 | 3 | 800 | Accept: 2,400 cents |
| A103 | “two” | 900 | Quarantine: quantity is text |
The initial accepted total is 2,500 + 2,400 = 4,900 cents ($49). Keep A103 out of the accepted amount and report the batch as incomplete. The supplier's new optional delivery note can remain in the raw record while approved fields are mapped to the destination.
After the source supplies quantity 2 for A103, the additional amount is 1,800 cents. The complete accepted batch totals 6,700 cents ($67). Download the JSON exercise with rules, inputs, correction, and expected results to reproduce the calculation. It contains invented records only.
Make correction and replay controlled operations
Use a stable business key and a defined update policy so retrying a batch does not create another copy of the same order. In this simple exercise, an already accepted identical order is a no-op. A changed accepted order should enter an explicit correction path with history, rather than silently replacing a prior record.
Commit validated destination writes with an appropriate transaction or checkpoint strategy for the system. Record what succeeded before a failure. Replaying only a failed record may be suitable for independent orders; a financial or inventory batch that must balance as a whole may need to remain unpublished until all checks pass.
Test a repeated batch, an interrupted write, a duplicate key with different values, and a corrected quarantined record. Verify the final state and history for each case. A successful retry message is weaker evidence than the correct destination records.
Reconcile meaning as well as counts
Track records received, accepted, held, and rejected using categories that sum to the received set. Keep duplicate attempts distinct from unique business records. Add useful control totals in consistent units and compare them with source-side totals where available.
Alert an owner when required fields change, quarantines rise, an expected batch is missing, or freshness exceeds the business limit. Include a sample reference and reason without exposing sensitive contents in notifications. Decide who authorizes a new mapping and how consumers learn that the data is partial.
Connect this work to IT Change Management for deployment controls and Corporate Performance Management for comparable reporting definitions.