I recently came across a replication scenario that sparked an interesting discussion around copy strategies and CDC adoption.
The environment involved a very large source table with a relatively small percentage of rows changing each day. Despite the low change volume, the process relied on periodic full copies of the entire dataset.
As copy windows became a concern, several optimisation ideas were explored:
- Can the data be partitioned and copied in parallel?
- Would multiple distributions improve throughput?
- Are there ways to tune the process to reduce overall copy time?
- Is there a better approach for handling large datasets with relatively low daily change rates?
What stood out to me wasn't the answer to any one of those questions. Instead, it was a broader architectural discussion.
At what point does a full copy stop being the most practical approach?
In large environments, the challenge is often not the amount of data changing, but the amount of data being moved repeatedly. A table may experience only a small percentage of daily changes, yet the entire dataset is still copied on a recurring basis.
As tables grow, copy windows can become harder to fit into maintenance schedules, even when change volume remains relatively stable.
At some point, the conversation starts shifting from "How do we make copies faster?" to "Should we still be doing full copies at all?"
I've seen environments where full-copy strategies worked successfully for years because they were simple and predictable. I've also seen situations where CDC significantly reduced data movement and made replication more manageable at scale.
Questions for the Community
- What factors typically drive your decision to move from periodic full copies to CDC?
- Is there a table size, copy duration, or change volume that becomes a tipping point?
- Have you implemented optimisations that significantly extended the life of a full-copy approach?
- How do you balance operational simplicity against the efficiency gains that CDC can provide?
- Are there situations where you would continue using full copies even when CDC is available?
Discussion question:
When only a small fraction of a large table changes each day, do you see continued full copies as a deliberate design choice, or as a sign that it's time to rethink the architecture?
I'd be interested to hear where others draw that line and what factors influence that decision in practice.
------------------------------
Shivang Joshi
*Precisely Software Inc.
------------------------------