Connect CDC

Discussions

Members

Resources

Events

 View Only

  • 1.  Adding a column in Connect CDC without a model update can break replication (and why column order matters)

    Employee
    Posted 7 days ago
    What I ran into
    I ran into a case where a new column was added to a table, but it was not even part of replication. It still caused replication to fail with data format errors.
    At first this felt unexpected. If the column is not being replicated, why would it matter?

    What is actually happening
    Connect CDC mappings depend on column order, not just column names.
    When a new column is added:
    • CDC still reads rows based on the structure it knows
    • The Change Selector expects the old column layout
    If a column is added in the middle of the table:
    • All columns after it shift position
    • CDC can start reading values into the wrong fields
    • This leads to data mismatches and format errors
    Even if that column is not mapped, the structure change affects everything.

    Why adding at the end behaves differently
    This was the key detail for me.
    • If a column is added at the end, existing column order stays the same
    • CDC mappings continue to align correctly
    • No model update is needed in that case (if the column is not used)
    So the placement of the column matters more than the column itself.

    What I now do for safe changes
    If a column is added in the middle:
    • Stop replication for the affected table
    • Let the queue drain completely
    • Refresh the table in Director
    • Validate and commit the model
    • Run a model update before restarting
    This ensures CDC understands the new structure before processing resumes.
    What changed for me
    I used to think CDC mappings were flexible as long as I did not touch mapped columns.
    This made it clear that CDC is tightly coupled to table structure layout, not just logic.
    Even unused columns can break replication if they change the physical order.
    Takeaway
    In Connect CDC, schema changes are not just about data usage. They are about structure alignment.
    • Add at the end → usually safe
    • Add in the middle → requires model update
    That small difference can decide whether replication runs smoothly or fails.

    Have you run into cases where a small schema change unexpectedly caused replication errors?


    ------------------------------
    Adhitya Maya
    *Precisely Software Inc.
    ------------------------------


  • 2.  RE: Adding a column in Connect CDC without a model update can break replication (and why column order matters)

    Employee
    Posted 3 hours ago
    Good point. I've seen similar cases where a seemingly harmless schema change caused replication failures because CDC relies on the physical table structure and column order. It's a great reminder that even non-replicated columns can impact replication when added in the middle of a table. Refreshing the table and performing a model update after such changes is definitely a best practice.


    ------------------------------
    Neha Goel
    *Precisely Software Inc.
    ------------------------------