What I ran into
I recently came across a recovery scenario that initially looked pretty straightforward.
A table had gone through a REORG operation, the Change Selector stopped as expected, and the standard recovery approach was followed. The plan was simple:
- Reload the affected table
- Restart replication
- Confirm processing resumes normally
The reload completed successfully, so naturally everyone expected replication to come back as well.
It didn't.
When the obvious answer isn't the answer
At first, it was tempting to assume the reload hadn't actually fixed the problem.
But the more we looked at it, the less the table seemed to be the issue.
The table was healthy. The reload had done its job.
So why was the Change Selector still stopping every time it restarted?
Looking beyond the table
The breakthrough came when attention shifted away from the table itself and toward the recovery position.
The Change Selector had originally stopped after encountering a journal entry related to the REORG event. Even after the table was recovered, the Change Selector was still trying to restart from a point before that same journal entry.
Which meant every restart looked something like this:
- Start the Change Selector
- Read the journal stream
- Encounter the same journal entry
- Stop again
The environment appeared to be stuck, even though the underlying issue had already been addressed.
What stood out to me
What I found interesting was that there were actually two separate recovery activities happening:
- Recovering the table itself
- Recovering the replication position
The first one was complete.
The second one wasn't.
Once the restart position moved beyond the event that originally triggered the stop condition, processing resumed normally and replication continued as expected.
Takeaway
The lesson for me was simple:
Fixing the object that caused replication to stop doesn't always mean replication is ready to continue.
Sometimes the real question isn't "Was the issue fixed?"
It's "Has replication moved past the point where it encountered that issue?"
Those sound similar, but they're not always the same thing.
Discussion
Have you run into situations where the original problem was resolved, but replication still couldn't move forward because of its recovery position?
When validating a recovery, do you focus first on the object that failed, the replication state, or both?
*Precisely Software Inc.