Connect

 View Only
  • 1.  Your expression looks right. But is Connect CDC evaluating it when you think it is?

    Employee
    Posted 10 days ago
    What I noticed
    I was mapping target columns in Connect CDC Director, selecting predefined methods, and everything validated cleanly. The model committed. Replication started. Then I looked at the actual values landing at the target and realised I had assumed evaluation happened at apply-time when it did not. That one assumption shaped the wrong data silently.

    Connect CDC ships a full library of Predefined Data Transformation methods: Copy input, Concat2, Concat3, Current date, Current timestamp, Date format, Upper Case, Lower Case, Job name, Row manipulation type, Row timestamp, Transaction ID, Transaction timestamp, Lookup Table, Substring, Trim, and more. You can also create your own compiled method called a procedure. It is a complex expression built with programming statements, given an internal procedure name, an explicit input parameter list, and an external display name for the Director. Column references are not allowed inside the procedure body. The current row is not an implied input you must explicitly declare every parameter the procedure needs.

    This is where the distinction matters. The Director validates the method at design time, syntax only. Actual execution happens at apply-time on the kernel when real row data flows through it. Those are two different moments with two different contexts.

    There is also a firm boundary on where user procedures can go. You cannot enter a user procedure in the Sending Table Properties Gate Condition tab, or in the Result expression box that appears when you select the Expression Method for a target column in the Mapping tab. That restriction keeps row filtering and value transformation in separate, well-defined layers.

    Why this design works
    • Predictability: Syntax errors surface at design time before a row moves. Value computation remains bound to the actual row context at apply time.
    • Reliability: Explicit parameter declarations mean the procedure never pulls silently from ambient row state. What goes in is declared. What comes out is deterministic.
    • Operational Safety: Blocking user procedures from Gate Condition contexts prevents expressions from evaluating where the row state is incomplete or ambiguous.
    What's different for user-defined vs. built-in methods
    There is one extra step.
    • Built-in methods are selected directly from the Method list. No compilation step required.
    • User-defined compiled methods require a Validate step in Director before they appear in the Method list for mapping. Also, the method name cannot match any existing built-in method name. Appending company initials or using mixed-case names avoids conflicts, since all Connect CDC built-in methods are coded in lower case.
    • A validated method has passed syntax checking not functional testing. Logic errors inside the body only surface at apply-time with real data.
    Why this matters in production
    If you use Current date or Current timestamp in a mapping, knowing when the method evaluates tells you whether you are capturing the source change time or the apply processing time. For audit trails, that difference is not minor.
    A procedure that validates correctly in Director can still produce wrong output or fail on unexpected values at runtime. Validation and correctness are not the same thing.
    If you ever see target column values that look structurally right but semantically wrong, do not assume the mapping is broken. It is Connect CDC doing exactly what you told it at exactly the time you may not have expected.

    What changed for me
    I used to treat the Validate button as confirmation that a method was correct. I now treat it as confirmation that the syntax is acceptable. I test user-defined compiled methods against representative source data before committing them to any production model. The separation between design-time and apply-time is intentional. Once I understood that boundary, the behavior became completely predictable.

    Takeaway
    Built-in methods map directly with no extra steps. User-defined compiled methods require explicit parameter declaration, a Director validation pass, and cannot be placed in Gate Condition contexts. Design-time validation and apply-time execution are separate layers and understanding which one you are working in prevents a class of silent data quality issues that are genuinely hard to trace after the fact.

    Have you built custom compiled methods in Connect CDC, and what was the most unexpected behavior when the procedure ran on real data versus how it looked during validation?


    ------------------------------
    Adhitya Maya
    *Precisely Software Inc.
    ------------------------------


  • 2.  RE: Your expression looks right. But is Connect CDC evaluating it when you think it is?

    Employee
    Posted 2 days ago

    Interesting perspective. It's easy to assume that a successful validation means everything will behave as expected, but real data often reveals scenarios that design time checks can't.



    ------------------------------
    Sathya LJK
    *Precisely Software Inc.
    ------------------------------



  • 3.  RE: Your expression looks right. But is Connect CDC evaluating it when you think it is?

    Employee
    Posted yesterday
    Very relatable. One of the biggest lessons with compiled methods is that "validates successfully" does not necessarily mean "behaves as expected" with real production data. Runtime testing is essential.


    ------------------------------
    Neha Goel
    *Precisely Software Inc.
    ------------------------------