首页 > 其他分享 >ODI CDC应用猜想

ODI CDC应用猜想

时间:2023-05-26 12:05:44浏览次数:31  
标签:猜想 CDC Subcriber 数据表 should ODI using operation changes


注:本文内容猜想属于个人意见,不保证内容的正确性。如有错误,欢迎指正。

 


1、Package中Model的选项设置为Journalizing Model


此配置无法正常完成cdc数据捕获)


参数选择Add Subscriber、Extend Window、Lock Subscriber/ Remove Subscriber、Purge Journal、Unlock Subcriber


Subcriber列表中添加导数据接口中对应的Subcriber Filter


 


结果:1)删除源数据表的数据,运行包,目标数据表的数据被删除


      2)新增或修改源数据表的数据,运行包,目标数据表的数据无变化,日记数据不存在,Subcriber不存在


 


猜想:Add Subcriber的动作在运行包的时候才进行,导致未捕获之前源数据表发生的数据变化信息,而结束后执行了Remove Subcriber,导致运行后查看Subcriber找不到对应的


      订阅者及日记数据。


      另外,根据官方文档描述,删除源数据表的数据,目标数据也跟着被删除(似乎没有被以上猜想影响),可能原因是:


Knowledge Module Options
When processing journalized data, the SYNC_JRN_DELETE option of the integration knowledge
module should be set carefully.
It invokes the deletion from the target datastore of the records
marked as deleted (D) in the journals and that are not excluded by the journalizing filter. If this
option is set to No, integration will only process inserts and updates.


 


官方推荐做法:


consistent set journalizing:


Operations before Using the Changed Data


Extend Window: The Consistency Window is a range of available changes in all the tables
of the consistency set for which the insert/update/delete are possible without violating
referential integrity. The extend window operation (re)computes this window to take into
account new changes captured since the latest Extend Window operation. This operation is
implemented using a package step with the Journalizing Model Type. This operation can be
scheduled separately from other journalizing operations.



 Lock Subscribers: Although the extend window is applied to the entire consistency set,
subscribers consume the changes separately. This operation performs a subscriber(s) specific
"snapshot" of the changes in the consistency window. This snapshot includes all the changes
within the consistency window that have not been consumed yet by the subscriber(s). This
operation is implemented using a package step with the Journalizing Model Type. It should
be always performed before the first interface using changes captured for the subscriber(s).


 


Operations after Using the Changed Data


After using the changed data, the following operations should be performed:
 Unlock Subscribers: This operation commits the use of the changes that where locked
during the Lock Subscribers operations for the subscribers. It should be processed only after
all the changes for the subscribers have been processed. This operation is implemented using
a package step with the Journalizing Model Type. It should be always performed after the
last interface using changes captured for the subscribers. If the changes need to be processed
again (for example, in case of an error), this operation should not be performed.
 Purge Journal: After all subscribers have consumed the changes they have subscribed to,
entries still remain in the journalizing tables and should be deleted. This is performed by the
Purge Journal operation. This operation is implemented using a package step with the
Journalizing Model Type. This operation can be scheduled separately from the other
journalizing operations.

标签:猜想,CDC,Subcriber,数据表,should,ODI,using,operation,changes
From: https://blog.51cto.com/u_16129500/6354821

相关文章

  • C# 操作List集合报错:集合被修改,枚举操作可能不会执行(Collection was modified, enume
    问题Ihavemultithreadsapplicationandigetthiserror**************ExceptionText**************System.InvalidOperationException:Collectionwasmodified;enumerationoperationmaynotexecute.atSystem.ThrowHelper.ThrowInvalidOperationExcepti......
  • SAP CODING BLOCK
    https://blog.csdn.net/m0_46145967/article/details/110954475  文章目录1.背景2.添加字段2.1普通模式2.2专家模式2.3新建MSEG的CDS视图3.添加字段到屏幕3.1需要更改的标准程序及屏幕3.2将拷贝出来的屏幕替换标准屏幕4.维护字段状态4.1维护字段状态变式4.2维护字段是......
  • list - 删除元素 ConcurrentModificationException
     前天看了公众号,说是三年开发都不会删除元素,看了一眼,没想到第二天就用上了........而我也是那个菜鸟哈哈哈哈哈哈.........记录一下吧 publicstaticvoidmain(String[]args){List<String>list=newArrayList<>();list.add("1");list.add("2");lis......
  • 哥德巴赫猜想
    一问题描述两千以内大于4的正偶数都可以变成两个素数的和的形式。二设计思路将偶数全部求出存入数组然后再将分出两个数据判断是否为素数三程序流程图 四伪代码实现#include<iostream>usingnamespacestd;intmain(){ intd[2000],j=0; for(inti=4;i<2000;i++){ if(i%......
  • (个人简介)Coding Home - 漂流瓶jz
    个人简介:华东师范大学计算机专业硕士。目前是Web前端开发。GitHub    jzplp(漂流瓶jz)·GitHub新浪博客   漂流瓶jz_新浪博客......
  • 【活动预告】数据集成引擎BitSail遇上CDC
    BitSail是字节跳动开源数据集成引擎,于2022年10月26日宣布开源,可支持多种异构数据源间的数据同步,并提供离线、实时、全量、增量场景下全域数据集成解决方案。BitSail支撑了字节内部众多的业务线,支持多种数据源之间的批式/流式/增量同步,日均数据传输量达上百PB。在开源后,CDC相关的诉......
  • 【活动预告】数据集成引擎BitSail遇上CDC
    BitSail是字节跳动开源数据集成引擎,于2022年10月26日宣布开源,可支持多种异构数据源间的数据同步,并提供离线、实时、全量、增量场景下全域数据集成解决方案。BitSail支撑了字节内部众多的业务线,支持多种数据源之间的批式/流式/增量同步,日均数据传输量达上百PB。在开源后,CDC相关的......
  • 双模齐下,提质增效:CODING 携手知微共创 BizDevOps 体系新篇章
    为了提升工作和管理效率,工具建设是许多企业不得不面对的现实,然而在工具建设落地过程中,往往存在一系列的问题。如不同组织、部门之间互不相通,各自为政,工具流程与实际工作所需不符,导致工具建设的结果是人去适应工具而不是工具来辅助人。由此可见,工具体系若建设不佳,非但无法起到提......
  • 专精特新 ︱ 腾讯云 CODING 助力消费电子类企业高速发展期的研运一体化
    ......
  • 异步CDC及异步FIFO分享
    分享两篇很棒的论文:1.《ClockDomainCrossing(CDC)Design&VerificationTechniquesUsingSystemVerilog》http://www.sunburst-design.com/papers/CummingsSNUG2008Boston_CDC.pdf2.《SimulationandSynthesisTechniquesforAsynchronousFIFO......