Home
Accessing Stages Through Enterprise Designer
Access different stages through Enterprise Designer.
DB Change Data Reader
The DB Change Data Reader stage allows you to select the columns to be included in the current jobflow, where the columns have the Change Data Capture feature enabled on them.

Welcome
Explore Data Integration. Connect different data sources and various stages used in Data Integration process.
Introduction
Learn about data management architecture and the star schema data warehouse design.
Connecting to Data Sources and Data Warehouses
Connect to different types of data sources.
Populating the Data Warehouse
Prepare your data for warehouse. Populate Time Dimension table, Dimension table, Fact table. Add time stamp to records in a Data Warehouse.
Updating the Data Warehouse
Define a Data Warehouse update schedule. Update a fact table. Use Global and Local cache for queries.
Accessing Stages Through Enterprise Designer
Access different stages through Enterprise Designer.
- Call Stored Procedure
- DB Change Data Reader
  The DB Change Data Reader stage allows you to select the columns to be included in the current jobflow, where the columns have the Change Data Capture feature enabled on them.
- DB Loader
  The DB Loader stage allows you to access and load data from/to databases configured in Spectrum Data Integration. This stage provides an interface to a high-speed data loading utility. Currently, the Spectrum Data Integration platform supports Oracle Loader, DB2 Loader, PostgreSQL Loader, and Teradata Loader.
- Field Parser
- Field Combiner
- Field Selector
- Generate Time Dimension
- Query Cache
- Query DB
- Query NoSQL DB
- Read From DB
- Read From File
- Read from Hadoop Sequence File
- Read From Hive File
- Read from HL7 File
- Read from NoSQL DB
- Read from SAP
- Read from Spreadsheet
- Read from Variable Format File
- Read From XML
- SQL Command
- Transposer
- Unique ID Generator
- Write to Cache
- Write to DB
- Write to File
- Write to Hadoop Sequence File
- Write to Hive File
- Write to NoSQL DB
- Write to Spreadsheet
- Write to Variable Format File
- Write to XML
- Date and Number Patterns
Configurations
Configure Oracle LogMiner, and HDFS connection for HA Cluster.
Optimizing Performance
Improve performance by determining an optimum fetch size. Learn about best practices for connecting to HDFS 3.x and Hive 2.1.1.

DB Change Data Reader

The DB Change Data Reader stage allows you to select the columns to be included in the current jobflow, where the columns have the Change Data Capture feature enabled on them.

In the stage, you can create a Change Data Capture (CDC) Resource, which is the required data source table. The columns of the CDC resource on which the Change Data Capture feature is enabled are reflected using checkboxes.

Change Data Capture

The Change Data Capture feature enables capture of all changes made in a column. For each selected column, all inserts, updates, and deletions are captured.

Supported Databases

Currently, Spectrum Technology Platform supports the Change Data Capture (CDC) feature for MS SQL and Oracle databases only.

MS SQL

For MS SQL data sources, the Change Data Capture feature can be enabled or disabled for columns of tables from the backend. Refer to here for the necessary steps.

Note: The Change Data Capture feature is not supported in the Express edition of SQL Server.

Oracle

For Oracle data sources, Spectrum Technology Platform tracks the data changes in the table columns using the LogMiner utility of Oracle. CDC cannot be enabled or disabled on columns of tables in an Oracle data source through Spectrum Technology Platform.

The data changes resulting from insert, update, and delete queries are tracked and captured from an entered start date and time up to the current date and time. This entered start time is applicable during the first query execution with CDC switched on.

In subsequent executions on the same Oracle connection, data changes are captured from the time of the last execution till the current time.

Note: This is an incremental process. In the first capture, the changes are captured from the entered start date and time to the current date and time. In subsequent captures, the changes are captured from after the end date and time of the previous capture to the current date and time.