Welcome
Explore Data Integration. Connect different data sources and various stages used in Data Integration process.
Introduction
Learn about data management architecture and the star schema data warehouse design.
Connecting to Data Sources and Data Warehouses
Connect to different types of data sources.
Populating the Data Warehouse
Prepare your data for warehouse. Populate Time Dimension table, Dimension table, Fact table. Add time stamp to records in a Data Warehouse.
Updating the Data Warehouse
Define a Data Warehouse update schedule. Update a fact table. Use Global and Local cache for queries.
Accessing Stages Through Enterprise Designer
Access different stages through Enterprise Designer.
- Call Stored Procedure
- DB Change Data Reader
  The DB Change Data Reader stage allows you to select the columns to be included in the current jobflow, where the columns have the Change Data Capture feature enabled on them.
- DB Loader
  The DB Loader stage allows you to access and load data from/to databases configured in Spectrum Data Integration. This stage provides an interface to a high-speed data loading utility. Currently, the Spectrum Data Integration platform supports Oracle Loader, DB2 Loader, PostgreSQL Loader, and Teradata Loader.
- Field Parser
- Field Combiner
- Field Selector
- Generate Time Dimension
- Query Cache
- Query DB
- Query NoSQL DB
- Read From DB
  - Visual Query Builder
  - Query Variables
- Read From File
- Read from Hadoop Sequence File
- Read From Hive File
- Read from HL7 File
- Read from NoSQL DB
- Read from SAP
- Read from Spreadsheet
- Read from Variable Format File
- Read From XML
- SQL Command
- Transposer
- Unique ID Generator
- Write to Cache
- Write to DB
- Write to File
- Write to Hadoop Sequence File
- Write to Hive File
- Write to NoSQL DB
- Write to Spreadsheet
- Write to Variable Format File
- Write to XML
- Date and Number Patterns
Configurations
Configure Oracle LogMiner, and HDFS connection for HA Cluster.
Optimizing Performance
Improve performance by determining an optimum fetch size. Learn about best practices for connecting to HDFS 3.x and Hive 2.1.1.

Query Variables

In the Read From DB stage, while defining the query, you can include variables instead of actual column names. Using variables in the query allows you to customize the query conditions at runtime (using the Dataflow Options) or through the Job Executor .

However, you can also provide Stage Options value in the Runtime tab and view the schema and sample of records to be fetched by the query by using the Regenerate Fields and Preview buttons respectively.

A variable is defined using the format #{variable}, and inserted in either the select or where clause of an SQL query.

Note: You can edit a query generated using the Visual Query Builder to include variables. However, the edited query will not be readable by the Visual Query Builder anymore. The Build SQL button is disabled when you include a variable in a manually written or generated SQL query.