Connectivity

7.1 Universal Load Operation

InfoLink provides a universal interface to load data from any data source (stored in any format) into any data source (in any format). The LoadTableFromSource (aka Load) operation represents the interface.

The Load operation unifies access to data sources by representing any data source as a set of named spaces consisting of tables with associated format specification as described in the Introduction.

The Load operation allows you to extract a table from any data source (called source) and load it into the same or another data source (called target). Loading into the same data source can be useful for file-based sources when you want to convert a file to another format (e.g. from XML to CSV).

Internally, the Load operation distinguishes two types of data sources:

• File-based data sources - low-level data sources that store files and provide access to the files as a stream of bytes without understanding their internal structure. Examples of this type of source include File System and Amazon S3.
• Record-based data sources - high-level data sources that provide an interface to access objects as a set of structured records (e.g. Salesforce CRM, PostgreSQL).

Any data source can be accessed at any of the levels listed above. InfoLink implements an optimization technique that chooses the access level for source and target data sources to achieve the best performance. If a data source supports a native mechanism to access data at the required level, then the native mechanism is used because native implementations usually provide the best performance. Otherwise, InfoLink’s mechanism is used. For example, both Amazon S3 and Azure Blob Storage provide only file-based access. To copy a CSV file from Amazon S3 onto Azure Blob Storage, the file will be opened as a stream without analyzing its content (i.e. file-based level of access) and copied onto Azure Blob Storage. If you want to convert the file from CSV to XML, Infolink will parse the content of the file into records (i.e. record-based access), convert the records into XML, compose the XML file, and store it onto Azure Blob Storage. Another example would be loading a table from PostgreSQL into MySQL. Both of these systems support native methods to load export and import CSV files (i.e. file-based level of access). The Load operation will use file-based access for the source and target because exporting and then importing a CSV file will be much faster than reading the source table by record and inserting the records into the target table.

Supported File formats

• CSV
• Excel
• JSON
• PDF
• PSV
• Tab-separated
• XML