Scheduling Imports

Scheduled Imports are comprised of several EnterWorks Server repositories and an EPX workflow. Together, they provide the means of processing imports on a scheduled basis as well as the means to pre-process the submitted files before actually importing them into EnterWorks.

Two EnterWorks Server repositories are used to support Scheduled Import processing: Scheduled Imports and Scheduled Import Jobs.

Note that when the repositories are opened in a Repository View, the default Preference View does not show all attributes.

System Log Directory Scheduled Imports Repository

All scheduled imports are defined in the Scheduled Imports repository. Each record in this repository represents one scheduled import.

Note that when the repository is opened in a Repository View, the default Preference View does not show all attributes.

Each attribute from this repository is described in the following table:

Attribute	Description
Category Attribute Association Name	The name of the Category Attribute Association object to be updated when the Import Type is “Cat Attr Assoc”.
Code Set Full Replacement	If set to “Yes”, the code set import file fully replaces the existing code set. This means that existing nodes not included in the import file will be deleted from the code set.
Code Set Name	Name of the code set to be updated when the Import Type is “Code Set”.
Delete Saved Set	If set to “Yes”, the specified Saved Set should be deleted before adding records for the current import to the saved set.
Dependent Keys	List of keys to be passed to each dependent operation. The keys must match attribute names in the dependent repository. The corresponding value in the Dependent Values attribute is then used to set the value of the key attribute when the Job is created. The value can reference attributes from the source job by surrounding the attribute name with double-pipe characters. For example, if an export is dependent on an import and that export should specify the same user, the Dependent Key would be set to “Manager Login” and the value would be “\|\|Manager Login\|\|”. Conversely, if the same dependent Export needs to be set to the admin user regardless of its initial definition, the Dependent Key would be set to “Manager Login” and the Dependent Values set to “admin”.
Dependent Operation	Name of dependent Import, Export, or Promotion to be initiated upon successful completion of this job.
Dependent Operation Immediate	Initiate the dependent operation immediately if “Yes”. Otherwise the operation will be queued and processed by the scheduler, which may take several minutes to start.
Dependent Operation Type	Type of dependent operation: Promotion, Import, or Export
Dependent Values	Values corresponding to the Dependent Keys that are passed to the dependent operations. Values may reference attributes by surround the attribute name with double-pipe characters. For example, to reference the Import Job number, the Dependent Values might contain: “Job\|\|Export Job\|\|”. If Export Job in the current record is set to 1234, the resulting value will be “Job1234”
Description	Description of scheduled import. It is strongly recommended that the Description be populated with all the details for the import, including any job that launches this job, any pre-processing performed, and what jobs this job launches. If ParameterX attributes are used, each should be detailed as to their content and purpose.
Display Attributes	[DEPRECATED] Comma-delimited list of attributes to include in e-mails
Effective Primary Key	[DEPRECATED] Comma-delimited list of attributes to serve as the primary key for this import.
Email Body	Override e-mail body for successful jobs. Attributes in the job record can be referenced by surrounding them with double-pipe characters (e.g., \|\|Manager Login\|\| is replaced with the value of the Manager Login attribute).
Email Notify Failure	Send an Import Failed e-mail to the owner if Yes and the import fails to process successfully.
Email Notify Success	Send an Import Successful e-mail to the owner if Yes and the import processes successfully.
Email Subject	Subject for success e-mail. May contain references to any other job attribute by surrounding it with double-pipe characters (e.g., \|\|Manager Login\|\| is replaced with the value of the attribute Manager Login).
Email To	Email address(es) to which success/failure notifications will be sent. If not defined, the e-mail address(es) associated with the Manager Login user/group will be used.
Error Dependent Keys	Optional keys for arguments to be passed to any Dependent jobs (as properties) when this job fails with an error.
Error Dependent Operation	List of Promotions, Imports, and/or Exports (based on Dependent Operation Type) that should be triggered on successful completion of this one.
Error Dependent Operation Immediate	Execute dependent operation immediately if Yes when this job fails with an error.
Error Dependent Operation Type	List of operation types for each Dependent Operation when this job fails with an error. Export – launch an export job Import – launch an import job Promotion – launch a promotion job
Error Dependent Values	Optional values for dependent keys when this job fails with an error.
Group Lock Action	Group locks allow for multiple concurrent jobs to all attach to the same lock. The lock is not freed until all processes have detached from it. Action to be taken with the lock identified in Group Lock Name: Attach – the import will attach to the specified lock. Processing for the import will begin immediately. Free – the import will detach from the named lock (presumably set by a previous job that invoked this import) once the import processing has completed. The lock is not completely freed until all attached processes have detached. Attach And Free – the import will attach to the named lock and begin processing and detach from the named lock once processing has completed. No Lock – the import will not attach to a group lock. Group Locks will always acquire the designated lock, even if another process has acquired or attached to the same lock. If multiple jobs attach to the same lock via Group Lock, the lock is not free until all of the processes have freed the lock.
Group Lock Name	Name of the group lock.
Ignore Missing File	When an Import is executed and the specified file cannot be found, the Failure e-mail will be suppressed if this flag is set to “Yes”.
Import Directory Name	Fully qualified path to directory where import file can be found.
Import File Name	Name of file to be imported.
Import File Sort	Specifies whether the matching files (if Import File Name includes a wildcard character) should be sorted before selecting a file. File Ascending – sort file names in ascending order File Descending – sort file names in descending order
Import Format	[DEPRECATED] Format of the import file.
Import Language	Specifies the language to be used for the import. If blank, the default language is used. If set to a non-default language, the multi-language fields for this language are updated.
Import Name	Logical name for the Import.
Import Now	The import file is to be processed immediately if set to “Yes”. This flag is immediately set to No once a record for this import has been created in the Scheduled Import Jobs repository. This flag will be set to Yes any time the import is to be processed outside of the scheduled time. The immediate processing of the import file will not alter the next scheduled date and time. For example, if the import is scheduled to be processed nightly at 1:30 a.m. and a user requests an immediate import at 11:00 p.m. by setting this flag to “Yes”, the file will be processed at 11 p.m. and then again at 1:30 a.m.
Import Option Name	Name of import option to be applied to this import. The code set Import Options contains a complete list of available options. The description for each code identifies what the option does and what values can be specified.
Import Option Value	Value for the corresponding import option to be applied to this import.
Import Priority	Priority of import controls order which jobs are checked. The lower the number, the higher the priority.
Import Type	Identifies the type of import: Template – the import uses an Import Template Repository – the import uses a View Mapping for a specific repository Code Set – the import is for a Code Set Custom – the import only performs the assigned pre-processing module
Inactive Records	Identifies action to be taken with Inactive records when the import type is Template and is configured as a full replacement import (which will flag records not present in the file as being inactive): No Action – don’t do anything Delete – delete inactive records from Staging and Production after the import has completed Reactivate – reactivate inactive records if they are present in the import file
Inactive View	Name of snapshot table view for the target repository. This view is used when processing inactive records in a full replacement template import.
Include Timestamp	Include date and time in Import_<job>.log file if Yes. This can be helpful in troubleshooting pre-processing for an import as it shows how long processing steps took (providing the appropriate messages are being reported).
Last Import Datetime	Date and time of when this import was last initiated (scheduled or manual).
Last Scheduled Import Datetime	Date and time of when scheduled import was last processed
Lock Action	Action to be taken with the lock identified in Lock Name: Acquire – the import will wait until the named lock is acquired. Processing for the import will not begin until then. Free – the import will free the named lock (presumably set by a previous job that invoked this import) once the import processing has completed. Acquire And Free – the import will wait until the named lock is acquired and will free the named lock once processing has completed. No Lock – the import will not wait for any lock before processing. Locks will only be acquired if they are currently free. If the designated lock has been attached to by multiple jobs using the Group Lock, the lock does not become free until all of those jobs have freed the lock.
Lock Name	Name of the lock to acquire or free (depending upon Lock Action). If more than one import shares the same lock, then they will be processed serially. The name may contain references to other properties by surrounding each property name with double-pipe characters. For example, to ensure only one instance of an import is processed at a time for each user, the Lock Name can be set to: “Import\|\|Manager Login\|\|”. This will produce a lock name of “Importjdoe” for John Doe and “Importjsmith” for John Smith. If the referenced lock is associated with the Group Lock in other jobs, the lock will not be considered free until all the Group Lock jobs have detached from the lock.
Manager Login	Login of the PIM user or EnterWorks role assigned to this import. This user (or group of users if a role is specified) will receive e-mails generated for processing imports.
Move Import File	The import file is moved to the designated import directory and renamed to include a date/time stamp if Yes. The import file is left in its original location and processed from there if No.
Next Import Datetime	Date and time of next scheduled import in the format “mm/dd/yyyy hh:mm:ss”. EnterWorks will initiate the processing of the import file when this value is in the past. The value is updated to the next scheduled time (using the Import Period) as soon as the processing commences.
Parameter1-5	Five attributes that can serve as parameters when jobs are connected through the Dependent Operation attributes. They are not used directly by the Import processing but can be referenced in other attributes (e.g., \|\|Parameter2\|\|).
Preprocess Class	Fully-qualified path, identifying the Preprocess class to be invoked for this import. It must be a subclass of com.enterworks.services.BaseCustomProcessFile
Preprocess File	Import file must be preprocessed using the designated class and arguments if “Yes”.
Preprocess Keys	Argument keys identifying the Preprocess Argument Values to be passed to the designated preprocess module.
Preprocess Values	Values for corresponding Preprocess Argument Keys to be passed to the designated preprocess module. Values can contain references to properties from this job by surrounding the property name with double-pipe characters. For example, to pass the Import Job, the value would be set to “\|\|Import Job\|\|”
Repository Name	Name of target repository.
Repository Timeout (Minutes)	Maximum number of minutes allowed for processing a Repository Import. This timeout specifies how long the Scheduled Import processing will wait for the import to complete processing. If a timeout is reached, the job is flagged with an error and the designated user or group is notified. The actual job may still complete successfully, but any dependent jobs will not be triggered.
Repository View Mapping	Name of mapping view for Repository Import. Mappings are either manually defined in the EnterWorks 10 UI or by manually performing an import using the EnterWorks 10 Repository interface and selecting the option to Save Mapping.
Run As Manager Login	Run the import using the login designated in Manager Login if Yes. When enabled, the Manager Login must be set to a user and not a group. This option ensures the import is subject to the security of the designated login and also identifies the records being created/updated by that login (vs. system).
Saved Set	Name of Saved Set to be created if specified.
Schedule Days	Identifies the days of the week if Schedule Type is “Days of Week”
Schedule Exclude Dates Code Set	If set to a code set listing dates as code values, the scheduling logic will skip to the next scheduled date if the calculated date is one of the dates in the code set.
Schedule Period	Number of time units (based on Schedule Type) after Last Scheduled Date before the next scheduled import is initiated. For example, an import that is to be processed daily could be set to type DAYS and have a period of 1 or be set to type MINUTES and have a period of 1440. The scheduled imports are not affected by any manually initiated imports
Schedule Type	Type of schedule: MINUTES – the Schedule Period identifies the number of minutes before re-invoking the import. DAYS – the Schedule Period identifies the number of days before re-invoking the import. DAYS OF WEEK – specific days of the week on which the import is to be executed (which is specified by the attribute Schedule Days). WEEKS – the Schedule Period identifies the number of weeks before re-invoking the import. MONTHS – the Schedule Period identifies the number of months before re-invoking the import.
Scheduled Folder	Logical folder in which the scheduled job is assigned for organization purposes. If set, and the Scheduled Folder hierarchy is selected in the repository list view, this import job can be found by clicking a node in the Scheduled Folder hierarchy.
Scheduled Folder Sequence	Used to control the order of scheduled jobs in the Scheduled Folders
Scheduled Import	Set to “Yes” if the import is to be scheduled. Each time the scheduled import is processed, the Next Import Datetime is updated based on the Schedule Type and Scheduled Period along with the previous value for Next Import Datetime. For example, if the Schedule Type is set to DAYS OF WEEK and Scheduled Days is set to Monday, Wednesday, and Friday and the Next Import Datetime is set to 10/3/2014 02:00:00 (2:00 am on Friday, October 3^rd), the Next Import Datetime attribute will be updated to 10/6/2014 02:00:00 (2:00 am on Monday, October 6^th).
Skip File Check	Skip checking for and moving a file if Yes. Requires Preprocess File = Yes and Preprocess Class is defined. The assigned class is responsible for ensuring a file is created.
Skip If Busy	Skip this Import if Scheduled and there is already an active job for the same import. This ensures only one scheduled import job is running at any given time.
Source FTP	FTP Server name if Source Type is FTP
Source FTP Delete File From Server	Delete file from FTP server if Yes (default)
Source FTP File Name	Name of source file if Source Type is FTP. The file name may contain wildcards (‘*’) which may result in multiple files being retrieved. If multiple files are expected, the Import File Name should also have wild cards and the Scheduled Import should be configured to call itself as a dependent operation to ensure each file is processed.
Source FTP List Gaps	Number of gaps in the data returned by the LIST command for the FTP server between the start of each line and the name of the file. This is used by EnterWorks to know how to extract the name of the file.
Source FTP Path	Fully qualified directory path on the FTP server if Source Type is FTP.
Source FTP Port	FTP Server port (21 default, 22 SFTP)
Source FTP User	FTP Server user name if Source Type is FTP
Source FTP User Password	FTP Server user password if Source Type is FTP
Source Type	Type of Source for Import (File or FTP). Default is File
Status	Status of import record: Active – record is active Retired/Inactive – record is not active
Template Name	Name of Import Template name if Import Type is “Template”
Template Timeout (Minutes)	Number of minutes before the processing for a Template import is flagged as timing out, at which point an e-mail will be sent to the designated user or group. The import template processing may still finish successfully, but any dependent jobs will not be initiated.

Scheduled Import Jobs Repository

The Scheduled Import Jobs repository contains records representing the currently active or recently completed import jobs. Whenever an import is initiated, the details for that import are copied from the Imports repository. This record is subsequently updated to reflect the current status of the import.

Note that when the repository is opened in a Repository View, the default Preference View does not show all attributes.

In addition to the attributes define in the Scheduled Imports repository, the Scheduled Import Jobs repository also has the following attributes:

Attribute	Description
# Created	The total number of records created
# Deleted	The total number of records deleted
# Errors	The total number of errors encountered during the import processing.
# Processed	The total number of records processed
# Updated	The total number of existing records that were updated
Import Errors	Lists the details for the cause of the import to fail (when the Import Status is “Error”).
Import Job	Unique number for the import job.
Import Status	Status of current or last import. Possible values include: Aborted – the import has been aborted by a user New – the import is new Processing – the import file is being processed. While in this state, no additional import can be initiated for this import record. Completed – import processing has completed successfully Error – the import failed. The Import Status must be changed to New or Completed for it to be processed again (if scheduled)
Job Log File	Name of the Scheduled Import job log file (e.g., Import_<job>.log)
Last Submitted By	Login ID of last user who submitted a file for this target.
System Log Directory	Contains the directory for where the EnterWorks import logs are created.
System Log File	Name of the EnterWorks import log file.

Scheduled Import Configuration Properties

The Scheduled Import behavior is controlled by several configuration properties:

Property	Description
import.baseDirectory	Full path to the directory where the Scheduled Import files will be placed (e.g., Enterworks/shared/Imports). If EnterWorks is configured in a distributed environment, a network directory path should be specified.
import.expireActiveDays	Number of days before import jobs that still show “Processing” are deleted from the Scheduled Import Jobs repository.
import.expireCompletedDays	Number of days before import jobs that have completed or failed are deleted from the Scheduled Import Jobs repository.
import.logDirectory	Full path to the directory where the EnterWorks import logs are placed (e.g., Enterworks/logs/EnableServer/import).