Implementing Pre- or Post-Processing

The Scheduled Imports provides the option to pre-process files before the files are actually imported and the Scheduled Exports provides the option to post-process files after they have been exported. In both cases, the actual processing is handled by a Java class that is an extension of the BaseCustomProcessFile class found in the Services.jar file (or in an application-specific JAR file). The following pre-defined pre/post-processing blocks are available:

Classpath

Description

com.enterworks.services.exports.MoveDamAssets

Copies the DAM assets for the records in the export file to the Portal PIM. The export file must include the FilePath attribute which is removed from the final file after processing.

com.enterworks.services.exports. MoveDamAssetsAndUpdateAttributesForItemsInFile

Moves the DAM assets to the Internal PIM and produces an import file to update the designated attributes to the specified values for each record in the file. The export file must include the FilePath attribute which is removed from the final file after processing.

com.enterworks.services.exports. ProcessPublicationMergeAutomatedSortTemplate

Converts a Publication Merge and Automated Sort template export in CSV format to XLS with the category attributes reduced to only those in the taxonomy node (assumes all records are for the same taxonomy node) and inserts placeholders for the Publication Merge and Automated Sort entries.

com.enterworks.services.imports. DAMLinksFullReplace

Treats the import file as a full replacement for the designated target repository (DAMLink repository may link assets to more than one repository). Any record found in the DAMLink repository but not in the file (for the designated repository) will be flagged as Inactive. If the import is configured to delete inactive records, those records will be physically removed from the DAMLink repository.

com.enterworks.services.imports.EncodeFile

Converts the import file from one encoding to another.

com.enterworks.services.imports. PreProcessAddFields

Adds attributes and values to the import file before loading.

com.enterworks.services.imports. PreProcessImageLinksFile

Preprocesses a generic DAM Link file that contains one or more business keys, the file name, and the image context, priority, and caption. The business keys are replaced with a single ID value.

com.enterworks.services.imports. ProcessImagePackage

Processes a single file or a zip file containing one or more image files. If the submitted file has the csv extension, it is passed on for import processing by ePIM. If the submitted file has the zip extension, the contents of the zip file are processed. Any valid image files are copied to the designated image directory. If the submitted file is a valid image file, it is copied to the designated image directory.

com.enterworks.services.imports. PromoteSavedSet

Promotes the valid records in the designated saved set to Production.

com.enterworks.services.imports. SplitDynamicAttributesFile

Preprocess import file containing category attributes, splitting the file into multiple parts to avoid exceeding database maximums. The file is split based on the part lists in the Enterworks.properties configuration file. Each part must include the primary key plus some attributes with a total column count of less than 1000.

com.enterworks.services.imports. SplitDynamicAttributesFromKeyValuePairFile

Reprocess import file containing category attributes in key/value pairs, combines multiple lines (if for the same target record), then splitting the file into multiple parts to avoid exceeding database maximums. The file is split based on the part lists in the configuration file. Each part must include the primary key plus some attributes with a total column count of less than 1000.

com.enterworks.services.imports. UncompressZipFile

Decompresses zip file before processing.

When configuring a Scheduled Import or Export with a pre/post-processing block,

  1. Open the Scheduled Import or Scheduled Export repository.
  2. Open the record for the Import or Export.
  3. Open the Import Details or Export Details tab, and open the Import Preprocess Options or Export PostProcess Options sub-tab.
  4. Set Preprocess File or Postprocess File to Yes.
  5. Enter the full class path for the processing block class and click the calculate button on the Pre-process Class or Post-process Class field.

  6. The define arguments window will open showing a description for what the block does along with what arguments can be set and the current values.

  7. The argument values can be changed and saved by clicking Update Attributes.

Pre/Post-Processing Block

Each processing block class must implement the processFile method. This method is called when there is an import or export file to be processed:

String processFile (String directoryName, String fileName, HashMap args, HashMap inactiveRecords, TreeMap primaryKey, and StringBuffer msgs)

Arguments:

Data Type

Argument

Description

String

directoryName

Fully-qualified path to the directory containing the file to be processed. The file to be returned must also be placed in this same directory.

String

fileName

Name of the file to be processed.

HashMap

args

Map of any pre/post processing arguments defined in the Scheduled Import/Export.

HashMap

inactiveRecords

Map containing the primary keys of any records in the repository having a Status of Inactive. This is only set for imports and only if the Inactive Records flag is set to Reactivate.

TreeMap

primaryKey

The primary key for the repository.

StringBuffer

msgs

Medium for returning error messages to be displayed with the job.

The method must return either the name of the processed file or null if the processing block failed.

If the processing block class has configurable arguments, there are three methods that must be implemented:

  • String getDescription() – returns a detailed description of what the processing block does.
  • void defineArguments() – builds the list of arguments that can be configured. Each argument is defined by calling the method:
  • void addArgument(String arg, String description) – adds an argument to the list of arguments/properties that can be set for the class in the Scheduled Import or Scheduled Export record:

Data Type

Argument

Description

String

Arg

Name of argument. This name will be used to retrieve the actual value for the argument. Each defined argument must be uniquely named

String

description

Detailed description for the argument. It should include possible values (if finite set) or range of values, default if left blank, etc.

The BaseCustomProcessFile class has a set of methods that help minimize the amount of coding required in a processing block class:

  • boolean doesFileExist(String directoryName, String fileName)
    • Returns true if the specified file in the specified directory exists.
  • void logDebug(String message)
    • Generates a message in the log file if debug logging is enabled (debugEnabled=true) in the Enterworks.properties file
  • void logReport(String message)
    • Adds a line to the import or export report file.
  • void logError(String message)
    • Adds a line to the EPX BIC log file.
  • void logError(StringBuffer msgs, String message)
    • Adds a line to the EPX BIC log file and to the Errors attribute for the Scheduled Import Job or Scheduled Export Job record.
  • void updateImportStatus(String recordsProcessed, String recordsUpdated, String recordsCreated, String recordsDeleted, String recordsWithErrors, String status, String importErrors)
    • Updates the Scheduled Import Job record with the specified details. This call should be made if the pre -processing is going to take a considerable amount of time to complete. The call should be made no more than once every several minutes.
  • void updateExportStatus(String recordsProcessed, String recordsWithErrors, String status, String exportErrors)
    • Updates the Scheduled Export Job record with the specified details. This call should be made if the post-processing is going to take a considerable amount of time to complete. The call should be made no more than once every several minutes.
  • void updateExportStatus(String recordsProcessed, String recordsWithErrors, String status, String downloadLink, String exportErrors)
    • Updates the Scheduled Export Job record with the specified details, including a URL for downloading the processed file. This call should be made after processing of the file has completed.
  • BufferedReader newInput(String directoryName, String fileName, String charSet)
    • Opens a CSV or TXT file for reading.
  • void closeInput(BufferedReader br)
    • Closes the opened CSV or TXT file.
  • PrintWriter newOutput(String directoryName, String fileName, String encoding)
    • Opens a CSV or TXT file for writing.
  • void closeOutput(PrintWriter output)
    • Closes the opened CSV or TXT file.
  • String[] simpleParseHeader(String headerLine, String delimiter)
    • Parses the header line using the designated delimiter. Delimiter is passed to the String.split() method.
  • String[] parseHeader(String headerLine, String delimiter)
    • Parses the header line using the designated delimiter. If the delimiter is a comma, then special processing is done for commas and quotes embedded in the header names.
  • HashMap parseLine(String line, String[] header, String delimiter)
    • Parses a line from the file using the defined header and delimiter. Returns a HashMap where each key matches a column name and its value is the corresponding value from the file.
  • void removeSpaces(HashMap parsedLine)
    • Removes leading and trailing white space from each parsed value.
  • void removeCharacter(HashMap parsedLine, String character)
    • Removes the designated character from each parsed value.
  • void convertToBoolean(HashMap parsedLine, String column)
    • Convers the values “Y” or “Yes” to 1 and everything else to 0 for the designated column.
  • void insertDecimal(HashMap parsedLine, String column, int decimalPosition)
    • Inserts a decimal point character in a value at the designated number of digits from the right.
  • void clearBadDate(HashMap parsedLine, String column)
    • Clears the date value if it is not 10 characters (in mm/dd/yyyy format) or is an invalid date (e.g., 00/00/0000).
  • void reactivateRecord(HashMap parsedLine, HashMap inactiveRecords, TreeMap primaryKey, String reactivateColumnName)
    • Reactivates a record that was previously inactivated but is now in the import file.
  • void outputHeaderLine(PrintWriter output, String[] columns, String delimiter)
    • Outputs the header line with each column separated using the specified delimiter.
  • void outputParsedLine(PrintWriter output, HashMap parsedLine, String[] columns, String delimiter)
    • Outputs a line using the parsed values and the designated delimiter.
  • ArrayList extractFiles(String directoryName, String fileName, String fileEncoding, StringBuffer msgs)
    • Extracts the contents of a zip file and returns a list of unzipped files.