Designing a Dataflow for Real-Time Revalidation

If you are using exception management in your dataflow, you can use the revalidation feature to rerun exception records through the validation process after they have been corrected in the Business Steward Portal. This enables you to determine if the change you made causes the record to process successfully in a real-time manner; you don't need to wait until the Read Exceptions batch job runs again to see the result.

The basic building blocks of a revalidation environment are:
  • A job or a service that reuses or contains an exposed subflow. It must also contain an input source, the subflow stage that processes the input, a Write Exceptions stage, and an output sink for successfully processed records.
  • An exposed subflow containing an Exception Monitor stage that points to a revalidation service and is configured for revalidation, including designating whether revalidated records should be reprocessed or approved.
  • An exposed service that also reuses or contains the exposed subflow. It processes records that were edited, saved, and sent for revalidation in the Business Steward Portal.

Here is an example scenario that helps illustrate a revalidation implementation:

In this example, there are three dataflows: a job, a subflow, and a service. The job runs input data through the subflow. The subflow contains an Exception Monitor stage, which determines if a record should be routed for manual review. Continuing with our example, that means any records with no data in the PostalCode field would be considered an exception and would be routed to the Write Exceptions stage; these exceptions are what appears in the Business Steward Portal. Records with anything else in that field would be routed to the Write to File stage.
Note: If your dataflow is also being configured to use best of breed functionality, you will need to manually add and expose the CollectionRecordType field in the revalidation Exception Monitor stage/subflow and the service itself. See Write Exceptions options and Creating a Best of Breed Record for more information on best of breed functionality.
The exception revalidation service that you designated when configuring the Exception Monitor stage is called when you edit one or more exception records in the Business Steward Portal Exception Editor and click Revalidate and Save. Like the job, the service contains the exception monitor subflow that uses the same business logic to reprocess the record(s). If the records fail one or more conditions set in the Exception Monitor stage, the exceptions will be updated in the repository. If the records pass the conditions set in the Exception Monitor stage, one of two actions will occur, depending on the selection made in the "Action after revalidation" field:
  • Reprocess records—Records will be deleted from the repository and reprocessed.
  • Approve records—Records will be marked as approved and sent back to the repository.
Follow these steps to create and use a real-time revalidation scenario:
  1. Open or create a job or service dataflow that contains an Exception Monitor stage, an input source (such as a Read from File or Input stage), an output sink (such as a Write to File or Output stage), and a Write Exceptions stage.
  2. Convert the Exception Monitor stage to a subflow and map the input and output fields to match those in the initial dataflow. Be sure to include the ExceptionMetadata field for the input source as well as the output stage that populates the Write Exceptions stage in the job. Expose the subflow so it can be used by the job and service.
  3. Create a service that contains an Input stage, the subflow you created in step 2, an Output stage, and an output sink (such as a Write to File or Write to DB stage). Map the input and output fields to match those in the initial dataflow; be sure to include the ExceptionMetadata field for the Input stage as well as the Output stage. Expose the service so it can be used by the subflow.
  4. Return to the subflow and open the Configuration tab of the Exception Monitor stage. Select the revalidation service you created in step 3 and specify which action to take after revalidation. Save and expose the subflow again.
  5. Return to the service, where a message will appear, notifying you of changes to the subflow and saying that the service will be refreshed. Click OK, then save and expose the service again.
  6. Return to the initial job or service, where a message will appear, notifying you of changes to the subflow and saying that the dataflow will be refreshed. Click OK, then save the dataflow.
  7. Run the job.
    Note: Even if you have run the initial job or service before, you must run it again after creating the revalidation scenario to populate the repository with records that are eligible for revalidation. You can identify whether records in the Exception Editor are eligible for revalidation because the "Revalidate & Save" button will be active for those records.