Distributed processing takes parts of your flow and distributes the processing of
those parts to a cluster of Spectrum Technology Platform servers. For example,
your flow may perform geocoding, and you might want to distribute the geocoding
processing among several Spectrum Technology Platform nodes in a cluster to
improve performance.
Decide which stages of your flow you want to distribute, then create a
subflow containing the stages that you want to distribute.
Do not use these stages in a subflow that will be used for distributed
processing:
Sorter
Unique ID Generator
Record Joiner
Interflow Match
These sets of stages must be used together in a subflow for distributed
processing:
Matching stages (Intraflow Match and Transactional Match) and
consolidation stages (Filter, Best of Breed and Duplicate
Synchronization).
Aggregator and Splitter
Do not include other subflows within the subflow (nested subflows).
Note these exceptions if you will be performing matching operations in a
subflow used for distributed processing:
Sorting must be done in the job and not in the subflow. You must turn
sort off in the stage and put the sort at job level.
Match Analysis is not supported in a distributed subflow
Collection numbers will be reused within a microflow batch group
Using a Write Exception stage in a subflow may produce unexpected results.
Instead, you should add this stage to your flow at the job level.
Once you have created your subflow for the portion of the flow you want to
distribute, add the subflow to the parent flow and connect it to an upstream
and downstream stage. Subflows used for distributed processing may have only one
input port.
Right-click the subflow and select Options.
Select Distributed.
Enter the number of microflows to be sent to each server.
Enter the number of records that should be in each microflow batch.
Optional: (Optional) Check Group field name and select the name of
the field by which the microflow batches should be grouped.
If you provide a group field, your batch sizes could be greater than the
number you specified in the Micro flow batch size
field because a group will not be split across multiple batches. For
example, if you specify a batch size of 100, but you have 108 records within
the same group, that batch will include 108 records. Similarly, if you
specify a batch size of 100, and a new group of 28 records with the same ID
starts at record 80, you will have 108 records in that batch.
This example shows a flow where a subflow named My Distributed
Subflow has been configured to run in distributed mode:
