Managing malformed input records
A malformed record is one that Spectrum Technology Platform cannot parse. When Spectrum Technology Platform encounters a malformed record, it can do one or more of these tasks:
- Terminate the job
- Continue processing
- Continue processing until a certain number of bad records are encountered
- Continue processing but write bad records to a log file (via an optional sink stage)
Note: Malformed records functionality is limited to sources configured to read from files local
to the server and that do not have sorting configured. When a source is configured
with either a remote file or with sort fields and the source encounters a malformed
record, the job will terminate regardless of the configuration for malformed
records.
To manage malformed records,
- Open the flow on the canvas.
-
Add a malformed records sink in your flow.
-
By default, processing stops at malformed records. This default behavior can be
changed in your Advanced configuration options or in Spectrum Management Console.
Regardless of your system's default behavior, you can override the default
behavior for a job by following these steps:
- Open the job in Spectrum Flow Designer.
- Within an open job, go to .
- Select either Do not terminate the job on a malformed record or select Terminate the job after encountering this many malformed records and enter the number of malformed records you will allow a job to encounter before terminating.