Write to File

Write to File writes dataflow output to a flat file.
Tip: You can copy your source and paste it as the sink into your dataflow to quickly set up the file and use the same fields as you defined in your source.
Prerequisite: To write a file to any of the file system connection types, such as FTP, Cloud, Amazon AWS S3, and HDFS, perform these steps:
  1. Create a connection to these file servers using Spectrum Management Console or Discovery. For details, see section Defining Connections.
  2. Select the required file path using the File name field in File Properties tab (described below).

File Properties Tab

Field Name

Description
Server name Indicates whether the file you select as input is located on the computer running Spectrum Enterprise Designer or on the Spectrum Technology Platform server. If you select a file on the local computer, the server name will be My Computer. If you select a file on the server the server name will be Spectrum Technology Platform.

File name

Specifies the path to the file. Click the ellipses button (...) to locate the file you want.

While writing a file to an HDFS file server, these compression formats are supported:
  1. GZIP (.gz)
  2. BZIP2 (.bz2)
Note: Include the appropriate extension in the file name, to indicate the desired compression format to be used while writing the file.
Attention: If the Spectrum Technology Platform server is running on Linux, remember that file names and paths on these platforms are case sensitive.
Record type The format of the records in the file. Select one of:
Line Sequential
A text file in which records are separated by an end-of-line (EOL) character such as a carriage return or line feed (CR or LF) and each field has a fixed starting and ending character position.
Fixed Width
A text file in which each record is a specific number of characters in length and each field has a fixed starting and ending character position.
Delimited
A text file in which records are separated by an end-of-line (EOL) character such as a carriage return or line feed (CR or LF) and each field is separated by a designated character such as a comma.
Character encoding The text file encoding. For more information, see Character Encodings.
Field separator Specifies the character used to separate fields in a delimited file. For example, this record uses a pipe (|) as a field separator:
7200 13TH ST|MIAMI|FL|33144

These characters available to define as field separators are:

  • Space
  • Tab
  • Comma
  • Period
  • Semicolon
  • Pipe

If the file uses a different character as a field separator, click the ellipses button to select another character as a delimiter.
Text qualifier

The character used to surround text values in a delimited file.

For example, this record uses double quotes (") as a text qualifier.

"7200 13TH ST"|"MIAMI"|"FL"|"33144"

The characters available to define as text qualifiers are:

  • Single quote (')
  • Double quote (")

If the file uses a different text qualifier, click the ellipses button to select another character as a text qualifier.
Record separator

Specifies the character used to separate records in line a sequential or delimited file. This field is not available if you check the Use default EOL check box.

The record separator settings available are:

Linux (U+000A)
A line feed character separates the records. This is the standard record separator for Linux systems.
Macintosh (U+000D)
A carriage return character separates the records. This is the standard record separator for Macintosh systems.
Windows (U+000D U+000A)
A carriage return followed by a line feed separates the records. This is the standard record separator for Windows systems.

If your file uses a different record separator, click the ellipses button to select another character as a record separator.
Use default EOL

Specifies that the file's record separator is the default end of line (EOL) character used on the operating system on which the Spectrum Technology Platform server is running.

Do not select this option if the file uses an EOL character that is different from the default EOL character used on the server's operating system. For example, if the file uses a Windows EOL but the server is running on Linux, do not check this option. Instead, select the Windows option in the Record separator field.
Record length

For fixed width files, specifies the exact number of characters in each record.

For line sequential files, specifies the length, in characters, of the longest record in the file.
Trim trailing spaces For line sequential files, select this checkbox to remove trailing spaces from the end of each record.
First row is header record

Specifies whether the first record in a delimited file contains header information and not data.

For example, this file snippet shows a header row in the first record.

"AddressLine1"|"City"|"StateProvince"|"PostalCode"
"7200 13TH ST"|"MIAMI"|"FL"|"33144"
"One Global View"|"Troy"|"NY"|12180
Treat records with fewer fields than defined as malformed

Delimited file records containing fewer fields than are defined on the Fields tab will be treated as malformed.
Import

Imports the file layout definition, encoding setting, and sort options from a settings file. The settings file is created by exporting settings from another Read from File or Write to File stage that used the same input file or a file that has the same layout as the file you are working with.
Export

Saves the file layout definition, encoding setting, and sort options to a settings file. You can then import these settings into other Read from File or Write to File stages that use the same input file or a file that has the same traits as the file you are working with now. You can also use the settings file with job executor to specify file settings at runtime.

For information about the settings file, see The File Definition Settings File.

Fields Tab

The Fields tab defines the names, positions, and, for fixed width and line sequential files, lengths of fields in the file. For more information, see these topics:

Sort Fields Tab

The Sort Fields tab defines fields by which to sort the output records before they are written to the output file. Sorting is optional. For more information, see Sorting Output Records.

Runtime Tab

Option Name

Description

File name

This displays the file defined on the File Properties tab.
Generate multiple files Select this option to write records to different files instead of writing all records to one file. The file to which each record is written is specified in the record itself. Each record must contain a field that specifies either a file name or the full file path of the file to which you want the record written. For example, if you want to send the stock prices of different companies (of various groups) to all the clients separately, this feature writes the stock prices of different companies into separate files that may be sent to each of the clients, if you so wish. If you enable the Generate multiple file option you must specify an output file on either the Spectrum Technology Platform server or on an FTP server. If you want to write data to a file on an FTP server you must define a connection to the file server using Spectrum Management Console.
Note: The records in the column you select in the File path field must be in sorted order. Use this feature when record contains either a file name or the full file path of the file.
File path field Selects the field that contains the path (either a file name or the full file path) of the file to which you want to write the record. This field is only enabled if you select Generate multiple files.

Write Mode

Specifies whether to add the dataflow's output to the end of the file or to delete the existing data in the file before writing the output:

Overwrite
Replaces the existing data in the output file each time the dataflow runs.
Append
Adds the dataflow's output to the end of the file without erasing the file's existing data.