Open Parser

Open Parser parses your input data from many cultures of the world using a simple but powerful parsing grammar. Using this grammar, you can define a sequence of expressions that represent domain patterns for parsing your input data. Open Parser also collects statistical data and scores the parsing matches to help you determine the effectiveness of your parsing grammars.

Use Open Parser to:

Parse input data using domain-specific and culture-specific parsing grammars that you define in Domain Editor.
Parse input data using domain-independent parsing grammars that you define in Open Parser using the same simple but powerful parsing grammar available in Domain Editor.
Parse input data using domain-independent parsing grammars at runtime that you define in Dataflow Options.
Preview parsing grammars to test how sample input data parses before running the job using the target input data file.
Trace parsing grammar results to view how tokens matched or did not match the expressions you defined and to better understand the matching process.

Configuration Files

These tables describe the parameters and the values you need to specify before you run the Open Parser job.

Table 1. inputFileConfig
Parameter	Description
pb.bdq.input.type	Input file type. The values can be: `TEXT`, `ORC` or `PARQUET`.
pb.bdq.inputfile.path	The path where you have placed the input file on HDFS. For example, /user/hduser/sampledata/opennameparser/ input/OpenParser_Input.csv
textinputformat.record.delimiter	File record delimiter used in the text type input file. For example, `LINUX`, `MACINTOSH`, or `WINDOWS`
pb.bdq.inputformat.field.delimiter	Field or column delimiter used in the input file, such as comma (`,`) or tab.
pb.bdq.inputformat.text.qualifier	Text qualifiers, if any, in the columns or fields of the input file.
pb.bdq.inputformat.file.header	Comma-separated value of the headers used in the input file.
pb.bdq.inputformat.skip.firstrow	If the first row is to be skipped from processing. The values can be `True` or `False`, where `True` indicates skip.

Table 2. openParserConfig
Parameter	Description
pb.bdq.job.type	This is a constant value that defines the job. The value for this job is: `OpenParser`.
pb.bdq.job.name	Name of the job. Default is `OpenParserSample`.
pb.bdq.dnm.openparser.configuration	Json string for defining open parser configuration. It specifies details, such as the parsing grammar to be used, and the language or culture of the data you want to parse.
pb.bdq.reference.data	The path where you have placed the reference data. For example, {"referenceDataPathLocation":"LocaltoDataNodes", "dataDir":"/home/data/referenceData"}

Table 3. openParserConfigHDFSRefData(DataDownloader)
Parameter	Description
pb.bdq.job.type	This is a constant value that defines the job. The value for this job is: `OpenParser`.
pb.bdq.job.name	Name of the job. Default is `OpenParserSample`.
pb.bdq.dnm.openparser.configuration	Json string for defining open parser configuration. It specifies details, such as the parsing grammar to be used, and the language or culture of the data you want to parse.
pb.bdq.reference.data	Path of reference data on HDFS and the data downloader path. For example, {"referenceDataPathLocation":"HDFS", "dataDir":"/home/data/dm/referenceData", "dataDownloader":{"dataDownloader":"HDFS", "localFSRepository":"/local/download"}}

Table 4. mapReduceConfig
Specifies the MapReduce configuration parameters
Customize MapReduce parameters, such as mapreduce.map.memory.mb, mapreduce.reduce.memory.mb and mapreduce.map.speculative, as needed for your job. Note: Use this file only for MapReduce jobs.

Table 5. OutputFileConfig
Parameter	Description
pb.bdq.output.type	Specify if the output is in: `TEXT`, `ORC`, or `PARQUET` format.
pb.bdq.outputfile.path	The path where you want the output file to be generated on HDFS. For example, /user/hduser/sampledata/ opennameparser/output
pb.bdq.outputformat.field.delimiter	Field or column delimiter in the output file, such as comma (`,`) or tab.
pb.bdq.output.overwrite	For a `true` value, the output folder is overwritten every time job is run.
pb.bdq.outputformat.headerfile.create	Specify `true`, if the output file needs to have a header.
Properties of Parquet file
parquet.compression	The compression algorithm used to compress pages. It is one of these: `UNCOMPRESSED`, `SNAPPY`, `GZIP`, or `LZO`. Default is `UNCOMPRESSED`.
parquet.block.size	The size of a row group being buffered in memory. Larger values improve the I/O when reading but consume more memory when writing. Default size is 134217728 bytes (= 128 * 1024 * 1024)
parquet.page.size	Page constitutes block and is the smallest unit that must be read fully to access a single record. Default size is 1048576 bytes (= 1 * 1024 * 1024) Note: A very small page size results in deterioration of compression.
parquet.dictionary.page.size	Default size is 1048576 bytes (= 1 * 1024 * 1024)
parquet.enable.dictionary	The boolean value (`True` or `False`) to enable or disable dictionary encoding. Default is `True`
parquet.validation	Default boolean value is `False`.
parquet.writer.version	Specifies the version of writer. It should be `PARQUET_1_0` or `PARQUET_2_0`. Default is `PARQUET_1_0`.
parquet.writer.max-padding	Default to no padding, 0% of the row group size
parquet.page.size.check.estimate	Default boolean value is `True`
parquet.page.size.row.check.min	Default is 100
parquet.page.size.row.check.max	Default is 10000