Entities ======== .. code-block:: http /api/repositories/{repository}/entities .. csv-table:: :header: Parameter, Type, Purpose :widths: 20, 20, 60 :stub-columns: 1 repository, string, The name of the :doc:`repository` .. _e_count: Count ~~~~~~~~ .. http:get:: /api/repositories/(string:repository)/entities/count :synopsis: Get the number of entities in the repository Returns the number of entities in the repository. .. csv-table:: :header: Response Code, Reason :widths: 25, 75 200, No error 500, Server error The response contains the number of entities in the repository. .. code-block:: json { "count": "23" } .. _e_show_all: Show All ~~~~~~~~ .. http:get:: /api/repositories/(string:repository)/entities :synopsis: List all entities in a repository Returns a list of all entities in a :doc:`repository` :ref:`Paging ` is supported for this GET request using :doc:`standard-url-query-string`. The underlying data rows can be returned in a :ref:`JSON key/value pair ` by setting the :ref:`external JSON format ` parameter. This allows access by different external tools that requires the JSON data in a key/value pair instead of standard array of row data with columns names defined separately. .. code-block:: http GET /api/repositories/jupiter/entities HTTP/1.1 Accept: application/json .. csv-table:: :header: Response Code, Reason :widths: 25, 75 200, No error The response will be a :doc:`standard-response`, with a list of the available entities in the :doc:`repository`. The ``columns`` array will contain the list of columns for each data row: .. csv-table:: :header: Human Name, Column Name, Description :widths: 25, 25, 50 "Name", "_humanised_name", "The name of the entity" "Ref", "ENTITY_ID", "Internal entity reference" "Schema Name", "SCHEMA_NAME", "The schema name for the entity" "in TSA or TSQ project", "_is_in_project", "Whether the entity is in a Quality or Time Series Analysis project" "Attributes", "_display_attributes", "The number of attributes, including overflow if present" "Rows", "LOADED_ROWS", "The number of loaded rows" "Values", "LOADED_VALUES", "The number of distinct values loaded (across all attributes)" "Row min len", "RECORD_MIN_LEN", "Minimum Row Length" "Row max len", "RECORD_MAX_LEN", "Maximum Row Length" "Business Rules", "_business_rules", "The number of business rules defined for this entity" "Enabled Business Rules", "_b_rules_enabled", "The number of enabled business rules" "Passing Business Rules", "_b_rules_passed", "The number of passing business rules" "Failing Business Rules", "_b_rules_failed", "The number of failing business rules" "Data Source", "__DATA_SOURCE", "File name of data source" "Connection Data", "__CONNECTION_DATA", "The connection parameters for the source of the data" "Encoding", "__ENCODING", "Encoding used when loading the data" "Schema", "__SCHEMA", "File name of Schema Source" "Permanent Keys", "KEYS_PERMANENT", "The number of permanent keys for this entity" "Discovered Keys", "KEYS_DISCOVERED", "The number of discovered keys for this entity" "Permanent Deps", "DEPENDENCIES_PERMANENT", "The number of permanent dependencies identified for this entity" "Discovered Deps", "DEPENDENCIES_DISCOVERED", "The number of discovered dependencies identified for this entity" "Permanent Joins", "_joins_permanent", "The number of permanent joins for this entity" "Discovered Joins", "_joins_discovered", "The number of discovered joins for this entity" "BR Compliance %", "_b_rules_compliance", "Measure of the entity compliance to the Business Rules" "Business description", "BUSINESS_DESCRIPTION", "Business description" "Source Type", "_source_type", "Source type of the loaded data" "Delimiter", "FIELD_DELIMITER", "Field Delimiter" "Quote char", "QUOTE_CHAR", "Quote character" "Entity Type", "__CREATION_TYPE", "Whether the entity is Virtual, Real, Ignored or Deleted" "Entity State", "__STATE", "The state of the entity, eg whether schema only, loaded or delete requested, etc." "Notes", "note.COUNT", "The number of notes for this entity" "Loaded Date", "_loaded_date", "Date of Data Load" "Loaded by user", "_loaded_by_user", "User who loaded the data" Each data row will contain the data for a single entity; the metadata will contain the URL to drill-down to the entity. The standard data rows will be returned as follows: .. code-block:: json { "dataRow": [ "Cloudy", "5", "input", "N", "10", "50", "262", "79", "138", "0", "0", "0", "0", "input.csv", "Delimited", "cp1252", "input.csv", "0", "6", "0", "14", "0", "0", "0.000", "", "Delimited file (with labels)", ",", "", "Real", "Fully Loaded", "", "2014/11/07 16:25:17", "bob" ], "metadata": { "url": "/api/repositories/jupiter/entities/5" } } If :ref:`external JSON format ` parameter was set then the data rows in a :ref:`JSON key/value pair ` will be returned as follows: .. code-block:: json { "Name": "Cloudy", "Ref": "5", "Schema Name": "input", "in TSA or TSQ project": "N", "Attributes": "10", "Rows": "50", "Values": "262", "Row min len": "79", "Row max len": "138", "Business Rules": "0", "Enabled Business Rules": "0", "Passing Business Rules": "0", "Failing Business Rules": "0", "Data Source": "input.csv", "Connection Data": "Delimited", "Encoding": "cp1252", "Schema": "input.csv", "Permanent Keys": "0", "Discovered Keys": "6", "Permanent Deps": "0", "Discovered Deps": "14", "Permanent Joins": "0", "Discovered Joins": "0", "BR Compliance %": "0.000", "Business description": "", "Source Type": "Delimited file (with labels)", "Delimiter": ",", "Quote char": "", "Entity Type": "Real", "Entity State": "Fully Loaded", "Notes": "", "Loaded Date": "2014/11/07 16:25:17", "Loaded by user": "bob" "metadata": { "url": "/api/repositories/jupiter/entities/5" } } If there are no entities, ``rows`` will be an empty string. Commands ~~~~~~~~ .. http:post:: /api/repositories/(string:repository)/entities :synopsis: Execute commands against entities Executes commands for the Entities in the repositories. The request requires a :doc:`standard-request` packet. Available commands are: .. csv-table:: :header: Command, Description :widths: 30, 70 get, Get a filtered list of entities from the repository create, Create a dynamic entity in the corresponding repository preview, Get a preview of the entity rawpreview, Get a raw preview of the entity export, Export all or a filtered list of entities from the repository The response is a :doc:`standard-response`; the datarows array will have an element for each row that matches the filter criteria. .. _e_show_some: Get --- .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "get", "options": { "keypatternlist": ["1", "5"] } } .. csv-table:: :header: Response Code, Reason :widths: 25, 75 200, Success 500, Server error The request packet must contain the ``command`` and ``options`` elements. The ``data`` element is not required. All fields are required. .. _e_create: Create ------ .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "create", "data": { "entityName": "Delimited entity", "entityDesc": "A Delimited entity", "source": "client", "connectInfo": { "type": "delimited", "username": "bob", "password": "bob", "sourceFile": "File1.txt", "schemaFile": "File1.ddl", "quote": "'", "delimiter": ",", "attr": "names", "encoding": "cp1252", "terminator": "crlf" }, "sample": "SAMPLE {first 100}" } } .. csv-table:: :header: Response Code, Reason :widths: 25, 75 204, No error 400, Invalid data passed with request 500, Server error The request packet must contain the ``command`` and ``data`` elements. The ``options`` element is not required, but may specify the columns to load. For example: .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "create", "data": { "entityName": "DBEntity", "source": "server", "connectInfo": { "connectionId": "1", "sourceFile": "input.csv", "quote": "\"", "delimiter": ",", "attr": "names", "encoding": "cp1252", "terminator": "crlf" } }, "options": { "columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number"] } } Note that the columnised name for each required column must be specified, not the humanised name. Columns may include virtual columns, which are specified using this syntax: ``{ {} }``. The third parameter is always an empty pair of braces. The recommended value for ``ATTRIBUTE_SCHEMA_LENGTH`` is 100. An example is shown below: .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "create", "data": { "entityName": "DBEntity", "source": "server", "connectInfo": { "connectionId": "1", "sourceFile": "input.csv", "quote": "\"", "delimiter": ",", "attr": "names", "encoding": "cp1252", "terminator": "crlf" } }, "options": { "columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number", "Branch", "{\"Branch plus 100\" \"Branch plus 100\" {} {This is a virtual column calculated from Branch plus 100} {Branch + 100Branch100} 100}"] } } The following table shows which fields are required in ``data`` element: .. csv-table:: :header: Field, Required :widths: 20, 10 entityName, Y entityDesc, N source, Y connectInfo, Y sample, N ``source`` field takes the following values: .. csv-table:: :header: Value, Description :widths: 20, 75 client, entity has to be created using the data from client filesystem server, entity has to be created using the data from server filesystem ``sample`` field denotes the filtering option to be performed on the entity. It takes the following values: .. csv-table:: :header: Value, Description :widths: 25, 75 SAMPLE all, all the samples. No filtering is performed SAMPLE {first x}, first x records SAMPLE {first x skip y}, Skips first y records and returns the next x records SAMPLE {random x}, x% random records SAMPLE {random x skip y}, Skips first y records and returns x% records randomly from the records remaining .. _e_create_parameters: The following table shows which fields are required in ``connectInfo`` element for `delimited` connection: .. csv-table:: :header: Field, Required for client source, Required for server source :widths: 25, 47, 45 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, N, N quote, N, N delimiter, N, N attr, Y, Y encoding, N, N terminator, N, N The following table shows which fields are required in ``connectInfo`` element for `cobol` connection: .. csv-table:: :header: Field, Required for client source, Required for server source :widths: 30, 53, 50 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, Y, Y endian, N, N terminator, N, N align, N, N redefines, N, N upacked, N, N encoding, N, N national, N, N The following table shows which fields are required in ``connectInfo`` element for `trillium` connection: .. csv-table:: :header: Field, Required for `client` source, Required for `server` source :widths: 30, 50, 50 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, Y, Y The following table shows which fields are required in ``connectInfo`` element for `odbc`, `generic`, `oracle` and `db2` connections: .. csv-table:: :header: Field, Required :widths: 15, 10 connectionId, Y username, N password, N sourceFile, Y sqlwhere, N ``type`` denotes the type of entity to be created. This field is only applicable for entities created from client datasources. Type supports ``cobol``, ``delimited``, ``hdfs_delimited``, ``ODBC``, and ``trillium`` values. Only TSS Servers installed in a Hadoop environment(EdgeNode/Cluster mode) support the `hdfs_delimited`` type. ``connectionId`` denotes the loader connection ID of the :doc:`loaderconnection`. That field is only applicable to entities created from :doc:`loader connections `. ``quote`` field denotes the quote character used in the data. It can have the following values - ``NUL, `, ", ', \, /`` ``delimiter`` field denotes the delimiter character used in the data. It can have the following values: ``NUL, ,, |, TAB, `, /, !, \", £, $, ^, &, *, (, ), -, _, +, =, {, }, ;, :, \', @, #, ~, <, >, ?, \, ¬`` ``attr`` field denotes the attribute information of the data used to create entity: .. csv-table:: :header: Value, Description :widths: 55, 75 none, No information names, Names on the first line ddl, ddl file is supplied ``terminator`` field indicates how records in the data file are terminated. The options are ``lf`` (linefeed), ``cr`` (carriage return), ``crlf`` (linefeed and carriage return), and ``none``. ``encoding`` field indicates the character encoding used by the data file. The following are the supported encodings: ``cp1252, ascii, big5, cp1250, cp1251, cp1253, cp1254, cp1255, cp1256, cp1257, cp1258, cp437, cp737, cp775, cp850, cp852, cp855, cp857, cp860, cp861, cp862, cp863, cp864, cp865, cp866, cp869, cp874, cp932, cp936, cp949, cp950, dingbats, ebcdic, euc-cn, euc-jp, euc-kr, gb12345, gb1988, gb2312, gb2312-raw, identity, iso2022, iso2022-jp, iso2022-kr, iso8859-1, iso8859-10, iso8859-13, iso8859-14, iso8859-15, iso8859-16, iso8859-2, iso8859-3 iso8859-4, iso8859-5, iso8859-6, iso8859-7, iso8859-8, iso8859-9, jis0201, jis0208, jis0212, koi8-r, koi8-u, ksc5601, macCentEuro, macCroatian macCyrillic, macDingbats, macGreek, macIceland, macJapan, macRoman, macRomania, macThai, macTurkish, macUkraine, shiftjis, symbol, tis-620, unicode, utf-8`` ``national`` field denotes the national character encoding used by the data file. It takes the same values as ``encoding`` field. ``endian`` indicates the byte order of the data. Only ``big`` (for Big Endian) and ``little`` (for Little Endian) are supported. ``align`` field indicates one- or two-byte data alignment. The options are ``one`` and ``two``. ``redefines`` field indicates how to manage REDEFINES clauses in a copybook. The options are ``all`` and ``first``. Set it to ``all`` to account for all REDEFINES clauses in a copybook. Set it to ``first`` to ignore all REDEFINES clauses in the copybook. ``upacked`` field indicates whether or not unsigned comp-3 fields in the copybook should be treated as comp-6. The options are ``0`` (for false) and ``1`` (for true). ``sqlWhere`` field contains a SQL where clause for odbc connections. It is blank if no ``sqlWhere`` value is provided, and accepts conditions that filter responses, such as ``"Cpac < 101000000"`` to limit a query to records where the Cpac column value is less than 101000000.

Please refer to the Repository Admin guide for further details about the connection parameters. If you are using data files from the local filesystem, you must first call :doc:`uploaddata` to upload the file to the server. See the :doc:`uploaddata` for more details. If the command is successful, the response will return a url link to the entity, along with the newly created entity Id as follows: .. code-block:: json { "url": "/api/repositories/jupiter/entities/17", "entityId": "17" } .. _e_preview: Preview ------- Returns a preview of the data for any Loader Connection type, taking into account the schema sent as part of connectInfo object. This can be used before creating an entity to see the entity structure. .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "preview", "data": { "source": "client", "connectInfo": { "type": "delimited", "username": "bob", "password": "bob", "sourceFile": "File1.txt", "schemaFile": "File1.ddl", "quote": "'", "delimiter": ",", "attr": "names", "encoding": "cp1252", "terminator": "crlf" } } } .. csv-table:: :header: Response Code, Reason :widths: 25, 75 204, No error 400, Invalid data passed with request 500, Server error The request packet must contain the ``command`` and ``data`` elements. The ``options`` element is not required, but may specify the columns to preview. For example: .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "preview", "data": { "entityName": "DBEntity", "source": "server", "connectInfo": { "connectionId": "1", "sourceFile": "input.csv", "quote": "\"", "delimiter": ",", "attr": "names", "encoding": "cp1252", "terminator": "crlf" } }, "options": { "columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number"] } } Note that the columnised name must be specified for the required columns, not the humanised name. The following table shows which fields are required in ``data`` element: .. csv-table:: :header: Field, Required :widths: 20, 10 source, Y connectInfo, Y ``source`` field takes the following values: .. csv-table:: :header: Value, Description :widths: 20, 75 client, entity preview using the data from client/local filesystem server, entity preview using the data from server filesystem Refer to :ref:`entity create connectInfo parameters ` for the ``connectInfo`` parameters. ``connectInfo`` parameters are same for entity create and preview command. .. _e_rawpreview: Raw Preview ----------- This will return a raw preview of the data without any schema. This can be used before creating an entity to see the contents of the data source. .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "rawpreview", "data": { "source": "client", "connectInfo": { "type": "delimited", "sourceFile": "File1.txt" } } } .. csv-table:: :header: Response Code, Reason :widths: 25, 75 204, No error 400, Invalid data passed with request 500, Server error The request packet must contain the ``command`` and ``data`` elements. The ``options`` element is not required. The following table shows which fields are required in ``data`` element: .. csv-table:: :header: Field, Required :widths: 20, 10 source, Y connectInfo, Y ``source`` field takes the following values: .. csv-table:: :header: Value, Description :widths: 20, 75 client, Raw preview of the data from client filesystem server, Raw preview of the data from server filesystem The following table shows which fields are required in ``connectInfo`` element for `delimited` connection: .. csv-table:: :header: Field, Required for client source, Required for server source :widths: 25, 47, 45 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, N, N encoding, N, N The following table shows which fields are required in ``connectInfo`` element for `cobol` connection: .. csv-table:: :header: Field, Required for client source, Required for server source :widths: 30, 53, 50 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, Y, Y The following table shows which fields are required in ``connectInfo`` element for `trillium` connection: .. csv-table:: :header: Field, Required for `client` source, Required for `server` source :widths: 30, 50, 50 type, Y, N connectionId, N, Y username, N, N password, N, N sourceFile, Y, Y schemaFile, Y, Y The following table shows which fields are required in ``connectInfo`` element for `odbc`, `generic`, `oracle` and `db2` connections: .. csv-table:: :header: Field, Required :widths: 15, 10 connectionId, Y username, N password, N sourceFile, Y sqlwhere, N ``type`` denotes the type of entity to be created. That field is applicable only for entities created from client datasources. ``delimited``, ``cobol`` and ``trillium`` are supported at the moment. ``connectionId`` denotes the loader connection ID of the :doc:`loaderconnection`. That field is only applicable to entities created from :doc:`loader connections `. Please refer to the Repository Admin guide for further details about the connection parameters. If you are using data files from the local filesystem, you must first call :doc:`uploaddata` to upload the file to the server. See the :doc:`uploaddata` for more details. .. _e_export: Export ------- Some or all of the rows may be exported as a scheduled task. .. code-block:: http POST /api/repositories/jupiter/entities HTTP/1.1 Content-Type: application/json { "command": "export", "data": { "filename": "filename.csv", "filetype": "csv", "encoding": "utf-8" }, "options": { "columns": ["_humanised_name", "ENTITY_ID"], "where": "Ref = 1" } } .. csv-table:: :header: Response Code, Reason :widths: 25, 75 200, No error 400, Invalid data passed with request 500, Server error The request packet must contain the ``command`` and ``data`` elements. The ``options`` element is not required, but may specify the columns to export, and/or the rows to filter. The following table shows which fields are required in the ``data`` element: .. csv-table:: :header: Field, Required :widths: 20, 10 filename, Y filetype, N encoding, N If ``filetype`` is not specified, it defaults to csv. If ``encoding`` is not specified, it defaults to the server native encoding. The response will contain the URL of the scheduled task and the link to download the exported file. .. code-block:: json { "job": "/api/repositories/jupiter/scheduler/1", "file": "/api/repositories/jupiter/download/filename.csv" }