Entities¶
/api/repositories/{repository}/entities
Parameter | Type | Purpose |
---|---|---|
repository | string | The name of the Repository |
Count¶
-
GET
/api/repositories/
(string: repository)/entities/count
¶ Returns the number of entities in the repository.
Response Code | Reason |
---|---|
200 | No error |
500 | Server error |
The response contains the number of entities in the repository.
{
"count": "23"
}
Show All¶
-
GET
/api/repositories/
(string: repository)/entities
¶ Returns a list of all entities in a Repository
Paging is supported for this GET request using Standard URL Query String.
The underlying data rows can be returned in a JSON key/value pair by setting the external JSON format parameter. This allows access by different external tools that requires the JSON data in a key/value pair instead of standard array of row data with columns names defined separately.
GET /api/repositories/jupiter/entities HTTP/1.1
Accept: application/json
Response Code | Reason |
---|---|
200 | No error |
The response will be a Standard Response, with a list of the available entities in the Repository.
The columns
array will contain the list of columns for each data row:
Human Name | Column Name | Description |
---|---|---|
Name | _humanised_name | The name of the entity |
Ref | ENTITY_ID | Internal entity reference |
Schema Name | SCHEMA_NAME | The schema name for the entity |
in TSA or TSQ project | _is_in_project | Whether the entity is in a Quality or Time Series Analysis project |
Attributes | _display_attributes | The number of attributes, including overflow if present |
Rows | LOADED_ROWS | The number of loaded rows |
Values | LOADED_VALUES | The number of distinct values loaded (across all attributes) |
Row min len | RECORD_MIN_LEN | Minimum Row Length |
Row max len | RECORD_MAX_LEN | Maximum Row Length |
Business Rules | _business_rules | The number of business rules defined for this entity |
Enabled Business Rules | _b_rules_enabled | The number of enabled business rules |
Passing Business Rules | _b_rules_passed | The number of passing business rules |
Failing Business Rules | _b_rules_failed | The number of failing business rules |
Data Source | __DATA_SOURCE | File name of data source |
Connection Data | __CONNECTION_DATA | The connection parameters for the source of the data |
Encoding | __ENCODING | Encoding used when loading the data |
Schema | __SCHEMA | File name of Schema Source |
Permanent Keys | KEYS_PERMANENT | The number of permanent keys for this entity |
Discovered Keys | KEYS_DISCOVERED | The number of discovered keys for this entity |
Permanent Deps | DEPENDENCIES_PERMANENT | The number of permanent dependencies identified for this entity |
Discovered Deps | DEPENDENCIES_DISCOVERED | The number of discovered dependencies identified for this entity |
Permanent Joins | _joins_permanent | The number of permanent joins for this entity |
Discovered Joins | _joins_discovered | The number of discovered joins for this entity |
BR Compliance % | _b_rules_compliance | Measure of the entity compliance to the Business Rules |
Business description | BUSINESS_DESCRIPTION | Business description |
Source Type | _source_type | Source type of the loaded data |
Delimiter | FIELD_DELIMITER | Field Delimiter |
Quote char | QUOTE_CHAR | Quote character |
Entity Type | __CREATION_TYPE | Whether the entity is Virtual, Real, Ignored or Deleted |
Entity State | __STATE | The state of the entity, eg whether schema only, loaded or delete requested, etc. |
Notes | note.COUNT | The number of notes for this entity |
Loaded Date | _loaded_date | Date of Data Load |
Loaded by user | _loaded_by_user | User who loaded the data |
Each data row will contain the data for a single entity; the metadata will contain the URL to drill-down to the entity.
The standard data rows will be returned as follows:
{
"dataRow": [
"Cloudy",
"5",
"input",
"N",
"10",
"50",
"262",
"79",
"138",
"0",
"0",
"0",
"0",
"input.csv",
"Delimited",
"cp1252",
"input.csv",
"0",
"6",
"0",
"14",
"0",
"0",
"0.000",
"",
"Delimited file (with labels)",
",",
"",
"Real",
"Fully Loaded",
"",
"2014/11/07 16:25:17",
"bob"
],
"metadata": {
"url": "/api/repositories/jupiter/entities/5"
}
}
If external JSON format parameter was set then the data rows in a JSON key/value pair will be returned as follows:
{
"Name": "Cloudy",
"Ref": "5",
"Schema Name": "input",
"in TSA or TSQ project": "N",
"Attributes": "10",
"Rows": "50",
"Values": "262",
"Row min len": "79",
"Row max len": "138",
"Business Rules": "0",
"Enabled Business Rules": "0",
"Passing Business Rules": "0",
"Failing Business Rules": "0",
"Data Source": "input.csv",
"Connection Data": "Delimited",
"Encoding": "cp1252",
"Schema": "input.csv",
"Permanent Keys": "0",
"Discovered Keys": "6",
"Permanent Deps": "0",
"Discovered Deps": "14",
"Permanent Joins": "0",
"Discovered Joins": "0",
"BR Compliance %": "0.000",
"Business description": "",
"Source Type": "Delimited file (with labels)",
"Delimiter": ",",
"Quote char": "",
"Entity Type": "Real",
"Entity State": "Fully Loaded",
"Notes": "",
"Loaded Date": "2014/11/07 16:25:17",
"Loaded by user": "bob"
"metadata": {
"url": "/api/repositories/jupiter/entities/5"
}
}
If there are no entities, rows
will be an empty string.
Commands¶
-
POST
/api/repositories/
(string: repository)/entities
¶ Executes commands for the Entities in the repositories.
The request requires a Standard Request packet. Available commands are:
Command | Description |
---|---|
get | Get a filtered list of entities from the repository |
create | Create a dynamic entity in the corresponding repository |
preview | Get a preview of the entity |
rawpreview | Get a raw preview of the entity |
export | Export all or a filtered list of entities from the repository |
The response is a Standard Response; the datarows array will have an element for each row that matches the filter criteria.
Get¶
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "get",
"options": {
"keypatternlist": ["1", "5"]
}
}
Response Code | Reason |
---|---|
200 | Success |
500 | Server error |
The request packet must contain the command
and options
elements. The data
element is not required.
All fields are required.
Create¶
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "create",
"data": {
"entityName": "Delimited entity",
"entityDesc": "A Delimited entity",
"source": "client",
"connectInfo": {
"type": "delimited",
"username": "bob",
"password": "bob",
"sourceFile": "File1.txt",
"schemaFile": "File1.ddl",
"quote": "'",
"delimiter": ",",
"attr": "names",
"encoding": "cp1252",
"terminator": "crlf"
},
"sample": "SAMPLE {first 100}"
}
}
Response Code | Reason |
---|---|
204 | No error |
400 | Invalid data passed with request |
500 | Server error |
The request packet must contain the command
and data
elements. The options
element is not required, but may specify the columns to load. For example:
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "create",
"data": {
"entityName": "DBEntity",
"source": "server",
"connectInfo": {
"connectionId": "1",
"sourceFile": "input.csv",
"quote": "\"",
"delimiter": ",",
"attr": "names",
"encoding": "cp1252",
"terminator": "crlf"
}
},
"options": {
"columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number"]
}
}
Note that the columnised name for each required column must be specified, not the humanised name.
Columns may include virtual columns, which are specified using this syntax: {<ATTRIBUTE_NAME> <ATTRIBUTE_NAME> {} <ATTRIBUTE_DESCRIPTION> <ATTRIBUTE_XML_EXPRESSION> <ATTRIBUTE_SCHEMA_LENGTH>}
. The third parameter is always an empty pair of braces. The recommended value for ATTRIBUTE_SCHEMA_LENGTH
is 100. An example is shown below:
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "create",
"data": {
"entityName": "DBEntity",
"source": "server",
"connectInfo": {
"connectionId": "1",
"sourceFile": "input.csv",
"quote": "\"",
"delimiter": ",",
"attr": "names",
"encoding": "cp1252",
"terminator": "crlf"
}
},
"options": {
"columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number", "Branch", "{\"Branch plus 100\" \"Branch plus 100\" {} {This is a virtual column calculated from Branch plus 100} {<expr><text>Branch + 100</text><plus><var>Branch</var><int>100</int></plus></expr>} 100}"]
}
}
The following table shows which fields are required in data
element:
Field | Required |
---|---|
entityName | Y |
entityDesc | N |
source | Y |
connectInfo | Y |
sample | N |
source
field takes the following values:
Value | Description |
---|---|
client | entity has to be created using the data from client filesystem |
server | entity has to be created using the data from server filesystem |
sample
field denotes the filtering option to be performed on the entity. It takes the following values:
Value | Description |
---|---|
SAMPLE all | all the samples. No filtering is performed |
SAMPLE {first x} | first x records |
SAMPLE {first x skip y} | Skips first y records and returns the next x records |
SAMPLE {random x} | x% random records |
SAMPLE {random x skip y} | Skips first y records and returns x% records randomly from the records remaining |
The following table shows which fields are required in connectInfo
element for delimited connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | N | N |
quote | N | N |
delimiter | N | N |
attr | Y | Y |
encoding | N | N |
terminator | N | N |
The following table shows which fields are required in connectInfo
element for cobol connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | Y | Y |
endian | N | N |
terminator | N | N |
align | N | N |
redefines | N | N |
upacked | N | N |
encoding | N | N |
national | N | N |
The following table shows which fields are required in connectInfo
element for trillium connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | Y | Y |
The following table shows which fields are required in connectInfo
element for odbc, generic, oracle and db2 connections:
Field | Required |
---|---|
connectionId | Y |
username | N |
password | N |
sourceFile | Y |
sqlwhere | N |
type
denotes the type of entity to be created. This field is only applicable for entities created from client datasources.
Type supports cobol
, delimited
, hdfs_delimited
, ODBC
, and trillium
values. Only TSS Servers installed in
a Hadoop environment(EdgeNode/Cluster mode) support the hdfs_delimited` type.
connectionId
denotes the loader connection ID of the Loader Connection. That field is only applicable to entities created from loader connections.
quote
field denotes the quote character used in the data. It can have the following values - NUL, `, ", ', \, /
delimiter
field denotes the delimiter character used in the data. It can have the following values:
NUL, ,, |, TAB, `, /, !, \", £, $, ^, &, *, (, ), -, _, +, =, {, }, ;, :, \', @, #, ~, <, >, ?, \, ¬
attr
field denotes the attribute information of the data used to create entity:
Value | Description |
---|---|
none | No information |
names | Names on the first line |
ddl | ddl file is supplied |
terminator
field indicates how records in the data file are terminated.
The options are lf
(linefeed), cr
(carriage return), crlf
(linefeed and carriage return), and none
.
encoding
field indicates the character encoding used by the data file. The following are the supported encodings:
cp1252, ascii, big5, cp1250, cp1251, cp1253, cp1254, cp1255, cp1256, cp1257, cp1258, cp437, cp737, cp775, cp850, cp852, cp855, cp857, cp860, cp861, cp862, cp863, cp864, cp865, cp866, cp869, cp874, cp932, cp936, cp949, cp950, dingbats, ebcdic, euc-cn, euc-jp, euc-kr, gb12345, gb1988, gb2312, gb2312-raw, identity, iso2022, iso2022-jp, iso2022-kr, iso8859-1, iso8859-10, iso8859-13, iso8859-14, iso8859-15, iso8859-16, iso8859-2, iso8859-3 iso8859-4, iso8859-5, iso8859-6, iso8859-7, iso8859-8, iso8859-9, jis0201, jis0208, jis0212, koi8-r, koi8-u, ksc5601, macCentEuro, macCroatian macCyrillic, macDingbats, macGreek, macIceland, macJapan, macRoman, macRomania, macThai, macTurkish, macUkraine, shiftjis, symbol, tis-620, unicode, utf-8
national
field denotes the national character encoding used by the data file. It takes the same values as encoding
field.
endian
indicates the byte order of the data. Only big
(for Big Endian) and little
(for Little Endian) are supported.
align
field indicates one- or two-byte data alignment. The options are one
and two
.
redefines
field indicates how to manage REDEFINES clauses in a copybook. The options are all
and first
.
Set it to all
to account for all REDEFINES clauses in a copybook. Set it to first
to ignore all REDEFINES clauses in the copybook.
upacked
field indicates whether or not unsigned comp-3 fields in the copybook should be treated as comp-6. The options are 0
(for false) and 1
(for true).
sqlWhere
field contains a SQL where clause for odbc connections. It is blank if no sqlWhere
value is provided, and accepts conditions that filter responses, such as "Cpac < 101000000"
to limit a query to records where the Cpac column value is less than 101000000.</p>
Please refer to the Repository Admin guide for further details about the connection parameters.
If you are using data files from the local filesystem, you must first call Data Upload to upload the file to the server. See the Data Upload for more details.
If the command is successful, the response will return a url link to the entity, along with the newly created entity Id as follows:
{
"url": "/api/repositories/jupiter/entities/17",
"entityId": "17"
}
Preview¶
Returns a preview of the data for any Loader Connection type, taking into account the schema sent as part of connectInfo object. This can be used before creating an entity to see the entity structure.
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "preview",
"data": {
"source": "client",
"connectInfo": {
"type": "delimited",
"username": "bob",
"password": "bob",
"sourceFile": "File1.txt",
"schemaFile": "File1.ddl",
"quote": "'",
"delimiter": ",",
"attr": "names",
"encoding": "cp1252",
"terminator": "crlf"
}
}
}
Response Code | Reason |
---|---|
204 | No error |
400 | Invalid data passed with request |
500 | Server error |
The request packet must contain the command
and data
elements. The options
element is not required, but may specify the columns to preview. For example:
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "preview",
"data": {
"entityName": "DBEntity",
"source": "server",
"connectInfo": {
"connectionId": "1",
"sourceFile": "input.csv",
"quote": "\"",
"delimiter": ",",
"attr": "names",
"encoding": "cp1252",
"terminator": "crlf"
}
},
"options": {
"columns": ["Line_01", "Line_02", "Line_03", "Line_04", "Vin_number"]
}
}
Note that the columnised name must be specified for the required columns, not the humanised name.
The following table shows which fields are required in data
element:
Field | Required |
---|---|
source | Y |
connectInfo | Y |
source
field takes the following values:
Value | Description |
---|---|
client | entity preview using the data from client/local filesystem |
server | entity preview using the data from server filesystem |
Refer to entity create connectInfo parameters for the connectInfo
parameters. connectInfo
parameters are same for entity create and preview command.
Raw Preview¶
This will return a raw preview of the data without any schema. This can be used before creating an entity to see the contents of the data source.
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "rawpreview",
"data": {
"source": "client",
"connectInfo": {
"type": "delimited",
"sourceFile": "File1.txt"
}
}
}
Response Code | Reason |
---|---|
204 | No error |
400 | Invalid data passed with request |
500 | Server error |
The request packet must contain the command
and data
elements. The options
element is not required.
The following table shows which fields are required in data
element:
Field | Required |
---|---|
source | Y |
connectInfo | Y |
source
field takes the following values:
Value | Description |
---|---|
client | Raw preview of the data from client filesystem |
server | Raw preview of the data from server filesystem |
The following table shows which fields are required in connectInfo
element for delimited connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | N | N |
encoding | N | N |
The following table shows which fields are required in connectInfo
element for cobol connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | Y | Y |
The following table shows which fields are required in connectInfo
element for trillium connection:
Field | Required for client source | Required for server source |
---|---|---|
type | Y | N |
connectionId | N | Y |
username | N | N |
password | N | N |
sourceFile | Y | Y |
schemaFile | Y | Y |
The following table shows which fields are required in connectInfo
element for odbc, generic, oracle and db2 connections:
Field | Required |
---|---|
connectionId | Y |
username | N |
password | N |
sourceFile | Y |
sqlwhere | N |
type
denotes the type of entity to be created. That field is applicable only for entities created from client datasources.
delimited
, cobol
and trillium
are supported at the moment.
connectionId
denotes the loader connection ID of the Loader Connection. That field is only applicable to entities created from loader connections.
Please refer to the Repository Admin guide for further details about the connection parameters.
If you are using data files from the local filesystem, you must first call Data Upload to upload the file to the server. See the Data Upload for more details.
Export¶
Some or all of the rows may be exported as a scheduled task.
POST /api/repositories/jupiter/entities HTTP/1.1
Content-Type: application/json
{
"command": "export",
"data": {
"filename": "filename.csv",
"filetype": "csv",
"encoding": "utf-8"
},
"options": {
"columns": ["_humanised_name", "ENTITY_ID"],
"where": "Ref = 1"
}
}
Response Code | Reason |
---|---|
200 | No error |
400 | Invalid data passed with request |
500 | Server error |
The request packet must contain the command
and data
elements. The options
element is not required, but may specify the columns to export, and/or the rows to filter.
The following table shows which fields are required in the data
element:
Field | Required |
---|---|
filename | Y |
filetype | N |
encoding | N |
If filetype
is not specified, it defaults to csv. If encoding
is not specified, it defaults to the server native encoding.
The response will contain the URL of the scheduled task and the link to download the exported file.
{
"job": "/api/repositories/jupiter/scheduler/1",
"file": "/api/repositories/jupiter/download/filename.csv"
}