The Upload Process and Limitations

After choosing files and clicking the upload button, the process has three steps. (1) The file(s) are compressed to a ZIP file on the client browser, (2) they are then uploaded to the Spectrum Spatial Analyst Server, and (3) they are passed to the Spectrum server for processing.

Each of the three steps has its performance characteristics and some limitations, as described below:

  1. Client-side file compression: The chosen files compress into a ZIP file, which happens on the client machine for the Spectrum Spatial Analyst user. TAB, Shape, CSV, XLSX, XLS, and GeoJSON are all zipped. Data with many columns will compress to a smaller file than those with fewer columns because data compresses better than geometry.
    Note: The ZIP file created in the client browser must be less than 512 MB. Spectrum Spatial Analyst will not upload larger files.
  2. Uploading: The ZIP file is uploaded to the server using a multipart upload REST API request. The performance of this step is entirely dependent on the size of the file and the upload bandwidth of the user’s connection to the Spectrum server. If the user is remote, then their bandwidth may be lower than when the user is within the same local area network (LAN) as the server. The table below gives estimates for how long an upload may take given different ZIP file sizes and bandwidth.
  3. Server-side processing: The uploaded file unzips. CSV, Excel, and GeoJSON data is scanned to auto-detect the column types. Then a table is created (a TAB file, SQL Server or in PostGIS), and the data inserted into the table. If both the source and destination formats are TAB, then the uploaded TAB file is used without performing the inserting step; this may have lower processing times.
    Note: There is a 300 second (5 minutes) timeout for processing the data after uploading it. If the Spectrum Spatial Analyst server is on a different machine than Spectrum Spatial, then this processing time includes the time required to send the file to the Spectrum over the internal network as well as the time to translate and insert data. If needed, the 300-second timeout for processing can be increased by adding the property multipart.upload.timeout, with a value in seconds, to the java options for the Precisely AnalystConnect service.

There is a balance between time spent compressing to a ZIP file and time spent uploading. Users on slow to medium bandwidth connections benefit from compression because it improves the upload speed. A user’s upload speed may take the longest of the three steps and varies between users.

The table below shows the variation in theoretical upload speed, in seconds, for different file sizes and bandwidth speeds. The table assumes a five percent (5%) overhead in transferring the file.

Table 1. Upload speed in seconds based on file size and upload bandwidth
File Size in Mbytes
1 2 4 8 16 32 64 128 256 512 1024
Upload Speed Mbits per second 4 2.1 4.2 8.4 16.8 33.6 67.1 134.2 268.4 536.9 1073.7 2147.5
8 1.0 2.1 4.2 8.4 16.8 33.6 67.1 134.2 268.4 536.9 1073.7
12 0.7 1.4 2.8 5.6 11.2 22.4 44.7 89.5 179.0 357.9 715.8
16 0.5 1.0 2.1 4.2 8.4 16.8 33.6 67.1 134.2 268.4 536.9
20 0.4 0.8 1.7 3.4 6.7 13.4 26.8 53.7 107.4 214.7 429.5
50 0.2 0.3 0.7 1.3 2.7 5.4 10.7 21.5 42.9 85.9 171.8
100 0.1 0.2 0.3 0.7 1.3 2.7 5.4 10.7 21.5 42.9 85.9
150 0.1 0.1 0.2 0.4 0.9 1.8 3.6 7.2 14.3 28.6 57.3