Does anyone know if there is a data limit on the number of rows in a data file upload to then be used for a map dashboard visualization?
Quote from: ssmith001 on 05 Dec 2018 11:33:31 AM
Does anyone know if there is a data limit on the number of rows in a data file upload to then be used for a map dashboard visualization?
Hi,
There are limits on file size, but not row count. The limits are set by your Administrator, and can be changed:
(https://i.ibb.co/9Wdr2pQ/Upload-limits.png)
Cheers!
MF.
Thank you, I didn't know that was a setting in the system. Can it be changed to any value and what are the concerns with setting it to a larger size?
Quote from: ssmith001 on 06 Dec 2018 08:25:06 AM
Thank you, I didn't know that was a setting in the system. Can it be changed to any value and what are the concerns with setting it to a larger size?
The underlying technology is using a Parquet file format - the file is loaded into memory when needed, and obviously the bigger the file, the more memory will be required and the longer the response times will be.
IBM made some changes for the 11.1.1 release and altered the format of the Parquet files used, as well as adding a new microservice utilising Spark SQL for querying data from the in-memory columnar data store. This has two benefits - faster access times all around, and also the ability to support larger data sets than 11.0.x would handle.
Cheers!
MF.