Large data table loaded on demand
I’m currently downloading a very large data table from an info link, and using DataOnDemand to limit the size of the resulting table.
This is is working fine; however, even though the resulting table is ~150k rows by 20 or so columns, this operation increases the file size by almost 600mb, which seems very unreasonable.
If it were just one table, I’d probably just leave it as is and not worry about it, but I need to repeat this operation to generate 5 more similar tables, and as I said, the file grows by 500-600mb with each table.
Why is the file growing so much in this scenario? It almost seems as if the source data from the original info link is kept somewhere inside the analysis. I checked the diagnostics information under the help menu and the resulting tables are listed as having a size of no more than 40-50Mb.
Any advise would be greatly appreciated.