How does TERR variable passing happen?

I am working on a large data set (30 million rows x 18 columns of text and numbers) in Spotfire professional 7.5.

I would like to use a data function locally e.g. using TERR to do some calculations on this data and returnn on the order of 50 rows and 10 columns as the result.

I have written and tested code to do this in open source R. Loading the data frame in open source R takes a while, but the calculations themselves take on the order of 5 seconds.

When putting the script code into Spotfire by registering a TERR (Local) script and conmecting input and output I obtain a very poor performance.

Specifically it takes several minutes of exporting rows (where I can tell that a large amount of writing to the disk happens) and then a long time of "executing" the script (Is there a way to get print statements from TERR into a console for debugging?).

It seems that when the data function is run, Spotfire first writes teh whole data table to a temporary file  "C:\Users\myusername\AppData\Local\Temp\TIBCO\Spotfire\7.5\bc604479-ebf9-4eb1-b637-9729d2dc5202.sbdf  

and then afterwards this is loaded into memory of TERR process. This is not the desired behaviour. Is it possible to use large data table with TERR without using the disk. My computer has 16 GB of RAM so it should not be necessary to write to disk.

What are some tools to debug the TERR tools and find memory problems?

All the best,



(1) Answer