How best to achieve a timely integration using Scribe Insight
I am trying to work out the most efficient way of achieving a timely integration using Scribe Insight. I am trying to achieve the closest to a real time integration as possible, but without using the MSMQ method, therefore using the traditional Time, File, Query process. The two options I'm comparing are as follows:
1) Using a Time based integration set to run every 60 seconds
2) Using a Query based integration set to run every 10 seconds and only process where there are rows in the source query to process
So having tried both of these options, option 1 is averaging running much quicker. I think this is because the job is running every 60 seconds regardless of whether there are source rows, the message processor is keeping the connections open, meaning when there are records to process, it runs much quicker. The reason option 2 is slower (even though set at 10 seconds), is because it takes quite a long time to re-establish the connections, if the message processor has been idle for over the 5 minute keep-alive period.
Therefore I need to know if there are any other ways of achieving this in a fast way but efficiently. The problem with option 1 is that it is running many times per day when it doesn't really need to.
Any suggestions or advice would be much appreciated.