Timeout error during Execute Query operation with large amount of results

[Alejandro Osorio]

I'am currently working on the development of a custom connector (CDK)which consumes information from a REST API with pagination.

In some cases,in whichExecute Query Operation takes more than 1 hour andreturns several thousands of records, On Demand / Scheduled executions fail indicating timeout error (Most of the times the number of processed records is 33575)

 

Can you please provide some advice or examples on how to avoid timeouts in Scribe Online

 

Thanks,

Alejandro.

[Leon Bernard 2]

There are 2 Receive and send timeouts. On the client side and on the web service side. Although it may be necessary to increase the receiveTimeout setting on the client side, it will not help if the server side timeout is lower. The question is are you timing out when getting the request from the request pool, when waiting for the response or while keeping the response open to process it. if is one of the latter 2, asking for only the fields you require or smaller data sets by decreasing the page size, will likely speed up your response time, and avoid the timeout. If you are timing out getting the request then you are likely not disposing of your requests and it is timing out for a request from being freed up in the request pool, which is limited. If you are timing out processing the response, I would recommend caching the response in a memory stream and disconnecting until you need the next page. if timing out receiving the response, increasing the timeout on your connectors binding is appropriate, if you are getting better results. I would consult the api's documentation on how they recommend dealing with timeouts, or put in a support ticket for the best approach.

[Leon Bernard 2]

Note that once timeouts are mitigated in the running of the job, you may still encounter timeouts in the preview window, as we have a seperate timeout for showing a preview. If so, you will also want to detect, in the connector if the request is a preview and limit your page size to 25 records as this is all we display. This is achieved by checking the IsTestQuery property of the ScribeQuery.

 

https://dev.scribesoft.com/en/sdk/html/p_scribe_core_connectorapi_query_...