Large file handling in File poller

[Manoj Chaurasia]

Hello,

 

Can you anyone suggest to handle large file using file poller activity I don't have File adapter license , hence I need to mange with it.

 

Thanks

Kali

[Manoj Chaurasia]

Hi Raja,

 

Your answer is somewhat helpful.

let me give you more details for clear answer.

 

1. What is the size and type of the file you are expecting for the file - 100 MB CSV file

2. Is time a criteria or just memory management is your concern - Memory management is concern

3. Also provide information about what you want to do after you read this file. -I will read each line and create a fixed flat file.

 

Now , can you give me a solid solution

 

Thanks

Kali

[Manoj Chaurasia]

Hi Kali,

 

If it is a 100 MB file, TIBCO can manage usually with sufficient heap allocation. We had a similar process in my project and we were able to successfully process and create flat files. But we have allocated enough memory ( 8/16 GB) to support such process. If you will not be able to afford such memory, then I would suggest doing a "exclude file content". TIBCO suggests the following for processing large files:

 

To process a large number of records, follow this procedure:

1. Create a Parse Data activity.

2. Specify the fields on the Configuration tab. Check the checkbox for the Manually Specify Start Record field.

3. Select the Parse Data activity and click the group icon on the toolbar to create a group containing the Parse Data activity. See TIBCO BusinessWorks Process Design Guide for more information about working with groups.

4. Specify "Repeat Until True Loop" as the Group action, and specify an index name (for example, "i").

5. The loop should exit when the done output item for the Parse Data activity is set to true. For example, the condition for the loop could be set to the following: string($ParseData/Output/done) = string(true())

6. Set the noOfRecords input item for the Parse Data activity to the number of records you wish to process for each execution of the loop.

7. If you do not check the Manually Specify Start Record field on the Configuration tab of the Parse Data activity, the loop processes the specified noOfRecords with each iteration until there are no more input records to parse.

You can optionally check the Manually Specify Start Record field to specify the startRecord on the Input tab. If you do this, you must create an XPath expression to properly specify the starting record to read with each iteration of the loop. For example, the count of records in the input starts at zero, so the startRecord input item could be set to the current value of the loop index minus one. For example, $i - 1.

The procedure above is a general guideline for creating a loop group for parsing a large set of input records in parts. You may want to modify the procedure to include additional processing of the records, or you may want to change the XPath expressions to suit your business process.

 

Hope this helps.

[Manoj Chaurasia]

Thank you Raja for your comprehensive answer

[Rajasegaran P]

Hi Kali,

 

Can you provide some additional information about what you are planning to do.

1. What is the size and type of the file you are expecting for the file

2. Is time a criteria or just memory management is your concern

3. Also provide information about what you want to do after you read this file.

 

For starters, you can "exclude the file content" and just poll for the file. Once a process is triggered, you can read the file in multiple batches through either a custom java code or you can also use MTOM attachments if this is intended for WebService use. I believe BW will be able to read a file which is around 50 MB if proper resource is allocated. Hope this helps.

[shivani jain]

How to avoid file of larger size in tibco.

I have to sent image as attachment using mail activity.

I'm receiving image in base64 format. I have to avoid base64 string more than 15 MB.

How can I acheive this