Introducing the all-new TIBCO Community site!

For current users, please click "Sign In" to reset your password and access the enhanced features. If you're a first-time visitor, we extend a warm welcome—click "Sign Up" to become a part of the TIBCO Community!

If you're seeking alternative community sites, explore ibi, Jaspersoft, and Spotfire.

Jump to content

Large file handling in File poller


Recommended Posts

Hi Raja,

 

Your answer is somewhat helpful.

let me give you more details for clear answer.

 

1. What is the size and type of the file you are expecting for the file - 100 MB CSV file

2. Is time a criteria or just memory management is your concern - Memory management is concern

3. Also provide information about what you want to do after you read this file. -I will read each line and create a fixed flat file.

 

Now , can you give me a solid solution

 

Thanks

Kali

Link to comment
Share on other sites

Hi Kali,

 

If it is a 100 MB file, TIBCO can manage usually with sufficient heap allocation. We had a similar process in my project and we were able to successfully process and create flat files. But we have allocated enough memory ( 8/16 GB) to support such process. If you will not be able to afford such memory, then I would suggest doing a "exclude file content". TIBCO suggests the following for processing large files:

 

To process a large number of records, follow this procedure:

1. Create a Parse Data activity.

2. Specify the fields on the Configuration tab. Check the checkbox for the Manually Specify Start Record field.

3. Select the Parse Data activity and click the group icon on the toolbar to create a group containing the Parse Data activity. See TIBCO BusinessWorks Process Design Guide for more information about working with groups.

4. Specify "Repeat Until True Loop" as the Group action, and specify an index name (for example, "i").

5. The loop should exit when the done output item for the Parse Data activity is set to true. For example, the condition for the loop could be set to the following: string($ParseData/Output/done) = string(true())

6. Set the noOfRecords input item for the Parse Data activity to the number of records you wish to process for each execution of the loop.

7. If you do not check the Manually Specify Start Record field on the Configuration tab of the Parse Data activity, the loop processes the specified noOfRecords with each iteration until there are no more input records to parse.

You can optionally check the Manually Specify Start Record field to specify the startRecord on the Input tab. If you do this, you must create an XPath expression to properly specify the starting record to read with each iteration of the loop. For example, the count of records in the input starts at zero, so the startRecord input item could be set to the current value of the loop index minus one. For example, $i - 1.

The procedure above is a general guideline for creating a loop group for parsing a large set of input records in parts. You may want to modify the procedure to include additional processing of the records, or you may want to change the XPath expressions to suit your business process.

 

Hope this helps.

Link to comment
Share on other sites

  • 4 years later...

Hi Kali,

 

Can you provide some additional information about what you are planning to do.

1. What is the size and type of the file you are expecting for the file

2. Is time a criteria or just memory management is your concern

3. Also provide information about what you want to do after you read this file.

 

For starters, you can "exclude the file content" and just poll for the file. Once a process is triggered, you can read the file in multiple batches through either a custom java code or you can also use MTOM attachments if this is intended for WebService use. I believe BW will be able to read a file which is around 50 MB if proper resource is allocated. Hope this helps.

Link to comment
Share on other sites

  • 3 years later...
×
×
  • Create New...