Large Amount of Data

[Jack]

I'll save them in a file for further processing.

"John Machin" <sjmachin at lexicon.net> wrote in message 
news:1180228756.574525.16110 at q19g2000prn.googlegroups.com...
> On May 26, 6:17 pm, "Jack" <nos... at invalid.com> wrote:
>> I have tens of millions (could be more) of document in files. Each of 
>> them
>> has other
>> properties in separate files. I need to check if they exist, update and
>> merge properties, etc.
>
> And then save the results where?
> Option (0) retain it in memory
> Option (1) a file
> Option (2) a database
>
> And why are you doing this agglomeration of information? Presumably so
> that it can be queried. Do you plan to load the whole file into memory
> in order to satisfy a simple query?