String concatenation - which is the fastest way ?

[Stefan Behnel]

Chris Angelico, 11.08.2011 12:59:
> On Thu, Aug 11, 2011 at 7:40 AM,<przemolicc at poczta.fm>  wrote:
>> I am not a database developer so I don't want to change the whole process
>> of data flow between applications in my company. Another process is
>> reading this XML from particular Oracle table so I have to put the final XML there.
>
> I think you may be looking at a submission to
> http://www.thedailywtf.com/ soon. You seem to be working in a rather
> weird dataflow. :( Under the circumstances, you're probably going to
> want to go with the original ''.join() option.
>
>> This server has 256 GB of RAM so memory is not a problem.
>> Also the select which fetches the data is sorted. That is why I have to
>> carefully divide into subtasks and then merge it in correct order.
>
> There's no guarantee that all of that 256GB is available to you, of course.
>
> What may be the easiest way is to do the select in a single process,
> then partition it and use the Python multiprocessing module to split
> the job into several parts. Then you need only concatenate the handful
> of strings.

Or join them using an n-way merge.

Stefan