Read from database, write to another database, simultaneously

[Laurent Pointal]

Bjoern Schliessmann a écrit :
> Sean Davis wrote:
> 
>> The author of one of the python database clients mentioned that
>> using one thread to retrieve the data from the oracle database and
>> another to insert the data into postgresql with something like a
>> pipe between the two threads might make sense, keeping both IO
>> streams busy.
> 
> IMHO he's wrong. Network interaction is quite slow compared with CPU
> performance, so there's no gain (maybe even overhead due to thread
> management and locking stuff). That's true even on multiprocessor
> machines, not only because there's almost nothing to compute but
> only IO traffic. CMIIW.


Not so sure, there is low CPU in the Python script, but there may be
CPU+disk activity on the database sides [with cache management and other
optimizations on disk access].
So, with a reader thread and a writer thread, he can have a select on a
database performed in parallel with an insert on the other database.
After, he must know if the two databases use same disks, same
controller, same host... or not.

But, if its only a do-once job, maybe the optimization is net really
necessary.