Why don't you start around 50 threads at a time to do the file writes? Threads are effective for IO. You open the source file, start a queue, and start sending data sets to be written to the queue. Your source file processing can go on while the writes are done in other threads.