Best approach to create humongous amount of files

[Peter Otten]

Mario R. Osorio wrote:

> On Wednesday, May 20, 2015 at 2:09:59 PM UTC-4, Denis McMahon wrote:
>> On Wed, 20 May 2015 17:14:15 +0530, Parul Mogra wrote:
>> 
>> > Hello everyone,
>> > My objective is to create large amount of data files (say a million
>> > *.json files), using a pre-existing template file (*.json). Each file
>> > would have a unique name, possibly by incorporating time stamp
>> > information. The files have to be generated in a folder specified.
>> 
>> > What is the best strategy to achieve this task, so that the files will
>> > be generated in the shortest possible time? Say within an hour.
>> 
>> timestamps are normally unixtime in seconds. There are 3600 seconds in an
>> hour. You'll have a hard job creating a million files with timestamp
>> based naming inside of an hour.
>> 
>> --
>> Denis McMahon, denismfmcmahon at gmail.com
> 
> I would use a combination of both, timestamp and a serial number, such as:
> 201505201425440000
> 201505201425440001
> 201505201425440002
> 201505201425440003
> 201505201425450000
> 201505201425450001
> 201505201425460000
> .. and so on ..

Like this?

import time
import itertools
from operator import itemgetter

try:
    from itertools import imap as map
except ImportError:
    pass

INDEX_TEMPLATE = "{}-{:02}-{:02}-{:02}-{:02}-{:02}-{i:03}"


def unique_names(template):
    return (
        template.format(INDEX_TEMPLATE.format(*t, i=i))
        for g in map(itemgetter(1), itertools.groupby(iter(time.gmtime, 
())))
        for i, t in enumerate(g, 1))


if __name__ == "__main__":
    import random

    for name in unique_names("foo-{}.txt"):
        print(name)
        time.sleep(random.random())

I mean, readability counts...