Best approach to create humongous amount of files

[Tim Chase]

On 2015-05-20 22:58, Chris Angelico wrote:
> On Wed, May 20, 2015 at 9:44 PM, Parul Mogra <scoria.799 at gmail.com>
> wrote:
> > My objective is to create large amount of data files (say a
> > million *.json files), using a pre-existing template file
> > (*.json). Each file would have a unique name, possibly by
> > incorporating time stamp information. The files have to be
> > generated in a folder specified.
[snip] 
> try a simple sequential integer.
> 
> All you'd need would be a loop that creates a bunch of files... most
> of your code will be figuring out what parts of the template need to
> change. Not too difficult.

If you store your template as a Python string-formatting template,
you can just use string-formatting to do your dirty work:


  import random
  HOW_MANY = 1000000
  template = """{
    "some_string": "%(string)s",
    "some_int": %(int)i
    }
    """

  wordlist = [
    word.rstrip()
    for word in open('/usr/share/dict/words')
    ]
  wordlist[:] = [ # just lowercase all-alpha words
    word
    for word in wordlist
    if word.isalpha() and word.islower()
    ]

  for i in xrange(HOW_MANY):
    fname = "data_%08i.json" % i
    with open(fname, "w") as f:
      f.write(template % {
        "string_value": random.choice(wordlist),
        "int_value": random.randint(0, 1000),
        })


-tkc