mimicking a file in memory

[p.]

On Nov 20, 3:14 pm, Grant Edwards <gra... at visi.com> wrote:
> On 2007-11-20, p. <ppetr... at gmail.com> wrote:
>
>
>
> >> By "memory" I presume you mean virtual memory?  RAM with
> >> disk-blocks as backing store? On any real OS, tempfiles are
> >> just RAM with disk-blocks as backing store.
>
> >> Sound similar? The only difference is the API used to access
> >> the bytes.  You want a file-I/O API, so you can either use the
> >> extensively tested and and highly optimized filesystem code in
> >> the OS to make disk-backed-RAM look like a file, or you can try
> >> to write Python code that does the same thing.
>
> >> Which do you think is going to work faster/better?
>
> >> [The kernel is generally better at knowing what needs to be in
> >> RAM than you are -- let it do its job.]
>
> >> IOW: just use a temp file.  Life will be simple. The bytes
> >> probably won't ever hit the platters (if they do, then that
> >> means they would have the other way too).
>
> > Grant, are temp files automatically put into ram for all linux
> > distros?
>
> All files are put into ram for all linux distros that use
> virtual memory.  (You'll know if you're not using virtual.)
>
> > at any rate, i could set up ram disk. much better solution
> > than using python...except that i've never done a ram disk
> > before. more reading to do...
>
> You don't have set up a ram disk.  You already have one.  All
> your disks are ram disks. It's just that some of them have
> magnetic platters as backing store so they get preserved during
> a reboot.  On some Linux distros, the /tmp directory is a
> filesystem without prmanent magnetic backing-store.  On others
> it does have a permanent backing store.  If you do a "mount"
> command, you'll probably see a "filesystem" who's type is
> "tmpfs". That's a filesystem with no permanent magnetic
> backing-store[1].
>
> Seehttp://en.wikipedia.org/wiki/TMPFS
>
> /tmp might or might not be in a tmpfs filesystem (depends on
> the distro).  In any case, you probably don't need to worry
> about it.
>
> Just call tempfile.NamedTemporaryFile() and tell it you want an
> unbuffered file (that way you don't have to remember to flush
> the file after writing to it).  It will return a file object:
>
>   f = tempfile.NamedTemporaryFile(bufsize=0)
>
> Write the data to that file object and flush it:
>
>   f.write(mydata)
>
> Pass the file's name to whatever broken library it is that
> insists on a file name instead of a file-like object:
>
>   brokenLib.brokenModule(f.name).
>
> When you're done, delete the file object:
>
>   del f
>
> NB: This particular approach won't work on Windows.  On Windows
>     you'll have to use tempfile.mktemp(), which can have race
>     conditions.  It returns a name, so you'll have to create
>     the file, write to it, and then pass the name to the broken
>     module.
>
> [1] Tmpfs pages use the swap partition for temporary backing
>     store the same as for all other memory pages.  If you're
>     using tmpfs for big stuff, make sure your swap partition is
>     large enough to hold whatever you're doing in tmpfs plus
>     whatever normal swapping capacity you need.
>
> ------------------------------demo.py------------------------------
> def brokenModule(filename):
>     f = file(filename)
>     d = f.read()
>     print d
>     f.close()
>
> import tempfile,os
>
> f = tempfile.NamedTemporaryFile(bufsize=0)
> n = f.name
> print f,":",n
> os.system("ls -l %s\n" % n)
>
> f.write("hello world")
> brokenModule(n)
>
> del f
> os.system("ls -l %s\n" % n)
> ------------------------------demo.py------------------------------
>
> If you run this you'll see something like this:
>
> $ python demo.py
> <open file '<fdopen>', mode 'w+b' at 0xb7c37728> : /tmp/tmpgqSj8p
> -rw------- 1 grante users 0 2007-11-20 17:11 /tmp/tmpgqSj8p
> hello world
> ls: cannot access /tmp/tmpgqSj8p: No such file or directory
>
> --
> Grant Edwards                   grante             Yow! I want to mail a
>                                   at               bronzed artichoke to
>                                visi.com            Nicaragua!

excellent. didn't know tempfile was a module. thanks so much.