Generate unique ID for URL

[Roy Smith]

In article <1ce88f36-bfc7-4a55-89f8-70d1645d27ad at googlegroups.com>,
 Richard <richardbp at gmail.com> wrote:

> So the use case - I'm storing webpages on disk and want a quick retrieval 
> system based on URL. 
> I can't store the files in a single directory because of OS limitations so 
> have been using a sub folder structure.
> For example to store data at URL "abc": a/b/c/index.html
> This data is also viewed locally through a web app.
> 
> If you can suggest a better approach I would welcome it. 

Ah, so basically, you're reinventing Varnish?

Maybe do what Varnish (and MongoDB, and a few other things) do?  Bypass 
the file system entirely.  Juar mmap() a chunk of memory large enough to 
hold everything and let the OS figure out how to page things to disk.