Suggestions on mechanism or existing code - maintain persistence of file download history

[jkn]

On Wednesday, January 29, 2020 at 8:27:03 PM UTC, Chris Angelico wrote:
> On Thu, Jan 30, 2020 at 7:06 AM jkn <jkn_gg at nicorp.f9.co.uk> wrote:
> >
> > Hi all
> >     I'm almost embarrassed to ask this as it's "so simple", but thought I'd give
> > it a go...
> 
> Hey, nothing wrong with that!
> 
> > I want to be a able to use a simple 'download manager' which I was going to write
> > (in Python), but then wondered if there was something suitable already out there.
> > I haven't found it, but thought people here might have some ideas for existing work, or approaches.
> >
> > The situation is this - I have a long list of file URLs and want to download these
> > as a 'background task'. I want this to process to be 'crudely persistent' - you
> > can CTRL-C out, and next time you run things it will pick up where it left off.
> 
> A decent project. I've done this before but in restricted ways.
> 
> > The download part is not difficult. Is is the persistence bit I am thinking about.
> > It is not easy to tell the name of the downloaded file from the URL.
> >
> > I could have a file with all the URLs listed and work through each line in turn.
> > But then I would have to rewrite the file (say, with the previously-successful
> > lines commented out) as I go.
> >
> 
> Hmm. The easiest way would be to have something from the URL in the
> file name. For instance, you could hash the URL and put the first few
> digits of the hash in the file name, so
> http://some.domain.example/some/path/filename.html might get saved
> into "a39321604c - filename.html". That way, if you want to know if
> it's been downloaded already, you just hash the URL and see if any
> file begins with those digits.
> 
> Would that kind of idea work?
> 
> ChrisA

Hi Chris
    Thanks for the idea. I should perhaps have said more clearly that it is not
easy (though perhaps not impossible) to infer the name of the downloaded data
from the URL - it is not a 'simple' file URL, more of a tag.

However I guess your scheme would work if I just hashed the URL and created
a marker file - "a39321604c.downloaded" once downloaded. The downloaded content
would be separately (and somewhat opaquely) named, but that doesn't matter.

MRAB's scheme does have the disadvantages to me that Chris has pointed out.

    Jon N