Suggestions on mechanism or existing code - maintain persistence of file download history

[R.Wieser]

DL,

>> Nothing that can't be countered by keeping copies of the last X number of
>> to-be-dowloaded-URLs files.
>
> That's a good idea, but how would the automated system 'know' to give-up
> on the current file and utilise generation n-1? Unable to open the file or
> ???

Well, that would be one reason for it.   But there was also talk about
integrity checking, both on the file and URL-entry level.   Failing any of
that could trigger it.

But to be honest, I would not even automate it.   Such a situation should
not occur often, if at all.  So having it is cause for concern, and should
be investigated.   Instead I would give the user an error message, and offer
him the list of older files to choose one from to continue with (or have him
specify the file on the commandline).

> Using ctrl+c is a VERY BAD idea.

To have it just exit the program ?  Yes, indeed.

Though you /could/ keep track of what needs to be finished and have the
ctrl-c handler do that for you (barf).

Another posibility is to capture the ctrl-c and set a flag, which than
instructs the program to terminate loops wherever possible - and thus have
the program finish as if there was no more to do.

> (see also 'sledgehammer to crack a nut')

While I agree with you there, I've been searching for other ways to detect a
keypress (in a console-based script) and have found none.   IOW, you do not
(seem to) have another option.

> Why do you need to abandon the process mid-way?

Take your pick.    Mostly because of "I need to leave *now*".   Or perhaps
because of an "you have X seconds before the device shuts down because of
the battery being low" situation.

> What is the OP's definition of "unlikely" or "acceptable risk"?

Good question, but one I had no wish for to try to ascertain.   I just gave
some "worst case" secenario based replies (aka, the "did you think of <this>
situation" kind).

> There are so many reasons why such won't work first-time, when they should
> every time; that it may be quite difficult to detect 'corruption'

:-) That was not even considered.  Just the "what do I still need to
download" datafile.

> Accordingly, there is no non-atomic transaction in the proposal

The problem there is that you are second-guessing to what the OP will be
writing, and that it will be good.   I didn't and I don't assume that.  I've
seen too much "it works, so its good" noobie code. :-)

> Do I want to deal with the complexities of managing files and corruptions,
> in that arena?
...
> Do you?

:-)   Its was-and-is not my choice to make.   I gave the OP some stuff to
think about, and left making the choice upto him.    All three of the
presented options are viable.

> Be aware that formation rules for URLs are not congruent with OS FS rules!

Yup.   Though I seldom see that happen.  Though I guess I should have
mentioned that ... :-|

Regards,
Rudy Wieser