Suggestions on mechanism or existing code - maintain persistence of file download history

[DL Neil]

On 31/01/20 9:53 PM, R.Wieser wrote:
>> Using ctrl+c is a VERY BAD idea.
> 
> To have it just exit the program ?  Yes, indeed.
> 
> Though you /could/ keep track of what needs to be finished and have the
> ctrl-c handler do that for you (barf).
> 
> Another posibility is to capture the ctrl-c and set a flag, which than
> instructs the program to terminate loops wherever possible - and thus have
> the program finish as if there was no more to do.
> 
>> (see also 'sledgehammer to crack a nut')
> 
> While I agree with you there, I've been searching for other ways to detect a
> keypress (in a console-based script) and have found none.   IOW, you do not
> (seem to) have another option.

Color me disappointed! I was hoping to be enlightened as to how one 
might code exactly that. (so I went looking...)

My first thought was a try...except looking for a keyboard error, but 
wouldn't that only work if there was an input() loop? The idea of 
putting, effectively the entire code-base, into a try...except block 
seemed crude and likely to cause side-effects.

https://stackoverflow.com/questions/1112343/how-do-i-capture-sigint-in-python 
put me on a good path. So, I opened a door in my memory, shut my ears to 
the intense screeching noise, and used a yard-broom to sweep away all 
the rust that flaked off the hinges...

PSL's signal — Set handlers for asynchronous events 
(https://docs.python.org/3/library/signal.html) appears to handle 
keyboard interrupts, *at least in MS-Windows*.

Rusty? I think, years ago, I used signal(s) to establish time-limits on 
systems that had a possibility of 'run-away' or over-refining a solution 
which would only ever be asymptotic. I've not used the library in this 
manner.

That said, I'm using one of Python's asynchronous features, and Internet 
activity is riddled with delay-periods which make such retrievals ripe 
for async (code) solutions...


>> Why do you need to abandon the process mid-way?
> 
> Take your pick.    Mostly because of "I need to leave *now*".   Or perhaps
> because of an "you have X seconds before the device shuts down because of
> the battery being low" situation.

I (personally) cannot make use of the above. However, even if my 
portable's battery threatens complete disaster, the system "hibernates", 
and thus averts major disaster, eg file system (or RDBMS) corruption. 
This seems enough for 99% of 'average uses'. Of course it is regarded as 
trivially-unacceptable in the non-stop world, eg banking. Would our OP 
classify this project as anything other than the first of those 
categories? Thus:

>> What is the OP's definition of "unlikely" or "acceptable risk"?
> 
> Good question, but one I had no wish for to try to ascertain.   I just gave
> some "worst case" secenario based replies (aka, the "did you think of <this>
> situation" kind).

Understood. However, your understanding of RDBMS interrupt 'security' 
appears either aged, or biased to a particular implementation (which (in 
my imagination) wouldn't match the high-standards I expect you hold).

Have already admitted that I (personally) may 'over use' RDBMS because 
it is 'easy' for me. When we do feel the justification to examine the 
details of "worst case scenarios"/"did you think of this?" RDBMS 
stand-out as a 'solution' simply because of the incredible amounts of 
time and depths of thought, highly-specialised minds (far superior to 
any input I could offer) have invested in providing highly refined 
services/servers. (which I dare-say informed @Chris' comments)

Neither MSFT nor the relevant GNU/Linux projects assure common file 
systems to such a degree (and particularly not in the mentioned 
circumstances). That's why there are FS 'rescue' procedures (Linux runs 
fsck automatically when 'trouble' is noted during system-start and every 
so-many reboots, regardless).

I'm sufficiently intrigued to wonder if anyone will dispute/correct such 
- but not sufficiently motivated to research any position other than the 
one currently held - so call me "Curious", but not "George" then...


>> There are so many reasons why such won't work first-time, when they should
>> every time; that it may be quite difficult to detect 'corruption'
> 
> :-) That was not even considered.  Just the "what do I still need to
> download" datafile.

Yes, we left the OP's (actual) spec, way-back somewhere!


>> Accordingly, there is no non-atomic transaction in the proposal
> 
> The problem there is that you are second-guessing to what the OP will be
> writing, and that it will be good.   I didn't and I don't assume that.  I've
> seen too much "it works, so its good" noobie code. :-)

Absolutely! One of the 'joys' of working text-only is that even when one 
knows (much) more about the background to a posted-question, it is still 
so easy to 'answer-wrong'.

Even in my courses, where the Discussion Lists (are supposedly) related 
to each week/chapter of a course, you'd think we'd have adequate 'hints' 
as to the level of the trainee and where his/her question is 
'coming-from'. Not true!

The secret is probably somewhere in the understanding of *that* 
potential problem and going from there...

Sigh, if I had a dollar for every time someone looked at a simple 
solution (MS-Excel I'm looking at you!) and uttered 'suggestions' such 
as "we could run the entire company off one of these spreadsheet things" 
or "if we captured all the company's transactions, couldn't we run it 
off one Python pgm?"...

The world has moved-on though. Forty years ago it was common for someone 
writing his/her first program(me) to come 'crying' "the computer made a 
mistake" because it performed some simple arithmetic/statistical 
calculation without yielding the expected answer!

That said, it continues to astonish me that computing-professionals 
still can't apply the 'attention to detail' coding requires to their 
typing - regardless of whether the input is code or a Discussion List 
post. What is that mind-set???
(this comment NOT aimed at you, or any other participant on this list, 
personally!)


>> Do I want to deal with the complexities of managing files and corruptions,
>> in that arena?
> ...
>> Do you?
> 
> :-)   Its was-and-is not my choice to make.   I gave the OP some stuff to
> think about, and left making the choice upto him.    All three of the
> presented options are viable.

Good stuff! Yet at the same time, that very lack of clarity is good 
reason to allow that 'another' solution might be 'right' - even if it is 
not 'better'.
(for any definitions of "right" or "better", or ...)

As sent to the OP. I appreciate these discussions, in the expectation of 
learning something-new. (and with rust-removal paints at the ready!)


>> Be aware that formation rules for URLs are not congruent with OS FS rules!
> 
> Yup.   Though I seldom see that happen.  Though I guess I should have
> mentioned that ... :-|

I'd show you the marks where that came back to bite me once - but 
fortunately for you, the list doesn't allow graphic attachments (nor 
should distressed (and distressing) images of such parts of one's 
anatomy EVER be seen, even by qualified medical professionals...)
-- 
Regards =dn