[Tutor] Debug help

[Cameron Simpson]

On 19Jul2020 15:25, jim <jf_byrnes at comcast.net> wrote:
[...]
>Here is the part of the code that generates the error, the remainder of 
>the script just manipulates the spreadsheet.
>
>    for stock in stocks:
>        # ~ count = 0
>        # ~ while count < total_num_of_stocks:
>        # ~ while count < 33:
>        while True:
>            # ~ count = count + 1
>            try:
>                # ~ print('Day = ' + day)
>                ticker = yf.Ticker(stock)
>                # ~ print('Ticker = ' + ticker)
>                hist = ticker.history(start=day)
>                # ~ print('History = ' + hist)
>                close_price = hist.loc[day]['Close']
>                # ~ print('Close = ' + close_price)
>                closing_prices.append(close_price)
>
>            except:
>                print ('yfinance JSONDecodeError DAILY, retyring')
>                # ~ print ('ticker: ' + hist)
>                continue
>            break
>
>    # Get stock values by multiplying the list elements by number of shares
>    # using zip()
>    value = [num_shares * stk for num_shares, stk in zip(shares, 
>closing_prices)]
>
>    # Put date in cell then go down one row
>    S(cell_loc).string = day
>    S().dispatch('GoDown')
>
>    # Fill column for MY100000 with stock values using the number of 
>stocks in MY100000
>    for stk in value[:total_num_MY100000]:
>        S().value = stk # Line 102
>        S().dispatch('GoDown')
>
>If I run the script with the print lines outside the except commented 
>out it will print the date in the spreadsheet, go down 1 cell and 
>throw the error and end. If I uncomment any of the print lines it 
>seems to go into a endless loop printing out the exception error msg.
>
>I wonder if anyone has an idea of how to figure out what Yahoo Finance 
>is doing to cause a problem and I am curious as to why uncommenting a 
>print seems to cause an endless loop.

I'm unfamiliar with Yahoo Finance, but I want to comment on your 
debugging issues.

Your code's control structure is like this:

    while True:
        try:
            [...]
            # ~ print('Close = ' + close_price)
            [...]
        except:
            print ('yfinance JSONDecodeError DAILY, retyring')
            # ~ print ('ticker: ' + hist)
            continue

First up, consider what happens when anything goes wrong inside the 
try/except: you print something and continue, restarting the loop. If 
that thing happens on every pass, you get an infinite loop.

See the commented out print(). Supposing close_price is not a string.  
The addition will raise a TypeError exception. And into the infinite 
loop you go.

Better, for debugging statements, to perform as few operations as 
possible. because if something is wrong, any operation might itself fail 
because things as not as you expect. So consider:

    print('Close = ', close_price)

Because print() converts all is arguments to strings, the form above 
does no addition and still produces a result.

However, the biggest problem here is what we call a "bare except": an 
except which catches _any_ exception. That's handy for reporting on 
_everything_ which explodes, but otherwise a bad practice.

The rule of thumb for catching excpetions is to _only_ catch what you 
expect and know how to handle _correctly_. Your code catches everything 
(again, great for reporting) but does not know what to do with it.

Because you do not print the exception or where it came from, you learn 
nothing about the underlying issue. Consider this:

    while True:
        try:
            [...]
            # ~ print('Close = ' + close_price)
            [...]
        except Exception as e:
            print("unexpection exception:", e)
            print ('yfinance JSONDecodeError DAILY, retyring')
            # ~ print ('ticker: ' + hist)
            continue

At least this will show you what is wrong. But better still is not to 
catch it at all. It is unexpected, so your print-and-continue action is 
probably inappropriate (because you do not know the context - _anything_ 
might be wrong). As a scarey example, consider:

    while True:
        try:
            x = closed_price
            [...]
            # ~ print('Close = ' + close_price)
            [...]
        except Exception as e:
            print("unexpection exception:", e)
            print ('yfinance JSONDecodeError DAILY, retyring')
            # ~ print ('ticker: ' + hist)
            continue

Here, "closed_price" is a misspelled variable name. Python will raise a 
NameError because it is an unknown name, and you won't have even printed 
out the exception, so you will be blind to a simple typo in the code.

The usual pattern with try/except is to catch exactly what you can 
handle, and let other exceptions out. That does abort your programme, 
but you get the raw exception _and_ a nice stacktrace showing where it 
came from:

    while True:
        try:
            x = closed_price
            [...]
            # ~ print('Close = ' + close_price)
            [...]
        except HTTPError as e:
            print("HTTP networking error", e)
            time.sleep(1)
            continue

Here we catch an expected failure (the network maybe went away, and the 
HTTP conencion did not work). We print that, and we sleep for 1 second 
to avoid a busy loop (spinning until the network works again). But this 
is an _expected_ exception with a well defined plan for accomodating it.  
So we catch it, report it, and continue.

Now, you may only know how to handle a specific type of an exception.  
Then the pattern is only slightly more elaborate:

    while True:
        try:
            x = closed_price
            [...]
            # ~ print('Close = ' + close_price)
            [...]
        except HTTPError as e:
            # I'm just making up some kind of test on the exception here
            if e.errno == errno.EIO:
                print("HTTP networking error", e)
                time.sleep(1)
                continue
            raise

Here, we know how to handle a specific type of HTTPError. Any others are 
just reraised, producing an error and a stack trace for you to examine.

Cheers,
Cameron Simpson <cs at cskk.id.au>