list slice and generators

[Peter Otten]

Pavlos Parissis wrote:

> Hi,
> 
> Do you see any possible dangerous hidden bug in the below code(using
> python2.7 and python3.4)?
> 
> My goal is to avoid go through the metrics list twice. But, I don't
> know if there will be a problem with doing in place replace of list
> elements using 2 generators.
> 
> # metrics = ['', '0', '10'....]
> metrics = [x.metric(name) for x in self._server_per_proc]
> metrics[:] = (converter(x) for x in metrics)
> metrics[:] = (x for x in metrics if x is not None)

Both generators are executed immediately, and the right side is always 
evaluated before the the slice assignment. Try

metrics = (x.metric(name) for x in self._server_per_proc)
metrics = (converter(x) for x in metrics)
metrics = [x for x in metrics if x is not None]

or

metrics = (converter(x.metric(name)) for x in self._server_per_proc)
metrics = [x for x in metrics if x is not None]

to get down to one intermediate list. Avoiding the last one is a bit tricky:

metrics = (converter(x.metric(name)) for x in self._server_per_proc)
metrics = (x for x in metrics if x is not None)
try:
    # if there is at least one item the generator is not empty
    first = next(metrics)
except StopIteration:
    metrics = ()
else:
    # put the first item back in
    metrics = itertools.chain([first], metrics)
    assert metrics

> return calculate(name, metrics)
> 
> 
> def calculate(name, metrics):
>      if not metrics:
>         return None
> 
>     if name in METRICS_SUM:
>         return sum(metrics)
>     elif name in METRICS_AVG:
          # writing a function that calculates the average without
          # materialising the list left as an exercise ;)
          metrics = list(metrics) 
>         return int(sum(metrics)/len(metrics))
>     else:
>         raise ValueError("Unknown type of calculation for
>         {}".format(name))
> 
> 
> def converter(value):
>     try:
>         return int(float(value))
>     except ValueError:
>         return value.strip() or None
>     except TypeError:
>         return None

Disclaimer: all code untested.