[Datetime-SIG] PEP-431/495

[Akira Li]

Tim Peters <tim.peters at gmail.com> writes:

> [Tim]
>>> ...
>>> Later you seem to say you'd prefer a 3-state flag instead, so not sure
>>> you really mean "boolean" here.
>
> [Stuart Bishop <stuart at stuartbishop.net>]
>> I write Python and SQL for a living. Booleans are 3 state to me ;)
>
> Got it!  Python is sooooooo behind the times :-)
>
>
>> In this case, I'm not fussed if the datetime instance has a 2 state or
>> 3 state flag. This is different to the various constructors which I
>> think need a 3 state flag in their arguments. True, False, None.
>
> As things seem to have progressed later, mapping pytz's explicit time
> checking into a more magical scheme sprayed all over the datetime
> internals is not straightforward,  So, as I concluded elsewhere, that
> may or may not be done someday, but it's out of scope for PEP 495.
> I'm a fan of making progress ("now is better than never", where the
> latter was PEP 431's fate waiting for perfection on all counts).
>

Even if datetime's or replace()'s *first* parameter would be 3-state
None|True|False; the internal flag can still be 2-state True|False.

first=None could cause a tzinfo callback (it implies that tzinfo must
not be None in this case) that sets *first* to True|False appropriately.

...
>> Do systems that rely on classic behavior actually exist?
>
> Of course.  A more-or-less subtle example appears later.  But we
> already mentioned dead-obvious uses:  things like "same time tomorrow"
> and "same time two weeks from now" are common as mud, and classic
> arithmetic implements them fine.  So do functions building on those
> primitives to implement more sophisticated calendar operations.  You
> might complain that naive time "same time tomorrow" makes no sense if
> someone is starting from 24 hours before what turns out to be a gap
> due to DST starting, but few in the real world schedule things at such
> times (e.g., DST transitions never occur during normal "business work
> hours"  if, e.g;, some app postpones a business meeting a week, it's
> not credible that they'll ever end up in a gap by adding
> timedelta(weeks=1) - unless they're trying to account for leap seconds
> too, and the Earth's rotation speeds up "a lot", and "same time next
> week" ends up exactly in the missing second).
>

There is a $5 wifi button that can be used to track baby data.
Python helps at various stages:
https://medium.com/@edwardbenson/how-i-hacked-amazon-s-5-wifi-button-to-track-baby-data-794214b0bdd8

Babies can poop at night and during DST transitions too. Sleep-deprived
parents should be able to see the tracking data in local time in
addition to UTC (doing timezone conversions is computer's job).

On the internet, people may cooperate while being in different time
zones i.e., even "business" software might have to work during DST
transitions. MMORPGs are probably also not limited to a single time
zone.

Non-pytz timezones make mistake on the order of an hour regularly.
It is *three orders of magnitude larger* than a second. It is a different
class of errors. The code that can't handle ~1s errors over short period
 of time should use time.monotonic() anyway.

...
>> It requires someone to have explicitly chosen to use daylight savings
>> capable timezones, without using pytz, while at the same time relying on
>> classic's surprising arithmetic. Maybe systems using dateutils without
>> using dateutils' implementation of datetime arithmetic.
>
> ? dateutil doesn't implement arithmetic that I know of, apart from
> "relative deltas".  It inherits Python's classic arithmetic for
> datetime - datetime, and datetime +/- timedelta, AFAICT.

dateutil doesn't work during DST transitions but PEP 495 might allow to
fix it.

As I understand, outside of DST transitions if dates are unique valid
local times; dateutil uses "same time tomorrow":

  (d_with_dateutil_tzinfo + DAY == 
   d.tzinfo.localize(d.replace(tzinfo=None) + DAY, is_dst=None))

while pytz uses "+24 hours":

   dt_add(d_with_dateutil_tzinfo, DAY) == d + DAY 

where dt_add() is defined below. The equility works but (d + DAY) may
have a wrong tzinfo object if the arithmetic crosses DST boundaries (but
it has correct timestamp/utc time anyway).  d.tzinfo.normalize(d + DAY)
should be used to get the correct tzinfo e.g. for displaying the result.

Both types of operations should be supported.

>...
>     def dt_add(dt, td):
>         return dt.tzinfo.fromutc(dt + (td - dt.utcoffset()))
>

>...
> Note:  my dt_add 1-liner may fail in cases starting or landing on a
> "problem time" (fold/gap).  I've never cared, because DST transitions
> are intentionally scheduled to occur "wee hours on a weekend", i.e.
> when few people are both awake and sober enough _to_ care.  But, after
> 495 tzinfos are available, the dt_add 1-liner will always work
> correctly.  That this implementation of timeline arithmetic _can_
> screw up now has nothing to do with its code, it's inherited from the
> inability of pure conversion to always work right now.
>

Such choices should make an application developer, not a library/language
developer. library/language should avoid silent errors as much as possible.

>> I think this is a bug worth fixing rather than entrenching, before
>> adding any dst aware tzinfo implementations to stdlib (including
>> 'local').
>
> datetime was released a dozen years ago.  There's nothing it does that
> wasn't already thoroughly entrenched a decade ago.

pytz is widely used. datetime objects with dateutil and pytz tzinfo
behave differently as shown above.

There are no non-fixed tzinfos in stdlib. dst-tzinfo in stdlib could
adopt either pytz or dateutil behavior.

If dateutil can be fixed to work correctly using the disambiguation flag
then its behavior is preferable because it eliminates localize,
normalize calls except localize() could be useful in __new__ if first
parameter is None to raise an exception for invalid input otherwise it
is equivalent to the default *first* value.

>> ...
>> However... this also means the new flag on the datetime instances is
>> largely irrelevant to pytz.  pytz' API will need to remain the same.
>
> My hope was that 495 alone would at least spare pytz's users from
> needing to do a `.normalize()` dance after `.astimezone()` anymore.
> Although I'm not clear on why it's needed even now.
>

As far as I know, normalize() is not necessary after astimezone() even
now https://answers.launchpad.net/pytz/+question/249229

>> ...
>> For pytz users, being able to write a function do tell if the data you
>> were given is broken is a step backwards. When constructing a datetime
>> instance with pytz, users have the choice of raising exceptions or
>> having pytz normalize the input. They are never given broken data (by
>> their definition), and there is no need to weed it out.
>
> Assuming they follow all "the rules", yes?  For example, if they
> forget to use .localize(), etc, it seems like anything could happen.
> What if they use .replace()?:  .combine()?  Unpickle a datetime
> representing a missing time?  Etc.  I don't see that pytz has anything
> magical to check datetimes created by those.

If people forget localize() then tzinfo is not attached and an exception
is raised later. It is like mixing bytes and Unicode: if you forget
decode() then an exception is raised later.

replace() is just a shortcut for a constructor.
combine() returns naive objects.

You can unpickle non-normalized datetime. replace(first=None) may force
normalization.

Your program may avoid producing non-normalized values. You can choose
to save utc+tzid (to use whatever tzdata is available) or
utc+tzid+tzdata-version (if you need the same local time) to restore it
later.

>> ...
>> I think all functions that can create datetime instances will need the
>> new optional flag and the flag should be tri-state, defaulting to not
>> whine.
>
> See the "PEP-495 - Strict Invalid Time Checking" thread for more.
> There seems to be increasing "feature creep" here.  Rewriting vast
> swaths of datetime internals to cater to this is at best impractical,
> especially compared to supplying a "check this datetime" function
> users who care can call when they care.  Nevertheless, it's a suitable
> subject for a different PEP.  I don't want to bog 495 down with it.
> If it had _stopped_ with asking for an optional check in the datetime
> constructor, it may have been implemented already ;-)

It may be a subject for another PEP but here's a possible implementation:

  class datetime:
    def __new__(...):
      if first is None and hasattr(tzinfo, 'localize'):
        self = tzinfo.localize(naive, is_dst=None) # may raise InvalidTime
  
note: self.first is never None i.e., utcoffset(), tzname(), dst() etc always
see either first=True or first=False.