From ncoghlan at gmail.com Wed Feb 1 15:40:11 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Feb 2017 21:40:11 +0100 Subject: [Python-ideas] [docs] https://docs.python.org/fr/ ? In-Reply-To: References: <56ED7EBD.5050804@palard.fr> Message-ID: On 30 January 2017 at 19:03, Brett Cannon wrote: > On Mon, 30 Jan 2017 at 07:16 Berker Peksa? wrote: >> +1 for b) or any idea that would indicate that the Python developers >> don't maintain translations of the official documentation. I don't >> have a strong opinion on naming the GitHub organization (maybe >> python-docs-translations?) but that can be discussed later. Another >> advantage of this approach is that you can have separate issue >> trackers for each language (e.g. python-docs-fr) so people can easily >> report documentation issues in their native languages. > > Does hosting on Read the Docs makes any of this easier/harder? RTD models translations as separate projects, so it should make it easier: http://docs.readthedocs.io/en/latest/localization.html#project-with-multiple-translations Most importantly, because they're separate projects, each translation will get to set its own project home page and source control repo reference. It will still be up to translation authors to make some of the manual changes that Terry mentions to point people to the right issue trackers, but it shouldn't be any worse than what we see with occasionally redirecting people to the Python Packaging Authority issue trackers for various purposes. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Feb 1 15:47:58 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Feb 2017 21:47:58 +0100 Subject: [Python-ideas] A decorator to call super() In-Reply-To: References: <6c998ac7-fb4c-5818-67a3-6a906538474e@nedbatchelder.com> Message-ID: On 31 January 2017 at 21:55, Joao S. O. Bueno wrote: > Sure - thanks - I did not even consider the descriptor mechanism, as > I got focused in getting the equivalent from the __class__ cell > inside the decorator code. > > And of course, now there is the "__init_subclass__" mechanism - a > mixin version using that was as straight forward as it can be as well. Folks that are interested in these kinds of ideas may also want to take a look at Fraser Tweedale's "Elk" project and see what would be involved in porting the "method modifiers" feature to Python 3: https://frasertweedale.github.io/elk/modifiers.html Cheers, Nick. P.S. I don't know of anyone actually using Elk in production, but it's a good project to explore for anyone interested in more structured approaches to managing method overrides -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From songofacandy at gmail.com Thu Feb 2 05:12:24 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 2 Feb 2017 19:12:24 +0900 Subject: [Python-ideas] [docs] https://docs.python.org/fr/ ? In-Reply-To: References: <56ED7EBD.5050804@palard.fr> Message-ID: >> Does hosting on Read the Docs makes any of this easier/harder? > > RTD models translations as separate projects, so it should make it > easier: http://docs.readthedocs.io/en/latest/localization.html#project-with-multiple-translations > > Most importantly, because they're separate projects, each translation > will get to set its own project home page and source control repo > reference. I haven't know RTD supports i18n! But we split repository of Python and *.po files. So we need Travis or other CI server. Current our build script is here. https://github.com/python-doc-ja/py35-locale/blob/master/.travis.yml From cory at lukasa.co.uk Thu Feb 2 05:38:25 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 2 Feb 2017 10:38:25 +0000 Subject: [Python-ideas] Unified TLS API for Python Message-ID: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> All, A recent change in the Fastly policy has led the distutils-sig to discover that there is a very real risk of many macOS installations of Python being entirely unable to access PyPI using pip, due to the fact that they are linked against the ancient system OpenSSL (see [1]). This problem ultimately stems from the fact that Python has never exposed the ability to use TLS backends other than OpenSSL in the standard library. As Christian and I discussed back at PyCon US 2015, the ssl module would more properly be called the openssl module, due to exposing many OpenSSL-specific concepts and behaviours in its API. This has meant that the Python ecosystem is overwhelmingly also an OpenSSL ecosystem, which is problematic on Windows and macOS (and Unices that for whatever reason aren?t interested in shipping an OpenSSL), as it has meant that Python needs to bring its own OpenSSL, and that it is troublesome to interact with the system trust database. The first step to resolving this would be to provide a new module that exposes TLS concepts in a much more generic manner. There are two possible approaches to this. The first is to have this module be a generic concrete implementation that can be compiled against multiple TLS backends (like curl). This would require that all the relevant bindings be built into the standard library from the get-go, which provides a substantial maintenance burden on a team of people that are already understaffed maintaining the ssl module alone. The second approach is to define a generic high-level TLS interface that provides a minimal usable interface that can be implemented by both first- and third-party modules. This would allow the standard library to continue to ship with only exactly what it needs (for example, OpenSSL, SecureTransport and SChannel), but would give those who want to use more esoteric TLS choices (NSS, GnuTLS, mbedTLS, and BoringSSL are some examples) an API that they can implement that will guarantee that complying modules can use the appropriate TLS backend. To that end, I?ve composed a draft PEP that would define this API. The current copy can be found on GitHub[2] and is also provided below. This draft PEP has been under discussion in the Python Security-SIG since the start of January, and has gone through a number of revisions. This is the next step in moving this forward: exposing the wider Python development community to the PEP for further revision before it is proposed to python-dev. Please note that this proposal is not intended to resolve the current problem pip has with TLSv1.2, and so is not subject to the tight time constraints associated with that problem. That problem will be solved by building a temporary shim into urllib3 that can use SecureTransport bindings on the Mac. This proposal is intended as a solution to the more-general problem of supporting different TLS backends in Python, and so is larger in scope and less urgent. I should also mention that there will be a tendency to want to make this API all things to all people from the get-go. I?m going to strongly resist attempts to extend this API too much, because each additional bit of API surface makes it harder for us to encourage module authors to conform to this API. I will encourage people to extend this API over time as needed, but to begin with I think it is most important that basic TLS clients and servers can be constructed with this API. More specialised features should be considered future enhancements, rather than being tacked on to this initial PEP. Please let me know what you think. Cory [1]: https://mail.python.org/pipermail/distutils-sig/2017-January/029970.html [2]: https://github.com/Lukasa/peps/pull/1 ? PEP: XXX Title: TLS Abstract Base Classes Version: $Revision$ Last-Modified: $Date$ Author: Cory Benfield Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 17-Oct-2016 Python-Version: 3.7 Post-History: 30-Aug-2002 Abstract ======== This PEP would define a standard TLS interface in the form of a collection of abstract base classes. This interface would allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL that can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL. Rationale ========= In the 21st century it has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the `ssl module`_, which provides a Python API to the `OpenSSL library`_. Because the ``ssl`` module is distributed with the Python standard library, it has become the overwhelmingly most-popular method for handling TLS in Python. An extraordinary majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ``ssl`` module for their TLS connectivity. Unfortunately, the preeminence of the ``ssl`` module has had a number of unforeseen side-effects that have had the effect of tying the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide "platform-native" experiences. Problems -------- The fact that the ``ssl`` module is build into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues: * It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers. * For Windows distributions of Python, they need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released. * For macOS distributions of Python, they need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows. * Many systems, including but not limited to Windows and macOS, do not make their system certificate stores available to OpenSSL. This forces users to either obtain their trust roots from elsewhere (e.g. `certifi`_) or to attempt to export their system trust stores in some form. Relying on `certifi`_ is less than ideal, as most system administrators do not expect to receive security-critical software updates from PyPI. Additionally, it is not easy to extend the `certifi`_ trust bundle to include custom roots, or to centrally manage trust using the `certifi`_ model. Even in situations where the system certificate stores are made available to OpenSSL in some form, the experience is still sub-standard, as OpenSSL will perform different validation checks than the platform-native TLS implementation. This can lead to users experiencing different behaviour on their browsers or other platform-native tools than they experience in Python, with little or no recourse to resolve the problem. * Users may wish to integrate with TLS libraries other than OpenSSL for many other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or because OpenSSL is simply too large and unweildy for the platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ``ssl`` module API. Additionally, the ``ssl`` module as implemented today limits the ability of CPython itself to add support for alternative TLS backends, or remove OpenSSL support entirely, should either of these become necessary or useful. The ``ssl`` module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS backend. Proposal ======== This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these abstract base classes wherever possible. There are three goals here: 1. To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations. 2. To provide an API that has few or no OpenSSL-specific concepts leak through. The ``ssl`` module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new ABCs would remove those specific concepts. 3. To provide a path for the core development team to make OpenSSL one of many possible TLS backends, rather than requiring that it be present on a system in order for Python to have TLS support. The proposed interface is laid out below. Abstract Base Classes --------------------- There are several interfaces that require standardisation. Those interfaces are: 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the ``ssl`` module. 2. Wrapping a socket object, currently implemented by the `SSLSocket`_ class in the ``ssl`` module. 3. Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the `SSLObject`_ class in the ``ssl`` module. 4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently this is also implemented by the `SSLContext`_ class in the ``ssl`` module. 5. Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings. 6. Specifying application-layer protocols that can be negotiated during the TLS handshake. 7. Specifying TLS versions. 8. Reporting errors to the caller, currently implemented by the `SSLError`_ class in the ``ssl`` module. 9. Specifying certificates to load, either as client or server certificates. 10. Specifying which trust database should be used to validate certificates presented by a remote peer. 11. Finding a way to get hold of these interfaces at run time. While it is technically possible to define (2) in terms of (3), for the sake of simplicity it is easier to define these as two separate ABCs. Implementations are of course free to implement the concrete subclasses however they see fit. Obviously, (5) doesn't require an abstract base class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations. (9) is a thorny problem, becuase in an ideal world the private keys associated with these certificates would never end up in-memory in the Python process (that is, the TLS library would collaborate with a Hardware Security Module (HSM) to provide the private key in such a way that it cannot be extracted from process memory). Thus, we need to provide an extensible model of providing certificates that allows concrete implementations the ability to provide this higher level of security, while also allowing a lower bar for those implementations that cannot. This lower bar would be the same as the status quo: that is, the certificate may be loaded from an in-memory buffer or from a file on disk. (10) also represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. Some implementations have specific trust store formats that only they can use (such as the OpenSSL CA directory format that is created by ``c_rehash``), and others may not allow you to specify a trust store that does not include their default trust store. For this reason, we need to provide a model that assumes very little about the form that trust stores take. The "Trust Store" section below goes into more detail about how this is achieved. Finally, this API will split the responsibilities currently assumed by the `SSLContext`_ object: specifically, the responsibility for holding and managing configuration and the responsibility for using that configuration to build wrapper objects. This is necessarily primarily for supporting functionality like Server Name Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server has the ability to modify the TLS configuration in response to the client telling the server what hostname it is trying to reach. This is mostly used to change certificate chain so as to present the correct TLS certificate chain for the given hostname. The specific mechanism by which this is done is by returning a new `SSLContext`_ object with the appropriate configuration. This is not a model that maps well to other TLS implementations. Instead, we need to make it possible to provide a return value from the SNI callback that can be used to indicate what configuration changes should be made. This means providing an object that can hold TLS configuration. This object needs to be applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. For this reason, we split the responsibility of `SSLContext`_ into two separate objects. The ``TLSConfiguration`` object is an object that acts as container for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects are objects that are instantiated with a ``TLSConfiguration`` object. Both objects would be immutable. Configuration ~~~~~~~~~~~~~ The ``TLSConfiguration`` concrete class defines an object that can hold and manage TLS configuration. The goals of this class are as follows: 1. To provide a method of specifying TLS configuration that avoids the risk of errors in typing (this excludes the use of a simple dictionary). 2. To provide an object that can be safely compared to other configuration objects to detect changes in TLS configuration, for use with the SNI callback. This class is not an ABC, primarily because it is not expected to have implementation-specific behaviour. The responsibility for transforming a ``TLSConfiguration`` object into a useful set of configuration for a given TLS implementation belongs to the Context objects discussed below. This class has one other notable property: it is immutable. This is a desirable trait for a few reasons. The most important one is that it allows these objects to be used as dictionary keys, which is potentially extremely valuable for certain TLS backends and their SNI configuration. On top of this, it frees implementations from needing to worry about their configuration objects being changed under their feet, which allows them to avoid needing to carefully synchronize changes between their concrete data structures and the configuration object. This object is extendable: that is, future releases of Python may add configuration fields to this object as they become useful. For backwards-compatibility purposes, new fields are only appended to this object. Existing fields will never be removed, renamed, or reordered. The ``TLSConfiguration`` object would be defined by the following code:: ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] _configuration_fields = [ 'validate_certificates', 'certificate_chain', 'ciphers', 'inner_protocols', 'lowest_supported_version', 'highest_supported_version', 'trust_store', 'sni_callback', ] _DEFAULT_VALUE = object() class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): """ An immutable TLS Configuration object. This object has the following properties: :param validate_certificates bool: Whether to validate the TLS certificates. This switch operates at a very broad scope: either validation is enabled, in which case all forms of validation are performed including hostname validation if possible, or validation is disabled, in which case no validation is performed. Not all backends support having their certificate validation disabled. If a backend does not support having their certificate validation disabled, attempting to set this property to ``False`` will throw a ``TLSError`` when this object is passed into a context object. :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The certificate, intermediate certificate, and the corresponding private key for the leaf certificate. These certificates will be offered to the remote peer during the handshake if required. The first Certificate in the list must be the leaf certificate. All subsequent certificates will be offered as intermediate additional certificates. :param ciphers Tuple[CipherSuite]: The available ciphers for TLS connections created with this configuration, in priority order. :param inner_protocols Tuple[Union[NextProtocol, bytes]]: Protocols that connections created with this configuration should advertise as supported during the TLS handshake. These may be advertised using either or both of ALPN or NPN. This list of protocols should be ordered by preference. :param lowest_supported_version TLSVersion: The minimum version of TLS that should be allowed on TLS connections using this configuration. :param highest_supported_version TLSVersion: The maximum version of TLS that should be allowed on TLS connections using this configuration. :param trust_store TrustStore: The trust store that connections using this configuration will use to validate certificates. :param sni_callback Optional[ServerNameCallback]: A callback function that will be called after the TLS Client Hello handshake message has been received by the TLS server when the TLS client specifies a server name indication. Only one callback can be set per ``TLSConfiguration``. If the ``sni_callback`` is ``None`` then the callback is disabled. If the ``TLSConfiguration`` is used for a ``ClientContext`` then this setting will be ignored. The ``callback`` function will be called with three arguments: the first will be the ``TLSBufferObject`` for the connection; the second will be a string that represents the server name that the client is intending to communicate (or ``None`` if the TLS Client Hello does not contain a server name); and the third argument will be the original ``Context``. The server name argument will be the IDNA *decoded* server name. The ``callback`` must return a ``TLSConfiguration`` to allow negotiation to continue. Other return values signal errors. Attempting to control what error is signaled by the underlying TLS implementation is not specified in this API, but is up to the concrete implementation to handle. The Context will do its best to apply the ``TLSConfiguration`` changes from its original configuration to the incoming connection. This will usually include changing the certificate chain, but may also include changes to allowable ciphers or any other configuration settings. """ __slots__ = () def __new__(cls, validate_certificates=None: Optional[bool], certificate_chain=None: Optional[Tuple[Tuple[Certificate], PrivateKey]], ciphers=None: Optional[Tuple[CipherSuite]], inner_protocols=None: Optional[Tuple[Union[NextProtocol, bytes]]], lowest_supported_version=None: Optional[TLSVersion], highest_supported_version=None: Optional[TLSVersion], trust_store=None: Optional[TrustStore], sni_callback=None: Optional[ServerNameCallback]): if validate_certificates is None: validate_certificates = True if ciphers is None: ciphers = DEFAULT_CIPHER_LIST if inner_protocols is None: inner_protocols = [] if lowest_supported_version is None: lowest_supported_version = TLSVersion.TLSv1 if highest_supported_version is None: highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED return super().__new__( cls, validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) def update(self, validate_certificates=_DEFAULT_VALUE, certificate_chain=_DEFAULT_VALUE, ciphers=_DEFAULT_VALUE, inner_protocols=_DEFAULT_VALUE, lowest_supported_version=_DEFAULT_VALUE, highest_supported_version=_DEFAULT_VALUE, trust_store=_DEFAULT_VALUE, sni_callback=_DEFAULT_VALUE): """ Create a new ``TLSConfiguration``, overriding some of the settings on the original configuration with the new settings. """ if validate_certificates is _DEFAULT_VALUE: validate_certificates = self.validate_certificates if certificate_chain is _DEFAULT_VALUE: certificate_chain = self.certificate_chain if ciphers is _DEFAULT_VALUE: ciphers = self.ciphers if inner_protocols is _DEFAULT_VALUE: inner_protocols = self.inner_protocols if lowest_supported_version is _DEFAULT_VALUE: lowest_supported_version = self.lowest_supported_version if highest_supported_version is _DEFAULT_VALUE: highest_supported_version = self.highest_supported_version if trust_store is _DEFAULT_VALUE: trust_store = self.trust_store if sni_callback is _DEFAULT_VALUE: sni_callback = self.sni_callback return self.__class__( validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) Context ~~~~~~~ We define two Context abstract base classes. These ABCs define objects that allow configuration of TLS to be applied to specific connections. They can be thought of as factories for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` objects. Unlike the current ``ssl`` module, we provide two context classes instead of one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` classes. This simplifies the APIs (for example, there is no sense in the server providing the ``server_hostname`` parameter to ``ssl.SSLContext.wrap_socket``, but because there is only one context class that parameter is still available), and ensures that implementations know as early as possible which side of a TLS connection they will serve. Additionally, it allows implementations to opt-out of one or either side of the connection. For example, SChannel on macOS is not really intended for server use and has an enormous amount of functionality missing for server-side use. This would allow SChannel implementations to simply not define a concrete subclass of ``ServerContext`` to signal their lack of support. As much as possible implementers should aim to make these classes immutable: that is, they should prefer not to allow users to mutate their internal state directly, instead preferring to create new contexts from new TLSConfiguration objects. Obviously, the ABCs cannot enforce this constraint, and so they do not attempt to. The ``Context`` abstract base class has the following class definition:: TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] class _BaseContext(metaclass=ABCMeta): @abstractmethod def __init__(self, configuration: TLSConfiguration): """ Create a new context object from a given TLS configuration. """ @property @abstractmethod def configuration(self) -> TLSConfiguration: """ Returns the TLS configuration that was used to create the context. """ class ClientContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, server_hostname: Optional[str], auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. This is also used to validate the TLS certificate for the given hostname. If hostname validation is not desired, then pass ``None`` for this parameter. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any, server_hostname: Optional[str]) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The buffer objects must be either file objects or objects that implement the buffer protocol. The ``server_hostname`` parameter has the same meaning as in ``wrap_socket``. """ class ServerContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.accept()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The buffer objects must be either file objects or objects that implement the buffer protocol. """ Socket ~~~~~~ The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which has the following definition:: class TLSWrappedSocket(metaclass=ABCMeta): # The various socket methods all must be implemented. Their definitions # have been elided from this class defintion in the PEP because they # aren't instructive. @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def unwrap(self) -> socket.socket: """ Cleanly terminate the TLS connection on this wrapped socket. Once called, this ``TLSWrappedSocket`` can no longer be used to transmit data. Returns the socket that was wrapped with TLS. """ Buffer ~~~~~~ The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which has the following definition:: class TLSWrappedBuffer(metaclass=ABCMeta): @abstractmethod def read(self, amt=None: int) -> bytes: """ Read up to ``amt`` bytes of data from the input buffer and return the result as a ``bytes`` instance. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``read()`` can also cause write operations. """ @abstractmethod def readinto(self, buffer: Any, amt=None: int) -> int: """ Read up to ``amt`` bytes of data from the input buffer into ``buffer``, which must be an object that implements the buffer protocol. Returns the number of bytes read. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``, or until the buffer is full. Raises ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``readinto()`` can also cause write operations. """ @abstractmethod def write(self, buf: Any) -> int: """ Write ``buf`` in encrypted form to the output buffer and return the number of bytes written. The ``buf`` argument must be an object supporting the buffer interface. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``write()`` can also cause read operations. """ @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def shutdown(self) -> None: """ Performs a clean TLS shut down. This should generally be used whenever possible to signal to the remote peer that the content is finished. """ Cipher Suites ~~~~~~~~~~~~~ Supporting cipher suites in a truly library-agnostic fashion is a remarkably difficult undertaking. Different TLS implementations often have *radically* different APIs for specifying cipher suites, but more problematically these APIs frequently differ in capability as well as in style. Some examples are shown below: OpenSSL ^^^^^^^ OpenSSL uses a well-known cipher string format. This format has been adopted as a configuration language by most products that use OpenSSL, including Python. This format is relatively easy to read, but has a number of downsides: it is a string, which makes it remarkably easy to provide bad inputs; it lacks much detailed validation, meaning that it is possible to configure OpenSSL in a way that doesn't allow it to negotiate any cipher at all; and it allows specifying cipher suites in a number of different ways that make it tricky to parse. The biggest problem with this format is that there is no formal specification for it, meaning that the only way to parse a given string the way OpenSSL would is to get OpenSSL to parse it. OpenSSL's cipher strings can look like this:: 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' This string demonstrates some of the complexity of the OpenSSL format. For example, it is possible for one entry to specify multiple cipher suites: the entry ``ECDH+AESGCM`` means "all ciphers suites that include both elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". More explicitly, that will expand to four cipher suites:: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" That makes parsing a complete OpenSSL cipher string extremely tricky. Add to the fact that there are other meta-characters, such as "!" (exclude all cipher suites that match this criterion, even if they would otherwise be included: "!MD5" means that no cipher suites using the MD5 hash algorithm should be included), "-" (exclude matching ciphers if they were already included, but allow them to be re-added later if they get included again), and "+" (include the matching ciphers, but place them at the end of the list), and you get an *extremely* complex format to parse. On top of this complexity it should be noted that the actual result depends on the OpenSSL version, as an OpenSSL cipher string is valid so long as it contains at least one cipher that OpenSSL recognises. OpenSSL also uses different names for its ciphers than the names used in the relevant specifications. See the manual page for ``ciphers(1)`` for more details. The actual API inside OpenSSL for the cipher string is simple:: char *cipher_list = ; int rc = SSL_CTX_set_cipher_list(context, cipher_list); This means that any format that is used by this module must be able to be converted to an OpenSSL cipher string for use with OpenSSL. SecureTransport ^^^^^^^^^^^^^^^ SecureTransport is the macOS system TLS library. This library is substantially more restricted than OpenSSL in many ways, as it has a much more restricted class of users. One of these substantial restrictions is in controlling supported cipher suites. Ciphers in SecureTransport are represented by a C ``enum``. This enum has one entry per cipher suite, with no aggregate entries, meaning that it is not possible to reproduce the meaning of an OpenSSL cipher string like "ECDH+AESGCM" without hand-coding which categories each enum member falls into. However, the names of most of the enum members are in line with the formal names of the cipher suites: that is, the cipher suite that OpenSSL calls "ECDHE-ECDSA-AES256-GCM-SHA384" is called "TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. The API for configuring cipher suites inside SecureTransport is simple:: SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ...}; OSStatus status = SSLSetEnabledCiphers(context, ciphers, sizeof(cphers)); SChannel ^^^^^^^^ SChannel is the Windows system TLS library. SChannel has extremely restrictive support for controlling available TLS cipher suites, and additionally adopts a third method of expressing what TLS cipher suites are supported. Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). Each of these constants does not refer to an entire cipher suite, but instead an individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, which refer to the bulk encryption algorithm used in a cipher suite, ``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key exchange algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which refer to the message authentication code used in a cipher suite, and ``CALG_ECDSA`` and ``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange algorithm. This can be thought of as the half of OpenSSL's functionality that SecureTransport doesn't have: SecureTransport only allows specifying exact cipher suites, while SChannel only allows specifying *parts* of the cipher suite, while OpenSSL allows both. Determining which cipher suites are allowed on a given connection is done by providing a pointer to an array of these ``ALG_ID`` constants. This means that any suitable API must allow the Python code to determine which ``ALG_ID`` constants must be provided. Proposed Interface ^^^^^^^^^^^^^^^^^^ The proposed interface for the new module is influenced by the combined set of limitations of the above implementations. Specifically, as every implementation *except* OpenSSL requires that each individual cipher be provided, there is no option but to provide that lowest-common denominator approach. The simplest approach is to provide an enumerated type that includes all of the cipher suites defined for TLS. The values of the enum members will be their two-octet cipher identifier as used in the TLS handshake, stored as a tuple of integers. The names of the enum members will be their IANA-registered cipher suite names. Rather than populate this enum by hand, it is likely that we'll define a script that can build it from Christian Heimes' `tlsdb JSON file`_ (warning: large file). This also opens up the possibility of extending the API with additional querying function, such as determining which TLS versions support which ciphers, if that functionality is found to be useful or necessary. If users find this approach to be onerous, a future extension to this API can provide helpers that can reintroduce OpenSSL's aggregation functionality. Because this enum would be enormous, the entire enum is not provided here. Instead, a small sample of entries is provided to give a flavor of how it will appear. :: class CipherSuite(Enum): ... TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = (0xC0, 0x12) ... TLS_ECDHE_ECDSA_WITH_AES_128_CCM = (0xC0, 0xAC) ... TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = (0xC0, 0x2B) ... Protocol Negotiation ~~~~~~~~~~~~~~~~~~~~ Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 handshake. While NPN and ALPN are, at their fundamental level, built on top of bytestrings, string-based APIs are frequently problematic as they allow for errors in typing that can be hard to detect. For this reason, this module would define a type that protocol negotiation implementations can pass and be passed. This type would wrap a bytestring to allow for aliases for well-known protocols. This allows us to avoid the problems inherent in typos for well-known protocols, while allowing the full extensibility of the protocol negotiation layer if needed by letting users pass byte strings directly. :: class NextProtocol(Enum): H2 = b'h2' H2C = b'h2c' HTTP1 = b'http/1.1' WEBRTC = b'webrtc' C_WEBRTC = b'c-webrtc' FTP = b'ftp' STUN = b'stun.nat-discovery' TURN = b'stun.turn' TLS Versions ~~~~~~~~~~~~ It is often useful to be able to restrict the versions of TLS you're willing to support. There are many security advantages in refusing to use old versions of TLS, and some misbehaving servers will mishandle TLS clients advertising support for newer versions. The following enumerated type can be used to gate TLS versions. Forward-looking applications should almost never set a maximum TLS version unless they absolutely must, as a TLS backend that is newer than the Python that uses it may support TLS versions that are not in this enumerated type. Additionally, this enumerated type defines two additional flags that can always be used to request either the lowest or highest TLS version supported by an implementation. :: class TLSVersion(Enum): MINIMUM_SUPPORTED SSLv2 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3 MAXIMUM_SUPPORTED Errors ~~~~~~ This module would define three base classes for use with error handling. Unlike many of the the other classes defined here, these classes are not abstract, as they have no behaviour. They exist simply to signal certain common behaviours. Backends should subclass these exceptions in their own packages, but needn't define any behaviour for them. In general, concrete implementations should subclass these exceptions rather than throw them directly. This makes it moderately easier to determine which concrete TLS implementation is in use during debugging of unexpected errors. However, this is not mandatory. The definitions of the errors are below:: class TLSError(Exception): """ The base exception for all TLS related errors from any backend. Catching this error should be sufficient to catch *all* TLS errors, regardless of what backend is used. """ class WantWriteError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is written to the network, or until the output buffer is drained. """ class WantReadError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is read from the network, or until more data is available in the input buffer. """ Certificates ~~~~~~~~~~~~ This module would define an abstract X509 certificate class. This class would have almost no behaviour, as the goal of this module is not to provide all possible relevant cryptographic functionality that could be provided by X509 certificates. Instead, all we need is the ability to signal the source of a certificate to a concrete implementation. For that reason, this certificate implementation defines only constructors. In essence, the certificate object in this module could be as abstract as a handle that can be used to locate a specific certificate. Concrete implementations may choose to provide alternative constructors, e.g. to load certificates from HSMs. If a common interface emerges for doing this, this module may be updated to provide a standard constructor for this use-case as well. Concrete implementations should aim to have Certificate objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class Certificate(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes) -> Certificate: """ Creates a Certificate object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, AnyStr]) -> Certificate: """ Creates a Certificate object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. """ Private Keys ~~~~~~~~~~~~ This module would define an abstract private key class. Much like the Certificate class, this class has almost no behaviour in order to give as much freedom as possible to the concrete implementations to treat keys carefully. This class has all the caveats of the ``Certificate`` class. :: class PrivateKey(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes, password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN", the key type, and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. The key may additionally be encrypted. If it is, the ``password`` argument can be used to decrypt the key. The ``password`` argument may be a function to call to get the password for decrypting the private key. It will only be called if the private key is encrypted and a password is necessary. It will be called with no arguments, and it should return either bytes or bytearray containing the password. Alternatively a bytes, or bytearray value may be supplied directly as the password argument. It will be ignored if the private key is not encrypted and no password is needed. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, bytes, str], password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. The ``password`` parameter behaves exactly as the equivalent parameter on ``from_buffer``. """ Trust Store ~~~~~~~~~~~ As discussed above, loading a trust store represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. For this reason, we need to provide a model that assumes very little about the form that trust stores take. This problem is the same as the one that the Certificate and PrivateKey types need to solve. For this reason, we use the exact same model, by creating an opaque type that can encapsulate the various means that TLS backends may open a trust store. A given TLS implementation is not required to implement all of the constructors. However, it is strongly recommended that a given TLS implementation provide the ``system`` constructor if at all possible, as this is the most common validation trust store that is used. Concrete implementations may also add their own constructors. Concrete implementations should aim to have TrustStore objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class TrustStore(metaclass=ABCMeta): @abstractclassmethod def system(cls) -> TrustStore: """ Returns a TrustStore object that represents the system trust database. """ @abstractclassmethod def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> TrustStore: """ Initializes a trust store from a single file full of PEMs. """ Runtime Access ~~~~~~~~~~~~~~ A not-uncommon use case for library users is to want to allow the library to control the TLS configuration, but to want to select what backend is in use. For example, users of Requests may want to be able to select between OpenSSL or a platform-native solution on Windows and macOS, or between OpenSSL and NSS on some Linux platforms. These users, however, may not care about exactly how their TLS configuration is done. This poses a problem: given an arbitrary concrete implementation, how can a library work out how to load certificates into the trust store? There are two options: either all concrete implementations can be required to fit into a specific naming scheme, or we can provide an API that makes it possible to grab these objects. This PEP proposes that we use the second approach. This grants the greatest freedom to concrete implementations to structure their code as they see fit, requiring only that they provide a single object that has the appropriate properties in place. Users can then pass this "backend" object to libraries that support it, and those libraries can take care of configuring and using the concrete implementation. All concrete implementations must provide a method of obtaining a ``Backend`` object. The ``Backend`` object can be a global singleton or can be created by a callable if there is an advantage in doing that. The ``Backend`` object has the following definition:: Backend = namedtuple( 'Backend', ['client_context', 'server_context', 'certificate', 'private_key', 'trust_store'] ) Each of the properties must provide the concrete implementation of the relevant ABC. This ensures that code like this will work for any backend:: trust_store = backend.trust_store.system() Changes to the Standard Library =============================== The portions of the standard library that interact with TLS should be revised to use these ABCs. This will allow them to function with other TLS backends. This includes the following modules: - asyncio - ftplib - http.client - imaplib - nntplib - poplib - smtplib Migration of the ssl module --------------------------- Naturally, we will need to extend the ``ssl`` module itself to conform to these ABCs. This extension will take the form of new classes, potentially in an entirely new module. This will allow applications that take advantage of the current ``ssl`` module to continue to do so, while enabling the new APIs for applications and libraries that want to use them. In general, migrating from the ``ssl`` module to the new ABCs is not expected to be one-to-one. This is normally acceptable: most tools that use the ``ssl`` module hide it from the user, and so refactoring to use the new module should be invisible. However, a specific problem comes from libraries or applications that leak exceptions from the ``ssl`` module, either as part of their defined API or by accident (which is easily done). Users of those tools may have written code that tolerates and handles exceptions from the ``ssl`` module being raised: migrating to the ABCs presented here would potentially cause the exceptions defined above to be thrown instead, and existing ``except`` blocks will not catch them. For this reason, part of the migration of the ``ssl`` module would require that the exceptions in the ``ssl`` module alias those defined above. That is, they would require the following statements to all succeed:: assert ssl.SSLError is tls.TLSError assert ssl.SSLWantReadError is tls.WantReadError assert ssl.SSLWantWriteError is tls.WantWriteError The exact mechanics of how this will be done are beyond the scope of this PEP, as they are made more complex due to the fact that the current ``ssl`` exceptions are defined in C code, but more details can be found in `an email sent to the Security-SIG by Christian Heimes`_. Future ====== Major future TLS features may require revisions of these ABCs. These revisions should be made cautiously: many backends may not be able to move forward swiftly, and will be invalidated by changes in these ABCs. This is acceptable, but wherever possible features that are specific to individual implementations should not be added to the ABCs. The ABCs should restrict themselves to high-level descriptions of IETF-specified features. However, well-justified extensions to this API absolutely should be made. The focus of this API is to provide a unifying lowest-common-denominator configuration option for the Python community. TLS is not a static target, and as TLS evolves so must this API. References ========== .. _ssl module: https://docs.python.org/3/library/ssl.html .. _OpenSSL Library: https://www.openssl.org/ .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ .. _certifi: https://pypi.org/project/certifi/ .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError .. _MSDN articles: https://msdn.microsoft.com/en-us/library/windows/desktop/mt490158(v=vs.85).aspx .. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json .. _an email sent to the Security-SIG by Christian Heimes: https://mail.python.org/pipermail/security-sig/2017-January/000213.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From robertc at robertcollins.net Thu Feb 2 06:03:22 2017 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 3 Feb 2017 00:03:22 +1300 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> Message-ID: Nice. What's needed to move this forward? On 2 Feb. 2017 11:38 pm, "Cory Benfield" wrote: > All, > > A recent change in the Fastly policy has led the distutils-sig to discover > that there is a very real risk of many macOS installations of Python being > entirely unable to access PyPI using pip, due to the fact that they are > linked against the ancient system OpenSSL (see [1]). This problem > ultimately stems from the fact that Python has never exposed the ability to > use TLS backends other than OpenSSL in the standard library. As Christian > and I discussed back at PyCon US 2015, the ssl module would more properly > be called the openssl module, due to exposing many OpenSSL-specific > concepts and behaviours in its API. This has meant that the Python > ecosystem is overwhelmingly also an OpenSSL ecosystem, which is problematic > on Windows and macOS (and Unices that for whatever reason aren?t interested > in shipping an OpenSSL), as it has meant that Python needs to bring its own > OpenSSL, and that it is troublesome to interact with the system trust > database. > > The first step to resolving this would be to provide a new module that > exposes TLS concepts in a much more generic manner. There are two possible > approaches to this. The first is to have this module be a generic concrete > implementation that can be compiled against multiple TLS backends (like > curl). This would require that all the relevant bindings be built into the > standard library from the get-go, which provides a substantial maintenance > burden on a team of people that are already understaffed maintaining the > ssl module alone. The second approach is to define a generic high-level TLS > interface that provides a minimal usable interface that can be implemented > by both first- and third-party modules. This would allow the standard > library to continue to ship with only exactly what it needs (for example, > OpenSSL, SecureTransport and SChannel), but would give those who want to > use more esoteric TLS choices (NSS, GnuTLS, mbedTLS, and BoringSSL are some > examples) an API that they can implement that will guarantee that complying > modules can use the appropriate TLS backend. > > To that end, I?ve composed a draft PEP that would define this API. The > current copy can be found on GitHub[2] and is also provided below. This > draft PEP has been under discussion in the Python Security-SIG since the > start of January, and has gone through a number of revisions. This is the > next step in moving this forward: exposing the wider Python development > community to the PEP for further revision before it is proposed to > python-dev. > > Please note that this proposal is not intended to resolve the current > problem pip has with TLSv1.2, and so is not subject to the tight time > constraints associated with that problem. That problem will be solved by > building a temporary shim into urllib3 that can use SecureTransport > bindings on the Mac. This proposal is intended as a solution to the > more-general problem of supporting different TLS backends in Python, and so > is larger in scope and less urgent. > > I should also mention that there will be a tendency to want to make this > API all things to all people from the get-go. I?m going to strongly resist > attempts to extend this API too much, because each additional bit of API > surface makes it harder for us to encourage module authors to conform to > this API. I will encourage people to extend this API over time as needed, > but to begin with I think it is most important that basic TLS clients and > servers can be constructed with this API. More specialised features should > be considered future enhancements, rather than being tacked on to this > initial PEP. > > Please let me know what you think. > > Cory > > [1]: https://mail.python.org/pipermail/distutils-sig/2017- > January/029970.html > [2]: https://github.com/Lukasa/peps/pull/1 > > ? > > PEP: XXX > Title: TLS Abstract Base Classes > Version: $Revision$ > Last-Modified: $Date$ > Author: Cory Benfield > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 17-Oct-2016 > Python-Version: 3.7 > Post-History: 30-Aug-2002 > > > Abstract > ======== > > This PEP would define a standard TLS interface in the form of a collection > of > abstract base classes. This interface would allow Python implementations > and > third-party libraries to provide bindings to TLS libraries other than > OpenSSL > that can be used by tools that expect the interface provided by the Python > standard library, with the goal of reducing the dependence of the Python > ecosystem on OpenSSL. > > > Rationale > ========= > > In the 21st century it has become increasingly clear that robust and > user-friendly TLS support is an extremely important part of the ecosystem > of > any popular programming language. For most of its lifetime, this role in > the > Python ecosystem has primarily been served by the `ssl module`_, which > provides > a Python API to the `OpenSSL library`_. > > Because the ``ssl`` module is distributed with the Python standard > library, it > has become the overwhelmingly most-popular method for handling TLS in > Python. > An extraordinary majority of Python libraries, both in the standard > library and > on the Python Package Index, rely on the ``ssl`` module for their TLS > connectivity. > > Unfortunately, the preeminence of the ``ssl`` module has had a number of > unforeseen side-effects that have had the effect of tying the entire Python > ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL > even > in situations where it may provide a worse user experience than > alternative TLS > implementations, which imposes a cognitive burden and makes it hard to > provide > "platform-native" experiences. > > > Problems > -------- > > The fact that the ``ssl`` module is build into the standard library has > meant > that all standard-library Python networking libraries are entirely reliant > on > the OpenSSL that the Python implementation has been linked against. This > leads to the following issues: > > * It is difficult to take advantage of new, higher-security TLS without > recompiling Python to get a new OpenSSL. While there are third-party > bindings > to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format > that > the standard library understands, forcing projects that want to use them > to > maintain substantial compatibility layers. > > * For Windows distributions of Python, they need to be shipped with a copy > of > OpenSSL. This puts the CPython development team in the position of being > OpenSSL redistributors, potentially needing to ship security updates to > the > Windows Python distributions when OpenSSL vulnerabilities are released. > > * For macOS distributions of Python, they need either to be shipped with a > copy > of OpenSSL or linked against the system OpenSSL library. Apple has > formally > deprecated linking against the system OpenSSL library, and even if they > had > not, that library version has been unsupported by upstream for nearly one > year as of the time of writing. The CPython development team has started > shipping newer OpenSSLs with the Python available from python.org, but > this > has the same problem as with Windows. > > * Many systems, including but not limited to Windows and macOS, do not make > their system certificate stores available to OpenSSL. This forces users > to > either obtain their trust roots from elsewhere (e.g. `certifi`_) or to > attempt to export their system trust stores in some form. > > Relying on `certifi`_ is less than ideal, as most system administrators > do > not expect to receive security-critical software updates from PyPI. > Additionally, it is not easy to extend the `certifi`_ trust bundle to > include > custom roots, or to centrally manage trust using the `certifi`_ model. > > Even in situations where the system certificate stores are made > available to > OpenSSL in some form, the experience is still sub-standard, as OpenSSL > will > perform different validation checks than the platform-native TLS > implementation. This can lead to users experiencing different behaviour > on > their browsers or other platform-native tools than they experience in > Python, > with little or no recourse to resolve the problem. > > * Users may wish to integrate with TLS libraries other than OpenSSL for > many > other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), > or > because OpenSSL is simply too large and unweildy for the platform (e.g. > for > embedded Python). Those users are left with the requirement to use > third-party networking libraries that can interact with their preferred > TLS > library or to shim their preferred library into the OpenSSL-specific > ``ssl`` > module API. > > Additionally, the ``ssl`` module as implemented today limits the ability of > CPython itself to add support for alternative TLS backends, or remove > OpenSSL > support entirely, should either of these become necessary or useful. The > ``ssl`` module exposes too many OpenSSL-specific function calls and > features to > easily map to an alternative TLS backend. > > > Proposal > ======== > > This PEP proposes to introduce a few new Abstract Base Classes in Python > 3.7 to > provide TLS functionality that is not so strongly tied to OpenSSL. It also > proposes to update standard library modules to use only the interface > exposed > by these abstract base classes wherever possible. There are three goals > here: > > 1. To provide a common API surface for both core and third-party > developers to > target their TLS implementations to. This allows TLS developers to > provide > interfaces that can be used by most Python code, and allows network > developers to have an interface that they can target that will work > with a > wide range of TLS implementations. > 2. To provide an API that has few or no OpenSSL-specific concepts leak > through. > The ``ssl`` module today has a number of warts caused by leaking OpenSSL > concepts through to the API: the new ABCs would remove those specific > concepts. > 3. To provide a path for the core development team to make OpenSSL one of > many > possible TLS backends, rather than requiring that it be present on a > system > in order for Python to have TLS support. > > The proposed interface is laid out below. > > > Abstract Base Classes > --------------------- > > There are several interfaces that require standardisation. Those interfaces > are: > > 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the > ``ssl`` module. > 2. Wrapping a socket object, currently implemented by the `SSLSocket`_ > class > in the ``ssl`` module. > 3. Providing an in-memory buffer for doing in-memory encryption or > decryption > with no actual I/O (necessary for asynchronous I/O models), currently > implemented by the `SSLObject`_ class in the ``ssl`` module. > 4. Applying TLS configuration to the wrapping objects in (2) and (3). > Currently > this is also implemented by the `SSLContext`_ class in the ``ssl`` > module. > 5. Specifying TLS cipher suites. There is currently no code for doing this > in > the standard library: instead, the standard library uses OpenSSL cipher > suite strings. > 6. Specifying application-layer protocols that can be negotiated during the > TLS handshake. > 7. Specifying TLS versions. > 8. Reporting errors to the caller, currently implemented by the `SSLError`_ > class in the ``ssl`` module. > 9. Specifying certificates to load, either as client or server > certificates. > 10. Specifying which trust database should be used to validate certificates > presented by a remote peer. > 11. Finding a way to get hold of these interfaces at run time. > > While it is technically possible to define (2) in terms of (3), for the > sake of > simplicity it is easier to define these as two separate ABCs. > Implementations > are of course free to implement the concrete subclasses however they see > fit. > > Obviously, (5) doesn't require an abstract base class: instead, it > requires a > richer API for configuring supported cipher suites that can be easily > updated > with supported cipher suites for different implementations. > > (9) is a thorny problem, becuase in an ideal world the private keys > associated > with these certificates would never end up in-memory in the Python process > (that is, the TLS library would collaborate with a Hardware Security Module > (HSM) to provide the private key in such a way that it cannot be extracted > from > process memory). Thus, we need to provide an extensible model of providing > certificates that allows concrete implementations the ability to provide > this > higher level of security, while also allowing a lower bar for those > implementations that cannot. This lower bar would be the same as the status > quo: that is, the certificate may be loaded from an in-memory buffer or > from a > file on disk. > > (10) also represents an issue because different TLS implementations vary > wildly > in how they allow users to select trust stores. Some implementations have > specific trust store formats that only they can use (such as the OpenSSL CA > directory format that is created by ``c_rehash``), and others may not > allow you > to specify a trust store that does not include their default trust store. > > For this reason, we need to provide a model that assumes very little about > the > form that trust stores take. The "Trust Store" section below goes into more > detail about how this is achieved. > > Finally, this API will split the responsibilities currently assumed by the > `SSLContext`_ object: specifically, the responsibility for holding and > managing > configuration and the responsibility for using that configuration to build > wrapper objects. > > This is necessarily primarily for supporting functionality like Server Name > Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server > has > the ability to modify the TLS configuration in response to the client > telling > the server what hostname it is trying to reach. This is mostly used to > change > certificate chain so as to present the correct TLS certificate chain for > the > given hostname. The specific mechanism by which this is done is by > returning > a new `SSLContext`_ object with the appropriate configuration. > > This is not a model that maps well to other TLS implementations. Instead, > we > need to make it possible to provide a return value from the SNI callback > that > can be used to indicate what configuration changes should be made. This > means > providing an object that can hold TLS configuration. This object needs to > be > applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. > > For this reason, we split the responsibility of `SSLContext`_ into two > separate > objects. The ``TLSConfiguration`` object is an object that acts as > container > for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects > are > objects that are instantiated with a ``TLSConfiguration`` object. Both > objects > would be immutable. > > Configuration > ~~~~~~~~~~~~~ > > The ``TLSConfiguration`` concrete class defines an object that can hold and > manage TLS configuration. The goals of this class are as follows: > > 1. To provide a method of specifying TLS configuration that avoids the > risk of > errors in typing (this excludes the use of a simple dictionary). > 2. To provide an object that can be safely compared to other configuration > objects to detect changes in TLS configuration, for use with the SNI > callback. > > This class is not an ABC, primarily because it is not expected to have > implementation-specific behaviour. The responsibility for transforming a > ``TLSConfiguration`` object into a useful set of configuration for a given > TLS > implementation belongs to the Context objects discussed below. > > This class has one other notable property: it is immutable. This is a > desirable > trait for a few reasons. The most important one is that it allows these > objects > to be used as dictionary keys, which is potentially extremely valuable for > certain TLS backends and their SNI configuration. On top of this, it frees > implementations from needing to worry about their configuration objects > being > changed under their feet, which allows them to avoid needing to carefully > synchronize changes between their concrete data structures and the > configuration object. > > This object is extendable: that is, future releases of Python may add > configuration fields to this object as they become useful. For > backwards-compatibility purposes, new fields are only appended to this > object. > Existing fields will never be removed, renamed, or reordered. > > The ``TLSConfiguration`` object would be defined by the following code:: > > ServerNameCallback = Callable[[TLSBufferObject, Optional[str], > TLSConfiguration], Any] > > > _configuration_fields = [ > 'validate_certificates', > 'certificate_chain', > 'ciphers', > 'inner_protocols', > 'lowest_supported_version', > 'highest_supported_version', > 'trust_store', > 'sni_callback', > ] > > > _DEFAULT_VALUE = object() > > > class TLSConfiguration(namedtuple('TLSConfiguration', > _configuration_fields)): > """ > An immutable TLS Configuration object. This object has the > following > properties: > > :param validate_certificates bool: Whether to validate the TLS > certificates. This switch operates at a very broad scope: > either > validation is enabled, in which case all forms of validation > are > performed including hostname validation if possible, or > validation > is disabled, in which case no validation is performed. > > Not all backends support having their certificate validation > disabled. If a backend does not support having their > certificate > validation disabled, attempting to set this property to > ``False`` > will throw a ``TLSError`` when this object is passed into a > context object. > > :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The > certificate, intermediate certificate, and the corresponding > private key for the leaf certificate. These certificates will > be > offered to the remote peer during the handshake if required. > > The first Certificate in the list must be the leaf > certificate. All > subsequent certificates will be offered as intermediate > additional > certificates. > > :param ciphers Tuple[CipherSuite]: > The available ciphers for TLS connections created with this > configuration, in priority order. > > :param inner_protocols Tuple[Union[NextProtocol, bytes]]: > Protocols that connections created with this configuration > should > advertise as supported during the TLS handshake. These may be > advertised using either or both of ALPN or NPN. This list of > protocols should be ordered by preference. > > :param lowest_supported_version TLSVersion: > The minimum version of TLS that should be allowed on TLS > connections using this configuration. > > :param highest_supported_version TLSVersion: > The maximum version of TLS that should be allowed on TLS > connections using this configuration. > > :param trust_store TrustStore: > The trust store that connections using this configuration will > use > to validate certificates. > > :param sni_callback Optional[ServerNameCallback]: > A callback function that will be called after the TLS Client > Hello > handshake message has been received by the TLS server when the > TLS > client specifies a server name indication. > > Only one callback can be set per ``TLSConfiguration``. If the > ``sni_callback`` is ``None`` then the callback is disabled. If > the > ``TLSConfiguration`` is used for a ``ClientContext`` then this > setting will be ignored. > > The ``callback`` function will be called with three arguments: > the > first will be the ``TLSBufferObject`` for the connection; the > second will be a string that represents the server name that > the > client is intending to communicate (or ``None`` if the TLS > Client > Hello does not contain a server name); and the third argument > will > be the original ``Context``. The server name argument will be > the > IDNA *decoded* server name. > > The ``callback`` must return a ``TLSConfiguration`` to allow > negotiation to continue. Other return values signal errors. > Attempting to control what error is signaled by the underlying > TLS > implementation is not specified in this API, but is up to the > concrete implementation to handle. > > The Context will do its best to apply the ``TLSConfiguration`` > changes from its original configuration to the incoming > connection. > This will usually include changing the certificate chain, but > may > also include changes to allowable ciphers or any other > configuration settings. > """ > __slots__ = () > > def __new__(cls, validate_certificates=None: Optional[bool], > certificate_chain=None: Optional[Tuple[Tuple[Certificate], > PrivateKey]], > ciphers=None: Optional[Tuple[CipherSuite]], > inner_protocols=None: Optional[Tuple[Union[NextProtocol, > bytes]]], > lowest_supported_version=None: > Optional[TLSVersion], > highest_supported_version=None: > Optional[TLSVersion], > trust_store=None: Optional[TrustStore], > sni_callback=None: Optional[ServerNameCallback]): > > if validate_certificates is None: > validate_certificates = True > > if ciphers is None: > ciphers = DEFAULT_CIPHER_LIST > > if inner_protocols is None: > inner_protocols = [] > > if lowest_supported_version is None: > lowest_supported_version = TLSVersion.TLSv1 > > if highest_supported_version is None: > highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED > > return super().__new__( > cls, validate_certificates, certificate_chain, ciphers, > inner_protocols, lowest_supported_version, > highest_supported_version, trust_store, sni_callback > ) > > def update(self, validate_certificates=_DEFAULT_VALUE, > certificate_chain=_DEFAULT_VALUE, > ciphers=_DEFAULT_VALUE, > inner_protocols=_DEFAULT_VALUE, > lowest_supported_version=_DEFAULT_VALUE, > highest_supported_version=_DEFAULT_VALUE, > trust_store=_DEFAULT_VALUE, > sni_callback=_DEFAULT_VALUE): > """ > Create a new ``TLSConfiguration``, overriding some of the > settings > on the original configuration with the new settings. > """ > if validate_certificates is _DEFAULT_VALUE: > validate_certificates = self.validate_certificates > > if certificate_chain is _DEFAULT_VALUE: > certificate_chain = self.certificate_chain > > if ciphers is _DEFAULT_VALUE: > ciphers = self.ciphers > > if inner_protocols is _DEFAULT_VALUE: > inner_protocols = self.inner_protocols > > if lowest_supported_version is _DEFAULT_VALUE: > lowest_supported_version = self.lowest_supported_version > > if highest_supported_version is _DEFAULT_VALUE: > highest_supported_version = self.highest_supported_version > > if trust_store is _DEFAULT_VALUE: > trust_store = self.trust_store > > if sni_callback is _DEFAULT_VALUE: > sni_callback = self.sni_callback > > return self.__class__( > validate_certificates, certificate_chain, ciphers, > inner_protocols, lowest_supported_version, > highest_supported_version, trust_store, sni_callback > ) > > > > Context > ~~~~~~~ > > We define two Context abstract base classes. These ABCs define objects that > allow configuration of TLS to be applied to specific connections. They can > be > thought of as factories for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` > objects. > > Unlike the current ``ssl`` module, we provide two context classes instead > of > one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` > classes. This simplifies the APIs (for example, there is no sense in the > server > providing the ``server_hostname`` parameter to > ``ssl.SSLContext.wrap_socket``, > but because there is only one context class that parameter is still > available), > and ensures that implementations know as early as possible which side of a > TLS > connection they will serve. Additionally, it allows implementations to > opt-out > of one or either side of the connection. For example, SChannel on macOS is > not > really intended for server use and has an enormous amount of functionality > missing for server-side use. This would allow SChannel implementations to > simply not define a concrete subclass of ``ServerContext`` to signal their > lack > of support. > > As much as possible implementers should aim to make these classes > immutable: > that is, they should prefer not to allow users to mutate their internal > state > directly, instead preferring to create new contexts from new > TLSConfiguration > objects. Obviously, the ABCs cannot enforce this constraint, and so they > do not > attempt to. > > The ``Context`` abstract base class has the following class definition:: > > TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] > > > class _BaseContext(metaclass=ABCMeta): > @abstractmethod > def __init__(self, configuration: TLSConfiguration): > """ > Create a new context object from a given TLS configuration. > """ > > @property > @abstractmethod > def configuration(self) -> TLSConfiguration: > """ > Returns the TLS configuration that was used to create the > context. > """ > > > class ClientContext(_BaseContext): > @abstractmethod > def wrap_socket(self, > socket: socket.socket, > server_hostname: Optional[str], > auto_handshake=True: bool) -> TLSWrappedSocket: > """ > Wrap an existing Python socket object ``socket`` and return a > ``TLSWrappedSocket`` object. ``socket`` must be a > ``SOCK_STREAM`` > socket: all other socket types are unsupported. > > The returned SSL socket is tied to the context, its settings > and > certificates. > > The parameter ``server_hostname`` specifies the hostname of the > service which we are connecting to. This allows a single > server to > host multiple SSL-based services with distinct certificates, > quite > similarly to HTTP virtual hosts. This is also used to validate > the > TLS certificate for the given hostname. If hostname validation > is > not desired, then pass ``None`` for this parameter. > > The parameter ``auto_handshake`` specifies whether to do the > SSL > handshake automatically after doing a ``socket.connect()``, or > whether the application program will call it explicitly, by > invoking the ``TLSWrappedSocket.do_handshake()`` method. > Calling > ``TLSWrappedSocket.do_handshake()`` explicitly gives the > program > control over the blocking behavior of the socket I/O involved > in > the handshake. > """ > > @abstractmethod > def wrap_buffers(self, incoming: Any, outgoing: Any, > server_hostname: Optional[str]) -> > TLSWrappedBuffer: > """ > Wrap a pair of buffer objects (``incoming`` and ``outgoing``) > to > create an in-memory stream for TLS. The SSL routines will read > data > from ``incoming`` and decrypt it, and write encrypted data to > ``outgoing``. > > The buffer objects must be either file objects or objects that > implement the buffer protocol. > > The ``server_hostname`` parameter has the same meaning as in > ``wrap_socket``. > """ > > > class ServerContext(_BaseContext): > @abstractmethod > def wrap_socket(self, socket: socket.socket, > auto_handshake=True: bool) -> TLSWrappedSocket: > """ > Wrap an existing Python socket object ``socket`` and return a > ``TLSWrappedSocket`` object. ``socket`` must be a > ``SOCK_STREAM`` > socket: all other socket types are unsupported. > > The returned SSL socket is tied to the context, its settings > and > certificates. > > The parameter ``auto_handshake`` specifies whether to do the > SSL > handshake automatically after doing a ``socket.accept()``, or > whether the application program will call it explicitly, by > invoking the ``TLSWrappedSocket.do_handshake()`` method. > Calling > ``TLSWrappedSocket.do_handshake()`` explicitly gives the > program > control over the blocking behavior of the socket I/O involved > in > the handshake. > """ > > @abstractmethod > def wrap_buffers(self, incoming: Any, outgoing: Any) -> > TLSWrappedBuffer: > """ > Wrap a pair of buffer objects (``incoming`` and ``outgoing``) > to > create an in-memory stream for TLS. The SSL routines will read > data > from ``incoming`` and decrypt it, and write encrypted data to > ``outgoing``. > > The buffer objects must be either file objects or objects that > implement the buffer protocol. > """ > > > Socket > ~~~~~~ > > The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, > which > has the following definition:: > > class TLSWrappedSocket(metaclass=ABCMeta): > # The various socket methods all must be implemented. Their > definitions > # have been elided from this class defintion in the PEP because > they > # aren't instructive. > @abstractmethod > def do_handshake(self) -> None: > """ > Performs the TLS handshake. Also performs certificate > validation > and hostname verification. > """ > > @abstractmethod > def cipher(self) -> Optional[CipherSuite]: > """ > Returns the CipherSuite entry for the cipher that has been > negotiated on the connection. If no connection has been > negotiated, > returns ``None``. > """ > > @abstractmethod > def negotiated_protocol(self) -> Optional[Union[NextProtocol, > bytes]]: > """ > Returns the protocol that was selected during the TLS > handshake. > This selection may have been made using ALPN, NPN, or some > future > negotiation mechanism. > > If the negotiated protocol is one of the protocols defined in > the > ``NextProtocol`` enum, the value from that enum will be > returned. > Otherwise, the raw bytestring of the negotiated protocol will > be > returned. > > If ``Context.set_inner_protocols()`` was not called, if the > other > party does not support protocol negotiation, if this socket > does > not support any of the peer's proposed protocols, or if the > handshake has not happened yet, ``None`` is returned. > """ > > @property > @abstractmethod > def context(self) -> Context: > """ > The ``Context`` object this socket is tied to. > """ > > @abstractproperty > def negotiated_tls_version(self) -> Optional[TLSVersion]: > """ > The version of TLS that has been negotiated on this connection. > """ > > @abstractmethod > def unwrap(self) -> socket.socket: > """ > Cleanly terminate the TLS connection on this wrapped socket. > Once > called, this ``TLSWrappedSocket`` can no longer be used to > transmit > data. Returns the socket that was wrapped with TLS. > """ > > > Buffer > ~~~~~~ > > The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, > which > has the following definition:: > > class TLSWrappedBuffer(metaclass=ABCMeta): > @abstractmethod > def read(self, amt=None: int) -> bytes: > """ > Read up to ``amt`` bytes of data from the input buffer and > return > the result as a ``bytes`` instance. If ``amt`` is ``None``, > will > attempt to read until either EOF is reached or until further > attempts to read would raise either ``WantReadError`` or > ``WantWriteError``. > > Raise ``WantReadError`` or ``WantWriteError`` if there is > insufficient data in either the input or output buffer and the > operation would have caused data to be written or read. > > As at any time a re-negotiation is possible, a call to > ``read()`` > can also cause write operations. > """ > > @abstractmethod > def readinto(self, buffer: Any, amt=None: int) -> int: > """ > Read up to ``amt`` bytes of data from the input buffer into > ``buffer``, which must be an object that implements the buffer > protocol. Returns the number of bytes read. If ``amt`` is > ``None``, > will attempt to read until either EOF is reached or until > further > attempts to read would raise either ``WantReadError`` or > ``WantWriteError``, or until the buffer is full. > > Raises ``WantReadError`` or ``WantWriteError`` if there is > insufficient data in either the input or output buffer and the > operation would have caused data to be written or read. > > As at any time a re-negotiation is possible, a call to > ``readinto()`` can also cause write operations. > """ > > @abstractmethod > def write(self, buf: Any) -> int: > """ > Write ``buf`` in encrypted form to the output buffer and > return the > number of bytes written. The ``buf`` argument must be an object > supporting the buffer interface. > > Raise ``WantReadError`` or ``WantWriteError`` if there is > insufficient data in either the input or output buffer and the > operation would have caused data to be written or read. > > As at any time a re-negotiation is possible, a call to > ``write()`` > can also cause read operations. > """ > > @abstractmethod > def do_handshake(self) -> None: > """ > Performs the TLS handshake. Also performs certificate > validation > and hostname verification. > """ > > @abstractmethod > def cipher(self) -> Optional[CipherSuite]: > """ > Returns the CipherSuite entry for the cipher that has been > negotiated on the connection. If no connection has been > negotiated, > returns ``None``. > """ > > @abstractmethod > def negotiated_protocol(self) -> Optional[Union[NextProtocol, > bytes]]: > """ > Returns the protocol that was selected during the TLS > handshake. > This selection may have been made using ALPN, NPN, or some > future > negotiation mechanism. > > If the negotiated protocol is one of the protocols defined in > the > ``NextProtocol`` enum, the value from that enum will be > returned. > Otherwise, the raw bytestring of the negotiated protocol will > be > returned. > > If ``Context.set_inner_protocols()`` was not called, if the > other > party does not support protocol negotiation, if this socket > does > not support any of the peer's proposed protocols, or if the > handshake has not happened yet, ``None`` is returned. > """ > > @property > @abstractmethod > def context(self) -> Context: > """ > The ``Context`` object this socket is tied to. > """ > > @abstractproperty > def negotiated_tls_version(self) -> Optional[TLSVersion]: > """ > The version of TLS that has been negotiated on this connection. > """ > > @abstractmethod > def shutdown(self) -> None: > """ > Performs a clean TLS shut down. This should generally be used > whenever possible to signal to the remote peer that the > content is > finished. > """ > > > Cipher Suites > ~~~~~~~~~~~~~ > > Supporting cipher suites in a truly library-agnostic fashion is a > remarkably > difficult undertaking. Different TLS implementations often have *radically* > different APIs for specifying cipher suites, but more problematically these > APIs frequently differ in capability as well as in style. Some examples are > shown below: > > OpenSSL > ^^^^^^^ > > OpenSSL uses a well-known cipher string format. This format has been > adopted as > a configuration language by most products that use OpenSSL, including > Python. > This format is relatively easy to read, but has a number of downsides: it > is > a string, which makes it remarkably easy to provide bad inputs; it lacks > much > detailed validation, meaning that it is possible to configure OpenSSL in a > way > that doesn't allow it to negotiate any cipher at all; and it allows > specifying > cipher suites in a number of different ways that make it tricky to parse. > The > biggest problem with this format is that there is no formal specification > for > it, meaning that the only way to parse a given string the way OpenSSL > would is > to get OpenSSL to parse it. > > OpenSSL's cipher strings can look like this:: > > 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+ > AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' > > This string demonstrates some of the complexity of the OpenSSL format. For > example, it is possible for one entry to specify multiple cipher suites: > the > entry ``ECDH+AESGCM`` means "all ciphers suites that include both > elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". > More explicitly, that will expand to four cipher suites:: > > "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM- > SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" > > That makes parsing a complete OpenSSL cipher string extremely tricky. Add > to > the fact that there are other meta-characters, such as "!" (exclude all > cipher > suites that match this criterion, even if they would otherwise be included: > "!MD5" means that no cipher suites using the MD5 hash algorithm should be > included), "-" (exclude matching ciphers if they were already included, but > allow them to be re-added later if they get included again), and "+" > (include > the matching ciphers, but place them at the end of the list), and you get > an > *extremely* complex format to parse. On top of this complexity it should be > noted that the actual result depends on the OpenSSL version, as an OpenSSL > cipher string is valid so long as it contains at least one cipher that > OpenSSL > recognises. > > OpenSSL also uses different names for its ciphers than the names used in > the > relevant specifications. See the manual page for ``ciphers(1)`` for more > details. > > The actual API inside OpenSSL for the cipher string is simple:: > > char *cipher_list = ; > int rc = SSL_CTX_set_cipher_list(context, cipher_list); > > This means that any format that is used by this module must be able to be > converted to an OpenSSL cipher string for use with OpenSSL. > > SecureTransport > ^^^^^^^^^^^^^^^ > > SecureTransport is the macOS system TLS library. This library is > substantially > more restricted than OpenSSL in many ways, as it has a much more restricted > class of users. One of these substantial restrictions is in controlling > supported cipher suites. > > Ciphers in SecureTransport are represented by a C ``enum``. This enum has > one > entry per cipher suite, with no aggregate entries, meaning that it is not > possible to reproduce the meaning of an OpenSSL cipher string like > "ECDH+AESGCM" without hand-coding which categories each enum member falls > into. > > However, the names of most of the enum members are in line with the formal > names of the cipher suites: that is, the cipher suite that OpenSSL calls > "ECDHE-ECDSA-AES256-GCM-SHA384" is called > "TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. > > The API for configuring cipher suites inside SecureTransport is simple:: > > SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, > ...}; > OSStatus status = SSLSetEnabledCiphers(context, ciphers, > sizeof(cphers)); > > SChannel > ^^^^^^^^ > > SChannel is the Windows system TLS library. > > SChannel has extremely restrictive support for controlling available TLS > cipher suites, and additionally adopts a third method of expressing what > TLS > cipher suites are supported. > > Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned > ints). > Each of these constants does not refer to an entire cipher suite, but > instead > an individual algorithm. Some examples are ``CALG_3DES`` and > ``CALG_AES_256``, > which refer to the bulk encryption algorithm used in a cipher suite, > ``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key > exchange > algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which > refer to > the message authentication code used in a cipher suite, and ``CALG_ECDSA`` > and > ``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange > algorithm. > > This can be thought of as the half of OpenSSL's functionality that > SecureTransport doesn't have: SecureTransport only allows specifying exact > cipher suites, while SChannel only allows specifying *parts* of the cipher > suite, while OpenSSL allows both. > > Determining which cipher suites are allowed on a given connection is done > by > providing a pointer to an array of these ``ALG_ID`` constants. This means > that > any suitable API must allow the Python code to determine which ``ALG_ID`` > constants must be provided. > > > Proposed Interface > ^^^^^^^^^^^^^^^^^^ > > The proposed interface for the new module is influenced by the combined > set of > limitations of the above implementations. Specifically, as every > implementation > *except* OpenSSL requires that each individual cipher be provided, there > is no > option but to provide that lowest-common denominator approach. > > The simplest approach is to provide an enumerated type that includes all > of the > cipher suites defined for TLS. The values of the enum members will be their > two-octet cipher identifier as used in the TLS handshake, stored as a > tuple of > integers. The names of the enum members will be their IANA-registered > cipher > suite names. > > Rather than populate this enum by hand, it is likely that we'll define a > script that can build it from Christian Heimes' `tlsdb JSON file`_ > (warning: > large file). This also opens up the possibility of extending the API with > additional querying function, such as determining which TLS versions > support > which ciphers, if that functionality is found to be useful or necessary. > > If users find this approach to be onerous, a future extension to this API > can > provide helpers that can reintroduce OpenSSL's aggregation functionality. > > Because this enum would be enormous, the entire enum is not provided here. > Instead, a small sample of entries is provided to give a flavor of how it > will > appear. > > :: > > class CipherSuite(Enum): > ... > TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = (0xC0, 0x12) > ... > TLS_ECDHE_ECDSA_WITH_AES_128_CCM = (0xC0, 0xAC) > ... > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = (0xC0, 0x2B) > ... > > > Protocol Negotiation > ~~~~~~~~~~~~~~~~~~~~ > > Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 > handshake. While NPN and ALPN are, at their fundamental level, built on > top of > bytestrings, string-based APIs are frequently problematic as they allow for > errors in typing that can be hard to detect. > > For this reason, this module would define a type that protocol negotiation > implementations can pass and be passed. This type would wrap a bytestring > to > allow for aliases for well-known protocols. This allows us to avoid the > problems inherent in typos for well-known protocols, while allowing the > full > extensibility of the protocol negotiation layer if needed by letting users > pass > byte strings directly. > > :: > > class NextProtocol(Enum): > H2 = b'h2' > H2C = b'h2c' > HTTP1 = b'http/1.1' > WEBRTC = b'webrtc' > C_WEBRTC = b'c-webrtc' > FTP = b'ftp' > STUN = b'stun.nat-discovery' > TURN = b'stun.turn' > > TLS Versions > ~~~~~~~~~~~~ > > It is often useful to be able to restrict the versions of TLS you're > willing to > support. There are many security advantages in refusing to use old > versions of > TLS, and some misbehaving servers will mishandle TLS clients advertising > support for newer versions. > > The following enumerated type can be used to gate TLS versions. > Forward-looking > applications should almost never set a maximum TLS version unless they > absolutely must, as a TLS backend that is newer than the Python that uses > it > may support TLS versions that are not in this enumerated type. > > Additionally, this enumerated type defines two additional flags that can > always > be used to request either the lowest or highest TLS version supported by an > implementation. > > :: > > class TLSVersion(Enum): > MINIMUM_SUPPORTED > SSLv2 > SSLv3 > TLSv1 > TLSv1_1 > TLSv1_2 > TLSv1_3 > MAXIMUM_SUPPORTED > > > Errors > ~~~~~~ > > This module would define three base classes for use with error handling. > Unlike > many of the the other classes defined here, these classes are not > abstract, as > they have no behaviour. They exist simply to signal certain common > behaviours. > Backends should subclass these exceptions in their own packages, but > needn't > define any behaviour for them. > > In general, concrete implementations should subclass these exceptions > rather > than throw them directly. This makes it moderately easier to determine > which > concrete TLS implementation is in use during debugging of unexpected > errors. > However, this is not mandatory. > > The definitions of the errors are below:: > > class TLSError(Exception): > """ > The base exception for all TLS related errors from any backend. > Catching this error should be sufficient to catch *all* TLS errors, > regardless of what backend is used. > """ > > class WantWriteError(TLSError): > """ > A special signaling exception used only when non-blocking or > buffer-only I/O is used. This error signals that the requested > operation cannot complete until more data is written to the > network, > or until the output buffer is drained. > """ > > class WantReadError(TLSError): > """ > A special signaling exception used only when non-blocking or > buffer-only I/O is used. This error signals that the requested > operation cannot complete until more data is read from the > network, or > until more data is available in the input buffer. > """ > > > Certificates > ~~~~~~~~~~~~ > > This module would define an abstract X509 certificate class. This class > would > have almost no behaviour, as the goal of this module is not to provide all > possible relevant cryptographic functionality that could be provided by > X509 > certificates. Instead, all we need is the ability to signal the source of a > certificate to a concrete implementation. > > For that reason, this certificate implementation defines only > constructors. In > essence, the certificate object in this module could be as abstract as a > handle > that can be used to locate a specific certificate. > > Concrete implementations may choose to provide alternative constructors, > e.g. > to load certificates from HSMs. If a common interface emerges for doing > this, > this module may be updated to provide a standard constructor for this > use-case > as well. > > Concrete implementations should aim to have Certificate objects be > hashable if > at all possible. This will help ensure that TLSConfiguration objects used > with > an individual concrete implementation are also hashable. > > :: > > class Certificate(metaclass=ABCMeta): > @abstractclassmethod > def from_buffer(cls, buffer: bytes) -> Certificate: > """ > Creates a Certificate object from a byte buffer. This byte > buffer > may be either PEM-encoded or DER-encoded. If the buffer is PEM > encoded it *must* begin with the standard PEM preamble (a > series of > dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and > another > series of dashes). In the absence of that preamble, the > implementation may assume that the certificate is DER-encoded > instead. > """ > > @abstractclassmethod > def from_file(cls, path: Union[pathlib.Path, AnyStr]) -> > Certificate: > """ > Creates a Certificate object from a file on disk. This method > may > be a convenience method that wraps ``open`` and > ``from_buffer``, > but some TLS implementations may be able to provide > more-secure or > faster methods of loading certificates that do not involve > Python > code. > """ > > > Private Keys > ~~~~~~~~~~~~ > > This module would define an abstract private key class. Much like the > Certificate class, this class has almost no behaviour in order to give as > much > freedom as possible to the concrete implementations to treat keys > carefully. > > This class has all the caveats of the ``Certificate`` class. > > :: > > class PrivateKey(metaclass=ABCMeta): > @abstractclassmethod > def from_buffer(cls, > buffer: bytes, > password=None: Optional[Union[Callable[[], > Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: > """ > Creates a PrivateKey object from a byte buffer. This byte > buffer > may be either PEM-encoded or DER-encoded. If the buffer is PEM > encoded it *must* begin with the standard PEM preamble (a > series of > dashes followed by the ASCII bytes "BEGIN", the key type, and > another series of dashes). In the absence of that preamble, the > implementation may assume that the certificate is DER-encoded > instead. > > The key may additionally be encrypted. If it is, the > ``password`` > argument can be used to decrypt the key. The ``password`` > argument > may be a function to call to get the password for decrypting > the > private key. It will only be called if the private key is > encrypted > and a password is necessary. It will be called with no > arguments, > and it should return either bytes or bytearray containing the > password. Alternatively a bytes, or bytearray value may be > supplied > directly as the password argument. It will be ignored if the > private key is not encrypted and no password is needed. > """ > > @abstractclassmethod > def from_file(cls, > path: Union[pathlib.Path, bytes, str], > password=None: Optional[Union[Callable[[], > Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: > """ > Creates a PrivateKey object from a file on disk. This method > may > be a convenience method that wraps ``open`` and > ``from_buffer``, > but some TLS implementations may be able to provide > more-secure or > faster methods of loading certificates that do not involve > Python > code. > > The ``password`` parameter behaves exactly as the equivalent > parameter on ``from_buffer``. > """ > > > Trust Store > ~~~~~~~~~~~ > > As discussed above, loading a trust store represents an issue because > different > TLS implementations vary wildly in how they allow users to select trust > stores. > For this reason, we need to provide a model that assumes very little about > the > form that trust stores take. > > This problem is the same as the one that the Certificate and PrivateKey > types > need to solve. For this reason, we use the exact same model, by creating an > opaque type that can encapsulate the various means that TLS backends may > open > a trust store. > > A given TLS implementation is not required to implement all of the > constructors. However, it is strongly recommended that a given TLS > implementation provide the ``system`` constructor if at all possible, as > this > is the most common validation trust store that is used. Concrete > implementations may also add their own constructors. > > Concrete implementations should aim to have TrustStore objects be hashable > if > at all possible. This will help ensure that TLSConfiguration objects used > with > an individual concrete implementation are also hashable. > > :: > > class TrustStore(metaclass=ABCMeta): > @abstractclassmethod > def system(cls) -> TrustStore: > """ > Returns a TrustStore object that represents the system trust > database. > """ > > @abstractclassmethod > def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> > TrustStore: > """ > Initializes a trust store from a single file full of PEMs. > """ > > > Runtime Access > ~~~~~~~~~~~~~~ > > A not-uncommon use case for library users is to want to allow the library > to > control the TLS configuration, but to want to select what backend is in > use. > For example, users of Requests may want to be able to select between > OpenSSL or > a platform-native solution on Windows and macOS, or between OpenSSL and > NSS on > some Linux platforms. These users, however, may not care about exactly how > their TLS configuration is done. > > This poses a problem: given an arbitrary concrete implementation, how can a > library work out how to load certificates into the trust store? There are > two > options: either all concrete implementations can be required to fit into a > specific naming scheme, or we can provide an API that makes it possible to > grab > these objects. > > This PEP proposes that we use the second approach. This grants the greatest > freedom to concrete implementations to structure their code as they see > fit, > requiring only that they provide a single object that has the appropriate > properties in place. Users can then pass this "backend" object to libraries > that support it, and those libraries can take care of configuring and > using the > concrete implementation. > > All concrete implementations must provide a method of obtaining a > ``Backend`` > object. The ``Backend`` object can be a global singleton or can be created > by a > callable if there is an advantage in doing that. > > The ``Backend`` object has the following definition:: > > Backend = namedtuple( > 'Backend', > ['client_context', 'server_context', > 'certificate', 'private_key', 'trust_store'] > ) > > Each of the properties must provide the concrete implementation of the > relevant > ABC. This ensures that code like this will work for any backend:: > > trust_store = backend.trust_store.system() > > > Changes to the Standard Library > =============================== > > The portions of the standard library that interact with TLS should be > revised > to use these ABCs. This will allow them to function with other TLS > backends. > This includes the following modules: > > - asyncio > - ftplib > - http.client > - imaplib > - nntplib > - poplib > - smtplib > > > Migration of the ssl module > --------------------------- > > Naturally, we will need to extend the ``ssl`` module itself to conform to > these > ABCs. This extension will take the form of new classes, potentially in an > entirely new module. This will allow applications that take advantage of > the > current ``ssl`` module to continue to do so, while enabling the new APIs > for > applications and libraries that want to use them. > > In general, migrating from the ``ssl`` module to the new ABCs is not > expected > to be one-to-one. This is normally acceptable: most tools that use the > ``ssl`` > module hide it from the user, and so refactoring to use the new module > should > be invisible. > > However, a specific problem comes from libraries or applications that leak > exceptions from the ``ssl`` module, either as part of their defined API or > by > accident (which is easily done). Users of those tools may have written code > that tolerates and handles exceptions from the ``ssl`` module being raised: > migrating to the ABCs presented here would potentially cause the exceptions > defined above to be thrown instead, and existing ``except`` blocks will not > catch them. > > For this reason, part of the migration of the ``ssl`` module would require > that > the exceptions in the ``ssl`` module alias those defined above. That is, > they > would require the following statements to all succeed:: > > assert ssl.SSLError is tls.TLSError > assert ssl.SSLWantReadError is tls.WantReadError > assert ssl.SSLWantWriteError is tls.WantWriteError > > The exact mechanics of how this will be done are beyond the scope of this > PEP, > as they are made more complex due to the fact that the current ``ssl`` > exceptions are defined in C code, but more details can be found in > `an email sent to the Security-SIG by Christian Heimes`_. > > > Future > ====== > > Major future TLS features may require revisions of these ABCs. These > revisions > should be made cautiously: many backends may not be able to move forward > swiftly, and will be invalidated by changes in these ABCs. This is > acceptable, > but wherever possible features that are specific to individual > implementations > should not be added to the ABCs. The ABCs should restrict themselves to > high-level descriptions of IETF-specified features. > > However, well-justified extensions to this API absolutely should be made. > The > focus of this API is to provide a unifying lowest-common-denominator > configuration option for the Python community. TLS is not a static target, > and > as TLS evolves so must this API. > > > References > ========== > > .. _ssl module: https://docs.python.org/3/library/ssl.html > .. _OpenSSL Library: https://www.openssl.org/ > .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ > .. _certifi: https://pypi.org/project/certifi/ > .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext > .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket > .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject > .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError > .. _MSDN articles: https://msdn.microsoft.com/en- > us/library/windows/desktop/mt490158(v=vs.85).aspx > .. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json > .. _an email sent to the Security-SIG by Christian Heimes: > https://mail.python.org/pipermail/security-sig/2017-January/000213.html > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 2 06:17:16 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Feb 2017 11:17:16 +0000 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> Message-ID: On 2 February 2017 at 10:38, Cory Benfield wrote: > This PEP would define a standard TLS interface in the form of a collection of > abstract base classes. This interface would allow Python implementations and > third-party libraries to provide bindings to TLS libraries other than OpenSSL > that can be used by tools that expect the interface provided by the Python > standard library, with the goal of reducing the dependence of the Python > ecosystem on OpenSSL. Most of the PEP went over my head, but what I understood looks well thought out and sensible. Overall I'm +1 on this. One thing that wasn't clear to me was the backward compatibility implications. Will the existing ssl module and its API be retained, or will it be modified/deprecated? Will Windows/OSX distributions of Python ultimately stop shipping with OpenSSL included? From an end user POV, I won't care (high level modules like asyncio, urllib[1], http.client etc will be updated to transparently use the new infrastructure) so I accept that this is something of an "internal" and/or library author concern. Thanks for putting the effort into this - people like me rely on "other people" getting the security stuff right, and I appreciate the work that those others like yourself put in for me. Paul [1] urllib wasn't actually a module noted as needing change. Was that an omission, or is it simply because it defers the TLS stuff to the lower levels like http.client? From himchanterry at gmail.com Thu Feb 2 07:01:34 2017 From: himchanterry at gmail.com (himchanterry at gmail.com) Date: Thu, 2 Feb 2017 20:01:34 +0800 Subject: [Python-ideas] New function to add into the "random" module Message-ID: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> Hi, my name is Terry, and I?d like to propose a small function that could be added into the Python?s ?random? module. This function is for generating a random float/decimal number. I would like to know if there?s such function that does the same thing, if there is, please point out to me. If not, I would like to ask you developers to add this function into future Python versions. The function code is as follow: from random import * def randfloat(x , y , maxfloatpt=None): ??? if x > y: ??????? x , y = y , x ??? lenx = len(str(x)) ??? leny = len(str(y)) ??? intx = int(x) ??? inty = int(y) ??? bigger = max(lenx,leny) ??? if maxfloatpt == None: ??????? if bigger == lenx: ??????????? intxlen = len(str(intx)) ??????????? maxfloatpt = len(str(x)[intxlen:]) ??????? elif bigger == leny: ??????????? intylen = len(str(inty)) ??????????? maxfloatpt = len(str(y)[intylen:]) ??????? else: ??????????? pass ???else: ??????? pass ??? num = randint(intx , inty) ??? num = str(num) ??? num = '%s.' % num ????for i in range(0 , maxfloatpt): ??????? flnum = randint(0 , 9) ??????? str(flnum) ??????? num = '%s%s' % (num , flnum) ??? return float(num) ??????? P.S.: If the indent has anything wrong, just correct me, thx! P.P.S.: The function is tested, it?s working ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmoisset at machinalis.com Thu Feb 2 07:35:13 2017 From: dmoisset at machinalis.com (Daniel Moisset) Date: Thu, 2 Feb 2017 12:35:13 +0000 Subject: [Python-ideas] New function to add into the "random" module In-Reply-To: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> References: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> Message-ID: Hi Terry, the functionality you have provided is already covered by the random.uniform function On 2 February 2017 at 12:01, wrote: > Hi, my name is Terry, and I?d like to propose a small function that could > be added into the Python?s ?random? module. > > > > This function is for generating a random float/decimal number. I would > like to know if there?s such function that does the same thing, if there > is, please point out to me. If not, I would like to ask you developers to > add this function into future Python versions. > > > > The function code is as follow: > > > > from random import * > > > > def randfloat(x , y , maxfloatpt=None): > > if x > y: > > x , y = y , x > > lenx = len(str(x)) > > leny = len(str(y)) > > intx = int(x) > > inty = int(y) > > bigger = max(lenx,leny) > > if maxfloatpt == None: > > if bigger == lenx: > > intxlen = len(str(intx)) > > maxfloatpt = len(str(x)[intxlen:]) > > elif bigger == leny: > > intylen = len(str(inty)) > > maxfloatpt = len(str(y)[intylen:]) > > else: > > pass > > else: > > pass > > num = randint(intx , inty) > > num = str(num) > > num = '%s.' % num > > for i in range(0 , maxfloatpt): > > flnum = randint(0 , 9) > > str(flnum) > > num = '%s%s' % (num , flnum) > > return float(num) > > > > P.S.: If the indent has anything wrong, just correct me, thx! > > > > P.P.S.: The function is tested, it?s working ? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Thu Feb 2 08:27:07 2017 From: george at fischhof.hu (George Fischhof) Date: Thu, 2 Feb 2017 14:27:07 +0100 Subject: [Python-ideas] OS related file operations (copy, move, delete, rename...) should be placed into one module In-Reply-To: References: <1682891988816860097@unknownmsgid> Message-ID: 2017-01-12 20:11 GMT+01:00 Todd : > On Thu, Jan 12, 2017 at 11:17 AM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> I agree that this has been a bit of a wart for a long time. >> >> While the old ?let?s treat strings as paths? modules are split up like >> you said, pathlib can do what they do and more: https://docs.python.org/ >> 3/library/pathlib.html >> >> >> Exactly -- this is The Solution. It combines paths themselves with things >> you are likely to do with paths. >> >> It may well lack some nice features. If so, suggestions for that would be >> the way to go. >> > > Can such suggestions go here or should someone start a new thread? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Hi Guys, I just would like to ask You whehet there is a step forward in this topic? BR, George -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Feb 2 09:01:22 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 2 Feb 2017 14:01:22 +0000 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> Message-ID: > On 2 Feb 2017, at 11:03, Robert Collins wrote: > > Nice. What's needed to move this forward? In the short term, for the PEP, nothing more than review and sanity checking. Antione has kindly provided some detailed review on GitHub (thanks Antoine!), and the more of that we get the better. Once review has quieted down for a few days, I?ll take that as something approximating consensus from this list, and I?ll kick it over to python-dev and start the formal PEP process. Longer term, the following chunks of work will be needed: 1. Providing a shim for the current ssl module. The ssl module has a number of objects that don?t quite cleave to this API, so we?d probably want to either add new objects that do that (likely by wrapping the current ones), or we?d want a whole-new module that cleaves to this API. I?ll discuss more in my response to Paul what I think we?d do with the ssl module longer-term. 2. Code needs to be written for the identified network libraries to have them use the new API. There?s no point doing this until (1) is done, as they?ll otherwise have no ability to use any TLS API at all, but transitioning over to the new API starts providing some real community value as third-party implementations can start being used with stdlib network modules. 3. Backends for SChannel and SecureTransport would need to be written. The most likely response I expect from Guido and others in python-dev is that they?d like to see example implementations for SChannel and SecureTransport that fit this API. I am likely to stub out a SecureTransport implementation sometime soon, as I have some experience with that API, and it?d make a good test-bed for the PEP. I?m a bad fit for doing SChannel though: I don?t know Windows well enough, and my ramp-up time would be high. I?ll scout out to see if there?s someone who?s willing to stub this out, though. 4. Eventually, integrating the two backends above into the standard library so that it becomes possible to reduce the reliance on OpenSSL. This would allow future Python implementations to ship with all of their network protocol libraries supporting platform-native TLS implementations on Windows and macOS. This will almost certainly require new PEPs. I?ll probably volunteer to maintain a SecureTransport library, and I have got verbal suggestions from some other people who?d be willing to step up and help with that. Again, we?d need help with SChannel (looking at you, Steve). These are all valuable things to do, and I?d accept help with any and all of them. Certainly we need to do (1) and (2) to start getting real value out of this PEP, with (3) being something that could be done third-party initially (and so don?t have to land in Python 3.7). Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Feb 2 09:08:27 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 2 Feb 2017 14:08:27 +0000 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> Message-ID: > On 2 Feb 2017, at 11:17, Paul Moore wrote: > > One thing that wasn't clear to me was the backward compatibility > implications. Will the existing ssl module and its API be retained, or > will it be modified/deprecated? Will Windows/OSX distributions of > Python ultimately stop shipping with OpenSSL included? From an end > user POV, I won't care (high level modules like asyncio, urllib[1], > http.client etc will be updated to transparently use the new > infrastructure) so I accept that this is something of an "internal" > and/or library author concern. These are all good questions. For the existing ssl module and its API, deprecating it is hard work: it?s used extremely widely, and I don?t think we?d ever remove it. I suspect we?d warn against using it in favour of programming to the new tls API, but we?d keep it in place partly because it is so heavily used today, and partly because it?ll form the basis of the OpenSSL implementation of the abstract API here. As to stopping shipping with OpenSSL, that?s unlikely. OpenSSL doesn?t just back the ssl module, it backs hashlib as well (via libcrypto). While it?s potentially possible to replace most of hashlib with constructs from platform-native implementations (CommonCrypto and CryptoNG), that?s a lot of work and hard to do, as the cryptography.io folks would tell you. So I suspect we?ll end up shipping with both, but the importance of OpenSSL for TLS would drop dramatically (and it would even potentially be possible to link only against libcrypto instead of libssl). > [1] urllib wasn't actually a module noted as needing change. Was that > an omission, or is it simply because it defers the TLS stuff to the > lower levels like http.client? That was an omission: I forgot to check whether it needed updating. It does. I?ve amended the PEP accordingly. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Thu Feb 2 09:21:42 2017 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Thu, 2 Feb 2017 09:21:42 -0500 Subject: [Python-ideas] New function to add into the "random" module In-Reply-To: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> References: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> Message-ID: On Thu, Feb 2, 2017 at 7:01 AM, wrote: > from random import * > > > > def randfloat(x , y , maxfloatpt=None): > > if x > y: > > x , y = y , x > > lenx = len(str(x)) > > leny = len(str(y)) > > intx = int(x) > > inty = int(y) > > bigger = max(lenx,leny) > > if maxfloatpt == None: > > if bigger == lenx: > > intxlen = len(str(intx)) > > maxfloatpt = len(str(x)[intxlen:]) > > elif bigger == leny: > > intylen = len(str(inty)) > > maxfloatpt = len(str(y)[intylen:]) > > else: > > pass > > else: > > pass > > num = randint(intx , inty) > > num = str(num) > > num = '%s.' % num > > for i in range(0 , maxfloatpt): > > flnum = randint(0 , 9) > > str(flnum) > > num = '%s%s' % (num , flnum) > > return float(num) > > > > P.S.: If the indent has anything wrong, just correct me, thx! > > > > P.P.S.: The function is tested, it?s working ? > It looks like this is generating a random float one digit at a time. Instead, consider the following: Say you have a function which generates a number between 0 and n. You want to generate a number between floats x and y, as fairly as you can. So stretch the range 0...n onto the range x...y, linearly. That means 0 goes to x, n goes to y, things close to 0 go to things close to x, etc. If x is 0, then it's simple: def stretch(num): return num / n * y We just need to make x = 0 temporarily, shifting y down as well: def stretch(num): return (num / n * (y - x)) + x Really, we care about the _length_ of the interval [x,y], relative to the length of [0,n]. Since we want to reach as many numbers as possible in [x,y], we just need n to be really big. The limit of n depends on how randint is implemented. I'm not sure whether Python uses extra randoms in the case of long ints, but let's say that 2^15 is both big enough for us and small enough for Python's randints to be pretty random. import random def randfloat(x, y): r = random.randint(0, 2**15) #our random int length = y - x #length of the interval f = (r / 2**15 * length) + x return f (Python 2 will do truncating division up there, so you'd need to make it do floating point division with a __future__ import or a cast.) To be responsible, we should probably up the limit to around 2^54 (which is the limit of precision for the 64-bit floats used by Python), and we'd need to learn about how random randint can be with large numbers, then generate larger random numbers using smaller ones. To be really responsible, we would not assume that floats are 64-bit. (Fortunately, we don't have to be responsible, because Daniel pointed out that this function exists in Python's random.) The idea of linearly mapping one range to another is useful, so please remember it. In fact, the stretch works even if x is bigger than y. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 2 09:43:44 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Feb 2017 14:43:44 +0000 Subject: [Python-ideas] New function to add into the "random" module In-Reply-To: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> References: <58931f9c.4d1c620a.3998e.e6fd@mx.google.com> Message-ID: On 2 February 2017 at 12:01, wrote: > P.P.S.: The function is tested, it?s working ? > First of all, thanks for your suggestion, and for taking the time to help improve Python. As noted, random.uniform probably does what you want, so we don't really need this function. But in case it's of interest to you, there's some additional considerations around proposed additions to the stdlib that we'd normally look at with a suggestion like this. If you find any other improvements you'd be interested in proposing, it might be helpful to you to be aware of them. The first thing that struck me was the performance of your approach. With multiple number<->string conversions it is likely to be extremely slow, making this function completely unsuitable for things like simulations, where the ability to generate many millions of results fast is critical. Python stdlib functions typically get used in a wide range of situations, and we need to make sure they are suitable for as large a range of applications as we can. Also, I'd be curious to know how you tested your function. Random number generators are notoriously difficult to get right, and you are here combining the results of multiple calls to the underlying PRNG. By doing so, you might (as far as I know, it's unlikely, but it's possible) have degraded the randomness of the result. Did your function pass the standard tests for randomness? Note: I don't know what those tests are, but I do know that such things exist, and I'd expect anything in the stdlib to have been validated by people who *do* know such things. Again, stdlib functions get used in a wide range of applications, and the authors typically rely on the Python developers to have "got the details right" on their behalf. Anyway, these are just a couple of thoughts for you. I hope they were of interest, and thanks again for contributing. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri Feb 3 13:30:04 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 3 Feb 2017 10:30:04 -0800 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> Message-ID: <3e4de1b8-76c7-d5de-44d1-e6b667c2b02c@python.org> On 02Feb2017 0601, Cory Benfield wrote: > > 4. Eventually, integrating the two backends above into the standard > library so that it becomes possible to reduce the reliance on OpenSSL. > This would allow future Python implementations to ship with all of their > network protocol libraries supporting platform-native TLS > implementations on Windows and macOS. This will almost certainly require > new PEPs. I?ll probably volunteer to maintain a SecureTransport library, > and I have got verbal suggestions from some other people who?d be > willing to step up and help with that. Again, we?d need help with > SChannel (looking at you, Steve). I'm always somewhat interested in learning a new API that I've literally never looked at before, so yeah, count me in :) (my other work was using the trust APIs directly, rather than the secure socket APIs). PyCon US sprints? It's not looking like I'll be able to set aside too much time before then, but I've already fenced off that time. Cheers, Steve From juraj.sukop at gmail.com Sat Feb 4 06:30:43 2017 From: juraj.sukop at gmail.com (Juraj Sukop) Date: Sat, 4 Feb 2017 12:30:43 +0100 Subject: [Python-ideas] math.nextafter Message-ID: Hello! Function `nextafter(x, y)` returns the next representable value of `x` in the direction of `y`, and if `x` equals to `y`, `y` is returned. [1] It is useful for incrementing/decrementing floating-point number by the smallest amount possible or for testing if two numbers are closest to each other without being the same. The following snippet written by Mark Dickinson emulates the functionality in pure Python [2]: import math import struct def next_up(x): # NaNs and positive infinity map to themselves. if math.isnan(x) or (math.isinf(x) and x > 0): return x # 0.0 and -0.0 both map to the smallest +ve float. if x == 0.0: x = 0.0 n = struct.unpack('= 0: n += 1 else: n -= 1 return struct.unpack(' x: return next_up(x) else: return next_down(x) Other implementations can be found at [3], [4] or [5], for example. It would be useful to have `math.nextafter` function available in standard library, it also is provided by C99 , and is similar in spirit to `math.copysign` or `math.isclose`. As to why to include it by default when the above snippet works just fine, a C implementation is likely to be much faster than using `struct.pack` and `struct.unpack`. Thank you for considering this proposal and any feedback is greatly welcomed! Juraj Sukop [1] http://en.cppreference.com/w/c/numeric/math/nextafter [2] http://stackoverflow.com/a/10426033 [3] http://git.musl-libc.org/cgit/musl/tree/src/math/nextafter.c [4] https://github.com/android/platform_bionic/blob/master/libm/upstream-freebsd/lib/msun/src/s_nextafter.c [5] https://github.com/golang/go/blob/master/src/math/nextafter.go -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sat Feb 4 06:52:18 2017 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Sat, 4 Feb 2017 06:52:18 -0500 Subject: [Python-ideas] math.nextafter In-Reply-To: References: Message-ID: <013401d27edd$234b8630$69e29290$@hotmail.com> The presence of the function in C99?s math.h isn?t strictly useful unless it is also in the MSVC math.h. MSVC only supports a subset of C99 From: Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Juraj Sukop Sent: Saturday, February 4, 2017 6:31 AM To: python-ideas at python.org Subject: [Python-ideas] math.nextafter Hello! Function `nextafter(x, y)` returns the next representable value of `x` in the direction of `y`, and if `x` equals to `y`, `y` is returned. [1] It is useful for incrementing/decrementing floating-point number by the smallest amount possible or for testing if two numbers are closest to each other without being the same. The following snippet written by Mark Dickinson emulates the functionality in pure Python [2]: import math import struct def next_up(x): # NaNs and positive infinity map to themselves. if math.isnan(x) or (math.isinf(x) and x > 0): return x # 0.0 and -0.0 both map to the smallest +ve float. if x == 0.0: x = 0.0 n = struct.unpack('= 0: n += 1 else: n -= 1 return struct.unpack(' x: return next_up(x) else: return next_down(x) Other implementations can be found at [3], [4] or [5], for example. It would be useful to have `math.nextafter` function available in standard library, it also is provided by C99 , and is similar in spirit to `math.copysign` or `math.isclose`. As to why to include it by default when the above snippet works just fine, a C implementation is likely to be much faster than using `struct.pack` and `struct.unpack`. Thank you for considering this proposal and any feedback is greatly welcomed! Juraj Sukop [1] http://en.cppreference.com/w/c/numeric/math/nextafter [2] http://stackoverflow.com/a/10426033 [3] http://git.musl-libc.org/cgit/musl/tree/src/math/nextafter.c [4] https://github.com/android/platform_bionic/blob/master/libm/upstream-freebsd/lib/msun/src/s_nextafter.c [5] https://github.com/golang/go/blob/master/src/math/nextafter.go -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Sat Feb 4 06:59:51 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sat, 4 Feb 2017 12:59:51 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: <013401d27edd$234b8630$69e29290$@hotmail.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> Message-ID: Hi all, Visual C++ 2015 supports this one: https://msdn.microsoft.com/en-us/library/h0dff77w.aspx In any case, this is easy to implement an efficient fallback in C, unlike the fma() function we discussed some time ago. To put this in a bit wider perspective: would it be useful to investigate how much of the C99 math library could be supported in Python in general? Stephan 2017-02-04 12:52 GMT+01:00 : > The presence of the function in C99?s math.h isn?t strictly useful unless > it is also in the MSVC math.h. MSVC only supports a subset of C99 > > > > *From:* Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com@ > python.org] *On Behalf Of *Juraj Sukop > *Sent:* Saturday, February 4, 2017 6:31 AM > *To:* python-ideas at python.org > *Subject:* [Python-ideas] math.nextafter > > > > Hello! > > > > Function `nextafter(x, y)` returns the next representable value of `x` in > the direction of `y`, and if `x` equals to `y`, `y` is returned. [1] > > > > It is useful for incrementing/decrementing floating-point number by the > smallest amount possible or for testing if two numbers are closest to each > other without being the same. > > > > The following snippet written by Mark Dickinson emulates the functionality > in pure Python [2]: > > > > import math > > import struct > > > > def next_up(x): > > # NaNs and positive infinity map to themselves. > > if math.isnan(x) or (math.isinf(x) and x > 0): > > return x > > > > # 0.0 and -0.0 both map to the smallest +ve float. > > if x == 0.0: > > x = 0.0 > > > > n = struct.unpack(' > if n >= 0: > > n += 1 > > else: > > n -= 1 > > return struct.unpack(' > > > def next_down(x): > > return -next_up(-x) > > > > def next_after(x, y): > > # If either argument is a NaN, return that argument. > > # This matches the implementation in decimal.Decimal > > if math.isnan(x): > > return x > > if math.isnan(y): > > return y > > > > if y == x: > > return y > > elif y > x: > > return next_up(x) > > else: > > return next_down(x) > > > > Other implementations can be found at [3], [4] or [5], for example. > > > > It would be useful to have `math.nextafter` function available in standard > library, it also is provided by C99 , and is similar in spirit to > `math.copysign` or `math.isclose`. > > > > As to why to include it by default when the above snippet works just fine, > a C implementation is likely to be much faster than using `struct.pack` and > `struct.unpack`. > > > > Thank you for considering this proposal and any feedback is greatly > welcomed! > > > > Juraj Sukop > > > > [1] http://en.cppreference.com/w/c/numeric/math/nextafter > > [2] http://stackoverflow.com/a/10426033 > > [3] http://git.musl-libc.org/cgit/musl/tree/src/math/nextafter.c > > [4] https://github.com/android/platform_bionic/blob/master/ > libm/upstream-freebsd/lib/msun/src/s_nextafter.c > > [5] https://github.com/golang/go/blob/master/src/math/nextafter.go > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Sat Feb 4 10:15:11 2017 From: steve.dower at python.org (Steve Dower) Date: Sat, 4 Feb 2017 07:15:11 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: <013401d27edd$234b8630$69e29290$@hotmail.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> Message-ID: These days, the subset of C99 supported by MSVC is "most" of it, so feel free to start off by assuming the best, at least for new features (the version we use for 2.7 obviously is not improving). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "tritium-list at sdamon.com" Sent: ?2/?4/?2017 3:53 To: "python-ideas at python.org" Subject: Re: [Python-ideas] math.nextafter The presence of the function in C99?s math.h isn?t strictly useful unless it is also in the MSVC math.h. MSVC only supports a subset of C99 From: Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Juraj Sukop Sent: Saturday, February 4, 2017 6:31 AM To: python-ideas at python.org Subject: [Python-ideas] math.nextafter Hello! Function `nextafter(x, y)` returns the next representable value of `x` in the direction of `y`, and if `x` equals to `y`, `y` is returned. [1] It is useful for incrementing/decrementing floating-point number by the smallest amount possible or for testing if two numbers are closest to each other without being the same. The following snippet written by Mark Dickinson emulates the functionality in pure Python [2]: import math import struct def next_up(x): # NaNs and positive infinity map to themselves. if math.isnan(x) or (math.isinf(x) and x > 0): return x # 0.0 and -0.0 both map to the smallest +ve float. if x == 0.0: x = 0.0 n = struct.unpack('= 0: n += 1 else: n -= 1 return struct.unpack(' x: return next_up(x) else: return next_down(x) Other implementations can be found at [3], [4] or [5], for example. It would be useful to have `math.nextafter` function available in standard library, it also is provided by C99 , and is similar in spirit to `math.copysign` or `math.isclose`. As to why to include it by default when the above snippet works just fine, a C implementation is likely to be much faster than using `struct.pack` and `struct.unpack`. Thank you for considering this proposal and any feedback is greatly welcomed! Juraj Sukop [1] http://en.cppreference.com/w/c/numeric/math/nextafter [2] http://stackoverflow.com/a/10426033 [3] http://git.musl-libc.org/cgit/musl/tree/src/math/nextafter.c [4] https://github.com/android/platform_bionic/blob/master/libm/upstream-freebsd/lib/msun/src/s_nextafter.c [5] https://github.com/golang/go/blob/master/src/math/nextafter.go -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Sat Feb 4 13:18:42 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sat, 4 Feb 2017 18:18:42 +0000 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: <3e4de1b8-76c7-d5de-44d1-e6b667c2b02c@python.org> References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> <3e4de1b8-76c7-d5de-44d1-e6b667c2b02c@python.org> Message-ID: <44468183-EA18-4CD9-9E31-D15C3F41E7C0@lukasa.co.uk> > On 3 Feb 2017, at 18:30, Steve Dower wrote: > > On 02Feb2017 0601, Cory Benfield wrote: >> >> 4. Eventually, integrating the two backends above into the standard >> library so that it becomes possible to reduce the reliance on OpenSSL. >> This would allow future Python implementations to ship with all of their >> network protocol libraries supporting platform-native TLS >> implementations on Windows and macOS. This will almost certainly require >> new PEPs. I?ll probably volunteer to maintain a SecureTransport library, >> and I have got verbal suggestions from some other people who?d be >> willing to step up and help with that. Again, we?d need help with >> SChannel (looking at you, Steve). > > I'm always somewhat interested in learning a new API that I've literally never looked at before, so yeah, count me in :) (my other work was using the trust APIs directly, rather than the secure socket APIs). > > PyCon US sprints? It's not looking like I'll be able to set aside too much time before then, but I've already fenced off that time. That might be a really good idea. With feedback from Nathaniel and Glyph I?m going back to the drawing board a bit with this PEP to see if we can reduce the amount of work needed from backends, so this may shrink down to something that can feasibly be done in a sprint. For those who are interested, the refactoring proposed by Nathaniel and Glyph is to drop the abstract TLSWrappedSocket class, and instead replace it with a *concrete* TLSWrappedSocket class that is given a socket and an implementation of TLSWrappedBuffers. This would essentially mean that implementations only need to write a TLSWrappedBuffers implementation and would get a TLSWrappedSocket essentially for free (with the freedom to provide a complete reimplementation of TLSWrappedSocket if they need to). I?m going to investigate the feasibility of that by writing a stub TLSWrappedBuffers for SecureTransport and then writing the TLSWrappedSocket implementation to validate what it looks like. Assuming the end result looks generally suitable, I?ll come back with a new draft. But the TL;DR is that if we do that all we need to implement on the Windows side is one-or-two classes, and we?d be ready to go. That?d be really nice. Cory From christian at python.org Sat Feb 4 16:45:03 2017 From: christian at python.org (Christian Heimes) Date: Sat, 4 Feb 2017 22:45:03 +0100 Subject: [Python-ideas] Unified TLS API for Python In-Reply-To: <44468183-EA18-4CD9-9E31-D15C3F41E7C0@lukasa.co.uk> References: <30102BA2-FBB0-4086-A1FA-36D098EF72CF@lukasa.co.uk> <3e4de1b8-76c7-d5de-44d1-e6b667c2b02c@python.org> <44468183-EA18-4CD9-9E31-D15C3F41E7C0@lukasa.co.uk> Message-ID: On 2017-02-04 19:18, Cory Benfield wrote: > >> On 3 Feb 2017, at 18:30, Steve Dower wrote: >> >> On 02Feb2017 0601, Cory Benfield wrote: >>> >>> 4. Eventually, integrating the two backends above into the standard >>> library so that it becomes possible to reduce the reliance on OpenSSL. >>> This would allow future Python implementations to ship with all of their >>> network protocol libraries supporting platform-native TLS >>> implementations on Windows and macOS. This will almost certainly require >>> new PEPs. I?ll probably volunteer to maintain a SecureTransport library, >>> and I have got verbal suggestions from some other people who?d be >>> willing to step up and help with that. Again, we?d need help with >>> SChannel (looking at you, Steve). >> >> I'm always somewhat interested in learning a new API that I've literally never looked at before, so yeah, count me in :) (my other work was using the trust APIs directly, rather than the secure socket APIs). >> >> PyCon US sprints? It's not looking like I'll be able to set aside too much time before then, but I've already fenced off that time. > > That might be a really good idea. > > With feedback from Nathaniel and Glyph I?m going back to the drawing board a bit with this PEP to see if we can reduce the amount of work needed from backends, so this may shrink down to something that can feasibly be done in a sprint. > > For those who are interested, the refactoring proposed by Nathaniel and Glyph is to drop the abstract TLSWrappedSocket class, and instead replace it with a *concrete* TLSWrappedSocket class that is given a socket and an implementation of TLSWrappedBuffers. This would essentially mean that implementations only need to write a TLSWrappedBuffers implementation and would get a TLSWrappedSocket essentially for free (with the freedom to provide a complete reimplementation of TLSWrappedSocket if they need to). > > I?m going to investigate the feasibility of that by writing a stub TLSWrappedBuffers for SecureTransport and then writing the TLSWrappedSocket implementation to validate what it looks like. Assuming the end result looks generally suitable, I?ll come back with a new draft. But the TL;DR is that if we do that all we need to implement on the Windows side is one-or-two classes, and we?d be ready to go. That?d be really nice. At first I was a bit worried that you were planning to chance the semantic of socket wrapping. The PEP doesn't get into detail how wrap_socket() works internally. How about you add a paragraph that the function wraps an OS-level file descriptors (POSIX) or socket handles (Windows). For some TLS libraries it's an optimization to reduce memory copies and overhead of extra GIL lock/unlock calls. Other TLS libraries (NSS, Linux kernel TLS with AF_KTLS) can only operator on file descriptors. In fact NSS can only operator on NSPR PRFileDesc, but NSPR has an API to wrap an OS-level fd in a PRFileDesc. By the way, how can a TLS implementation announce that it does not provide buffer wrapping? Christian From mal at egenix.com Mon Feb 6 05:29:17 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 6 Feb 2017 11:29:17 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> Message-ID: <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> On 04.02.2017 12:59, Stephan Houben wrote: > Hi all, > > Visual C++ 2015 supports this one: > > https://msdn.microsoft.com/en-us/library/h0dff77w.aspx > > In any case, this is easy to implement an efficient fallback in C, unlike > the fma() function we discussed some time ago. > > To put this in a bit wider perspective: would it be useful to investigate > how much of the C99 math library could > be supported in Python in general? +1 from me for those features which can be emulated for platforms which don't have the math lib function available and are not too esoteric (though many of those have already been added), e.g. cbt() may be useful. Now, with respect to the one mentioned in the subject, I'm not sure how useful this would be in the stdlib, since it's very much tied to whatever float type Python happens to use on a platform. Juraj: Could you provide some use cases, where such a function would help in Python applications ? (I can see use cases written in C, but due to the low level, find it hard to believe that people would use this at the Python level) Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 06 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From juraj.sukop at gmail.com Mon Feb 6 07:22:02 2017 From: juraj.sukop at gmail.com (Juraj Sukop) Date: Mon, 6 Feb 2017 13:22:02 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> Message-ID: On Mon, Feb 6, 2017 at 11:29 AM, M.-A. Lemburg wrote: > > Juraj: Could you provide some use cases, where such a function > would help in Python applications ? (I can see use cases > written in C, but due to the low level, find it hard to > believe that people would use this at the Python level) > In my case, `nextafter` would be used to check if a number is close to polynomial zero, e.g.: def f(x): return 2.0*x**3 - 3.0*x**2 + 5.0*x - 7.0 # x = 1.4455284586795218 x = 1.445528458679522 # x = 1.4455284586795223 # x = 1.4455284586795225 left = nextafter(x, -float('inf')) right = nextafter(x, float('inf')) print((f(left) < 0.0) != (f(x) < 0.0) or (f(x) < 0.0) != (f(right) < 0.0)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 6 08:16:56 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 6 Feb 2017 14:16:56 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> Message-ID: <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> On 06.02.2017 13:22, Juraj Sukop wrote: > On Mon, Feb 6, 2017 at 11:29 AM, M.-A. Lemburg wrote: > >> >> Juraj: Could you provide some use cases, where such a function >> would help in Python applications ? (I can see use cases >> written in C, but due to the low level, find it hard to >> believe that people would use this at the Python level) >> > > In my case, `nextafter` would be used to check if a number is close to > polynomial zero, e.g.: > > def f(x): > return 2.0*x**3 - 3.0*x**2 + 5.0*x - 7.0 > > # x = 1.4455284586795218 > x = 1.445528458679522 > # x = 1.4455284586795223 > # x = 1.4455284586795225 > > left = nextafter(x, -float('inf')) > right = nextafter(x, float('inf')) > > print((f(left) < 0.0) != (f(x) < 0.0) or (f(x) < 0.0) != (f(right) < > 0.0)) Isn't this something you can do with math.isclose() ? This would even give you a predefined error range, not a dynamic one. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 06 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From juraj.sukop at gmail.com Mon Feb 6 08:42:09 2017 From: juraj.sukop at gmail.com (Juraj Sukop) Date: Mon, 6 Feb 2017 14:42:09 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: Do you mean something like: isclose(f(x), 0.0, rel_tol, abs_tol) If so, what should `rel_tol` and `abs_tol` be? On Mon, Feb 6, 2017 at 2:16 PM, M.-A. Lemburg wrote: > On 06.02.2017 13:22, Juraj Sukop wrote: > > On Mon, Feb 6, 2017 at 11:29 AM, M.-A. Lemburg wrote: > > > >> > >> Juraj: Could you provide some use cases, where such a function > >> would help in Python applications ? (I can see use cases > >> written in C, but due to the low level, find it hard to > >> believe that people would use this at the Python level) > >> > > > > In my case, `nextafter` would be used to check if a number is close to > > polynomial zero, e.g.: > > > > def f(x): > > return 2.0*x**3 - 3.0*x**2 + 5.0*x - 7.0 > > > > # x = 1.4455284586795218 > > x = 1.445528458679522 > > # x = 1.4455284586795223 > > # x = 1.4455284586795225 > > > > left = nextafter(x, -float('inf')) > > right = nextafter(x, float('inf')) > > > > print((f(left) < 0.0) != (f(x) < 0.0) or (f(x) < 0.0) != (f(right) < > > 0.0)) > > Isn't this something you can do with math.isclose() ? > > This would even give you a predefined error range, > not a dynamic one. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Feb 06 2017) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Feb 6 18:46:49 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 7 Feb 2017 10:46:49 +1100 Subject: [Python-ideas] math.nextafter In-Reply-To: <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> Message-ID: <20170206234649.GU7345@ando.pearwood.info> On Mon, Feb 06, 2017 at 11:29:17AM +0100, M.-A. Lemburg wrote: > I'm not sure how useful this would be in the stdlib, > since it's very much tied to whatever float type Python > happens to use on a platform. With the possible exception of ?Py, are there any Python implementations which don't use a C double as float? -- Steve From mal at egenix.com Mon Feb 6 20:44:31 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 7 Feb 2017 02:44:31 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: <20170206234649.GU7345@ando.pearwood.info> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <20170206234649.GU7345@ando.pearwood.info> Message-ID: <6491259b-37ee-aa18-8b39-e98d73a4f1f4@egenix.com> On 07.02.2017 00:46, Steven D'Aprano wrote: > On Mon, Feb 06, 2017 at 11:29:17AM +0100, M.-A. Lemburg wrote: > >> I'm not sure how useful this would be in the stdlib, >> since it's very much tied to whatever float type Python >> happens to use on a platform. > > With the possible exception of ?Py, are there any Python implementations > which don't use a C double as float? (I'm not sure what you mean with "?Py") By far most implementations will use IEEE 754 64-bit doubles: https://en.wikipedia.org/wiki/Double-precision_floating-point_format MicroPython also supports single precision float configurations or no floats at all (depends on the availability of an FPU). What I wanted to say with the above comment is that Python itself does not make guarantees on how floats are represented internally (in CPython it's a C double, which only has minimum requirements). In theory, it would be possible to have a Python implementation using e.g. binary128 floats or MPFR (http://www.mpfr.org/). You are right in saying that this is a rather unlikely case :-) However, using the internal representation of floats to define a "valid" interval for roots doesn't sound like a good concept. Those intervals will depend on where the roots are. Close to 0, the intervals will be very small, near the outer range limits, very large: https://blogs.msdn.microsoft.com/dwayneneed/2010/05/06/fun-with-floating-point/ I guess there is some use in trying to determine the error intervals of calculations, though. Perhaps that's the main intent of those functions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 07 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From anthony at xtfx.me Tue Feb 7 01:05:49 2017 From: anthony at xtfx.me (C Anthony Risinger) Date: Tue, 7 Feb 2017 00:05:49 -0600 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: Message-ID: On Sat, Jan 28, 2017 at 5:26 AM, Paul Moore wrote: > On 28 January 2017 at 02:11, C Anthony Risinger wrote: > > I can't articulate it we'll, or even fully isolate the reasons for it. > All I > > really know is how I feel when peers ask me about Python or the reading I > > get when others speak about their experience using it. Python is > absolutely > > one of my favorite languages to write, yet I find myself recommending > > against it, and watching others do the same. Python comes with caveats > and > > detailed explanations out the gate and people simply perceive higher > > barriers and more chores. > > Picking up on this and the comment you made in the original post > > > With a still difficult distribution/compatibility story, I've watched > dozens of instances > > where people choose something else, usually Node or Golang. > > Can you explain why you recommend against Python, in a bit more > detail? If you are an enthusiastic Python user, but you are steering > people away from Python, then it would be worth understanding why. > > As you mention end user applications and distribution, one of my first > questions would be what platform you work on. Following on from that, > what sort of end user applications are you looking at? If we're > talking here about games for iOS, then that's a much different > situation than GUI apps for Windows or command line tools for Linux. > I'm working on Linux myself, with my typical audience being other developers or occasionally leadership. Builds usually include Windows, but always Linux and OSX. I'd like a one-click solution to: How do I redistribute and successfully install Python, dependencies, and an application with the least possible steps for the end user? For any platform or persona? I prefer dynamic applications redistribute their runtime, and not depend on the system in any significant way aside from major libraries or support files. It's more obnoxious sometimes but I believe it lowers the number of unexpected incidents. I also don't want to setup support infrastructure (pypi proxy cache or self hosted pypi) myself or require coordination with ops teams. Simple authenticated download for publishing is ideal, and direct execution of that download is even better. Containers are have greatly impacted how people think about distribution. There is a trend to use fast/small containers as CLI tools. I think there is still benefit to empowering a way to "remix python" with your local environment and produce a redistributable binary for downstream users. I have far less experience with Go but I like how easy it is to generate such executables. My personal feeling is that Python happily works in the "Command line > tools for Linux" area (except possibly with regard to C extensions > where the plethora of Linux ABIs makes things hard). But other areas > less so. I've been having good experiences making standalone > applications with the new Windows "embedded" distributions, but that > is relatively new, and still has a lot of rough edges. I'm working on > a project to bundle a working zipapp with the embedded distribution to > make a standalone exe - would having something like that make any > difference in your environment? > Yes I believe it would. I implemented something to this end for Linux (defunct, no Python 3, uses abandoned PEP 425): https://github.com/anthonyrisinger/zippy about 4-5 years ago for this purpose, before similar technologies such as PEX and zipapp existed. It worked well for the time it was used. The project is dead, but it had a few ideas I think might be worth further exploration. In a nutshell, "zippy" was a build tool (waf and distlib because no interface to pip) that always output a single python binary built from source. All python code (stdlib, deps, app) was appended in a zipfile. All C-extensions (including dotted) were observed during .so generation and re-linked statically into the binary as builtins instead (more on this below). The end result was a self-contained executable that would only recognize it's own stdlib and could properly run embedded C-modules. The implementation also changed entrypoints based on argv[0], so this could thing could perform as a multicall binary like BusyBox (the symlink name was resolved to an entrypoint). It was easy to send this to other developers where they could unpack it for development or use it as-is. A future goal was to allow respinning an unpacked environment into a new redistributable. To make this work, I changed python in a few specific ways: Set LANDMARK file at build time to something other than `os.py`. Python uses this file to recognize the stdlib during early boot. Customizing it [zippy-SOMEHASH.json] ensures a different unpacked build's stdlib is unrecognizable (this happens *often* if you've "installed" a build by unpacking to the default path, then do another build). Add `sys.executable` to the default search path. This ensures python first scans the filesystem for an unpacked environment but always falls back to its own appended zipfile. Combined with custom LANDMARK this made the python installation executable from any on-disk location. Build python locally, install app and all deps into the local build, re-link all C-extensions statically, append zipfile to python binary. I *may* have went overboard, but I generalized a method for statically linking numpy into python, I believe originally described at http://yt.enzotools.org/wiki/CrayXT5Installation . In short, distutils is monkeypatched to trace calls to `unixccompiler.UnixCCompiler.link_shared_object`. A normal object file with strict symbol exports, similar to a shared object, is written alongside the shared object, and Modules/Setup is updated. After deps are installed from pypi, python is re-linked with all the newly traced C-ext static objects. The process exposed a few interesting bugs in python and elsewhere (possibly since fixed) but was known to work for all stdlib C-extensions and every popular library we used throughout the company (psycopg2, librabbitmq, sqlite3, and dozens others were all successfully linked into the same process). This is approximately when Golang starting looking attractive to me. The past few years have seen serious improvements to this space for Python but I think there is a ways to go yet. Python has different challenges and advantages because it's older and dynamic, but I think some activity around all this might make it easier to customize Python builds. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Tue Feb 7 01:35:01 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 6 Feb 2017 22:35:01 -0800 Subject: [Python-ideas] Is it Python 3 yet? In-Reply-To: References: Message-ID: <3a474d92-06bc-5969-a154-5d50da300d5d@mgmiller.net> Hmm, agreed. BTW, I think the current download page is *way* too complicated for new comers. There should be a giant button for the latest 3.x/64 (platform sniffed), and below it a more subtle button for the "LTS" 2.X/32. The rest of the choices and text should be pushed to another page marked "advanced," including the PGP (boggle) section. The Licenses through History section could be moved to About. -Mike On 2017-01-26 08:11, Victor Stinner wrote: > The download button of https://www.python.org/ currently gives the From victor.stinner at gmail.com Tue Feb 7 04:15:07 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 7 Feb 2017 10:15:07 +0100 Subject: [Python-ideas] Is it Python 3 yet? Results on my poll on Twitter Message-ID: Hi, I created the following poll on Twitter with a duration of 7 days: """ Is it Python 3 yet? https://mail.python.org/pipermail/python-ideas/2017-January/044441.html I proposed to hide Python 2 by default from the http://python.org download page. ( ) It's Python 3 O'Clock! ( ) Have some legacy py2 code """ Results on 692 votes: - 84% It's Python 3 O'Clock! - 16% Have some legacy py2 code I didn't expect so many votes! Victor From thomas at kluyver.me.uk Tue Feb 7 07:33:29 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 07 Feb 2017 12:33:29 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: Message-ID: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> On Sat, Jan 28, 2017, at 11:26 AM, Paul Moore wrote: > I'm working on > a project to bundle a working zipapp with the embedded distribution to > make a standalone exe - would having something like that make any > difference in your environment? I'd be interested in this, and whether there's any potential for code sharing with Pynsist - which does something similar, but producing an installer rather than a standalone exe. Thomas From p.f.moore at gmail.com Tue Feb 7 08:02:10 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Feb 2017 13:02:10 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: On 7 February 2017 at 12:33, Thomas Kluyver wrote: > On Sat, Jan 28, 2017, at 11:26 AM, Paul Moore wrote: >> I'm working on >> a project to bundle a working zipapp with the embedded distribution to >> make a standalone exe - would having something like that make any >> difference in your environment? > > I'd be interested in this, and whether there's any potential for code > sharing with Pynsist - which does something similar, but producing an > installer rather than a standalone exe. https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul From steve.dower at python.org Tue Feb 7 09:29:05 2017 From: steve.dower at python.org (Steve Dower) Date: Tue, 7 Feb 2017 06:29:05 -0800 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: You can leave python.exe out of your distribution to avoid it showing up on PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then python36.dll you should be able to put them wherever you like. I think what we really want is a self-extractor that "installs" into the user's AppData directory without prompting for admin. I'm seeing more apps about like this and I think it's the right model for Python. We probably want to include the Electron shell as well, or make it trivially easy to add (I've been using a built-in Windows tool that is similar, but it's not ideal). There's a real chance we could have very modern, cross-platform Python apps with this approach. I don't think the tools for my current project will ever see public release (though the project itself should), but the experience has been useful. Thomas's work with pynsist is also informative, at least for the issues on Windows (we're already at the point where things would be easiest if Windows 7 just went away :) ). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Paul Moore" Sent: ?2/?7/?2017 5:03 To: "Thomas Kluyver" Cc: "Python-Ideas" Subject: Re: [Python-ideas] Using Python for end user applications On 7 February 2017 at 12:33, Thomas Kluyver wrote: > On Sat, Jan 28, 2017, at 11:26 AM, Paul Moore wrote: >> I'm working on >> a project to bundle a working zipapp with the embedded distribution to >> make a standalone exe - would having something like that make any >> difference in your environment? > > I'd be interested in this, and whether there's any potential for code > sharing with Pynsist - which does something similar, but producing an > installer rather than a standalone exe. https://github.com/pfmoore/pylaunch It's a very simple C stub that can be prepended to a zipapp, which then runs it with an embedded distribution. At the moment it's a bit stalled because the manifest hacking needed for putting the embedded distribution in a subdirectory doesn't work with Python 3.5, and there's a bug in 3.6 that breaks using Py_Main() with a zipapp. But it works, all I really need to do is to write some management code to automate the process of building the exe from a script. (I don't want to require the Python distribution to be in the same directory as the application, as then there would be a "python.exe" in that directory, which messes up my use case of having the apps on PATH). The fact that it's likely to end up being 3.6.1+ only means I'm not feeling in any particular rush to make progress. At the moment I'm occasionally dabbling with it to see if I can find workarounds for the issues with 3.6.0 and 3.5, but I'm not *really* interested in maintaining significantly more complex C code just to deal with backward compatible bug workarounds :-) Paul _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at kluyver.me.uk Tue Feb 7 09:47:26 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Tue, 07 Feb 2017 14:47:26 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: <1486478846.2137807.873141704.4492B401@webmail.messagingengine.com> On Tue, Feb 7, 2017, at 02:29 PM, Steve Dower wrote: > I think what we really want is a self-extractor that "installs" into > the user's AppData directory without prompting for admin. There's a PR in the works for Pynsist that will add a non-admin per-user install into AppData: https://github.com/takluyver/pynsist/pull/100 Because of the way permissions work on Windows, if you do have admin privileges, the installer will still request them so that it can offer you the choice of a systemwide installation. But if you don't, it should happily do a user installation without them. > We probably want to include the Electron shell as well, or make it > trivially easy to add (I've been using a built-in Windows tool that is > similar, but it's not ideal). There's a real chance we could have very > modern, cross-platform Python apps with this approach. I've been thinking for a while about Python apps using Electron (Positron? ;-). It's an interesting idea from the Python side, but I struggle to come up with reasons why developing an Electron+Python app would be easier than developing a regular Electron app. I prefer writing Python to Javascript, but you'd need quite a bit of Javascript anyway, you don't have to care about browser compatibility, and there would inevitably be some extra friction in using two languages. I'm sure there are use cases where it makes sense, like if you use Python's scientific computing ecosystem. But I don't know how broad they are. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Feb 7 10:27:56 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Feb 2017 15:27:56 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: On 7 February 2017 at 14:29, Steve Dower wrote: > You can leave python.exe out of your distribution to avoid it showing up on > PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then > python36.dll you should be able to put them wherever you like. Understood, but I may need python.exe present if the script uses multiprocessing, so I'm trying to avoid doing that (while I'm doing things manually, I can do what I like, obviously, but a generic "build my app" tool has to be a bit more cautious). LoadLibrary might work (I'm only calling Py_Main). I seem to recall trying this before and having issues but that might have been an earlier iteration which made more complex use of the C API. Also, I want to load python3.dll (the stable ABI) as I don't want to have to rebuild the stub once for each Python version, or have to search for the correct DLL in C. But I'll definitely give that a go. > I think what we really want is a self-extractor that "installs" into the > user's AppData directory without prompting for admin. I'm seeing more apps > about like this and I think it's the right model for Python. My goal is explicitly self-contained applications - the "xcopy deployment" model. I'm looking for things that can be used in *very* constrained environments (no install ability, possibly not even any access to locations outside of My Documents). > We probably > want to include the Electron shell as well, or make it trivially easy to add > (I've been using a built-in Windows tool that is similar, but it's not > ideal). There's a real chance we could have very modern, cross-platform > Python apps with this approach. I presume you mean http://electron.atom.io/? I've never used it, but again my goal is essentially command line apps here. Definitely "pure Python" apps. I don't know how I'd use electron from a Python script, but I'm guessing you're looking more at targeting GUI apps with traditional installers? It's an interesting area (GUI interfaces have always been a rough edge in Python development) but not one I am looking at at the moment. (Offtopic, but are there any tutorials on building apps using Python and Electron, as it sounds like an interesting thing to investigate). Paul From srkunze at mail.de Tue Feb 7 11:38:28 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 7 Feb 2017 17:38:28 +0100 Subject: [Python-ideas] Is it Python 3 yet? In-Reply-To: <3a474d92-06bc-5969-a154-5d50da300d5d@mgmiller.net> References: <3a474d92-06bc-5969-a154-5d50da300d5d@mgmiller.net> Message-ID: Agree with all of this. That's state of the art for many projects. On 07.02.2017 07:35, Mike Miller wrote: > Hmm, agreed. BTW, I think the current download page is *way* too > complicated for new comers. > > There should be a giant button for the latest 3.x/64 (platform sniffed), > and below it a more subtle button for the "LTS" 2.X/32. > > The rest of the choices and text should be pushed to another page > marked "advanced," including the PGP (boggle) section. > > The Licenses through History section could be moved to About. > > -Mike > > > On 2017-01-26 08:11, Victor Stinner wrote: >> The download button of https://www.python.org/ currently gives the > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From random832 at fastmail.com Tue Feb 7 14:54:18 2017 From: random832 at fastmail.com (Random832) Date: Tue, 07 Feb 2017 14:54:18 -0500 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> Message-ID: <1486497258.224033.873530488.19584ADA@webmail.messagingengine.com> On Sat, Feb 4, 2017, at 10:15, Steve Dower wrote: > These days, the subset of C99 supported by MSVC is "most" of it, so feel > free to start off by assuming the best, at least for new features (the > version we use for 2.7 obviously is not improving). The subset of the C *library*, anyway, since it was incorporated wholesale into the C++ standard. Probably worth taking more care with syntactic features. From pavol.lisy at gmail.com Tue Feb 7 16:00:59 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Tue, 7 Feb 2017 22:00:59 +0100 Subject: [Python-ideas] Is it Python 3 yet? In-Reply-To: <3a474d92-06bc-5969-a154-5d50da300d5d@mgmiller.net> References: <3a474d92-06bc-5969-a154-5d50da300d5d@mgmiller.net> Message-ID: On 2/7/17, Mike Miller wrote: > Hmm, agreed. BTW, I think the current download page is *way* too > complicated > for new comers. > > There should be a giant button for the latest 3.x/64 (platform sniffed), > and below it a more subtle button for the "LTS" 2.X/32. I am afraid that "LTS" could confuse some people because python has not standard LTS versions ( PEP-407 is deffered ) I rather propose to add end-of-life info (which is 2020-01-01 for 2.7 and 2021-12-23 for 3.6) See: https://docs.python.org/devguide/#status-of-python-branches (BTW link to this table could be useful to add on download page too) One could probably also want to consider that some packages plan to drop support for python2 before it's EOL (for example see: http://www.python3statement.org/ ) but I think that download page has not to be overcomplicated. From mahmoud at hatnote.com Tue Feb 7 16:17:55 2017 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Tue, 7 Feb 2017 13:17:55 -0800 Subject: [Python-ideas] Is it Python 3 yet? Results on my poll on Twitter In-Reply-To: References: Message-ID: Let the dissection of the Twitter-based biases and bubbles begin. :) On Tue, Feb 7, 2017 at 1:15 AM, Victor Stinner wrote: > Hi, > > I created the following poll on Twitter with a duration of 7 days: > """ > Is it Python 3 yet? > https://mail.python.org/pipermail/python-ideas/2017-January/044441.html > I proposed to hide Python 2 by default from the http://python.org > download page. > > ( ) It's Python 3 O'Clock! > ( ) Have some legacy py2 code > """ > > Results on 692 votes: > > - 84% It's Python 3 O'Clock! > - 16% Have some legacy py2 code > > I didn't expect so many votes! > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eryksun at gmail.com Tue Feb 7 19:49:44 2017 From: eryksun at gmail.com (eryk sun) Date: Wed, 8 Feb 2017 00:49:44 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: On Tue, Feb 7, 2017 at 3:27 PM, Paul Moore wrote: > On 7 February 2017 at 14:29, Steve Dower wrote: >> You can leave python.exe out of your distribution to avoid it showing up on >> PATH, or if your stub explicitly LoadLibrary's vcruntime140.dll and then >> python36.dll you should be able to put them wherever you like. > > Understood, but I may need python.exe present if the script uses > multiprocessing, so I'm trying to avoid doing that (while I'm doing > things manually, I can do what I like, obviously, but a generic "build > my app" tool has to be a bit more cautious). > > LoadLibrary might work (I'm only calling Py_Main). I seem to recall > trying this before and having issues but that might have been an > earlier iteration which made more complex use of the C API. Also, I > want to load python3.dll (the stable ABI) as I don't want to have to > rebuild the stub once for each Python version, or have to search for > the correct DLL in C. But I'll definitely give that a go. LoadLibrary and GetProcAddress will work, but that would get tedious if a program needed a lot of Python's API. It's also a bit of a kludge having to manually call LoadLibrary with a given DLL order. For the latter, I wish we could simply load python3.dll using LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH, but it doesn't work in good old Windows 7. python3.dll doesn't depend on python3x.dll in its DLL import table. I discovered in issue 29399 that in this case the loader in Windows 7 doesn't use the altered search path of python3.dll to load python3x.dll and vcruntime140.dll. As you're currently doing (as we discussed last September), creating an assembly in a subdirectory works in all supported Windows versions, and it's the most convenient way to access all of Python's limited API. From p.f.moore at gmail.com Wed Feb 8 04:07:37 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Feb 2017 09:07:37 +0000 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> Message-ID: On 8 February 2017 at 00:49, eryk sun wrote: >> LoadLibrary might work (I'm only calling Py_Main). I seem to recall >> trying this before and having issues but that might have been an >> earlier iteration which made more complex use of the C API. Also, I >> want to load python3.dll (the stable ABI) as I don't want to have to >> rebuild the stub once for each Python version, or have to search for >> the correct DLL in C. But I'll definitely give that a go. > > LoadLibrary and GetProcAddress will work, but that would get tedious > if a program needed a lot of Python's API. It's also a bit of a kludge > having to manually call LoadLibrary with a given DLL order. Tell me about it :-) That's what the Vim Python interface does (not so much to load when not on PATH, as to degrade gracefully when Python isn't installed), and it's stunningly painful to set up :-) > For the latter, I wish we could simply load python3.dll using > LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH, but it doesn't work > in good old Windows 7. python3.dll doesn't depend on python3x.dll in > its DLL import table. I discovered in issue 29399 that in this case > the loader in Windows 7 doesn't use the altered search path of > python3.dll to load python3x.dll and vcruntime140.dll. That was my problem. I still have to support Windows 7. Thanks for the reminder. > As you're currently doing (as we discussed last September), creating > an assembly in a subdirectory works in all supported Windows versions, > and it's the most convenient way to access all of Python's limited > API. Yes, it's a really good method, although it doesn't work 100% for Python 3.5, as the "pyvenv.cfg" approach for isolating the Python environment has a glitch. Offhand I can't recall the details (and as it's fixed with the new approach in 3.6, it's not really relevant any more) but IIRC I was getting some issues with 3.5. Hence why my solution is likely to be 3.6+ only. Paul From ncoghlan at gmail.com Thu Feb 9 13:56:14 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Feb 2017 19:56:14 +0100 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: <1486478846.2137807.873141704.4492B401@webmail.messagingengine.com> References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> <1486478846.2137807.873141704.4492B401@webmail.messagingengine.com> Message-ID: On 7 February 2017 at 15:47, Thomas Kluyver wrote: > I've been thinking for a while about Python apps using Electron (Positron? > ;-). It's an interesting idea from the Python side, but I struggle to come > up with reasons why developing an Electron+Python app would be easier than > developing a regular Electron app. I prefer writing Python to Javascript, > but you'd need quite a bit of Javascript anyway, you don't have to care > about browser compatibility, and there would inevitably be some extra > friction in using two languages. > > I'm sure there are use cases where it makes sense, like if you use Python's > scientific computing ecosystem. But I don't know how broad they are. I'd say the rationale for Electron/Python apps is the same as that for any JS frontend/Python backend configuration - JS/CSS/HTML5 is a great suite of technologies for defining user interfaces, but you don't necessarily want to be writing all your application logic in it. (You certainly *can*, you just may not want to) The trade-offs are different for client-side apps (since shipping two different language runtimes is kinda horrible, given neither V8 nor CPython is particularly lightweight), but it's not *that* different from the traditional Python GUI app development model of depending on a C/C++ toolkit like Tcl/Tk, Gtk, Qt, or wxWidgets. It's just that the modern GUI toolkit is called V8, most of the actual GUI bits are written in JavaScript rather than C/C++, and the language independent in-process bindings got fairly dramatically worse along the way :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From markusmeskanen at gmail.com Fri Feb 10 04:13:47 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 11:13:47 +0200 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator Message-ID: I'm suggesting the addition of support to using a dot notation when defining a function to be a method of a class, or a callback attribute. For example: def foo(self): pass Foo.foo = foo Becomes: def Foo.foo(self): pass Other syntaxes can also be used if the dot itself is an issue, although I dislike these: def Foo:foo(self): def foo at Foo(self): def Foo>foo(self): def Foo&foo(self): This functionality would be useful in the few rare cases where the class itself needs to be accessed in the function's definition (decorator, typing, etc.): @barify(Foo) def Foo.method(self, other: Foo) -> Foo: pass And when an object needs a callback as an attribute: class Menu: def __init__(self, items=None, select_callback=None): self.items = items if items is not None else [] self.select_callback = select_callback my_menu = Menu([item1, item2]) def my_menu.select_callback(self, item_index): print(self.items[item_index]) As opposed to: my_menu = Menu([item1, item2]) def select_callback(self, item_index): print(self.items[item_index]) my_menu.select_callback = select_callback Or defining them in "unnatural" order: def select_callback(self, item_index): print(self.items[item_index]) my_menu = Menu([item1, item2], select_callback) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 10 05:13:27 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 10 Feb 2017 21:13:27 +1100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: <20170210101327.GA4796@ando.pearwood.info> On Fri, Feb 10, 2017 at 11:13:47AM +0200, Markus Meskanen wrote: > I'm suggesting the addition of support to using a dot notation when > defining a function to be a method of a class, or a callback attribute. For > example: > > def foo(self): > pass > Foo.foo = foo > > Becomes: > > def Foo.foo(self): > pass That saves one line, at the cost of introducing even more complexity to the language. Are these use-cases common enough to justify the extra syntax? > Other syntaxes can also be used if the dot itself is an issue, although I > dislike these: > > def Foo:foo(self): > def foo at Foo(self): > def Foo>foo(self): > def Foo&foo(self): That's just putting arbitrary symbols into the statement. I know that *ultimately* all symbols are arbitrary, but . dot means attribute access in Python, so Foo.foo at least has some connection to creating an attribute of Foo called foo. In what way does "Foo greater than foo" suggest to the reader that foo ends up as an attribute of Foo? > This functionality would be useful in the few rare cases where the class > itself needs to be accessed in the function's definition (decorator, > typing, etc.): I'm sure that there are a number of use-cases for injecting a method into an existing class. But we already have a way of doing that: def method(self): ... Spam.method = method What advantage does this proposed syntax have? Since that's not actually a rhetorical question, I'll answer it myself: def Spam.method(self) not only saves the line Spam.method = method but it also avoids leaving a global name "method" in the namespace (no need to follow with `del method`); it makes it explicit from the beginning that the function is an attribute of Spam. If the implementation is clever enough, it can avoid clashing with a global of the same name: eggs = "something" def Spam.eggs(self): ... def Cheese.eggs(self): ... assert eggs == "something" So the idea isn't without merit; but the question in my mind is whether the advantages outweigh the extra complexity. -- Steve From boekewurm at gmail.com Fri Feb 10 05:15:38 2017 From: boekewurm at gmail.com (Matthias welp) Date: Fri, 10 Feb 2017 11:15:38 +0100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: Hi Markus, Thanks for writing this up, as I've had this same very valid problem before. On 10 February 2017 at 10:13, Markus Meskanen wrote: > I'm suggesting the addition of support to using a dot notation when defining > a function to be a method of a class, or a callback attribute. Your solution to me seems like a 'hack': class monkey-patching during runtime is already available if you really need it, and your proposal only makes it easier, which I don't really like. > This functionality would be useful in the few rare cases where the class > itself needs to be accessed in the function's definition (decorator, typing, > etc This problem could just as well be solved by allowing access to a scope-level variable (__class__? __type__?) which is available in the class body at construction time, which points to the (eventual) class type object, or evaluating the type hints in a class only after the class is created, which then allows for that class to be pointed to in the type annotations. E.G. this does not work right now: class A: def foo(self: A): pass as it fails with NameError: A is not defined, whereas you'd expect it to work. The problem is very annoying when you're trying to work with the dunder methods for e.g. numerical objects, as you cannot say '+ is only allowed for these types', but it is not limited to this scope only. -Matthias From stephanh42 at gmail.com Fri Feb 10 05:29:32 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 10 Feb 2017 11:29:32 +0100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: What about using a simple decorator instead? def monkey_patch(cls): return lambda func: setattr(cls, func.__name__, func) class Foo: pass @monkey_patch(Foo) def bar(self): return 42 Foo().bar() # gives 42 2017-02-10 11:15 GMT+01:00 Matthias welp : > Hi Markus, > > Thanks for writing this up, as I've had this same very valid problem > before. > > On 10 February 2017 at 10:13, Markus Meskanen > wrote: > > I'm suggesting the addition of support to using a dot notation when > defining > > a function to be a method of a class, or a callback attribute. > > Your solution to me seems like a 'hack': class monkey-patching during > runtime > is already available if you really need it, and your proposal only > makes it easier, > which I don't really like. > > > This functionality would be useful in the few rare cases where the class > > itself needs to be accessed in the function's definition (decorator, > typing, > > etc > > This problem could just as well be solved by allowing access to a > scope-level > variable (__class__? __type__?) which is available in the class body at > construction time, which points to the (eventual) class type object, > or evaluating > the type hints in a class only after the class is created, which then > allows for that > class to be pointed to in the type annotations. > > E.G. this does not work right now: > > class A: > def foo(self: A): > pass > > as it fails with NameError: A is not defined, whereas you'd expect it to > work. > > The problem is very annoying when you're trying to work with the dunder > methods for e.g. numerical objects, as you cannot say '+ is only allowed > for > these types', but it is not limited to this scope only. > > > -Matthias > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Fri Feb 10 05:44:07 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 12:44:07 +0200 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: On Fri, Feb 10, 2017 at 12:29 PM, Stephan Houben wrote: > What about using a simple decorator instead? > > def monkey_patch(cls): > return lambda func: setattr(cls, func.__name__, func) > > class Foo: > pass > > @monkey_patch(Foo) > def bar(self): > return 42 > > Foo().bar() > # gives 42 > This would work, but I still believe the proposed method is much shorter and easier to follow. Decorator approach is no different from doing `Foo.bar = bar` under the function definition I think, except it requires one to figure out what the decorator does first. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Fri Feb 10 05:45:27 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 12:45:27 +0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: > > That saves one line, at the cost of introducing even more complexity to > the language. > > Are these use-cases common enough to justify the extra syntax? > Keep in mind that the extra syntax is *very* minor, and goes hand-to-hand with the existing attribute access syntax. Basically it's taking the existing syntax to one more place, where it in my opinion should have been since long ago. > What advantage does this proposed syntax have? > > Since that's not actually a rhetorical question, I'll answer it myself: > > def Spam.method(self) not only saves the line > > Spam.method = method > > but it also avoids leaving a global name "method" in the namespace (no > need to follow with `del method`); it makes it explicit from the > beginning that the function is an attribute of Spam. > Yeah this is a major reason why I want this, and the reason I mentioned "unnatural order" in the original mail. Having the class's name in the beginning just makes it feel right.. > If the implementation is clever enough, it can avoid clashing with a > global of the same name: > > eggs = "something" > > def Spam.eggs(self): > ... > > def Cheese.eggs(self): > ... > > assert eggs == "something" > > > So the idea isn't without merit; but the question in my mind is whether > the advantages outweigh the extra complexity. I didn't even realize you would avoid the global namespace issue too, this makes me fall in love with the idea even more. I really think the added complexity isn't much. One thing to consider, however, is what would happen if someone attempted to use this in a class definition: class Foo: ... class Bar: def Foo.meth(self): ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 10 06:05:30 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 10 Feb 2017 22:05:30 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On Fri, Feb 10, 2017 at 9:45 PM, Markus Meskanen wrote: >> That saves one line, at the cost of introducing even more complexity to >> the language. >> >> Are these use-cases common enough to justify the extra syntax? > > > Keep in mind that the extra syntax is *very* minor, and goes hand-to-hand > with the existing attribute access syntax. Basically it's taking the > existing syntax to one more place, where it in my opinion should have been > since long ago. > Every now and then, we get a proposal along these lines. I think it's about time a PEP got written. The usual way this is explained is that a function name can be anything you can assign to. Currently, a function has to have a simple name, and it then gets created with that as its __name__ and bound to that name in the current namespace (module, class, or function). To achieve what you're looking for, the syntax would be defined in terms of assignment, same as a 'for' loop's iteration variable is: # Perfectly legal for spam.ham in iter: pass # Not currently legal def ham.spam(): pass Markus, do you want to head this up? I'll help out with any editorial work you have trouble with (as a PEP editor, I can assign it a number and so on). Considerations: * What would the __name__ be? In "def ham.spam():", is the name "spam" or "ham.spam"? Or say you have "def x[0]():" - is the name "x[0]" or something else? * Are there any syntactic ambiguities? Any special restrictions? * Exactly what grammar token would be used? Currently NAME; might become 'test'? * Will there be any possible backward incompatibilities? Create a pull request against https://github.com/python/peps - looks like the next number is 542. Any questions, I'm happy to help. ChrisA From p.f.moore at gmail.com Fri Feb 10 06:13:51 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Feb 2017 11:13:51 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 10:45, Markus Meskanen wrote: > Keep in mind that the extra syntax is *very* minor, and goes hand-to-hand > with the existing attribute access syntax. Basically it's taking the > existing syntax to one more place, where it in my opinion should have been > since long ago. In implementation terms, the syntax change is not as minor as you suggest. At the moment, the syntax for a "def" statement is: funcdef ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite funcname ::= identifier You're proposing replacing "identifier" as the definition of a "funcname" with... what? dotted_name might work, but that opens up the possibility of class Foo: pass foo = Foo def foo.a(self): pass (note I'm defining a method on the *instance*, not on the class). Do you want to allow that? What about "def a.b.c.d.e(): pass" (no self as argument, deeply nested instance attribute). Furthermore, once we open up this possibility, I would expect requests for things like func_table = {} func_table["foo"] = lambda a, b: a+b def func_table["bar"] (a,b): return a-b pretty quickly. How would you respond to those? (Setting up function tables is a much more common and reasonable need than monkeypatching classes). Your proposal is clear enough in terms of your intent, but the implementation details are non-trivial. Paul PS Personally, I'm slightly in favour of the idea in principle, but I don't think it's a useful enough addition to warrant having to deal with all the questions I note above. From rosuav at gmail.com Fri Feb 10 06:16:35 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 10 Feb 2017 22:16:35 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On Fri, Feb 10, 2017 at 10:13 PM, Paul Moore wrote: > Furthermore, once we open up this possibility, I would expect requests > for things like > > func_table = {} > func_table["foo"] = lambda a, b: a+b > def func_table["bar"] (a,b): > return a-b > > pretty quickly. How would you respond to those? (Setting up function > tables is a much more common and reasonable need than monkeypatching > classes). Which is why these proposals always seem to gravitate to "anything you can assign to", which is at least easy enough to explain. All your examples would be completely acceptable.under that system. ChrisA From p.f.moore at gmail.com Fri Feb 10 06:32:15 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Feb 2017 11:32:15 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 11:16, Chris Angelico wrote: > On Fri, Feb 10, 2017 at 10:13 PM, Paul Moore wrote: >> Furthermore, once we open up this possibility, I would expect requests >> for things like >> >> func_table = {} >> func_table["foo"] = lambda a, b: a+b >> def func_table["bar"] (a,b): >> return a-b >> >> pretty quickly. How would you respond to those? (Setting up function >> tables is a much more common and reasonable need than monkeypatching >> classes). > > Which is why these proposals always seem to gravitate to "anything you > can assign to", which is at least easy enough to explain. All your > examples would be completely acceptable.under that system. Precisely. Paul From steve at pearwood.info Fri Feb 10 06:51:59 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 10 Feb 2017 22:51:59 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: <20170210115149.GC4796@ando.pearwood.info> On Fri, Feb 10, 2017 at 10:05:30PM +1100, Chris Angelico wrote: > * What would the __name__ be? In "def ham.spam():", is the name "spam" > or "ham.spam"? "spam" of course, just like it is now: py> class Ham: ... def spam(self): ... ... ... py> py> Ham.spam.__name__ 'spam' You might be thinking of __qualname__: py> Ham.spam.__qualname__ 'Ham.spam' > Or say you have "def x[0]():" - is the name "x[0]" or > something else? I wouldn't allow that. I feel that "any assignment target at all" is an over-generalisation, a case of YAGNI. It is relatively easy to change our mind and add additional cases in the future, but very difficult to remove them if they turn out to be a mistake. My intuition tells me that we should allow : def name dot name (args): possibly even more than one dot: def name dot name dot name ... (args): but no additional cases: # syntax error def spam[0]function(): ... -- Steve From thomas at kluyver.me.uk Fri Feb 10 07:02:56 2017 From: thomas at kluyver.me.uk (Thomas Kluyver) Date: Fri, 10 Feb 2017 12:02:56 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: <1486728176.2746369.876712312.54C14179@webmail.messagingengine.com> On Fri, Feb 10, 2017, at 11:05 AM, Chris Angelico wrote: > * What would the __name__ be? In "def ham.spam():", is the name "spam" > or "ham.spam"? Or say you have "def x[0]():" - is the name "x[0]" or > something else? I'd say 'spam' in the first case, and a special value like '' in the latter. You already can't rely on __name__ being a usable name in arbitrary callables: x[0] = lambda: 0 x[0].__name__ == '' From rosuav at gmail.com Fri Feb 10 07:12:53 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 10 Feb 2017 23:12:53 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <1486728176.2746369.876712312.54C14179@webmail.messagingengine.com> References: <20170210101327.GA4796@ando.pearwood.info> <1486728176.2746369.876712312.54C14179@webmail.messagingengine.com> Message-ID: On Fri, Feb 10, 2017 at 11:02 PM, Thomas Kluyver wrote: > On Fri, Feb 10, 2017, at 11:05 AM, Chris Angelico wrote: >> * What would the __name__ be? In "def ham.spam():", is the name "spam" >> or "ham.spam"? Or say you have "def x[0]():" - is the name "x[0]" or >> something else? > > I'd say 'spam' in the first case, and a special value like ' function>' in the latter. You already can't rely on __name__ being a > usable name in arbitrary callables: > > x[0] = lambda: 0 > x[0].__name__ == '' And this is exactly why this wants a PEP - to collect decisions and arguments on all of these bikesheds. ChrisA From jsbueno at python.org.br Fri Feb 10 07:13:46 2017 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 10 Feb 2017 10:13:46 -0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170210115149.GC4796@ando.pearwood.info> References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: I am definetelly -1 to this idea. But since you are discussing this seriously, one nice thing is to recall how Javascript does that: `function () ` is an expression that returns the created function, and thus can be assigned to anything on the left side. Of course, that would throw us back to a way of thinking of inline definition of multiline functions - which is another requested and unresolved thing in Python. (But we might require the `def` statement to still be aligned, at least style-wise, and require people to write Foo.foo =\ def (self, ...): ... ) That said, this possibility in Javascript is the source of severe inconsistencies in how functions are declared across different libraries and projects, and IMHO, makes reading (and writting) a real pain. (And, as stated above, a two line decorator could make for the patching - it does not need to have such an ugly name as "monkey_patch" - it could be just "assign" instead) js -><- On 10 February 2017 at 09:51, Steven D'Aprano wrote: > On Fri, Feb 10, 2017 at 10:05:30PM +1100, Chris Angelico wrote: > >> * What would the __name__ be? In "def ham.spam():", is the name "spam" >> or "ham.spam"? > > "spam" of course, just like it is now: > > py> class Ham: > ... def spam(self): > ... ... > ... > py> > py> Ham.spam.__name__ > 'spam' > > > You might be thinking of __qualname__: > > py> Ham.spam.__qualname__ > 'Ham.spam' > > >> Or say you have "def x[0]():" - is the name "x[0]" or >> something else? > > I wouldn't allow that. I feel that "any assignment target at all" is an > over-generalisation, a case of YAGNI. > > It is relatively easy to change our mind and add additional cases in the > future, but very difficult to remove them if they turn out to be a > mistake. > > My intuition tells me that we should allow : > > def name dot name (args): > > possibly even more than one dot: > > def name dot name dot name ... (args): > > > but no additional cases: > > # syntax error > def spam[0]function(): ... > > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From markusmeskanen at gmail.com Fri Feb 10 07:28:25 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 14:28:25 +0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: I've started working on a PEP for this since most people seem to be for it. Will see how it turns out. On Fri, Feb 10, 2017 at 2:13 PM, Joao S. O. Bueno wrote: > I am definetelly -1 to this idea. > > But since you are discussing this seriously, one nice thing is to > recall how Javascript does that: > `function () ` is an expression that returns the created > function, and thus can be assigned to anything on > the left side. > > Of course, that would throw us back to a way of thinking of inline > definition of multiline functions - > which is another requested and unresolved thing in Python. > > (But we might require the `def` statement to still be aligned, at > least style-wise, and require > people to write > > Foo.foo =\ > def (self, ...): ... > > ) > > That said, this possibility in Javascript is the source of severe > inconsistencies in how functions are declared across different > libraries and projects, and IMHO, makes reading (and writting) a real pain. > > (And, as stated above, a two line decorator could make for the patching - > it does not need to have such an ugly name as "monkey_patch" - it > could be just "assign" instead) > > js > -><- > > On 10 February 2017 at 09:51, Steven D'Aprano wrote: > > On Fri, Feb 10, 2017 at 10:05:30PM +1100, Chris Angelico wrote: > > > >> * What would the __name__ be? In "def ham.spam():", is the name "spam" > >> or "ham.spam"? > > > > "spam" of course, just like it is now: > > > > py> class Ham: > > ... def spam(self): > > ... ... > > ... > > py> > > py> Ham.spam.__name__ > > 'spam' > > > > > > You might be thinking of __qualname__: > > > > py> Ham.spam.__qualname__ > > 'Ham.spam' > > > > > >> Or say you have "def x[0]():" - is the name "x[0]" or > >> something else? > > > > I wouldn't allow that. I feel that "any assignment target at all" is an > > over-generalisation, a case of YAGNI. > > > > It is relatively easy to change our mind and add additional cases in the > > future, but very difficult to remove them if they turn out to be a > > mistake. > > > > My intuition tells me that we should allow : > > > > def name dot name (args): > > > > possibly even more than one dot: > > > > def name dot name dot name ... (args): > > > > > > but no additional cases: > > > > # syntax error > > def spam[0]function(): ... > > > > > > > > > > -- > > Steve > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 10 08:42:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 11 Feb 2017 00:42:57 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: <20170210134256.GD4796@ando.pearwood.info> On Fri, Feb 10, 2017 at 02:28:25PM +0200, Markus Meskanen wrote: > I've started working on a PEP for this since most people seem to be for it. I don't know how you get "most people" -- there's only been a handful of responses in the few hours since the original post. And apart from one explicit -1, I read most of them as neutral, not in favour. Of course you are perfectly entitled to start work on a PEP at any time, but don't get your hopes up. I'm one of the neutral parties, perhaps just a tiny bit positive +0, but only for the original proposal. I am -1000 on allowing arbitrary assignment targets. I believe that the cost in readability far outweighs the usefulness of allowing things like: def mydict[key].attr[-1](arg): ... -- Steve From stephanh42 at gmail.com Fri Feb 10 08:55:31 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 10 Feb 2017 14:55:31 +0100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: Hi all, I would like to add one more generic remark about syntax extensions, regarding something Markus said and which has bothered me before, also related to other syntax proposals. "Decorator approach is no different from doing `Foo.bar = bar` under the function definition I think, except it requires one to figure out what the decorator does first." My point would be that the new syntax *also* requires one to figure out what the new syntax does. And unfortunately, syntax is much less discoverable than decorators. For a decorator, I can do `help(decorator)' or search the python library reference or probably just mouse-hover over the name in my favourite editor/IDE. But if I don't understand the dot in `class foo.bar:', then what? It's probably somewhere buried in the language spec for `class' but realistically I am now going to blight Stackoverflow with my questions. Stephan 2017-02-10 13:13 GMT+01:00 Joao S. O. Bueno : > I am definetelly -1 to this idea. > > But since you are discussing this seriously, one nice thing is to > recall how Javascript does that: > `function () ` is an expression that returns the created > function, and thus can be assigned to anything on > the left side. > > Of course, that would throw us back to a way of thinking of inline > definition of multiline functions - > which is another requested and unresolved thing in Python. > > (But we might require the `def` statement to still be aligned, at > least style-wise, and require > people to write > > Foo.foo =\ > def (self, ...): ... > > ) > > That said, this possibility in Javascript is the source of severe > inconsistencies in how functions are declared across different > libraries and projects, and IMHO, makes reading (and writting) a real pain. > > (And, as stated above, a two line decorator could make for the patching - > it does not need to have such an ugly name as "monkey_patch" - it > could be just "assign" instead) > > js > -><- > > On 10 February 2017 at 09:51, Steven D'Aprano wrote: > > On Fri, Feb 10, 2017 at 10:05:30PM +1100, Chris Angelico wrote: > > > >> * What would the __name__ be? In "def ham.spam():", is the name "spam" > >> or "ham.spam"? > > > > "spam" of course, just like it is now: > > > > py> class Ham: > > ... def spam(self): > > ... ... > > ... > > py> > > py> Ham.spam.__name__ > > 'spam' > > > > > > You might be thinking of __qualname__: > > > > py> Ham.spam.__qualname__ > > 'Ham.spam' > > > > > >> Or say you have "def x[0]():" - is the name "x[0]" or > >> something else? > > > > I wouldn't allow that. I feel that "any assignment target at all" is an > > over-generalisation, a case of YAGNI. > > > > It is relatively easy to change our mind and add additional cases in the > > future, but very difficult to remove them if they turn out to be a > > mistake. > > > > My intuition tells me that we should allow : > > > > def name dot name (args): > > > > possibly even more than one dot: > > > > def name dot name dot name ... (args): > > > > > > but no additional cases: > > > > # syntax error > > def spam[0]function(): ... > > > > > > > > > > -- > > Steve > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Fri Feb 10 09:00:26 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 16:00:26 +0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170210134256.GD4796@ando.pearwood.info> References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> <20170210134256.GD4796@ando.pearwood.info> Message-ID: > > > I've started working on a PEP for this since most people seem to be for > it. > > I don't know how you get "most people" -- there's only been a handful of > responses in the few hours since the original post. And apart from one > explicit -1, I read most of them as neutral, not in favour. > Yeah I worded that poorly, more like most people didn't turn me down which I was a bit afraid of. > I'm one of the neutral parties, perhaps just a tiny bit positive +0, but > only for the original proposal. > > I am -1000 on allowing arbitrary assignment targets. I believe that the > cost in readability far outweighs the usefulness of allowing things > like: > > def mydict[key].attr[-1](arg): ... Do not worry, I will not propose the advanced method, only dot notation! That being said, I don't think it's up to the language if someone wants to write ugly code like that, you can already do way uglier stuff with the existing features. I don't really see people doing this either: mydict[key].attr[-1].append(my_func) So why would they if we suddenly introduce this to functions? Anyways that's not a worry of this to-be PEP. - Markus -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Feb 10 09:06:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Feb 2017 14:06:04 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 13:55, Stephan Houben wrote: > My point would be that the new syntax *also* requires one to figure out what > the new syntax does. This is an extremely good point. It is mentioned when new syntax is proposed (the term often used is "discoverability") but the idea never seems to stick, as people keep forgetting to consider it when proposing new ideas. With this proposal the only thing you can search for is "def", and you're going to mostly find sites that explain the current syntax. So anyone looking for understanding of the new construct will likely end up even more confused after searching than they were before. Markus - if you do write up a PEP, please make sure this point is noted and addressed. Paul From p.f.moore at gmail.com Fri Feb 10 09:09:48 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Feb 2017 14:09:48 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> <20170210134256.GD4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 14:00, Markus Meskanen wrote: >> > I've started working on a PEP for this since most people seem to be for >> > it. >> >> I don't know how you get "most people" -- there's only been a handful of >> responses in the few hours since the original post. And apart from one >> explicit -1, I read most of them as neutral, not in favour. > > > Yeah I worded that poorly, more like most people didn't turn me down which I > was a bit afraid of. To be clear, I'm -1 on the proposal. As I said, I can see a small attraction in the *idea*, but for me, the practical considerations seem too great. Paul From markusmeskanen at gmail.com Fri Feb 10 09:11:18 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 16:11:18 +0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: > > I would like to add one more generic remark about syntax extensions, > regarding something > Markus said and which has bothered me before, also related to other syntax > proposals. > > "Decorator approach is no different from doing `Foo.bar = bar` under the > function definition I think, except it requires one to figure out what the > decorator does first." > > My point would be that the new syntax *also* requires one to figure out > what the new syntax does. > And unfortunately, syntax is much less discoverable than decorators. > For a decorator, I can do `help(decorator)' or search the python library > reference or probably > just mouse-hover over the name in my favourite editor/IDE. > > But if I don't understand the dot in `class foo.bar:', then what? > It's probably somewhere buried in the language spec for `class' but > realistically > I am now going to blight Stackoverflow with my questions. > I deeply believe the dot notation is very simple to understand (for the record, it's the default in JS ja Lua and they're not having any issues with it), and I can't think of a situation where someone knows Python well enough to understand decorators with arguments but wouldn't understand the dot notation. We already use the dot notation for normal attributes, so why not use it for attributes in function def? I think it'll be easier to StackOverflow the dot notation as opposed to argumented decorators. And what I meant by "they have to figure out what the decorator does first" is that it varies between every project. You absolutely cannot know for sure what the decorator does until you read through it, meaning you have to go look it up everytime. - Markus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 10 09:16:17 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Feb 2017 15:16:17 +0100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 12:16, Chris Angelico wrote: > On Fri, Feb 10, 2017 at 10:13 PM, Paul Moore wrote: >> Furthermore, once we open up this possibility, I would expect requests >> for things like >> >> func_table = {} >> func_table["foo"] = lambda a, b: a+b >> def func_table["bar"] (a,b): >> return a-b >> >> pretty quickly. How would you respond to those? (Setting up function >> tables is a much more common and reasonable need than monkeypatching >> classes). > > Which is why these proposals always seem to gravitate to "anything you > can assign to", which is at least easy enough to explain. All your > examples would be completely acceptable.under that system. But what do __name__ and __qualname__ get set to? What happens if you do this at class scope, rather than at module level or inside another function? What happens to the zero-argument super() support at class scope? What happens if you attempt to use zero-argument super() when *not* at class scope? These are *answerable* questions (and injecting the right __class__ cell reference for zero-argument super() support is a compelling technical argument in favour of this feature over ordinary attribute binding operations), but there's a lot more to the proposal than just relaxing a syntactic restriction in the language grammar. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 10 09:25:40 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Feb 2017 01:25:40 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On Sat, Feb 11, 2017 at 1:16 AM, Nick Coghlan wrote: > But what do __name__ and __qualname__ get set to? > > What happens if you do this at class scope, rather than at module > level or inside another function? > > What happens to the zero-argument super() support at class scope? > > What happens if you attempt to use zero-argument super() when *not* at > class scope? > > These are *answerable* questions... ... and are exactly why I asked the OP to write up a PEP. This isn't my proposal, so it's not up to me to make the decisions. For what it's worth, my answers would be: __name__ would be the textual representation of exactly what you typed between "def" and the open parenthesis. __qualname__ would be built the exact same way it currently is, based on that __name__. Zero-argument super() would behave exactly the way it would if you used a simple name. This just changes the assignment, not the creation of the function. So if you're inside a class, you could populate a lookup dictionary with method-like functions. Abuse this, and you're only shooting your own foot. Zero-argument super() outside of a class, just as currently, would be an error. (Whatever kind of error it currently is.) Maybe there are better answers to these questions, I don't know. That's what the PEP's for. ChrisA From z+py+pyideas at m0g.net Fri Feb 10 09:53:17 2017 From: z+py+pyideas at m0g.net (Guyzmo) Date: Fri, 10 Feb 2017 15:53:17 +0100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210115149.GC4796@ando.pearwood.info> Message-ID: <20170210145317.ygobvxiez7pngxew@BuGz.eclipse.m0g.net> Hi list, I'm quite neutral to this proposition, as it's not a use case I see often myself needing. On Fri, Feb 10, 2017 at 02:55:31PM +0100, Stephan Houben wrote: [?] > But if I don't understand the dot in `class foo.bar:', then what? > It's probably somewhere buried in the language spec for `class' but > realistically > I am now going to blight Stackoverflow with my questions. [?] but this is definitely not a reason to dismiss a proposal. A language is aimed at evolves and introduce new syntax features, and yes, stackoverflow will get questions about it, blog articles written and RTFW updated, so you'll get the info you'll need fastly. Cheers, -- Guyzmo From klahnakoski at mozilla.com Fri Feb 10 09:59:03 2017 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Fri, 10 Feb 2017 09:59:03 -0500 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: On 2017-02-10 05:44, Markus Meskanen wrote: > > > On Fri, Feb 10, 2017 at 12:29 PM, Stephan Houben > wrote: > > What about using a simple decorator instead? > > def monkey_patch(cls): > return lambda func: setattr(cls, func.__name__, func) > I suggest naming the decorator with something less generic than "monkey_patch", like "extension_method". -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 10 10:25:07 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 11 Feb 2017 02:25:07 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: <20170210152506.GE4796@ando.pearwood.info> On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: > For what it's worth, my answers would be: > > __name__ would be the textual representation of exactly what you typed > between "def" and the open parenthesis. __qualname__ would be built > the exact same way it currently is, based on that __name__. If I'm reading this right, you want this behaviour: class Spam: pass def Spam.func(self): pass assert 'Spam.func' not in Spam.__dict__ assert 'func' in Spam.__dict__ assert Spam.func.__name__ == 'Spam.func' assert Spam.func.__qualname__ == 'Spam.Spam.func' If that's the case, I can only ask... what advantage do you see from this? Because I can see plenty of opportunity for confusion, and no advantage. For what its worth, Lua already has this feature: http://www.lua.org/pil/6.2.html Lib = {} function Lib.foo (x,y) return x + y end If we define that function foo inside the Lib table, and then cause an error, the Lua interpreter tells us the function name: > Lib.foo('a', 1) stdin:2: attempt to perform arithmetic on local 'x' (a string value) stack traceback: stdin:2: in function 'foo' stdin:1: in main chunk [C]: in ? -- Steve From rosuav at gmail.com Fri Feb 10 10:34:36 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Feb 2017 02:34:36 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170210152506.GE4796@ando.pearwood.info> References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: On Sat, Feb 11, 2017 at 2:25 AM, Steven D'Aprano wrote: > If I'm reading this right, you want this behaviour: > > class Spam: > pass > > def Spam.func(self): pass > > assert 'Spam.func' not in Spam.__dict__ > assert 'func' in Spam.__dict__ > > assert Spam.func.__name__ == 'Spam.func' > assert Spam.func.__qualname__ == 'Spam.Spam.func' > > If that's the case, I can only ask... what advantage do you see from > this? Because I can see plenty of opportunity for confusion, and no > advantage. > > I might be wrong about the __name__; that was a response that came from the massively extensive research of "hmm, I think this would be what I'd do". It seems the simplest way to cope with the many possibilities; having __name__ be "func" would work in the dot form, but not others. But that's bikeshedding. ChrisA From steve.dower at python.org Fri Feb 10 10:38:18 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 10 Feb 2017 09:38:18 -0600 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: Since votes seem to be being counted and used for debate purposes, I am -1 to anything that encourages or condones people adding functionality to classes outside of the class definition. (Monkeypatching in my mind neither condones or encourages, and most descriptions come with plenty of caveats about how it should be avoided.) My favourite description of object-oriented programming is that it's like "reading a road map through a drinking(/soda/pop) straw". We do not need to tell people that it's okay to make this problem worse by providing first-class tools to do it. Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" Sent: ?2/?10/?2017 8:27 To: "Python-Ideas" Subject: Re: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator On Sat, Feb 11, 2017 at 1:16 AM, Nick Coghlan wrote: > But what do __name__ and __qualname__ get set to? > > What happens if you do this at class scope, rather than at module > level or inside another function? > > What happens to the zero-argument super() support at class scope? > > What happens if you attempt to use zero-argument super() when *not* at > class scope? > > These are *answerable* questions... ... and are exactly why I asked the OP to write up a PEP. This isn't my proposal, so it's not up to me to make the decisions. For what it's worth, my answers would be: __name__ would be the textual representation of exactly what you typed between "def" and the open parenthesis. __qualname__ would be built the exact same way it currently is, based on that __name__. Zero-argument super() would behave exactly the way it would if you used a simple name. This just changes the assignment, not the creation of the function. So if you're inside a class, you could populate a lookup dictionary with method-like functions. Abuse this, and you're only shooting your own foot. Zero-argument super() outside of a class, just as currently, would be an error. (Whatever kind of error it currently is.) Maybe there are better answers to these questions, I don't know. That's what the PEP's for. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Fri Feb 10 11:09:25 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 18:09:25 +0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: But if people are gonna do it anyways with the tools provided (monkey patching), why not provide them with better tools? And this wouldn't only be for classes, but for setting instance attributes too (see the Menu example in original mail). - Markus On Fri, Feb 10, 2017 at 5:38 PM, Steve Dower wrote: > Since votes seem to be being counted and used for debate purposes, I am -1 > to anything that encourages or condones people adding functionality to > classes outside of the class definition. (Monkeypatching in my mind neither > condones or encourages, and most descriptions come with plenty of caveats > about how it should be avoided.) > > My favourite description of object-oriented programming is that it's like > "reading a road map through a drinking(/soda/pop) straw". We do not need to > tell people that it's okay to make this problem worse by providing > first-class tools to do it. > > Top-posted from my Windows Phone > ------------------------------ > From: Chris Angelico > Sent: ?2/?10/?2017 8:27 > To: Python-Ideas > Subject: Re: [Python-ideas] Fwd: Define a method or function > attributeoutside of a class with the dot operator > > On Sat, Feb 11, 2017 at 1:16 AM, Nick Coghlan wrote: > > But what do __name__ and __qualname__ get set to? > > > > What happens if you do this at class scope, rather than at module > > level or inside another function? > > > > What happens to the zero-argument super() support at class scope? > > > > What happens if you attempt to use zero-argument super() when *not* at > > class scope? > > > > These are *answerable* questions... > > ... and are exactly why I asked the OP to write up a PEP. This isn't > my proposal, so it's not up to me to make the decisions. > > For what it's worth, my answers would be: > > __name__ would be the textual representation of exactly what you typed > between "def" and the open parenthesis. __qualname__ would be built > the exact same way it currently is, based on that __name__. > > Zero-argument super() would behave exactly the way it would if you > used a simple name. This just changes the assignment, not the creation > of the function. So if you're inside a class, you could populate a > lookup dictionary with method-like functions. Abuse this, and you're > only shooting your own foot. > > Zero-argument super() outside of a class, just as currently, would be > an error. (Whatever kind of error it currently is.) > > Maybe there are better answers to these questions, I don't know. > That's what the PEP's for. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Feb 10 11:15:23 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Feb 2017 16:15:23 +0000 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 16:09, Markus Meskanen wrote: > But if people are gonna do it anyways with the tools provided (monkey > patching), why not provide them with better tools? Because encouraging and making it easier for people to make mistakes is the wrong thing to do, surely? Paul From markusmeskanen at gmail.com Fri Feb 10 11:17:54 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 18:17:54 +0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: Well yes, but I think you're a bit too fast on labeling it a mistake to use monkey patching... On Feb 10, 2017 18:15, "Paul Moore" wrote: On 10 February 2017 at 16:09, Markus Meskanen wrote: > But if people are gonna do it anyways with the tools provided (monkey > patching), why not provide them with better tools? Because encouraging and making it easier for people to make mistakes is the wrong thing to do, surely? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Fri Feb 10 11:35:20 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 10 Feb 2017 17:35:20 +0100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: Another point of view: Some call it monkeypatching. Others call it configuration. There's room for both views and I don't see anything wrong with configuration using this kind of feature. Sven On 10.02.2017 17:17, Markus Meskanen wrote: > Well yes, but I think you're a bit too fast on labeling it a mistake > to use monkey patching... > > > On Feb 10, 2017 18:15, "Paul Moore" > wrote: > > On 10 February 2017 at 16:09, Markus Meskanen > > wrote: > > But if people are gonna do it anyways with the tools provided > (monkey > > patching), why not provide them with better tools? > > Because encouraging and making it easier for people to make mistakes > is the wrong thing to do, surely? > > Paul > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 10 12:20:03 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Feb 2017 18:20:03 +0100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170210152506.GE4796@ando.pearwood.info> References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: On 10 February 2017 at 16:25, Steven D'Aprano wrote: > On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: > >> For what it's worth, my answers would be: >> >> __name__ would be the textual representation of exactly what you typed >> between "def" and the open parenthesis. __qualname__ would be built >> the exact same way it currently is, based on that __name__. > > If I'm reading this right, you want this behaviour: > > class Spam: > pass > > def Spam.func(self): pass > > assert 'Spam.func' not in Spam.__dict__ > assert 'func' in Spam.__dict__ > > assert Spam.func.__name__ == 'Spam.func' > assert Spam.func.__qualname__ == 'Spam.Spam.func' > > If that's the case, I can only ask... what advantage do you see from > this? Because I can see plenty of opportunity for confusion, and no > advantage. What I would personally hope to see from the proposal is that given: class Spam: pass def Spam.func(self): return __class__ the effective runtime behaviour would be semantically identical to: class Spam: def func(self): return __class__ such that: * __name__ is set based on the method name after the dot * __qualname__ is set based on the __name__ of the given class * __set_owner__ is called after any function decorators are applied * zero-argument super() and other __class__ references work properly from the injected method Potentially, RuntimeError could be raised if the reference before the dot is not to a type instance. If it *doesn't* do that, then I'd be -1 on the proposal, since it doesn't add enough expressiveness to the language to be worth the extra syntax. By contrast, if it *does* do it, then it makes class definitions more decomposable, by providing post-definition access to parts of the machinery that are currently only accessible during the process of defining the class. The use case would be to make it easier to inject descriptors when writing class decorators such that they behave essentially the same as they do when defined in the class body: def my_class_decorator(cls): def cls.injected_method(self): # Just write injected methods the same way you would in a class body return __class__ return cls (Actually doing this may require elevating super and __class__ to true keyword expressions, rather than the pseudo-keywords they are now) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gerald.britton at gmail.com Fri Feb 10 13:01:37 2017 From: gerald.britton at gmail.com (Gerald Britton) Date: Fri, 10 Feb 2017 13:01:37 -0500 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator. In-Reply-To: References: Message-ID: This is looking familiar. .Net extension methods anyone? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri Feb 10 13:11:46 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 10 Feb 2017 12:11:46 -0600 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: When you apply the "what if everyone did this" rule, it looks like a bad idea (or alternatively, what if two people who weren't expecting anyone else to do this did it). Monkeypatching is fairly blatantly taking advantage of the object model in a way that is not "supported" and cannot behave well in the context of everyone doing it, whereas inheritance or mixins are safe. Making a dedicated syntax or decorator for patching is saying that we (the language) think you should do it. (The extension_method decorator sends exactly the wrong message about what it's doing.) Enabling a __class__ variable within the scope of the definition would also solve the motivating example, and is less likely to lead to code where you need to review multiple modules and determine whole-program import order to figure out why your calls do not work. Top-posted from my Windows Phone -----Original Message----- From: "Markus Meskanen" Sent: ?2/?10/?2017 10:18 To: "Paul Moore" Cc: "Python-Ideas" ; "Steve Dower" Subject: Re: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator Well yes, but I think you're a bit too fast on labeling it a mistake to use monkey patching... On Feb 10, 2017 18:15, "Paul Moore" wrote: On 10 February 2017 at 16:09, Markus Meskanen wrote: > But if people are gonna do it anyways with the tools provided (monkey > patching), why not provide them with better tools? Because encouraging and making it easier for people to make mistakes is the wrong thing to do, surely? Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 10 13:15:30 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 10 Feb 2017 18:15:30 +0000 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: One thing that I don't think has been mentioned, but that brings me from a +0 to a more negative outlook, is the interaction between this proposal and some of python's existing class-related features, metaclasses and descriptors. That is currently we know that function definition, and even method definition, will not have side effects. This potentially changes that since def Foo.foo(self): ... could be a descriptor. Even if it doesn't, its possible that `Foo.foo` is actually resolved from `Foo._foo`, and so this potentially further confuses the naming considerations. Then we have metaclasses. Prior to this change, it would be fully the monkeypatcher's responsibility to do any metaclass level changes if they were necessary when monkeypatching. However, since we are potentially adding a first class support for certain monkeypatches, It raises a question about some first class way to handle monkeypatched methods. Do we need to provide some kind of method to a metaclass writer that allows them to handle methods that are patched on later? Or does the language still ignore it? --Josh On Fri, Feb 10, 2017 at 12:20 PM Nick Coghlan wrote: > On 10 February 2017 at 16:25, Steven D'Aprano wrote: > > On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: > > > >> For what it's worth, my answers would be: > >> > >> __name__ would be the textual representation of exactly what you typed > >> between "def" and the open parenthesis. __qualname__ would be built > >> the exact same way it currently is, based on that __name__. > > > > If I'm reading this right, you want this behaviour: > > > > class Spam: > > pass > > > > def Spam.func(self): pass > > > > assert 'Spam.func' not in Spam.__dict__ > > assert 'func' in Spam.__dict__ > > > > assert Spam.func.__name__ == 'Spam.func' > > assert Spam.func.__qualname__ == 'Spam.Spam.func' > > > > If that's the case, I can only ask... what advantage do you see from > > this? Because I can see plenty of opportunity for confusion, and no > > advantage. > > What I would personally hope to see from the proposal is that given: > > class Spam: > pass > > def Spam.func(self): > return __class__ > > the effective runtime behaviour would be semantically identical to: > > class Spam: > def func(self): > return __class__ > > such that: > > * __name__ is set based on the method name after the dot > * __qualname__ is set based on the __name__ of the given class > * __set_owner__ is called after any function decorators are applied > * zero-argument super() and other __class__ references work properly > from the injected method > > Potentially, RuntimeError could be raised if the reference before the > dot is not to a type instance. > > If it *doesn't* do that, then I'd be -1 on the proposal, since it > doesn't add enough expressiveness to the language to be worth the > extra syntax. By contrast, if it *does* do it, then it makes class > definitions more decomposable, by providing post-definition access to > parts of the machinery that are currently only accessible during the > process of defining the class. > > The use case would be to make it easier to inject descriptors when > writing class decorators such that they behave essentially the same as > they do when defined in the class body: > > def my_class_decorator(cls): > def cls.injected_method(self): > # Just write injected methods the same way you would in a > class body > return __class__ > return cls > > (Actually doing this may require elevating super and __class__ to true > keyword expressions, rather than the pseudo-keywords they are now) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Fri Feb 10 13:48:37 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Fri, 10 Feb 2017 12:48:37 -0600 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: If everything was contained right in the same file, this is sanctioning another way to do it (when there should only be one obvious way). If you have multiple modules/packages, horrors can evolve where a class method could be patched in an unknown location by any loaded module (or you could even introduce order-of-import sensitivities). For testing, this can be a necessary evil which is OK so long as the patch is limited/apparent, and some other very narrow cases (setuptools something something?). That said, I don't want their use condoned or eased for fear of proliferation of these "antiprogrammer land mines" that I might trip over in the future. On Fri, Feb 10, 2017 at 12:15 PM, Joshua Morton wrote: > One thing that I don't think has been mentioned, but that brings me from > a +0 to a more negative outlook, is the interaction between this proposal > and some of python's existing class-related features, metaclasses and > descriptors. That is currently we know that function definition, and even > method definition, will not have side effects. This potentially changes > that since > > def Foo.foo(self): > ... > > could be a descriptor. Even if it doesn't, its possible that `Foo.foo` is > actually resolved from `Foo._foo`, and so this potentially further confuses > the naming considerations. > > Then we have metaclasses. Prior to this change, it would be fully the > monkeypatcher's responsibility to do any metaclass level changes if they > were necessary when monkeypatching. However, since we are potentially > adding a first class support for certain monkeypatches, It raises a > question about some first class way to handle monkeypatched methods. Do we > need to provide some kind of method to a metaclass writer that allows them > to handle methods that are patched on later? Or does the language still > ignore it? > > --Josh > > On Fri, Feb 10, 2017 at 12:20 PM Nick Coghlan wrote: > >> On 10 February 2017 at 16:25, Steven D'Aprano >> wrote: >> > On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: >> > >> >> For what it's worth, my answers would be: >> >> >> >> __name__ would be the textual representation of exactly what you typed >> >> between "def" and the open parenthesis. __qualname__ would be built >> >> the exact same way it currently is, based on that __name__. >> > >> > If I'm reading this right, you want this behaviour: >> > >> > class Spam: >> > pass >> > >> > def Spam.func(self): pass >> > >> > assert 'Spam.func' not in Spam.__dict__ >> > assert 'func' in Spam.__dict__ >> > >> > assert Spam.func.__name__ == 'Spam.func' >> > assert Spam.func.__qualname__ == 'Spam.Spam.func' >> > >> > If that's the case, I can only ask... what advantage do you see from >> > this? Because I can see plenty of opportunity for confusion, and no >> > advantage. >> >> What I would personally hope to see from the proposal is that given: >> >> class Spam: >> pass >> >> def Spam.func(self): >> return __class__ >> >> the effective runtime behaviour would be semantically identical to: >> >> class Spam: >> def func(self): >> return __class__ >> >> such that: >> >> * __name__ is set based on the method name after the dot >> * __qualname__ is set based on the __name__ of the given class >> * __set_owner__ is called after any function decorators are applied >> * zero-argument super() and other __class__ references work properly >> from the injected method >> >> Potentially, RuntimeError could be raised if the reference before the >> dot is not to a type instance. >> >> If it *doesn't* do that, then I'd be -1 on the proposal, since it >> doesn't add enough expressiveness to the language to be worth the >> extra syntax. By contrast, if it *does* do it, then it makes class >> definitions more decomposable, by providing post-definition access to >> parts of the machinery that are currently only accessible during the >> process of defining the class. >> >> The use case would be to make it easier to inject descriptors when >> writing class decorators such that they behave essentially the same as >> they do when defined in the class body: >> >> def my_class_decorator(cls): >> def cls.injected_method(self): >> # Just write injected methods the same way you would in a >> class body >> return __class__ >> return cls >> >> (Actually doing this may require elevating super and __class__ to true >> keyword expressions, rather than the pseudo-keywords they are now) >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Fri Feb 10 13:55:16 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Fri, 10 Feb 2017 20:55:16 +0200 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: Please keep in mind that this idea was not created to improve monkey patching, it just happens to be one of the side effects due to classes being objects. The main use case is the ability to set an instance's callback function (see the Menu example), and to allow the class being referenced in the function's header; for example in a decorator and during typing. No additional "fancy" features are intended, it would simply replace this: foo = Bar() def f(): ... foo.f = f With syntax sugar, similar to how decorators replaced this: def f(): ... f = decorate(f) On Feb 10, 2017 20:50, "Nick Timkovich" wrote: > If everything was contained right in the same file, this is sanctioning > another way to do it (when there should only be one obvious way). If you > have multiple modules/packages, horrors can evolve where a class method > could be patched in an unknown location by any loaded module (or you could > even introduce order-of-import sensitivities). > > For testing, this can be a necessary evil which is OK so long as the patch > is limited/apparent, and some other very narrow cases (setuptools something > something?). That said, I don't want their use condoned or eased for fear > of proliferation of these "antiprogrammer land mines" that I might trip > over in the future. > > On Fri, Feb 10, 2017 at 12:15 PM, Joshua Morton > wrote: > >> One thing that I don't think has been mentioned, but that brings me from >> a +0 to a more negative outlook, is the interaction between this proposal >> and some of python's existing class-related features, metaclasses and >> descriptors. That is currently we know that function definition, and even >> method definition, will not have side effects. This potentially changes >> that since >> >> def Foo.foo(self): >> ... >> >> could be a descriptor. Even if it doesn't, its possible that `Foo.foo` is >> actually resolved from `Foo._foo`, and so this potentially further confuses >> the naming considerations. >> >> Then we have metaclasses. Prior to this change, it would be fully the >> monkeypatcher's responsibility to do any metaclass level changes if they >> were necessary when monkeypatching. However, since we are potentially >> adding a first class support for certain monkeypatches, It raises a >> question about some first class way to handle monkeypatched methods. Do we >> need to provide some kind of method to a metaclass writer that allows them >> to handle methods that are patched on later? Or does the language still >> ignore it? >> >> --Josh >> >> On Fri, Feb 10, 2017 at 12:20 PM Nick Coghlan wrote: >> >>> On 10 February 2017 at 16:25, Steven D'Aprano >>> wrote: >>> > On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: >>> > >>> >> For what it's worth, my answers would be: >>> >> >>> >> __name__ would be the textual representation of exactly what you typed >>> >> between "def" and the open parenthesis. __qualname__ would be built >>> >> the exact same way it currently is, based on that __name__. >>> > >>> > If I'm reading this right, you want this behaviour: >>> > >>> > class Spam: >>> > pass >>> > >>> > def Spam.func(self): pass >>> > >>> > assert 'Spam.func' not in Spam.__dict__ >>> > assert 'func' in Spam.__dict__ >>> > >>> > assert Spam.func.__name__ == 'Spam.func' >>> > assert Spam.func.__qualname__ == 'Spam.Spam.func' >>> > >>> > If that's the case, I can only ask... what advantage do you see from >>> > this? Because I can see plenty of opportunity for confusion, and no >>> > advantage. >>> >>> What I would personally hope to see from the proposal is that given: >>> >>> class Spam: >>> pass >>> >>> def Spam.func(self): >>> return __class__ >>> >>> the effective runtime behaviour would be semantically identical to: >>> >>> class Spam: >>> def func(self): >>> return __class__ >>> >>> such that: >>> >>> * __name__ is set based on the method name after the dot >>> * __qualname__ is set based on the __name__ of the given class >>> * __set_owner__ is called after any function decorators are applied >>> * zero-argument super() and other __class__ references work properly >>> from the injected method >>> >>> Potentially, RuntimeError could be raised if the reference before the >>> dot is not to a type instance. >>> >>> If it *doesn't* do that, then I'd be -1 on the proposal, since it >>> doesn't add enough expressiveness to the language to be worth the >>> extra syntax. By contrast, if it *does* do it, then it makes class >>> definitions more decomposable, by providing post-definition access to >>> parts of the machinery that are currently only accessible during the >>> process of defining the class. >>> >>> The use case would be to make it easier to inject descriptors when >>> writing class decorators such that they behave essentially the same as >>> they do when defined in the class body: >>> >>> def my_class_decorator(cls): >>> def cls.injected_method(self): >>> # Just write injected methods the same way you would in a >>> class body >>> return __class__ >>> return cls >>> >>> (Actually doing this may require elevating super and __class__ to true >>> keyword expressions, rather than the pseudo-keywords they are now) >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 10 14:02:29 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 10 Feb 2017 11:02:29 -0800 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: <589E0E45.4070504@stoneleaf.us> On 02/10/2017 10:48 AM, Nick Timkovich wrote: > If everything was contained right in the same file, this is sanctioning > another way to do it (when there should only be one obvious way). No worries, this way is not obvious. > If you have multiple modules/packages, horrors can evolve where a class > method could be patched in an unknown location by any loaded module (or > you could even introduce order-of-import sensitivities). Folks can still do that nightmare right now. I'm -0.5 on it -- I don't think the payoff is worth the pain. But I'm +1 on writing a PEP -- collect all these pros and cons in one place to save on future discussion. (And (good) PEP writing is a way to earn valuable Python Points!) -- ~Ethan~ From stephanh42 at gmail.com Fri Feb 10 14:10:37 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 10 Feb 2017 20:10:37 +0100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: Hi all, For what it's worth, I believe that the "class extension" scenario from Nick can be supported using plain ol' metaclasses. Not sure if this covers all desired capabilities, but at least the super() mechanism works correctly. Syntax is like this: class Foo(metaclass=class_extend(Foo)): ... See: https://gist.github.com/stephanh42/97b47506e5e416f97f5790c070be7878 Stephan 2017-02-10 19:48 GMT+01:00 Nick Timkovich : > If everything was contained right in the same file, this is sanctioning > another way to do it (when there should only be one obvious way). If you > have multiple modules/packages, horrors can evolve where a class method > could be patched in an unknown location by any loaded module (or you could > even introduce order-of-import sensitivities). > > For testing, this can be a necessary evil which is OK so long as the patch > is limited/apparent, and some other very narrow cases (setuptools something > something?). That said, I don't want their use condoned or eased for fear > of proliferation of these "antiprogrammer land mines" that I might trip > over in the future. > > On Fri, Feb 10, 2017 at 12:15 PM, Joshua Morton > wrote: > >> One thing that I don't think has been mentioned, but that brings me from >> a +0 to a more negative outlook, is the interaction between this proposal >> and some of python's existing class-related features, metaclasses and >> descriptors. That is currently we know that function definition, and even >> method definition, will not have side effects. This potentially changes >> that since >> >> def Foo.foo(self): >> ... >> >> could be a descriptor. Even if it doesn't, its possible that `Foo.foo` is >> actually resolved from `Foo._foo`, and so this potentially further confuses >> the naming considerations. >> >> Then we have metaclasses. Prior to this change, it would be fully the >> monkeypatcher's responsibility to do any metaclass level changes if they >> were necessary when monkeypatching. However, since we are potentially >> adding a first class support for certain monkeypatches, It raises a >> question about some first class way to handle monkeypatched methods. Do we >> need to provide some kind of method to a metaclass writer that allows them >> to handle methods that are patched on later? Or does the language still >> ignore it? >> >> --Josh >> >> On Fri, Feb 10, 2017 at 12:20 PM Nick Coghlan wrote: >> >>> On 10 February 2017 at 16:25, Steven D'Aprano >>> wrote: >>> > On Sat, Feb 11, 2017 at 01:25:40AM +1100, Chris Angelico wrote: >>> > >>> >> For what it's worth, my answers would be: >>> >> >>> >> __name__ would be the textual representation of exactly what you typed >>> >> between "def" and the open parenthesis. __qualname__ would be built >>> >> the exact same way it currently is, based on that __name__. >>> > >>> > If I'm reading this right, you want this behaviour: >>> > >>> > class Spam: >>> > pass >>> > >>> > def Spam.func(self): pass >>> > >>> > assert 'Spam.func' not in Spam.__dict__ >>> > assert 'func' in Spam.__dict__ >>> > >>> > assert Spam.func.__name__ == 'Spam.func' >>> > assert Spam.func.__qualname__ == 'Spam.Spam.func' >>> > >>> > If that's the case, I can only ask... what advantage do you see from >>> > this? Because I can see plenty of opportunity for confusion, and no >>> > advantage. >>> >>> What I would personally hope to see from the proposal is that given: >>> >>> class Spam: >>> pass >>> >>> def Spam.func(self): >>> return __class__ >>> >>> the effective runtime behaviour would be semantically identical to: >>> >>> class Spam: >>> def func(self): >>> return __class__ >>> >>> such that: >>> >>> * __name__ is set based on the method name after the dot >>> * __qualname__ is set based on the __name__ of the given class >>> * __set_owner__ is called after any function decorators are applied >>> * zero-argument super() and other __class__ references work properly >>> from the injected method >>> >>> Potentially, RuntimeError could be raised if the reference before the >>> dot is not to a type instance. >>> >>> If it *doesn't* do that, then I'd be -1 on the proposal, since it >>> doesn't add enough expressiveness to the language to be worth the >>> extra syntax. By contrast, if it *does* do it, then it makes class >>> definitions more decomposable, by providing post-definition access to >>> parts of the machinery that are currently only accessible during the >>> process of defining the class. >>> >>> The use case would be to make it easier to inject descriptors when >>> writing class decorators such that they behave essentially the same as >>> they do when defined in the class body: >>> >>> def my_class_decorator(cls): >>> def cls.injected_method(self): >>> # Just write injected methods the same way you would in a >>> class body >>> return __class__ >>> return cls >>> >>> (Actually doing this may require elevating super and __class__ to true >>> keyword expressions, rather than the pseudo-keywords they are now) >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shoyer at gmail.com Fri Feb 10 15:00:42 2017 From: shoyer at gmail.com (Stephan Hoyer) Date: Fri, 10 Feb 2017 12:00:42 -0800 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: On Fri, Feb 10, 2017 at 9:20 AM, Nick Coghlan wrote: > What I would personally hope to see from the proposal is that given: > > class Spam: > pass > > def Spam.func(self): > return __class__ > > the effective runtime behaviour would be semantically identical to: > > class Spam: > def func(self): > return __class__ > Yes, this is exactly what I would hope/expect to see. One use case for this functionality is defining classes with an extensive method-based API with a sane dependency graph. For example, consider writing a class like numpy.ndarray or pandas.DataFrame with dozens of methods. You could argue that using so many methods is an anti-pattern, but nonetheless it's pretty common and hard to avoid in some cases (e.g., for making number-like classes that support arithmetic and comparisons). For obvious reasons, the functionality for these classes does not all live in a single module. But the modules that define helper functions for most methods also depend on the base class, so many of them need to get imported inside method definitions to avoid circular imports. The result is pretty ugly, and files defining the class still get gigantic. An important note is that ideally, we would still have way of indicating that Spam.func should exists in on the Spam class itself, even if it doesn't define the implementation. I suppose an abstractmethod overwritten by the later definition might do the trick, e.g., class Spam(metaclass=ABCMeta): @abstractmethod def func(self): pass def Spam.func(self): return __class__ And finally, it's quite possible that there's a clean metaclass based solution for extending Spam in another file, I just don't know it yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri Feb 10 15:42:54 2017 From: steve.dower at python.org (Steve Dower) Date: Fri, 10 Feb 2017 14:42:54 -0600 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> Message-ID: <6b8de1d7-6438-f768-9a38-232f779ce041@python.org> On 10Feb2017 1400, Stephan Hoyer wrote: > An important note is that ideally, we would still have way of indicating > that Spam.func should exists in on the Spam class itself, even if it > doesn't define the implementation. I suppose an abstractmethod > overwritten by the later definition might do the trick, e.g., > > class Spam(metaclass=ABCMeta): > @abstractmethod > def func(self): > pass > > def Spam.func(self): > return __class__ An abstractfunction should not become a concrete function on the abstract class - the right way to do this is to use a subclass. class SpamBase(metaclass=ABCMeta): @abstractmethod def func(self): pass class Spam(SpamBase): def func(self): return __class__ If you want to define parts of the class in separate modules, use mixins: from myarray.transforms import MyArrayTransformMixin from myarray.arithmetic import MyArrayArithmeticMixin from myarray.constructors import MyArrayConstructorsMixin class MyArray(MyArrayConstructorsMixin, MyArrayArithmeticMixin, MyArrayTransformMixin): pass The big different between these approaches and the proposal is that the proposal does not require both parties to agree on the approach. This is actually a terrible idea, as subclassing or mixing in a class that wasn't meant for it leads to all sorts of trouble unless the end user is very careful. Providing first-class syntax or methods for this discourages carefulness. (Another way of saying it is that directly overriding class members should feel a bit dirty because it *is* a bit dirty.) As Paul said in an earlier email, the best use of non-direct assignment in function definitions is putting it into a dispatch dictionary, and in this case making a decorator is likely cleaner than adding new syntax. But by all means, let's have a PEP. It will simplify the discussion when it comes up in six months again (or whenever the last time this came up was - less than a year, I'm sure). Cheers, Steve From desmoulinmichel at gmail.com Fri Feb 10 19:11:11 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 11 Feb 2017 01:11:11 +0100 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: <1486470809.2106043.873015032.16344842@webmail.messagingengine.com> <1486478846.2137807.873141704.4492B401@webmail.messagingengine.com> Message-ID: This could change when webassembly is stable. If we manage to make a Python => webassembly compiler, I doubt it will make Python in the browser happen. But it certainly can make Python in NodeJS happen, and so in Electron apps. Le 09/02/2017 ? 19:56, Nick Coghlan a ?crit : > On 7 February 2017 at 15:47, Thomas Kluyver wrote: >> I've been thinking for a while about Python apps using Electron (Positron? >> ;-). It's an interesting idea from the Python side, but I struggle to come >> up with reasons why developing an Electron+Python app would be easier than >> developing a regular Electron app. I prefer writing Python to Javascript, >> but you'd need quite a bit of Javascript anyway, you don't have to care >> about browser compatibility, and there would inevitably be some extra >> friction in using two languages. >> >> I'm sure there are use cases where it makes sense, like if you use Python's >> scientific computing ecosystem. But I don't know how broad they are. > > I'd say the rationale for Electron/Python apps is the same as that for > any JS frontend/Python backend configuration - JS/CSS/HTML5 is a great > suite of technologies for defining user interfaces, but you don't > necessarily want to be writing all your application logic in it. (You > certainly *can*, you just may not want to) > > The trade-offs are different for client-side apps (since shipping two > different language runtimes is kinda horrible, given neither V8 nor > CPython is particularly lightweight), but it's not *that* different > from the traditional Python GUI app development model of depending on > a C/C++ toolkit like Tcl/Tk, Gtk, Qt, or wxWidgets. > > It's just that the modern GUI toolkit is called V8, most of the actual > GUI bits are written in JavaScript rather than C/C++, and the language > independent in-process bindings got fairly dramatically worse along > the way :) > > Cheers, > Nick. > From tjreedy at udel.edu Fri Feb 10 21:05:49 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 10 Feb 2017 21:05:49 -0500 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: On 2/10/2017 4:13 AM, Markus Meskanen wrote: > I'm suggesting the addition of support to using a dot notation when > defining a function to be a method of a class, or a callback attribute. My default starting position for every proposed syntax addition is -1. 1. Additions usually make Python more complicated and harder to learn. (This can be argued in particular cases.) 2. Syntax changes, when used, cut code off from past versions. Consequently, to me, there should be a non-trivial gain to compensate for the above. > For example: > > def foo(self): > pass > Foo.foo = foo > > Becomes: > > def Foo.foo(self): > pass Saving about 10 keystrokes is close to trivial. If Foo is defined in the same file, then putting the def in the class statement would save even more. I am not enthusiastic about enablin the following style of class definition. class Foo: "Object that implement the Foo protocol. def Foo.__init__(self, x): self.x = s def Foo.__getitem__(self, key): return vars(Foo)[key] ... > And when an object needs a callback as an attribute: > > class Menu: > def __init__(self, items=None, select_callback=None): > self.items = items if items is not None else [] > self.select_callback = select_callback > > my_menu = Menu([item1, item2]) > > def my_menu.select_callback(self, item_index): > print(self.items[item_index]) A function with a 'self' parameter is normally an instance method (a function attribute of the class). As an instance attribute, it will have be called as inst.select_callback(inst, index). But I suppose you could find an example function that I like better as an instance attribute. > As opposed to: > > my_menu = Menu([item1, item2]) > > def select_callback(self, item_index): > print(self.items[item_index]) > my_menu.select_callback = select_callback or my_menu.select_callback = (lambda self, item_index: print(self.items[item_index])) The problem with two-phase initialization is that one temporarily has a partially initialized and likely useless object. I am not enthusiastic about encouraging this. > Or defining them in "unnatural" order: To me, this is the natural and proper order: create all the objects needed to initialize an instance before creating it. When __init__ returns, the instance is ready to go. In tkinter programming, callbacks must be defined before they are used in a bind or after call, which passes them on to tk, where they are not directly accessible as attributes. > def select_callback(self, item_index): > print(self.items[item_index]) > > my_menu = Menu([item1, item2], select_callback) Looks good to me ;-) -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Feb 11 00:33:39 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 11 Feb 2017 18:33:39 +1300 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> Message-ID: <589EA233.2090609@canterbury.ac.nz> Chris Angelico wrote: > Which is why these proposals always seem to gravitate to "anything you > can assign to", There might be some parsing difficulties with that, e.g. def foo(x)[5](y, z): ... That should be acceptable, because foo(x)[5] is something assignable, but foo(x) looks like the beginning of the definition of a function called foo. I'm not sure whether the parser would cope with that. -- Greg From random832 at fastmail.com Sat Feb 11 15:17:41 2017 From: random832 at fastmail.com (Random832) Date: Sat, 11 Feb 2017 15:17:41 -0500 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <589EA233.2090609@canterbury.ac.nz> References: <20170210101327.GA4796@ando.pearwood.info> <589EA233.2090609@canterbury.ac.nz> Message-ID: <1486844261.4158493.877954672.469D1407@webmail.messagingengine.com> On Sat, Feb 11, 2017, at 00:33, Greg Ewing wrote: > Chris Angelico wrote: > > Which is why these proposals always seem to gravitate to "anything you > > can assign to", > > There might be some parsing difficulties with that, e.g. > > def foo(x)[5](y, z): > ... > > That should be acceptable, because foo(x)[5] is something > assignable, but foo(x) looks like the beginning of the > definition of a function called foo. I'm not sure whether > the parser would cope with that. We could require parentheses to be used anywhere the grammar otherwise couldn't handle it, like yielding a tuple from a generator expression. def (whatever)(args): This does raise the question though of what the function's name/qualname would be. It's cosmetic, but it's also the only real difference between def and an assignment *now*, so it's worth considering. In the case where the last element of the expression is an attribute, the name would simply be the attribute, but would the class portion of the qualname (and the name when it's not an attribute) need to depend on the runtime value of what is being assigned, or would it simply use a string of exactly "foo(x)[5]"? From steve at pearwood.info Sat Feb 11 23:10:52 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Feb 2017 15:10:52 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutside of a class with the dot operator In-Reply-To: References: Message-ID: <20170212041050.GK4796@ando.pearwood.info> On Fri, Feb 10, 2017 at 06:17:54PM +0200, Markus Meskanen wrote: > Well yes, but I think you're a bit too fast on labeling it a mistake to use > monkey patching... More importantly, I think we're being a bit too quick to label this technique "monkey-patching" at all. Monkey-patching (or MP for brevity) implies making modifications to some arbitrary *uncooperative* class (or instance). When you're plugging electrodes into a monkey's brain, the monkey has no say in it. This proposed syntax can, of course, be used that way, but Python is already a "consenting adults" language and already has setattr: setattr(some_class, 'get_shrubbery', get_shrubbery) which is all you need to enable MP for good or evil. There have been a few times where I would have used this syntax if it had been available, and none of them were MP. They were injecting methods into classes I controlled. I suspect that this technique wouldn't feel so bad if we had a proper, respectable sounding "design pattern" name for it, like "method injection" or something. I expect that the only reason there is no name for this is that Java doesn't allow it. (I think.) So I'm going to call it "method injection". I don't think there's any evidence that slightly cleaner syntax for method injection will encourage MP. We already have clean syntax to inject arbitrary attributes (including methods made with lambda): TheClass.method = lambda self: ... and I don't think there's an epidemic of MP going on. -- Steve From steve at pearwood.info Sat Feb 11 23:37:47 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Feb 2017 15:37:47 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: Message-ID: <20170212043746.GL4796@ando.pearwood.info> On Fri, Feb 10, 2017 at 12:11:46PM -0600, Steve Dower wrote: > When you apply the "what if everyone did this" rule, it looks like a > bad idea (or alternatively, what if two people who weren't expecting > anyone else to do this did it). When you apply that rule, Python generally fails badly. In theory, Python is the worst possible language to be programming in, because the introspection capabilities are so powerful, the data hiding so feeble, the dynamicism of the language so great that almost anything written in pure-Python can be poked and prodded, bits deleted and new bits inserted. Python doesn't even have constants!!! import math math.pi = 3.0 # change the very geometry of spacetime And yet, in practice this is a problem more in theory than in practice. While you are right to raise this as a possible disadvantage of the proposal ("may ever-so-slightly encourage monkey-patching, by making it seem ever-so-slightly less mucky") I don't think you are right to weigh it as heavily as you appear to be doing. Python has had setattr() forever, and the great majority of Python programmers manage to avoid abusing it. > Monkeypatching is fairly blatantly taking advantage of the object > model in a way that is not "supported" and cannot behave well in the > context of everyone doing it, whereas inheritance or mixins are safe. That's an extremely optimistic view of things. Guido has frequently eluded to the problems with inheritance (you can't just inherit from anything and expect your code to work), and he's hardly the only one that has pointed out that inheritance and OOP hasn't turned out to be the panacea that people hoped. As for mixins, Michele Simionato has written a series of blog posts about the dangers of mixins and multiple inheritance, and suggesting traits as a more restricted and safer alternative. Start here: http://www.artima.com/weblogs/viewpost.jsp?thread=246488 > Making a dedicated syntax or decorator for patching is saying that we > (the language) think you should do it. We already have that syntax: anything.name = thing > (The extension_method decorator > sends exactly the wrong message about what it's doing.) Are you refering to a decorator something like this? @extend(TheClass) def method(self, arg): ... assert TheClass.method is method Arguments about what it should be called aside, what is the wrong message you see here? > Enabling a __class__ variable within the scope of the definition would > also solve the motivating example, Can you elaborate on that a bit more? Given the current idiom for injecting a method: class MyClass: ... # Later on... def method(self, arg): ... MyClass.method = method del method where does the __class__ variable fit into this? -- Steve From steve at pearwood.info Sat Feb 11 23:38:58 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Feb 2017 15:38:58 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <589EA233.2090609@canterbury.ac.nz> References: <20170210101327.GA4796@ando.pearwood.info> <589EA233.2090609@canterbury.ac.nz> Message-ID: <20170212043858.GM4796@ando.pearwood.info> On Sat, Feb 11, 2017 at 06:33:39PM +1300, Greg Ewing wrote: > Chris Angelico wrote: > >Which is why these proposals always seem to gravitate to "anything you > >can assign to", > > There might be some parsing difficulties with that, e.g. > > def foo(x)[5](y, z): > ... > > That should be acceptable, because foo(x)[5] is something > assignable, but foo(x) looks like the beginning of the > definition of a function called foo. I'm not sure whether > the parser would cope with that. Forget the parser. I know *I* can't cope with that. *wink* -- Steve From rosuav at gmail.com Sat Feb 11 23:50:03 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 12 Feb 2017 15:50:03 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170212043858.GM4796@ando.pearwood.info> References: <20170210101327.GA4796@ando.pearwood.info> <589EA233.2090609@canterbury.ac.nz> <20170212043858.GM4796@ando.pearwood.info> Message-ID: On Sun, Feb 12, 2017 at 3:38 PM, Steven D'Aprano wrote: > On Sat, Feb 11, 2017 at 06:33:39PM +1300, Greg Ewing wrote: >> Chris Angelico wrote: >> >Which is why these proposals always seem to gravitate to "anything you >> >can assign to", >> >> There might be some parsing difficulties with that, e.g. >> >> def foo(x)[5](y, z): >> ... >> >> That should be acceptable, because foo(x)[5] is something >> assignable, but foo(x) looks like the beginning of the >> definition of a function called foo. I'm not sure whether >> the parser would cope with that. > > Forget the parser. I know *I* can't cope with that. > > *wink* So you think the language should prevent silly assignments? >>> stuff = [None] * 10 >>> def foo(): return stuff ... >>> for x, foo()[x] in enumerate(range(len(stuff))): pass ... >>> stuff [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Given that Python is happy to do these kinds of assignments in 'for' statements, I don't see any reason to prevent them in 'def' statements. It's not the language's job to prevent abuse; at best, that's a job for a style guide. ChrisA From steve at pearwood.info Sun Feb 12 05:20:03 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Feb 2017 21:20:03 +1100 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170210101327.GA4796@ando.pearwood.info> <589EA233.2090609@canterbury.ac.nz> <20170212043858.GM4796@ando.pearwood.info> Message-ID: <20170212102003.GN4796@ando.pearwood.info> On Sun, Feb 12, 2017 at 03:50:03PM +1100, Chris Angelico wrote: > >> def foo(x)[5](y, z): > >> ... [...] > > Forget the parser. I know *I* can't cope with that. > > > > *wink* > > So you think the language should prevent silly assignments? On a case-by-case basis, of course. > >>> stuff = [None] * 10 > >>> def foo(): return stuff > ... > >>> for x, foo()[x] in enumerate(range(len(stuff))): pass > ... I have no idea what that does except by studying the code with great care and essentially running it in my own mental Python interpreter. Should it be prohibited? Not now, that would break backwards compatibility. If it were 1991 or thereabouts again, and Python 0.1 was newly released, and somebody suggested an enhancement to the language that would specifically allow that awfulness, would you be in favour of allowing it? If it were 1991, I'd seriously consider arguing that the loop assignment target should be restricted to a simple name or tuple of names. It's one thing to say "this abomination is allowed because of historical reasons", and another to say "I think your proposal isn't general enough. We should generalise your nice, clean, simple proposal to something nobody in their right mind would ever use!" In fact, if I were more cynical, I'd wonder whether you were trying to sabotage this proposal by over-generalising it to something that has no good use-cases. *multiple smileys* There's a lot of this sort of thing on Python-Ideas: "I think it would be good if Python included a stapler, as a lightweight, quick and easy way to join sheets of paper." "Excellent idea! I think the stapler should include a drill attachment, a sledge hammer and a crowbar, in case you wish to staple the paper to a concrete slab. You could use the drill with a masonry bit to drill into the concrete slab, then you hammer the extra-giant-size staple through the paper and the drill holes. Of course you'll need to use the crowbar to flip the slab upside down so you can hammer the other side of the staple flat. The slab might be thicker than your drill bit, so the stapler also needs X-ray imaging equipment so you can line up the holes you drill from each side and ensure they meet up correctly." *wink* > Given that Python is happy to do these kinds of assignments in 'for' > statements, I don't see any reason to prevent them in 'def' > statements. It's not the language's job to prevent abuse; at best, > that's a job for a style guide. For many years, preventing this sort of abuse is exactly what the language has done. This proposal is to introduce a *very slight* loosening of the restriction, not tear the whole thing down. There is plenty of good precedent for restricting assignment targets: py> errors = [None] py> try: ... pass ... except Exception as errors[0]: File "", line 3 except Exception as errors[0]: ^ SyntaxError: invalid syntax py> import math as mymodules[-1] File "", line 1 import math as mymodules[-1] ^ SyntaxError: invalid syntax And similar restrictions on decorators: py> @decorators['key'] File "", line 1 @decorators['key'] ^ SyntaxError: invalid syntax Its easy to loosen the restriction later if necessary, and all but impossible to tighten it up again if the original decision turns out to be a mistake. My view regarding syntax changes is, what is the *smallest* change to syntax that will satisfy the use-case? Not the broadest or most general. It would be different if you had concrete use-cases for the generalisation to any arbitrary assignment target. But as it stands, it is a clear case of YAGNI, and it complicates the question of what __name__ and __qualname__ should be set to. -- Steve From p.f.moore at gmail.com Sun Feb 12 06:38:46 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Feb 2017 11:38:46 +0000 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: <20170212043746.GL4796@ando.pearwood.info> References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On 12 February 2017 at 04:37, Steven D'Aprano wrote: >> Making a dedicated syntax or decorator for patching is saying that we >> (the language) think you should do it. > > We already have that syntax: > > anything.name = thing And the point here is that we don't need to extend def, because we already have that syntax. Adding new syntax for something that we can already do is generally accepted when the "thing we can already do" is deemed sufficiently important that it's worth making it a language feature in its own right. Decorators are a prime example of this - before the decorator syntax was added, decorating functions was just something that people occasionally did, but it wasn't a specific "concept". I'd argue that method injection (to use your phrase) isn't sufficiently important to warrant promotion to language syntax. I will say, though, that you're right that we've over-reacted a bit to the monkeypatching use case. Although maybe that's because no-one can think of many *other* use cases that they'd need the new syntax for :-) Paul From tritium-list at sdamon.com Sun Feb 12 07:26:29 2017 From: tritium-list at sdamon.com (Alex Walters) Date: Sun, 12 Feb 2017 07:26:29 -0500 Subject: [Python-ideas] site.py uses os.sep to determine platform Message-ID: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> On august 7th, 1998, Guido committed https://github.com/python/cpython/commit/d89fa0c5761254c970af72e5abcea420fd2 3e893 to python, adding the quit() and exit() built-ins. He decided to determine the platform python was running on by checking os.sep. I don't understand the rationale behind this choice in 1998, but I assume that there was one. It's 2017. We have sys.platform, and the standard library is tested against everything that vanilla cpython (unpatched by a vendor) will ever put in cpython. Is there any reason not to change os.sep == '\\' to sys.platform == 'win32' in 2017? From phd at phdru.name Sun Feb 12 08:10:32 2017 From: phd at phdru.name (Oleg Broytman) Date: Sun, 12 Feb 2017 14:10:32 +0100 Subject: [Python-ideas] site.py uses os.sep to determine platform In-Reply-To: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> References: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> Message-ID: <20170212131032.GA3082@phdru.name> On Sun, Feb 12, 2017 at 07:26:29AM -0500, Alex Walters wrote: > Is there any reason not to change... Do not change things that work for the sake of a change. Isn't it a good reason? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From steve.dower at python.org Sun Feb 12 10:05:44 2017 From: steve.dower at python.org (Steve Dower) Date: Sun, 12 Feb 2017 07:05:44 -0800 Subject: [Python-ideas] site.py uses os.sep to determine platform In-Reply-To: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> References: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> Message-ID: It depends on what the code beneath the if is doing. "Windows" may be a shorthand for "uses backslash", in which case the check is correct. We should certainly match the check to the intent, but we need to determine the intent first. Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Alex Walters" Sent: ?2/?12/?2017 4:35 To: "python-ideas at python.org" Subject: [Python-ideas] site.py uses os.sep to determine platform On august 7th, 1998, Guido committed https://github.com/python/cpython/commit/d89fa0c5761254c970af72e5abcea420fd2 3e893 to python, adding the quit() and exit() built-ins. He decided to determine the platform python was running on by checking os.sep. I don't understand the rationale behind this choice in 1998, but I assume that there was one. It's 2017. We have sys.platform, and the standard library is tested against everything that vanilla cpython (unpatched by a vendor) will ever put in cpython. Is there any reason not to change os.sep == '\\' to sys.platform == 'win32' in 2017? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Sun Feb 12 11:03:33 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 12 Feb 2017 17:03:33 +0100 Subject: [Python-ideas] site.py uses os.sep to determine platform In-Reply-To: References: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> Message-ID: Well to be pedantic, the code is obviously wrong since it assumes EOF is generated by ctrl-D, but the user could have changed that with stty. It should query the terminal settings instead... Yeah I know, people who do that don't need the message. Op 12 feb. 2017 4:26 p.m. schreef "Steve Dower" : > It depends on what the code beneath the if is doing. "Windows" may be a > shorthand for "uses backslash", in which case the check is correct. > > We should certainly match the check to the intent, but we need to > determine the intent first. > > Cheers, > Steve > > Top-posted from my Windows Phone > ------------------------------ > From: Alex Walters > Sent: ?2/?12/?2017 4:35 > To: python-ideas at python.org > Subject: [Python-ideas] site.py uses os.sep to determine platform > > On august 7th, 1998, Guido committed > https://github.com/python/cpython/commit/d89fa0c5761254c970af72e5abcea4 > 20fd2 > 3e893 to python, adding the quit() and exit() built-ins. He decided to > determine the platform python was running on by checking os.sep. I don't > understand the rationale behind this choice in 1998, but I assume that > there > was one. > > It's 2017. We have sys.platform, and the standard library is tested > against > everything that vanilla cpython (unpatched by a vendor) will ever put in > cpython. > > Is there any reason not to change os.sep == '\\' to sys.platform == 'win32' > in 2017? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markusmeskanen at gmail.com Sun Feb 12 11:51:25 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Sun, 12 Feb 2017 18:51:25 +0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: I will say, though, that you're right that we've over-reacted a bit to the monkeypatching use case. Although maybe that's because no-one can think of many *other* use cases that they'd need the new syntax for :-) Paul Hi Paul, I believe at least two other use cases than monkey patching have been mentioned already: 1. Allowing the class to be used in the method's header, f.e. for typing and decorators: @decorate(MyClass) def MyClass.method(self, other: MyClass) -> List[MyClass]: ... This is useful since you can't refer the class itself inside of its body. At the moment the way to use typing is to write the class's name as a string... It feels awful. 2. To register callbacks to objects, i.e. plain out set an attribute for an instance. I've used the menu example above: class Menu: def __init__(self, items=None, select_callback=None): self.items = items if items is not None else [] self.select_callback = select_callback my_menu = Menu(['Pizza', 'Cake', 'Pasta']) def my_menu.select_callback(item_index): if item_index == 0: # Pizza serve_food(pizza) else: # Cake or Pasta ... This is just one example of using it to set an instance's variable to a callback. It's just shorthand for: def select_callback(item_index): ... my_menu.select_callback = select_callback This reads much easier and saves us from typing the same thing three times (see decorators). -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Feb 12 12:30:18 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 12 Feb 2017 09:30:18 -0800 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: I think the proposal, so far, seems to confuse two separate things. One is attaching a method to a class after definition. The second is attaching a method to an instance after creation. Or at least it is unclear to me which of those is the intention, since both seem to occur in the examples. Or maybe it's both, but those feel like fairly different use cases. Which is to say, we really need a PEP. As it stands, I'm somewhere around -0.75 on the idea. @decorate(MyClass) > def MyClass.method(self, other: MyClass) -> List[MyClass]: > In this case, the syntax is 100% superfluous. One can simply write: @decorate(MyClass) def method(self, other: MyClass) -> List[MyClass]: The class is already mentioned in the decorator. If the intention is to add the method to the class, that's fine, and something a decorator can do. Perhaps the spelling for this decorator-factory could be `enhance`. Or more verbosely `inject_method`. Spelling aside, the syntax adds nothing. my_menu = Menu(['Pizza', 'Cake', 'Pasta']) > def my_menu.select_callback(item_index): > if item_index == 0: # Pizza > serve_food(pizza) > else: # Cake or Pasta > ... > Attaching to the instance is fine too. But I prefer the current spelling so far: my_menu1 = Menu(['Pizza', 'Cake', 'Pasta']) my_menu2 = Menu(...) def callback1(self, ...): ... def callback2(self, ...): ... my_menu1.callback = callback2 my_menu2.callback = callback1 Under the current approach, you can flexibly define callbacks outside of the scope of any particular instance or class, and attach them as needed to instances. Obviously the new syntax would not *remove* this option, but it would cover only a narrow subset of what we can already do... and the way we do it now feels much better self-documenting as to intent. Yours, David... -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Feb 12 12:49:43 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Feb 2017 17:49:43 +0000 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On 12 February 2017 at 16:51, Markus Meskanen wrote: >> I will say, though, that you're right that we've over-reacted a bit to >> the monkeypatching use case. Although maybe that's because no-one can >> think of many *other* use cases that they'd need the new syntax for >> :-) > Hi Paul, I believe at least two other use cases than monkey patching have > been mentioned already: My point was that people couldn't think of use cases *they* would need the syntax for. Personally, I'd never use the new syntax for the 2 examples you gave. I don't know if your examples are from real-world code, but they feel artificial to me (the callback one less so, but people have been using callbacks for years without missing this syntax). This is just one example of using it to set an instance's variable to a callback. It's just shorthand for: > def select_callback(item_index): > ... > my_menu.select_callback = select_callback > This reads much easier That's personal opinion, and there's a lot of disagreement on this point. > and saves us from typing the same thing three times > (see decorators). That's an objective benefit, sure. I don't think it's major in itself, but that's just *my* opinion :-) You could of course use a shorter name for the function, if it matters to you (it doesn't *have* to be the same as the attribute name). Anyway, let's wait for a PEP that addresses all of the points raised in this thread. Paul From markusmeskanen at gmail.com Sun Feb 12 12:55:57 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Sun, 12 Feb 2017 19:55:57 +0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: I think the proposal, so far, seems to confuse two separate things. One is attaching a method to a class after definition. The second is attaching a method to an instance after creation. Or at least it is unclear to me which of those is the intention, since both seem to occur in the examples. Or maybe it's both, but those feel like fairly different use cases. Aren't they the same though? Remember that classes are instances of type and methods are just their attributes. We're simply using setattr() in both cases: with instances, and with classes (=instances). @decorate(MyClass) > def MyClass.method(self, other: MyClass) -> List[MyClass]: > In this case, the syntax is 100% superfluous. One can simply write: @decorate(MyClass) def method(self, other: MyClass) -> List[MyClass]: The class is already mentioned in the decorator. If the intention is to add the method to the class, that's fine, and something a decorator can do. Perhaps the spelling for this decorator-factory could be `enhance`. Or more verbosely `inject_method`. Spelling aside, the syntax adds nothing. I think you missed the point, the decorator was just an example and has arbitary functionality. The point is that you can not refer the class itself in its body, so you can't do either of these methods: class Foo: def concenate(self, other: Foo) -> Foo: ... @just_an_example_decorator(mapper=Foo) def map(self) -> dict: ... Because Foo is not defined at the time of executing the function headers. The proposed feature would allow you to easily define these after the class definition and allow refering to the class directly. my_menu = Menu(['Pizza', 'Cake', 'Pasta']) > def my_menu.select_callback(item_index): > if item_index == 0: # Pizza > serve_food(pizza) > else: # Cake or Pasta > ... > Attaching to the instance is fine too. But I prefer the current spelling so far: my_menu1 = Menu(['Pizza', 'Cake', 'Pasta']) my_menu2 = Menu(...) def callback1(self, ...): ... def callback2(self, ...): ... my_menu1.callback = callback2 my_menu2.callback = callback1 I don't, it is repeating the variable name three times. I don't see how this differs from decorator syntax, do you prefer the old way on that too, or am I missing something? Under the current approach, you can flexibly define callbacks outside of the scope of any particular instance or class, and attach them as needed to instances. Obviously the new syntax would not *remove* this option, but it would cover only a narrow subset of what we can already do... and the way we do it now feels much better self-documenting I think you answered yourself here, this would not remove the existing flexible way. Just like @decorator syntax didn't remove the more flexible way. Honestly this is in my opinion almost one-to-one comparable with decorator syntax, and I don't think anyone here dares to claim decorators aren't awesome. - Markus -------------- next part -------------- An HTML attachment was scrubbed... URL: From mehaase at gmail.com Sun Feb 12 12:56:38 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Sun, 12 Feb 2017 12:56:38 -0500 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On Sun, Feb 12, 2017 at 11:51 AM, Markus Meskanen wrote: > > 2. To register callbacks to objects, i.e. plain out set an attribute for > an instance. I've used the menu example above: > > class Menu: > def __init__(self, items=None, select_callback=None): > self.items = items if items is not None else [] > self.select_callback = select_callback > > my_menu = Menu(['Pizza', 'Cake', 'Pasta']) > def my_menu.select_callback(item_index): > if item_index == 0: # Pizza > serve_food(pizza) > else: # Cake or Pasta > ... > This is just one example of using it to set an instance's variable to a > callback. It's just shorthand for: > > def select_callback(item_index): > ... > my_menu.select_callback = select_callback > One issue that has been overlooked so far in this thread is that hypothetical use cases are not as important as real-world use cases. One way that PEPs can demonstrate real-world relevance is by demonstrating the effect on some important libraries, e.g. the standard library. For example, asyncio.Future (and concurrent.futures.Future) has a list of callbacks and the API has add_done_callback() and remove_done_callback() functions for manipulating the callback list. The proposed syntax doesn't cooperate with these callbacks: f = asyncio.Future() def my_callback(x): ... f.add_done_callback(my_callback) How should I write this using the proposed syntax? If the proposal doesn't generalize well enough to cover existing callback patterns in Python's own standard library, then that is a significant weakness. Please keep this in mind as you write the PEP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Feb 12 13:19:39 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 12 Feb 2017 10:19:39 -0800 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: > > Attaching to the instance is fine too. But I prefer the current spelling > so far: > > > my_menu1 = Menu(['Pizza', 'Cake', 'Pasta']) > > my_menu2 = Menu(...) > > > def callback1(self, ...): > ... > def callback2(self, ...): > ... > > my_menu1.callback = callback2 > > my_menu2.callback = callback1 > > > I don't, it is repeating the variable name three times. I don't see how > this differs from decorator syntax, do you prefer the old way on that too, > or am I missing something? > I haven't repeated any name. Notice that '.callback' is different from 'callback1' or 'callback2'. That's exactly the idea?I can attach *arbitrary* callbacks later on to the '.callback' attribute. > I think > > you answered yourself here, this would not remove the existing flexible > way. Just like @decorator syntax didn't remove the more flexible way. > Honestly this is in my opinion almost one-to-one comparable with decorator > syntax, and I don't think anyone here dares to claim decorators aren't > awesome. > But we already *have* decorators! Here's a nice factory for them: def attach_to(thing, name=None): def decorator(fn): if name is None: name = fn.__name__ setattr(thing, name, fn) return decorator This does everything you are asking for, e.g.: my_menu = Menu() @attach_to(my_menu) def callback(self, ...) ... I got extra fancy with two lines to allow you to either use the same name as the function itself or pick a custom name for the attribute. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Feb 12 13:26:13 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 12 Feb 2017 10:26:13 -0800 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: Oh, I probably want `return fn` inside my inner decorator. Otherwise, the defined name gets bound to None in the global scope. I'm not sure, maybe that's better... but most likely we should leave the name for other users. I just wrote it without testing. On Sun, Feb 12, 2017 at 10:19 AM, David Mertz wrote: > > But we already *have* decorators! Here's a nice factory for them: > > def attach_to(thing, name=None): > > def decorator(fn): > > if name is None: > > name = fn.__name__ > > setattr(thing, name, fn) > > return decorator > > > This does everything you are asking for, e.g.: > > my_menu = Menu() > > @attach_to(my_menu) > def callback(self, ...) > ... > > > I got extra fancy with two lines to allow you to either use the same name > as the function itself or pick a custom name for the attribute. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Sun Feb 12 14:01:58 2017 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sun, 12 Feb 2017 17:01:58 -0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On 12 February 2017 at 14:51, Markus Meskanen wrote: > 1. Allowing the class to be used in the method's header, f.e. for typing and > decorators: > > @decorate(MyClass) > def MyClass.method(self, other: MyClass) -> List[MyClass]: > ... > > This is useful since you can't refer the class itself inside of its body. At > the moment the way to use typing is to write the class's name as a string... > It feels awful. You realize now that if we accept this change, and given your example, any "well behaved" Python code with markup will in a couple months required to be like class MyClass: """Docstring.""" def MyClass.__init__(self: MyClass, ...) -> None: ... # add other methods here. And all it will take is some bureaucratic minded person to put that as default option in some highly used linter, like the one that used-to-be-known-as-pep8. (And hint: what do you think is the mind orientation of contributors to linter code? :-) ) As a developer constrained to silly rules in automatic linters (like nazi-counting the number of blank lines everywhere) due to project manager "it's simples to just stand by the linters defaults" I feel quire worried about that. So, no, strings for type hinting are much less awful than effectively killing the class body in big projects. Not that this is much more serious than the worries about def x["fnord"][5]["gnorts"]method(self, bla): ... Which will never be used in sane code anyways. It is the real, present, danger of of having mandates in whole projects that all methods be defined outside the class body just because of "clean type-hinting". I now am much, much, more scared of this proposal than before, and I was already at -1 . Please, just let this R.I.P. js -><- From random832 at fastmail.com Sun Feb 12 15:51:42 2017 From: random832 at fastmail.com (Random832) Date: Sun, 12 Feb 2017 15:51:42 -0500 Subject: [Python-ideas] site.py uses os.sep to determine platform In-Reply-To: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> References: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> Message-ID: <1486932702.2133446.878645944.75ED9D39@webmail.messagingengine.com> On Sun, Feb 12, 2017, at 07:26, Alex Walters wrote: > On august 7th, 1998, Guido committed > https://github.com/python/cpython/commit/d89fa0c5761254c970af72e5abcea420fd2 > 3e893 to python, adding the quit() and exit() built-ins. He decided to > determine the platform python was running on by checking os.sep. I don't > understand the rationale behind this choice in 1998, but I assume that > there > was one. My guess would be that the rationale was to group Windows together with DOS and OS/2. From ncoghlan at gmail.com Sun Feb 12 16:29:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Feb 2017 22:29:10 +0100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On 12 February 2017 at 12:38, Paul Moore wrote: > On 12 February 2017 at 04:37, Steven D'Aprano wrote: >>> Making a dedicated syntax or decorator for patching is saying that we >>> (the language) think you should do it. >> >> We already have that syntax: >> >> anything.name = thing > > And the point here is that we don't need to extend def, because we > already have that syntax. Adding new syntax for something that we can > already do is generally accepted when the "thing we can already do" is > deemed sufficiently important that it's worth making it a language > feature in its own right. Decorators are a prime example of this - > before the decorator syntax was added, decorating functions was just > something that people occasionally did, but it wasn't a specific > "concept". Note that true method injection would *NOT* be the same as binding a callable as a class attribute after the fact: - attribute assignment doesn't modify __name__ - attribute assignment doesn't modify __qualname__ - attribute assignment doesn't call __set_owner__ - attribute assignment doesn't adjust the __class__ cell reference Any method injection syntax worthy of the name would need to do those things (probably via a new __setdescriptor__ magic method that is a counterpart to PEP 447's __getdescriptor__). > I'd argue that method injection (to use your phrase) isn't > sufficiently important to warrant promotion to language syntax. There's a lot to be said for implementing mixin behaviour by way of definition time method injection rather than via MRO traversal when looking up method names (although __init_subclass__ took away one of the arguments in favour of it, since mixins can check their invariants at definition time now). > I will say, though, that you're right that we've over-reacted a bit to > the monkeypatching use case. Although maybe that's because no-one can > think of many *other* use cases that they'd need the new syntax for > :-) Method injection is most attractive to me as a potential alternative to mixin classes that has fewer runtime side effects by moving more of the work to class definition time. More philosophically though, it offends my language design sensibilities that we have so much magic bound up in class definitions that we don't expose for procedural access post-definition time - there's a whole lot of behaviours that "just happen" when a method is defined lexically inside a class body that can't readily be emulated for callables that are defined outside it. However, even with that, I'm still only +0 on the idea - if folks really want it, `types.new_class` can already be used to creatively to address most of these things, and it's not exactly a problem that comes up very often in practice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 12 16:36:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Feb 2017 22:36:20 +0100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: On 12 February 2017 at 22:29, Nick Coghlan wrote: > However, even with that, I'm still only +0 on the idea - if folks > really want it, `types.new_class` can already be used to creatively to > address most of these things, and it's not exactly a problem that > comes up very often in practice. I'll also note that much of what I'm talking about there could be exposed as a types.bind_descriptor() function that implemented the various adjustments (rebinding __class__ references is tricky though, since the function with a bound closure variable might be hidden inside another descriptor, like property) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 12 16:47:41 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Feb 2017 22:47:41 +0100 Subject: [Python-ideas] site.py uses os.sep to determine platform In-Reply-To: References: <044301d2852b$3d8ffc20$b8aff460$@sdamon.com> Message-ID: On 12 February 2017 at 16:05, Steve Dower wrote: > It depends on what the code beneath the if is doing. "Windows" may be a > shorthand for "uses backslash", in which case the check is correct. In this case, it's to decide whether the default way of entering EOF at the terminal is by pressing Ctrl-D (POSIX-style) or Ctrl-Z+Enter (Windows-style). There's another case in site.py where "os.sep='/'" is used to decide whether or not the site-packages path should be qualified with the python version. In the cases where it is used, the separator is likely to give you the right answer under things like Cygwin, MinGW and WSL (all of which should use "/" for the default path separator and Ctrl-D to enter EOF even though you're actually running on a Windows machine), but I wouldn't place any bets on it being easy to replicate the separator setting (and os.path module selection) logic that figures that out in the first place. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Sun Feb 12 21:55:50 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 13 Feb 2017 13:55:50 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: <20170213025550.GO4796@ando.pearwood.info> On Sun, Feb 12, 2017 at 05:01:58PM -0200, Joao S. O. Bueno wrote: > You realize now that if we accept this change, and given your example, > any "well behaved" Python code with markup will in a couple months > required to be like > > class MyClass: > """Docstring.""" > > def MyClass.__init__(self: MyClass, ...) -> None: > ... > > # add other methods here. This is pure and unadulterated FUD. Nobody is going to use this as the standard way of writing classes. That would be silly: you end up repeating the class name over and over and over again. And to say that this will happen "in a couple [of] months" is totally unrealistic. Although, I suppose that if the entire Python community did drop 2.7-3.6 and move to 3.7 within just one or two months so they could use this syntax, that would certainly vindicate the (hypothetical) decision to add this syntax. But honestly, no. This is not going to happen. .Net VB and C# have something like this, as does Lua, and people still write classes the ordinary way 99.99% of the time. The chances of this becoming the required, or even the recommended, way to write methods is much less than the chances of President Trump introducing Sharia law to the United States. > And all it will take is some bureaucratic minded person to put that as default > option in some highly used linter, like the one that used-to-be-known-as-pep8. Do you *really* think that a linter that used to be called "PEP8" is going to require as a default syntax which (1) doesn't work before Python 3.7 at the earliest, and (2) has no support in PEP-8? It's one thing to question whether this feature is useful enough to be worth adding. It's another to make panicky claims that the Sky Will Fall if it is accepted. -- Steve From jsbueno at python.org.br Sun Feb 12 22:42:42 2017 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 13 Feb 2017 01:42:42 -0200 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: <20170213025550.GO4796@ando.pearwood.info> References: <20170212043746.GL4796@ando.pearwood.info> <20170213025550.GO4796@ando.pearwood.info> Message-ID: On 13 February 2017 at 00:55, Steven D'Aprano wrote: > On Sun, Feb 12, 2017 at 05:01:58PM -0200, Joao S. O. Bueno wrote: > >> You realize now that if we accept this change, and given your example, >> any "well behaved" Python code with markup will in a couple months >> required to be like >> >> class MyClass: >> """Docstring.""" >> >> def MyClass.__init__(self: MyClass, ...) -> None: >> ... >> >> # add other methods here. > > This is pure and unadulterated FUD. > > Nobody is going to use this as the standard way of writing classes. That > would be silly: you end up repeating the class name over and over and > over again. Sorry - but the I just pointed the effect. The person saying that would start writing classes this way is the grand-parent poster: On 12 February 2017 at 14:51, Markus Meskanen wrote: > 1. Allowing the class to be used in the method's header, f.e. for typing and > decorators: > > @decorate(MyClass) > def MyClass.method(self, other: MyClass) -> List[MyClass]: > ... > > This is useful since you can't refer the class itself inside of its body. At > the moment the way to use typing is to write the class's name as a string... > It feels awful. You are correct in your message, and thank you for calming me down, - but one thing remains: I was really scared by the grand parent poster - and I still prefer this possibility would not exist. (And yes, I have code in which I needed doing what is proposed: the extra assignment line did not hurt me at all) js -><- > > And to say that this will happen "in a couple [of] months" is totally > unrealistic. Although, I suppose that if the entire Python community did > drop 2.7-3.6 and move to 3.7 within just one or two months so they could > use this syntax, that would certainly vindicate the (hypothetical) > decision to add this syntax. > > But honestly, no. This is not going to happen. .Net VB and C# have > something like this, as does Lua, and people still write classes the > ordinary way 99.99% of the time. > > The chances of this becoming the required, or even the recommended, way > to write methods is much less than the chances of President Trump > introducing Sharia law to the United States. > > >> And all it will take is some bureaucratic minded person to put that as default >> option in some highly used linter, like the one that used-to-be-known-as-pep8. > > Do you *really* think that a linter that used to be called "PEP8" is > going to require as a default syntax which (1) doesn't work before > Python 3.7 at the earliest, and (2) has no support in PEP-8? > > It's one thing to question whether this feature is useful enough to be > worth adding. It's another to make panicky claims that the Sky Will Fall > if it is accepted. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From george at fischhof.hu Mon Feb 13 07:23:22 2017 From: george at fischhof.hu (George Fischhof) Date: Mon, 13 Feb 2017 13:23:22 +0100 Subject: [Python-ideas] Python package index: query and rating Message-ID: Hi There, I several times found that it is quite difficult to find an appropriate library in package index. I have two ides to improve the package index: 1.) Query like in a webshop: checkboxes should use to select from attributes. (topic, environment, category etc) It would br good to put a "do not want" column as well. For example right now I would like to search for a cryptographic library and I have the following page https://pypi.python.org/pypi?:action=browse&c=401 It contains 424 libraries. I would like to select for example desktop, security, sofware development from topic, and do NOT want any framwork reated libraries (django, flask etc) 2.) It would would be good to implement some user rating mechanism like in the webshops or software stores. It would make the choice easier. Best regards, George Fischhof -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Mon Feb 13 13:25:00 2017 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Mon, 13 Feb 2017 13:25:00 -0500 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> On 2017-02-12 14:01, Joao S. O. Bueno wrote: > On 12 February 2017 at 14:51, Markus Meskanen wrote: >> 1. Allowing the class to be used in the method's header, f.e. for typing and >> decorators: >> >> @decorate(MyClass) >> def MyClass.method(self, other: MyClass) -> List[MyClass]: >> ... >> >> This is useful since you can't refer the class itself inside of its body. At >> the moment the way to use typing is to write the class's name as a string... >> It feels awful. > You realize now that if we accept this change, and given your example, > any "well behaved" Python code with markup will in a couple months > required to be like > > class MyClass: > """Docstring.""" > > def MyClass.__init__(self: MyClass, ...) -> None: > ... > > # add other methods here. I am for method-outside-of-class form: If it is allowed, I will use it extensively: * instance methods and extension methods have the same form * less lines between the line you are looking at and the name of the class * explicit class name helps with searching for methods * reduces indentation thanks From josephhackman at gmail.com Mon Feb 13 14:32:33 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Mon, 13 Feb 2017 14:32:33 -0500 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> References: <20170212043746.GL4796@ando.pearwood.info> <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> Message-ID: Generally speaking, I'm +1 on this idea, I think it would make code more readable, especially for tools like IDEs. I just wanted to ask: can someone point me to the reason Python doesn't support referencing a class inside it's own definition? It seems like that would solve some of the cases discussed here, and with Type hinting that seems like something that maybe should be considered? From matt at getpattern.com Mon Feb 13 14:50:09 2017 From: matt at getpattern.com (Matt Gilson) Date: Mon, 13 Feb 2017 11:50:09 -0800 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> Message-ID: For whatever weight my opinion holds, I'm +0 on this one. In my estimation, in an ideal world it seems like: class Foo(object): def bar(self): """Bar!""" # Identical to: class Foo(object): pass def Foo.bar(self): """Bar!""" But I think that's going to be hard to achieve given implicit binding of `super` (as some have already mentioned) and more mind-bendingly when user-defined metaclasses are in play. Indeed, with metaclasses, it seems like it become impossible to actually guarantee the equality of the above two blocks of code. Maybe the PEP writers are OK with that, but that should be decided at the outset... Also note that if users start adopting this as their default mode of class creation (rather than just *class extending*), code-folding in a lot of IDEs won't handle it gracefully (at least not for quite a while). On Mon, Feb 13, 2017 at 11:32 AM, Joseph Hackman wrote: > Generally speaking, I'm +1 on this idea, I think it would make code more > readable, especially for tools like IDEs. > > I just wanted to ask: can someone point me to the reason Python doesn't > support referencing a class inside it's own definition? It seems like that > would solve some of the cases discussed here, and with Type hinting that > seems like something that maybe should be considered? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt at getpattern.com // P: 603.892.7736 We?re looking for beta testers. Go here to sign up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 13 15:12:13 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 13 Feb 2017 21:12:13 +0100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> Message-ID: <81c45835-cfe9-a5ea-9fb5-34712ac9e5ca@egenix.com> On 13.02.2017 20:32, Joseph Hackman wrote: > I just wanted to ask: can someone point me to the reason Python doesn't support referencing a class inside it's own definition? It seems like that would solve some of the cases discussed here, and with Type hinting that seems like something that maybe should be considered? The class doesn't exist yet, while Python is running the code in its definition block. You can play some tricks with meta classes exposing a .__prepare__() method. This will receive the name of the to-be-created class and allows returning a custom namespace in which the code is run. https://docs.python.org/3.6/reference/datamodel.html#preparing-the-class-namespace The meta class docs have more details on how all this works: https://docs.python.org/3.6/reference/datamodel.html#metaclasses -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 13 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From steve at pearwood.info Mon Feb 13 19:26:12 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Feb 2017 11:26:12 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> Message-ID: <20170214002611.GB29990@ando.pearwood.info> On Mon, Feb 13, 2017 at 02:32:33PM -0500, Joseph Hackman wrote: > Generally speaking, I'm +1 on this idea, I think it would make code > more readable, especially for tools like IDEs. > > I just wanted to ask: can someone point me to the reason Python > doesn't support referencing a class inside it's own definition? It > seems like that would solve some of the cases discussed here, and with > Type hinting that seems like something that maybe should be > considered? The simple answer is: since the class doesn't exist yet, you cannot refer to it. The class name is just a regular name: py> MyClass = 'something else' py> class MyClass: ... print(MyClass) ... something else so the interpreter would need to provide some special-cased magic inside the class body to make it work as you expect. That may be a good idea, but it is a separate issue from this. -- Steve From steve at pearwood.info Mon Feb 13 19:44:14 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Feb 2017 11:44:14 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> <38e219f2-8374-6619-b586-57d5ec341cbd@mozilla.com> Message-ID: <20170214004414.GC29990@ando.pearwood.info> On Mon, Feb 13, 2017 at 11:50:09AM -0800, Matt Gilson wrote: > For whatever weight my opinion holds, I'm +0 on this one. In my > estimation, in an ideal world it seems like: > > class Foo(object): > def bar(self): > """Bar!""" > > > # Identical to: > > class Foo(object): pass > > def Foo.bar(self): > """Bar!""" > > But I think that's going to be hard to achieve given implicit binding of > `super` (as some have already mentioned) and more mind-bendingly when > user-defined metaclasses are in play. I think that this is too high a bar to reach (pun not intended). A metaclass can do anything it likes to the methods in the class, and injecting a method after the class already exists is not necessarily the same as including it in the initial namespace argument passed to the metaclass. I think a more reasonable bar is to have def Foo.bar(self): ... equivalent to def bar(self): ... Foo.bar = bar # Foo is a class del bar except that the usual class magic like setting __qualname__, super() etc will work. That feels doable. For instances, the invariant should be slightly different: def bar(self): ... foo.bar = types.MethodType(bar, foo) # foo is an instance del bar > Indeed, with metaclasses, it seems > like it become impossible to actually guarantee the equality of the above > two blocks of code. Maybe the PEP writers are OK with that, but that > should be decided at the outset... Indeed. > Also note that if users start adopting this as their default mode of class > creation (rather than just *class extending*), code-folding in a lot of > IDEs won't handle it gracefully (at least not for quite a while). Why would people use this as the default mode of class creation? I mean, sure there's always that *one guy* who insists on their own weird idiosyncratic way of doing things. I know somebody who refuses to use for loops, and writes all his loops using while. But I can't see this becoming a widespread practice. We all have our quirks, but most of our quirks are not that quirky. -- Steve From steve at pearwood.info Mon Feb 13 20:06:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Feb 2017 12:06:57 +1100 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: References: <20170212043746.GL4796@ando.pearwood.info> Message-ID: <20170214010657.GD29990@ando.pearwood.info> On Sun, Feb 12, 2017 at 10:29:10PM +0100, Nick Coghlan wrote: [...] > Method injection is most attractive to me as a potential alternative > to mixin classes that has fewer runtime side effects by moving more of > the work to class definition time. > > More philosophically though, it offends my language design > sensibilities that we have so much magic bound up in class definitions > that we don't expose for procedural access post-definition time - > there's a whole lot of behaviours that "just happen" when a method is > defined lexically inside a class body that can't readily be emulated > for callables that are defined outside it. If the OP is willing to write a PEP, I think it is worth taking a three-part approach: - expose the class definition magic that Nick refers to; - which will allow writing a proper inject_method() decorator; - or allow def Class.method syntax. I think I would prefer def Class.method ... over @inject_method(Class) def method ... del method but given how high the barrier to new syntax is, perhaps we should be willing to take the decorator approach and leave the syntax for the future, once people have got used to the idea that extension methods won't cause the fall of civilization as we know it :-) > However, even with that, I'm still only +0 on the idea - if folks > really want it, `types.new_class` can already be used to creatively to > address most of these things, and it's not exactly a problem that > comes up very often in practice. Swift and Objective-C users might, I think, disagree with that: they even have a term for this, "swizzling". This is part of the "Interceptor" design pattern: https://en.wikipedia.org/wiki/Interceptor_pattern -- Steve From steve at pearwood.info Mon Feb 13 20:48:15 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Feb 2017 12:48:15 +1100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: Message-ID: <20170214014814.GE29990@ando.pearwood.info> On Fri, Feb 10, 2017 at 09:05:49PM -0500, Terry Reedy wrote: > > def foo(self): > > pass > > Foo.foo = foo > > > >Becomes: > > > > def Foo.foo(self): > > pass > > Saving about 10 keystrokes is close to trivial. The same argument can be made for @decorator syntax. And, if I understand correctly, the same argument *was* made against decorator syntax: that it was trivial, unnecessary and confusing. In a realistic example, the function definition line: def method(self, arg): and the line which actually injects it into the class: MyClass.method = method may be separated by *many lines of code*. Yes, short methods are better than long methods, but even so, sometimes you have a long method. Arguably the most important part of the definition of this method (that it belongs to MyClass) may be lost way down the end, past the implementation. Just as with decorators, the current idiom for method injecting repeats the method name too many times: # the bad old way of using decorators def function(): ... ... ... ... function = decorator(function) # the nice clean way @decorator def function(): ... ... ... ... Likewise this proposal: # the bad current way repeats the method name at least three times: def method(self): ... ... ... ... MyClass.method = method del method # and for injecting into an instance, the instance twice: def method(self): ... ... ... ... instance.method = types.MethodType(method, instance) del method # the clean proposed way: def instance.method(self): # or MyClass.method ... ... ... ... We can reasonably argue that the benefit is not worth the cost; we can even reasonably argue that we don't wish to encourage the Interceptor design pattern, method swizzling, extension methods or method injection, whatever name you want to call it. Even monkey-patching. But I don't think that we can reasonably argue that the suggested syntax isn't a clear, non-trivial win over the status quo, not unless we're also going to argue that introducing decorator syntax was a waste of time. > I am not enthusiastic about enablin the following style of > class definition. > > class Foo: "Object that implement the Foo protocol. > > def Foo.__init__(self, x): > self.x = s > > def Foo.__getitem__(self, key): > return vars(Foo)[key] Neither am I, but why would anyone write classes that way? Do you have evidence that people in the C#, VB, Objective-C, Swift or Lua communities routinely do that? > The problem with two-phase initialization is that one temporarily has a > partially initialized and likely useless object. I am not enthusiastic > about encouraging this. Indeed. But in practice, I don't believe that two-phase initialization is the usual reason for using method injection. It seems to be more commonly used for "introspection, overriding default behavior, or maybe even dynamic method loading". https://blog.newrelic.com/2014/04/16/right-way-to-swizzle/ -- Steve From mertz at gnosis.cx Mon Feb 13 21:14:34 2017 From: mertz at gnosis.cx (David Mertz) Date: Mon, 13 Feb 2017 18:14:34 -0800 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170214014814.GE29990@ando.pearwood.info> References: <20170214014814.GE29990@ando.pearwood.info> Message-ID: On Mon, Feb 13, 2017 at 5:48 PM, Steven D'Aprano wrote: > # the clean proposed way: > def instance.method(self): # or MyClass.method > ... > > But I don't think that we can reasonably argue that the suggested syntax > isn't a clear, non-trivial win over the status quo, not unless we're > also going to argue that introducing decorator syntax was a waste of > time. > I argue it's not a win specifically because we HAVE decorators already. My off-the-cuff `@attach_to(thing)` decorator factory missed some details one would want, I now recognize. We might want to attach a `.__class__` attribute, futz with `.__qualname__` and a few other things Nick points to. You've been spelling this as `@inject_method(Class)`. Apart from the name of the decorator, the idea of having a *uniform* syntax for "put the decoration" at the top because the function might be long" is an advantage. I agree that decorators are just syntax sugar, of course. But the new syntax sugar is just *too duplicative* of that we already have. Moreover, I think your spelling of what it is sugar for is slightly off. The `del method` at the end feels wrong to me. Specifically, in the example I repeated of attaching callbacks, the reason I'd want a function defined outside any particular class (or instance) scope is because I might want to use the same function as a method of various classes. An `attach_to()` decorator can leave the defined function untouched, and only muck with the version that gets attached. Hence you can even chain decorators: @attach_to(this_menu) @attach_to(that_menu) def callback(self, ...): ... Then later on: yet_another_menu.callback = callback -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Tue Feb 14 10:51:05 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Tue, 14 Feb 2017 15:51:05 +0000 Subject: [Python-ideas] Efficient debug logging Message-ID: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> A common pattern I use is to have logging calls for debug and information with my applications. The logging calls can be separately enabled and disabled. For example: debug_log_enabled = False def debugLog( msg ): If debug_log_enabled: print( ?Debug: %s? % (msg,) ) Then the caller can simple write: def main(): debugLog( ?Start of main? ) This is fine until the evaluation of the msg becomes expensive. debugLog( ?info is %r? % (expensiveFunction(),) ) What would be nice is to be able to avoid evaluation the tuple of arguments if debug is disabled as this can be expensive. I can write this: if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) But that is a more code then I would like to write. And if the debug code is a performance problem cannot be left in the production code. I could combine the boolean and the log function by using a class to tidy up the implementation. class DebugLog: def __init__( self, enabled = False ): self.enabled = enabled def __bool__( self ): return self.enabled def __call__( self, msg ): if self.enabled: print( ?Debug: %s? % (msg,) ) And call like this: dbg_log = DebugLog() If dbg_log: dbg_log( ?a debug message? ) But I?d like to only write: dbg_log( ?a debug message? ) And have the evaluation of the argument skipped unless its dbg_log is enabled. I cannot see how to do this with python as it stands. Something would have to be added to allow python to short circuit the argument tuple evaluation. Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? Thoughts? Barry From cory at lukasa.co.uk Tue Feb 14 12:03:22 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 14 Feb 2017 17:03:22 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: > On 14 Feb 2017, at 15:51, Barry Scott wrote: > And have the evaluation of the argument skipped unless its dbg_log is enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? > Is there any reason logger.isEnabledFor(level), as shown in the docs https://docs.python.org/2/library/logging.html#logging.Logger.isEnabledFor , is not sufficient for this? Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Tue Feb 14 12:39:02 2017 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Tue, 14 Feb 2017 12:39:02 -0500 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> Can you wrap the expensive functions in lambdas? And have your logger evaluate it, only if required? > debugLog( ?info is %r? % (lambda: expensiveFunction(),) ) On 2017-02-14 10:51, Barry Scott wrote: > A common pattern I use is to have logging calls for debug and information with my applications. > The logging calls can be separately enabled and disabled. > > For example: > > debug_log_enabled = False > def debugLog( msg ): > If debug_log_enabled: > print( ?Debug: %s? % (msg,) ) > > Then the caller can simple write: > > def main(): > debugLog( ?Start of main? ) > > This is fine until the evaluation of the msg becomes expensive. > > debugLog( ?info is %r? % (expensiveFunction(),) ) > > What would be nice is to be able to avoid evaluation the tuple of arguments if debug is > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) > > But that is a more code then I would like to write. And if the debug code is a performance problem cannot > be left in the production code. > > I could combine the boolean and the log function by using a class to tidy up the implementation. > > class DebugLog: > def __init__( self, enabled = False ): > self.enabled = enabled > > def __bool__( self ): > return self.enabled > > def __call__( self, msg ): > if self.enabled: print( ?Debug: %s? % (msg,) ) > > And call like this: > > dbg_log = DebugLog() > > If dbg_log: dbg_log( ?a debug message? ) > > But I?d like to only write: > > dbg_log( ?a debug message? ) > > And have the evaluation of the argument skipped unless its dbg_log is enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? > > Thoughts? > > Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From stephanh42 at gmail.com Tue Feb 14 12:48:43 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 14 Feb 2017 18:48:43 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> Message-ID: Seems slightly simpler to just make debugLog accept a callable as an alternative to a string. debugLog(lambda:( ?info is %s? % expensiveFunction()) ) Op 14 feb. 2017 18:42 schreef "Kyle Lahnakoski" : Can you wrap the expensive functions in lambdas? And have your logger evaluate it, only if required? > debugLog( ?info is %r? % (lambda: expensiveFunction(),) ) On 2017-02-14 10:51, Barry Scott wrote: > A common pattern I use is to have logging calls for debug and information with my applications. > The logging calls can be separately enabled and disabled. > > For example: > > debug_log_enabled = False > def debugLog( msg ): > If debug_log_enabled: > print( ?Debug: %s? % (msg,) ) > > Then the caller can simple write: > > def main(): > debugLog( ?Start of main? ) > > This is fine until the evaluation of the msg becomes expensive. > > debugLog( ?info is %r? % (expensiveFunction(),) ) > > What would be nice is to be able to avoid evaluation the tuple of arguments if debug is > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) > > But that is a more code then I would like to write. And if the debug code is a performance problem cannot > be left in the production code. > > I could combine the boolean and the log function by using a class to tidy up the implementation. > > class DebugLog: > def __init__( self, enabled = False ): > self.enabled = enabled > > def __bool__( self ): > return self.enabled > > def __call__( self, msg ): > if self.enabled: print( ?Debug: %s? % (msg,) ) > > And call like this: > > dbg_log = DebugLog() > > If dbg_log: dbg_log( ?a debug message? ) > > But I?d like to only write: > > dbg_log( ?a debug message? ) > > And have the evaluation of the argument skipped unless its dbg_log is enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? > > Thoughts? > > Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Feb 14 12:51:29 2017 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 14 Feb 2017 17:51:29 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: <92928575-6931-c320-c1b1-9c826bda0f2f@mrabarnett.plus.com> On 2017-02-14 15:51, Barry Scott wrote: > A common pattern I use is to have logging calls for debug and information with my applications. > The logging calls can be separately enabled and disabled. > > For example: > > debug_log_enabled = False > def debugLog( msg ): > If debug_log_enabled: > print( ?Debug: %s? % (msg,) ) > > Then the caller can simple write: > > def main(): > debugLog( ?Start of main? ) > > This is fine until the evaluation of the msg becomes expensive. > > debugLog( ?info is %r? % (expensiveFunction(),) ) > > What would be nice is to be able to avoid evaluation the tuple of arguments if debug is > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) > > But that is a more code then I would like to write. And if the debug code is a performance problem cannot > be left in the production code. > > I could combine the boolean and the log function by using a class to tidy up the implementation. > > class DebugLog: > def __init__( self, enabled = False ): > self.enabled = enabled > > def __bool__( self ): > return self.enabled > > def __call__( self, msg ): > if self.enabled: print( ?Debug: %s? % (msg,) ) > > And call like this: > > dbg_log = DebugLog() > > If dbg_log: dbg_log( ?a debug message? ) > > But I?d like to only write: > > dbg_log( ?a debug message? ) > > And have the evaluation of the argument skipped unless its dbg_log is enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? > > Thoughts? > You could let your debugging function accept a callable and use lambda to delay execution: def debugLog(msg): if debug_log_enabled: if callable(msg): msg = msg() print('Debug: %s' % (msg, )) debugLog('Start of main') debugLog(lambda: 'info is %r' % (expensiveFunction(), )) From mikhailwas at gmail.com Tue Feb 14 16:06:03 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 14 Feb 2017 22:06:03 +0100 Subject: [Python-ideas] Contraction for "for x in range()" Message-ID: I have a small syntax idea. In short, contraction of for x in range(a,b,c) : to for x in a,b,c : I really think there is something cute in it. So like a shortcut for range() which works only in for-in statement. So from syntactical POV, do you find it nice syntax? Visually it seems to me less bulky than range(). Example: for x in 0,5 : print (x) for y in 0,10,2 : print (y) for z in 0, y+8 : print (z) Which would be short for: for x in range(0,5): print (x) for y in range(0,10,2): print (y) for z in range(0, y+8): print (z) Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pyideas at gmail.com Tue Feb 14 16:09:45 2017 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Tue, 14 Feb 2017 15:09:45 -0600 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: Message-ID: On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: > I have a small syntax idea. > In short, contraction of > > for x in range(a,b,c) : > > to > > for x in a,b,c : > > I really think there is something cute in it. > So like a shortcut for range() which works only in for-in statement. > So from syntactical POV, do you find it nice syntax? > Visually it seems to me less bulky than range(). > > Example: > > for x in 0,5 : > print (x) > for y in 0,10,2 : > print (y) > for z in 0, y+8 : > print (z) This is already valid and useful syntax, and thus a non-starter. >>> for x in 0,5 : ... print (x) ... for y in 0,10,2 : ... print (y) ... for z in 0, y+8 : ... print (z) ... 0 0 0 8 10 0 18 2 0 10 5 0 0 8 10 0 18 2 0 10 -- Zach From abedillon at gmail.com Tue Feb 14 16:25:23 2017 From: abedillon at gmail.com (Abe Dillon) Date: Tue, 14 Feb 2017 15:25:23 -0600 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: Message-ID: I would be surprised if this hasn't been suggested many times before. It's similar to Matlab's syntax: for x = start:step:finish end Any such change would represent a large departure from normal python syntax for dubious gain. In general, you can put any expression after the `in` keyword so long as it evaluates to an iterable or an iterator. Your specific proposal would break some perfectly valid code. You could likely come up with some symbols that couldn't show up in existing code, but it's usually better to use words instead of obscure symbol notation to preserve the readability of Python rather than letting it devolve into a concise, yet unreadable soup of symbols like Perl. On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: > I have a small syntax idea. > In short, contraction of > > for x in range(a,b,c) : > > to > > for x in a,b,c : > > I really think there is something cute in it. > So like a shortcut for range() which works only in for-in statement. > So from syntactical POV, do you find it nice syntax? > Visually it seems to me less bulky than range(). > > Example: > > for x in 0,5 : > print (x) > for y in 0,10,2 : > print (y) > for z in 0, y+8 : > print (z) > > Which would be short for: > > for x in range(0,5): > print (x) > for y in range(0,10,2): > print (y) > for z in range(0, y+8): > print (z) > > > Mikhail > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Feb 14 16:34:09 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 14 Feb 2017 16:34:09 -0500 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator References: <20170214014814.GE29990@ando.pearwood.info> Message-ID: <20170214163409.08bbecff@subdivisions.wooz.org> On Feb 14, 2017, at 12:48 PM, Steven D'Aprano wrote: >On Fri, Feb 10, 2017 at 09:05:49PM -0500, Terry Reedy wrote: >> Saving about 10 keystrokes is close to trivial. > >The same argument can be made for @decorator syntax. > >And, if I understand correctly, the same argument *was* made against >decorator syntax: that it was trivial, unnecessary and confusing. Well, not exactly. Remember that the semantics, and common decorators like property, existed well before the decorator syntax was added. We had a lot of experience writing post-definition "decorators", which taught us that the behavior was useful but the syntax was painful. And adding the syntax made a huge improvement in readability. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From python at mrabarnett.plus.com Tue Feb 14 16:41:02 2017 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 14 Feb 2017 21:41:02 +0000 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: Message-ID: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> On 2017-02-14 21:09, Zachary Ware wrote: > On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: >> I have a small syntax idea. >> In short, contraction of >> >> for x in range(a,b,c) : >> >> to >> >> for x in a,b,c : >> >> I really think there is something cute in it. >> So like a shortcut for range() which works only in for-in statement. >> So from syntactical POV, do you find it nice syntax? >> Visually it seems to me less bulky than range(). >> >> Example: >> >> for x in 0,5 : >> print (x) >> for y in 0,10,2 : >> print (y) >> for z in 0, y+8 : >> print (z) > > This is already valid and useful syntax, and thus a non-starter. > > >>> for x in 0,5 : > ... print (x) > ... for y in 0,10,2 : > ... print (y) > ... for z in 0, y+8 : > ... print (z) > ... > 0 > 0 > 0 > 8 > 10 > 0 > 18 > 2 > 0 > 10 > 5 > 0 > 0 > 8 > 10 > 0 > 18 > 2 > 0 > 10 > The closest you could get without breaking existing code is [a:b:c]: for x in [0:5]: print(x) for y in [0:10:2]: print(y) for z in [0:y+8]: print(z) What's more, that could be valid outside the 'for' loop too. From toddrjen at gmail.com Tue Feb 14 17:03:37 2017 From: toddrjen at gmail.com (Todd) Date: Tue, 14 Feb 2017 17:03:37 -0500 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> References: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> Message-ID: On Tue, Feb 14, 2017 at 4:41 PM, MRAB wrote: > On 2017-02-14 21:09, Zachary Ware wrote: > >> On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: >> >>> I have a small syntax idea. >>> In short, contraction of >>> >>> for x in range(a,b,c) : >>> >>> to >>> >>> for x in a,b,c : >>> >>> I really think there is something cute in it. >>> So like a shortcut for range() which works only in for-in statement. >>> So from syntactical POV, do you find it nice syntax? >>> Visually it seems to me less bulky than range(). >>> >>> Example: >>> >>> for x in 0,5 : >>> print (x) >>> for y in 0,10,2 : >>> print (y) >>> for z in 0, y+8 : >>> print (z) >>> >> >> This is already valid and useful syntax, and thus a non-starter. >> >> >>> for x in 0,5 : >> ... print (x) >> ... for y in 0,10,2 : >> ... print (y) >> ... for z in 0, y+8 : >> ... print (z) >> ... >> 0 >> 0 >> 0 >> 8 >> 10 >> 0 >> 18 >> 2 >> 0 >> 10 >> 5 >> 0 >> 0 >> 8 >> 10 >> 0 >> 18 >> 2 >> 0 >> 10 >> >> The closest you could get without breaking existing code is [a:b:c]: > > for x in [0:5]: > print(x) > for y in [0:10:2]: > print(y) > for z in [0:y+8]: > print(z) > > What's more, that could be valid outside the 'for' loop too. > > Guido has already rejected this syntax and several others. -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Feb 14 17:19:57 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 14 Feb 2017 16:19:57 -0600 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> Message-ID: for i in range(...) is *sometimes* indicative of code smell, especially when then doing x[i], though it has its uses. I've never had a need to shorten a for...range line though. Other than it being "cute", do you have an example where it's definitively better? On Tue, Feb 14, 2017 at 4:03 PM, Todd wrote: > On Tue, Feb 14, 2017 at 4:41 PM, MRAB wrote: > >> On 2017-02-14 21:09, Zachary Ware wrote: >> >>> On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: >>> >>>> I have a small syntax idea. >>>> In short, contraction of >>>> >>>> for x in range(a,b,c) : >>>> >>>> to >>>> >>>> for x in a,b,c : >>>> >>>> I really think there is something cute in it. >>>> So like a shortcut for range() which works only in for-in statement. >>>> So from syntactical POV, do you find it nice syntax? >>>> Visually it seems to me less bulky than range(). >>>> >>>> Example: >>>> >>>> for x in 0,5 : >>>> print (x) >>>> for y in 0,10,2 : >>>> print (y) >>>> for z in 0, y+8 : >>>> print (z) >>>> >>> >>> This is already valid and useful syntax, and thus a non-starter. >>> >>> >>> for x in 0,5 : >>> ... print (x) >>> ... for y in 0,10,2 : >>> ... print (y) >>> ... for z in 0, y+8 : >>> ... print (z) >>> ... >>> 0 >>> 0 >>> 0 >>> 8 >>> 10 >>> 0 >>> 18 >>> 2 >>> 0 >>> 10 >>> 5 >>> 0 >>> 0 >>> 8 >>> 10 >>> 0 >>> 18 >>> 2 >>> 0 >>> 10 >>> >>> The closest you could get without breaking existing code is [a:b:c]: >> >> for x in [0:5]: >> print(x) >> for y in [0:10:2]: >> print(y) >> for z in [0:y+8]: >> print(z) >> >> What's more, that could be valid outside the 'for' loop too. >> >> > Guido has already rejected this syntax and several others. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Tue Feb 14 17:20:24 2017 From: abedillon at gmail.com (Abe Dillon) Date: Tue, 14 Feb 2017 16:20:24 -0600 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <92928575-6931-c320-c1b1-9c826bda0f2f@mrabarnett.plus.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <92928575-6931-c320-c1b1-9c826bda0f2f@mrabarnett.plus.com> Message-ID: There are several mechanisms in the logging module to handle this use-case. First, note that logging functions can take multiple arguments ( https://docs.python.org/3/library/logging.html?highlight=logging#logging.Logger.debug ): >>> import logging >>> msg = "hello, %s! %s to %s you!" >>> args = ("world", "pleased", "meet") >>> logging.warning(msg, *args) WARNING:root:hello, world! pleased to meet you! None of the string formatting will take place if the log call is below the level of the logger. It might be worth while to add a class that stores an unevaluated function, then evaluates the function when __str__ or __repr__ is called: >>> class MessageFunction: ... def __init__(self, func, *args, **kwargs): ... self.func = func ... self.args = args ... self.kwags = kwargs ... def __str__(self): ... return str(self.func(*self.args, **self.kwargs)) ... def __repr__(self): ... return repr(self.func(*self.args, **self.kwargs)) >>> import logging >>> logging.debug("result = %s", MessageFunction(expensive_func, *args, **kwargs)) You can also add Filters to your logger: https://docs.python.org/3/library/logging.html?highlight=logging#logging.Filter On Tue, Feb 14, 2017 at 11:51 AM, MRAB wrote: > On 2017-02-14 15:51, Barry Scott wrote: > >> A common pattern I use is to have logging calls for debug and information >> with my applications. >> The logging calls can be separately enabled and disabled. >> >> For example: >> >> debug_log_enabled = False >> def debugLog( msg ): >> If debug_log_enabled: >> print( ?Debug: %s? % (msg,) ) >> >> Then the caller can simple write: >> >> def main(): >> debugLog( ?Start of main? ) >> >> This is fine until the evaluation of the msg becomes expensive. >> >> debugLog( ?info is %r? % (expensiveFunction(),) ) >> >> What would be nice is to be able to avoid evaluation the tuple of >> arguments if debug is >> disabled as this can be expensive. I can write this: >> >> if debug_log_enabled: debugLog( ?info is %r? % >> (expensiveFunction(),) ) >> >> But that is a more code then I would like to write. And if the debug code >> is a performance problem cannot >> be left in the production code. >> >> I could combine the boolean and the log function by using a class to tidy >> up the implementation. >> >> class DebugLog: >> def __init__( self, enabled = False ): >> self.enabled = enabled >> >> def __bool__( self ): >> return self.enabled >> >> def __call__( self, msg ): >> if self.enabled: print( ?Debug: %s? % (msg,) ) >> >> And call like this: >> >> dbg_log = DebugLog() >> >> If dbg_log: dbg_log( ?a debug message? ) >> >> But I?d like to only write: >> >> dbg_log( ?a debug message? ) >> >> And have the evaluation of the argument skipped unless its dbg_log is >> enabled. >> >> I cannot see how to do this with python as it stands. >> >> Something would have to be added to allow python to short circuit the >> argument tuple evaluation. >> >> Maybe python can check for a special dunder on the class that know how to >> do this idiom, __if_true_call__? >> >> Thoughts? >> >> You could let your debugging function accept a callable and use lambda to > delay execution: > > def debugLog(msg): > if debug_log_enabled: > if callable(msg): > msg = msg() > > print('Debug: %s' % (msg, )) > > > debugLog('Start of main') > > debugLog(lambda: 'info is %r' % (expensiveFunction(), )) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Tue Feb 14 17:00:31 2017 From: barry at barrys-emacs.org (Barry) Date: Tue, 14 Feb 2017 22:00:31 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> Message-ID: <7F91D4A9-7EA2-4762-9861-7E9C72030BF7@barrys-emacs.org> > On 14 Feb 2017, at 17:39, Kyle Lahnakoski wrote: > > > Can you wrap the expensive functions in lambdas? And have your logger > evaluate it, only if required? > >> debugLog( ?info is %r? % (lambda: expensiveFunction(),) ) Interesting idea. I will bench mark and see what the cost of the lamba version is. It would still be nice to have the code look cleaner without the lambda. Barry > > >> On 2017-02-14 10:51, Barry Scott wrote: >> A common pattern I use is to have logging calls for debug and information with my applications. >> The logging calls can be separately enabled and disabled. >> >> For example: >> >> debug_log_enabled = False >> def debugLog( msg ): >> If debug_log_enabled: >> print( ?Debug: %s? % (msg,) ) >> >> Then the caller can simple write: >> >> def main(): >> debugLog( ?Start of main? ) >> >> This is fine until the evaluation of the msg becomes expensive. >> >> debugLog( ?info is %r? % (expensiveFunction(),) ) >> >> What would be nice is to be able to avoid evaluation the tuple of arguments if debug is >> disabled as this can be expensive. I can write this: >> >> if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) >> >> But that is a more code then I would like to write. And if the debug code is a performance problem cannot >> be left in the production code. >> >> I could combine the boolean and the log function by using a class to tidy up the implementation. >> >> class DebugLog: >> def __init__( self, enabled = False ): >> self.enabled = enabled >> >> def __bool__( self ): >> return self.enabled >> >> def __call__( self, msg ): >> if self.enabled: print( ?Debug: %s? % (msg,) ) >> >> And call like this: >> >> dbg_log = DebugLog() >> >> If dbg_log: dbg_log( ?a debug message? ) >> >> But I?d like to only write: >> >> dbg_log( ?a debug message? ) >> >> And have the evaluation of the argument skipped unless its dbg_log is enabled. >> >> I cannot see how to do this with python as it stands. >> >> Something would have to be added to allow python to short circuit the argument tuple evaluation. >> >> Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? >> >> Thoughts? >> >> Barry >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From barry at barrys-emacs.org Tue Feb 14 16:55:27 2017 From: barry at barrys-emacs.org (Barry) Date: Tue, 14 Feb 2017 21:55:27 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: > On 14 Feb 2017, at 17:03, Cory Benfield wrote: > > >> On 14 Feb 2017, at 15:51, Barry Scott wrote: >> And have the evaluation of the argument skipped unless its dbg_log is enabled. >> >> I cannot see how to do this with python as it stands. >> >> Something would have to be added to allow python to short circuit the argument tuple evaluation. >> >> Maybe python can check for a special dunder on the class that know how to do this idiom, __if_true_call__? >> > > Is there any reason logger.isEnabledFor(level), as shown in the docs https://docs.python.org/2/library/logging.html#logging.Logger.isEnabledFor, is not sufficient for this? The point is that the cost of creating the msg argument can be very high. At the point that logging decides to skip output it is to late to save the cost of creating the arg tuple. Not that it is relivent for this idea bit logging's levels are too course for logging in complex applications. The app I am working on at the moment has 20 seperate debug categories that are independently enabled. Barry > Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Tue Feb 14 18:22:27 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 15 Feb 2017 00:22:27 +0100 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> References: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> Message-ID: On 14 February 2017 at 22:41, MRAB wrote: > On 2017-02-14 21:09, Zachary Ware wrote: > >> On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: >> >>> I have a small syntax idea. >>> In short, contraction of >>> >>> for x in range(a,b,c) : >>> >>> to >>> >>> for x in a,b,c : >>> >>> I really think there is something cute in it. >>> So like a shortcut for range() which works only in for-in statement. >>> So from syntactical POV, do you find it nice syntax? >>> Visually it seems to me less bulky than range(). >>> >>> Example: >>> >>> for x in 0,5 : >>> print (x) >>> for y in 0,10,2 : >>> print (y) >>> for z in 0, y+8 : >>> print (z) >>> >> >> This is already valid and useful syntax, and thus a non-starter. >> >> >> >> The closest you could get without breaking existing code is [a:b:c]: > > for x in [0:5]: > print(x) > for y in [0:10:2]: > print(y) > for z in [0:y+8]: > print(z) > > What's more, that could be valid outside the 'for' loop too. > > This looks really good, at first glance I would even expect this will work, comes so common from lists and numpy index ranges. -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Feb 14 18:41:34 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 14 Feb 2017 17:41:34 -0600 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> Message-ID: Make some shim object that you can index into to get that functionality, could even call it Z (for the set of all integers). Short, and requires no new syntax. class IndexableRange: def __getitem__(self, item): if isinstance(item, slice): start = item.start if item.start is not None else 0 step = item.step if item.step is not None else 1 if item.stop is None: return itertools.count(start, step) else: return range(start, item.stop, step) else: return item Z = IndexableRange() for y in Z[0:10:2]: print(y) On Tue, Feb 14, 2017 at 5:22 PM, Mikhail V wrote: > > On 14 February 2017 at 22:41, MRAB wrote: > >> On 2017-02-14 21:09, Zachary Ware wrote: >> >>> On Tue, Feb 14, 2017 at 3:06 PM, Mikhail V wrote: >>> >>>> I have a small syntax idea. >>>> In short, contraction of >>>> >>>> for x in range(a,b,c) : >>>> >>>> to >>>> >>>> for x in a,b,c : >>>> >>>> I really think there is something cute in it. >>>> So like a shortcut for range() which works only in for-in statement. >>>> So from syntactical POV, do you find it nice syntax? >>>> Visually it seems to me less bulky than range(). >>>> >>>> Example: >>>> >>>> for x in 0,5 : >>>> print (x) >>>> for y in 0,10,2 : >>>> print (y) >>>> for z in 0, y+8 : >>>> print (z) >>>> >>> >>> This is already valid and useful syntax, and thus a non-starter. >>> >>> >>> >>> The closest you could get without breaking existing code is [a:b:c]: >> >> for x in [0:5]: >> print(x) >> for y in [0:10:2]: >> print(y) >> for z in [0:y+8]: >> print(z) >> >> What's more, that could be valid outside the 'for' loop too. >> >> > This looks really good, at first glance I would even expect this will > work, > comes so common from lists and numpy index ranges. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Tue Feb 14 19:20:37 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Wed, 15 Feb 2017 01:20:37 +0100 Subject: [Python-ideas] Contraction for "for x in range()" In-Reply-To: References: <802e9c27-0d98-bf74-4245-c727b88827ba@mrabarnett.plus.com> Message-ID: On 15 February 2017 at 00:41, Nick Timkovich wrote: > Make some shim object that you can index into to get that functionality, > could even call it Z (for the set of all integers). Short, and requires no > new syntax. > > class IndexableRange: > def __getitem__(self, item): > if isinstance(item, slice): > start = item.start if item.start is not None else 0 > step = item.step if item.step is not None else 1 > if item.stop is None: > return itertools.count(start, step) > else: > return range(start, item.stop, step) > else: > return item > > Z = IndexableRange() > > for y in Z[0:10:2]: > print(y) > > > l can, also just make function r(a,b,c) : return range(a,b,c) for example, will look similar. Initially I was by the idea to remove brackets from for statement. Now after looking more at examples I realize what the real issue with the look is. Namely it is the word "in" itself. It is simply too short and that makes a lot of space holes in the lines. And letters 'i' and 'n' sort of suck from readability POV. E.g. compare: for x in 1,10,2: and for x over 1,10,2 : So the latter actually would make it look nicer. But that would probably be even less probable to be implemented. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Tue Feb 14 19:51:08 2017 From: abedillon at gmail.com (Abe Dillon) Date: Tue, 14 Feb 2017 18:51:08 -0600 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: The point is that the cost of creating the msg argument can be very high. At the point that logging decides to skip output it is to late to save the cost of creating the arg tuple. This sounds like an optimization that's sufficiently rare and complex to warrant a custom fix or a 3rd party library. Not that it is relivent for this idea bit logging's levels are too course for logging in complete applications. The app I am working on at the moment has 20 seperate debug categories that are independently enabled. Holy balls! That sounds like a tortured use of log levels! On Tue, Feb 14, 2017 at 3:55 PM, Barry wrote: > > On 14 Feb 2017, at 17:03, Cory Benfield wrote: > > > On 14 Feb 2017, at 15:51, Barry Scott wrote: > And have the evaluation of the argument skipped unless its dbg_log is > enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the > argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to > do this idiom, __if_true_call__? > > > Is there any reason logger.isEnabledFor(level), as shown in the docs > https://docs.python.org/2/library/logging.html#logging.Logger.isEnabledFor, > is not sufficient for this? > > > The point is that the cost of creating the msg argument can be very high. > > At the point that logging decides to skip output it is to late to save the > cost of creating the arg tuple. > > Not that it is relivent for this idea bit logging's levels are too course > for logging in complex applications. The app I am working on at the moment > has 20 seperate debug categories > that are independently enabled. > > Barry > > Cory > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Tue Feb 14 21:39:27 2017 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Tue, 14 Feb 2017 21:39:27 -0500 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: <153b54dd-70bb-0a5f-c152-1e9d10bb9fbb@mozilla.com> On 2017-02-14 19:51, Abe Dillon wrote: > The point is that the cost of creating the msg argument can be very > high. > > At the point that logging decides to skip output it is to late to > save the cost of creating the arg tuple. > > This sounds like an optimization that's sufficiently rare and complex > to warrant a custom fix or a 3rd party library. > > Not that it is relivent for this idea bit logging's levels are too > course for logging in complete > applications. The app I am working on at the moment has 20 > seperate debug categories > that are independently enabled. > > Holy balls! That sounds like a tortured use of log levels! That's a funny expression. :) I have not got to the point of 20 debug categories in a single file, but I easily have 20+ debug categories in an application. I previously suggested wrapping the expensiveFunction in a lambda, but that is not what I do. I prefer using explicit logging switches: ____if DEBUG_FEATURE_A: ________debugLog(msg, expensiveFunction() ) ____if DEBUG_FEATURE_B: ________debugLog(msg2, expensiveFunction2() ) The benefit is clarity at the expense of more lines, while still avoiding the expensiveFunction() when I can. Using switches result in a smaller log file, because logging is focused on the feature, plus, these switches can still be dynamically set from outside the module. Log "levels" never made sense to me; how can a single dimension be useful substitute for a number of binary switches? With log "levels", you either don't have enough logging, or you drown in too much logging (or you manage a number of loggers, which is worse than logging switches). But to stick to the theme of language features: I believe there is a previous thread that touches on a solution to the original posters problem: "Alternative to PEP 532: delayed evaluation of expressions" -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Feb 15 00:28:15 2017 From: random832 at fastmail.com (Random832) Date: Wed, 15 Feb 2017 00:28:15 -0500 Subject: [Python-ideas] Fwd: Define a method or function attributeoutsideof a class with the dot operator In-Reply-To: <20170213025550.GO4796@ando.pearwood.info> References: <20170212043746.GL4796@ando.pearwood.info> <20170213025550.GO4796@ando.pearwood.info> Message-ID: <1487136495.1415077.881460440.7B9B5104@webmail.messagingengine.com> On Sun, Feb 12, 2017, at 21:55, Steven D'Aprano wrote: > But honestly, no. This is not going to happen. .Net VB and C# have > something like this, as does Lua, and people still write classes the > ordinary way 99.99% of the time. The VB/C# thing you are referring to is, I assume, extension methods. But they're really very different when you look at it. Extension methods are only used when the namespace containing them has been imported, and are based on the static type of the object they are being called on. They also have no access to the object's private members. Python doesn't have static types and doesn't have private members, and using this would make a real modification to the type the method is being added to rather than relying on namespaces being imported, so there would be fewer barriers to "use this for everything" than "use extension methods for everything in C#". From barry at barrys-emacs.org Wed Feb 15 05:45:47 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 15 Feb 2017 10:45:47 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <153b54dd-70bb-0a5f-c152-1e9d10bb9fbb@mozilla.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <153b54dd-70bb-0a5f-c152-1e9d10bb9fbb@mozilla.com> Message-ID: <6DF06674-FED7-4D7B-B3C5-23FA76A160D8@barrys-emacs.org> > On 15 Feb 2017, at 02:39, Kyle Lahnakoski wrote: > > > On 2017-02-14 19:51, Abe Dillon wrote: >> The point is that the cost of creating the msg argument can be very high. >> >> At the point that logging decides to skip output it is to late to save the cost of creating the arg tuple. >> >> This sounds like an optimization that's sufficiently rare and complex to warrant a custom fix or a 3rd party library. >> >> Not that it is relivent for this idea bit logging's levels are too course for logging in complete >> applications. The app I am working on at the moment has 20 seperate debug categories >> that are independently enabled. >> >> Holy balls! That sounds like a tortured use of log levels! > > That's a funny expression. :) > > I have not got to the point of 20 debug categories in a single file, but I easily have 20+ debug categories in an application. I previously suggested wrapping the expensiveFunction in a lambda, but that is not what I do. I prefer using explicit logging switches: > > ____if DEBUG_FEATURE_A: > ________debugLog(msg, expensiveFunction() ) > ____if DEBUG_FEATURE_B: > ________debugLog(msg2, expensiveFunction2() ) > > The benefit is clarity at the expense of more lines, while still avoiding the expensiveFunction() when I can. Using switches result in a smaller log file, because logging is focused on the feature, plus, these switches can still be dynamically set from outside the module. I would like to avoid the if so that the code is more compact if possible. > > Log "levels" never made sense to me; how can a single dimension be useful substitute for a number of binary switches? With log "levels", you either don't have enough logging, or you drown in too much logging (or you manage a number of loggers, which is worse than logging switches). Well said that is indeed the issue with the logger module as is. > > But to stick to the theme of language features: I believe there is a previous thread that touches on a solution to the original posters problem: "Alternative to PEP 532: delayed evaluation of expressions It is not clear to me from one reading is PEP 532 will allow a class with __then__ and __else to do what I want. Would it be something like this: if not debugLog and (expensive(), args()) Does debugLog get passed the tuple in the __then__? Barry > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Feb 15 06:07:19 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Feb 2017 22:07:19 +1100 Subject: [Python-ideas] Python package index: query and rating In-Reply-To: References: Message-ID: <20170215110719.GA5689@ando.pearwood.info> Hello George, and welcome! On Mon, Feb 13, 2017 at 01:23:22PM +0100, George Fischhof wrote: > Hi There, > > I several times found that it is quite difficult to find an appropriate > library in package index. > > I have two ides to improve the package index: Thank you for your ideas, but I don't think this is the right place for suggesting improvements to PyPI. You could try looking through the list of other Python mailing lists: https://mail.python.org/mailman/listinfo Or perhaps somebody else will suggest a better place to raise these issues. Regards, Steve From steve at pearwood.info Wed Feb 15 06:18:31 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Feb 2017 22:18:31 +1100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: <20170215111831.GB5689@ando.pearwood.info> On Tue, Feb 14, 2017 at 03:51:05PM +0000, Barry Scott wrote: > A common pattern I use is to have logging calls for debug and information with my applications. > The logging calls can be separately enabled and disabled. [...] > What would be nice is to be able to avoid evaluation the tuple of arguments if debug is > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) > > But that is a more code then I would like to write. And if the debug code is a performance problem cannot > be left in the production code. Occasionally people (often me... :-) will suggest that it would be good if Python had some sort of lightweight "thunk" for delayed evaluation. I don't have good syntax for that, but let's pretend it was "<# ... #>" just for the sake of having a way to talk about it. We can think of a thunk as being something like syntax for lazy evaluation, so you could write: debugLog( ?info is %r? % (<#expensiveFunction()#>, ) ) and the expensive function call would only actually be made if it was required. Python doesn't have thunks, but there is a relatively heavyweight solution for delayed evaluation: wrap the code in a function. debugLog( ?info is %r?, lambda: expensiveFunction() ) and then adjust debugLog so that if the argument is a function, it will call the function only when needed: def debugLog(message, value): if debug_log_enabled: if isinstance(value, types.FunctionType): value = value() log(message % value) > But I?d like to only write: > > dbg_log( ?a debug message? ) > > And have the evaluation of the argument skipped unless its dbg_log is enabled. Indeed. That's what thunks could give us, if only we had a clear picture of how they would work, when they would be evaluated, and what syntax they should use. In the meantime, there are other solutions, not as (hypothetically) lightweight, but they'll work *now* rather than needing to wait for Python 3.7 or 3.9 or 4.5 :-) But if you have some good, concrete suggestions for how to do thunks in Python, please tell us! -- Steve From barry at barrys-emacs.org Wed Feb 15 06:32:44 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 15 Feb 2017 11:32:44 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <7F91D4A9-7EA2-4762-9861-7E9C72030BF7@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <33b8d1f0-aa94-de1f-d6c4-8b9decd41f08@mozilla.com> <7F91D4A9-7EA2-4762-9861-7E9C72030BF7@barrys-emacs.org> Message-ID: <13772878.SWZjqmpXdB@varric.chelsea.private> On Tuesday, 14 February 2017 22:00:31 GMT Barry wrote: > > On 14 Feb 2017, at 17:39, Kyle Lahnakoski wrote: > > > > > > Can you wrap the expensive functions in lambdas? And have your logger > > evaluate it, only if required? > > > >> debugLog( ?info is %r? % (lambda: expensiveFunction(),) ) > > Interesting idea. I will bench mark and see what the cost of the lamba > version is. It would still be nice to have the code look cleaner without > the lambda. > Benchmarks show that the lambda idea works well. Its not that much more expensive the the if enabled version. Barry From steve at pearwood.info Wed Feb 15 06:32:48 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Feb 2017 22:32:48 +1100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: <20170214163409.08bbecff@subdivisions.wooz.org> References: <20170214014814.GE29990@ando.pearwood.info> <20170214163409.08bbecff@subdivisions.wooz.org> Message-ID: <20170215113248.GC5689@ando.pearwood.info> On Tue, Feb 14, 2017 at 04:34:09PM -0500, Barry Warsaw wrote: > On Feb 14, 2017, at 12:48 PM, Steven D'Aprano wrote: > > >On Fri, Feb 10, 2017 at 09:05:49PM -0500, Terry Reedy wrote: > >> Saving about 10 keystrokes is close to trivial. > > > >The same argument can be made for @decorator syntax. > > > >And, if I understand correctly, the same argument *was* made against > >decorator syntax: that it was trivial, unnecessary and confusing. > > Well, not exactly. Remember that the semantics, and common decorators like > property, existed well before the decorator syntax was added. We had a lot of > experience writing post-definition "decorators", which taught us that the > behavior was useful but the syntax was painful. And adding the syntax made a > huge improvement in readability. Your memory of the discussion may be more accurate than mine, but the PEP does suggest that general unfamiliarity with the concept was still a large issue: https://www.python.org/dev/peps/pep-0318/#why-is-this-so-hard -- Steve From rosuav at gmail.com Wed Feb 15 06:33:42 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 15 Feb 2017 22:33:42 +1100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <20170215111831.GB5689@ando.pearwood.info> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> Message-ID: On Wed, Feb 15, 2017 at 10:18 PM, Steven D'Aprano wrote: > Python doesn't have thunks, but there is a relatively heavyweight > solution for delayed evaluation: wrap the code in a function. > > debugLog( ?info is %r?, lambda: expensiveFunction() ) > > > and then adjust debugLog so that if the argument is a function, it will > call the function only when needed: > > def debugLog(message, value): > if debug_log_enabled: > if isinstance(value, types.FunctionType): > value = value() > log(message % value) Or use the function as the __repr__ of some object, which comes to the same thing without requiring special code inside debugLog. ChrisA From p.f.moore at gmail.com Wed Feb 15 06:35:16 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Feb 2017 11:35:16 +0000 Subject: [Python-ideas] Python package index: query and rating In-Reply-To: <20170215110719.GA5689@ando.pearwood.info> References: <20170215110719.GA5689@ando.pearwood.info> Message-ID: On 15 February 2017 at 11:07, Steven D'Aprano wrote: >> I have two ides to improve the package index: > > Thank you for your ideas, but I don't think this is the right place for > suggesting improvements to PyPI. You could try looking through the list > of other Python mailing lists: > > https://mail.python.org/mailman/listinfo > > Or perhaps somebody else will suggest a better place to raise these > issues. Hi George, The PyPI codebase is pretty old and difficult to maintain, so there is currently a project under way to replace it with a new implementation. The project is hosted at https://github.com/pypa/warehouse and if you're interested in getting involved that's probably the best starting point for you. With regard to your two specific points: > 1. Query like in a webshop: checkboxes should use to select from attributes The PyPI search interface is known to be pretty limited. I don't know what changes have been put into the warehouse interface so far, but I'd suggest checking out the state of their search, and if you still feel there's scope for improvements, I'm sure they would be interested in suggestions/contributions. > 2. It would would be good to implement some user rating mechanism like in the webshops or software stores. This has been debated, both here and (I believe) on other lists in the past. The general consensus is that it's not immediately obvious that this would be beneficial. I'd suggest you do some searches to dig up the history of this suggestion (sorry, I don't have any direct links) and see if you feel there's something that has been missed - if so, again the best place to propose this would be with the Warehouse guys. Paul From steve at pearwood.info Wed Feb 15 06:43:12 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Feb 2017 22:43:12 +1100 Subject: [Python-ideas] Define a method or function attribute outside of a class with the dot operator In-Reply-To: References: <20170214014814.GE29990@ando.pearwood.info> Message-ID: <20170215114312.GD5689@ando.pearwood.info> On Mon, Feb 13, 2017 at 06:14:34PM -0800, David Mertz wrote: > On Mon, Feb 13, 2017 at 5:48 PM, Steven D'Aprano > wrote: > > > # the clean proposed way: > > def instance.method(self): # or MyClass.method > > ... > > > > But I don't think that we can reasonably argue that the suggested syntax > > isn't a clear, non-trivial win over the status quo, not unless we're > > also going to argue that introducing decorator syntax was a waste of > > time. > > > > I argue it's not a win specifically because we HAVE decorators already. I wouldn't word it quite like that, but I'll certainly agree that a decorator may be able to (almost) solve the method-injection use-case. Being able to write: @inject(Class) def method ... is almost as nice as def Class.method ... The one thing a decorator can't do is clean up after itself and avoid leaving an unneeded and unnecessary 'method' name in the current namespace. [...] > Moreover, I think your spelling of what it is sugar for is slightly off. > The `del method` at the end feels wrong to me. Specifically, in the > example I repeated of attaching callbacks, the reason I'd want a function > defined outside any particular class (or instance) scope is because I might > want to use the same function as a method of various classes. The existing syntax won't be going away, so anything you can do now, you will still be able to do :-) My gut feeling is that `def Class.method` should avoid polluting the current namespace. If you need access to the method again, you can always grab it from `Class.method`. -- Steve From barry at barrys-emacs.org Wed Feb 15 06:47:11 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 15 Feb 2017 11:47:11 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> Message-ID: <1964845.4ySRCeR4W4@varric.chelsea.private> On Wednesday, 15 February 2017 22:33:42 GMT Chris Angelico wrote: > On Wed, Feb 15, 2017 at 10:18 PM, Steven D'Aprano wrote: > > Python doesn't have thunks, but there is a relatively heavyweight > > solution for delayed evaluation: wrap the code in a function. > > > > debugLog( ?info is %r?, lambda: expensiveFunction() ) > > > > > > and then adjust debugLog so that if the argument is a function, it will > > call the function only when needed: > > > > def debugLog(message, value): > > if debug_log_enabled: > > if isinstance(value, types.FunctionType): > > value = value() > > > > log(message % value) > > Or use the function as the __repr__ of some object, which comes to the > same thing without requiring special code inside debugLog. __repr__ is interesting however. Typically I describe in a string why the value is being logged. That means that the object is always a string. I cannot recall using debugLog( obj ) in production. dlog('This is the state of obj at the start of event processing: %r' % (obj,)) Barry From rosuav at gmail.com Wed Feb 15 06:48:59 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 15 Feb 2017 22:48:59 +1100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <1964845.4ySRCeR4W4@varric.chelsea.private> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1964845.4ySRCeR4W4@varric.chelsea.private> Message-ID: On Wed, Feb 15, 2017 at 10:47 PM, Barry Scott wrote: > __repr__ is interesting however. > > Typically I describe in a string why the value is being logged. That means > that the object is always a string. I cannot recall using debugLog( obj ) > in production. > > dlog('This is the state of obj at the start of event processing: %r' % (obj,)) Right. I would have no hesitation whatsoever in dumping out a crucial state object, even if it means writing a custom (and expensive) repr function for it. It's clean, simple, and immensely helpful in debugging. ChrisA From steve at pearwood.info Wed Feb 15 07:10:09 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Feb 2017 23:10:09 +1100 Subject: [Python-ideas] Using Python for end user applications In-Reply-To: References: Message-ID: <20170215121009.GE5689@ando.pearwood.info> On Tue, Feb 07, 2017 at 12:05:49AM -0600, C Anthony Risinger wrote: > How do I redistribute and successfully install Python, dependencies, and an > application with the least possible steps for the end user? For any > platform or persona? For Linux, Unix and Mac, where you can generally expect Python is already installed (although perhaps not the most recent version you would like), I think the best way to distribute Python apps is as a zip file: https://www.python.org/dev/peps/pep-0441/ For Windows, it's not *quite* so simple, as the end-user needs to install Python themselves. > I prefer dynamic applications redistribute their runtime, I don't. That means that when there's a serious security bug in Python, instead of patching it once (the system Python), I have to patch two, ten, a hundred applications, some of which I don't even know are written in Python, some of which may never have a patch released, or if they do, it may be weeks or months later. > Containers are have greatly impacted how people think about distribution. > There is a trend to use fast/small containers as CLI tools. It's the future! :-) https://circleci.com/blog/its-the-future/ On the other hand: https://circleci.com/blog/it-really-is-the-future/ -- Steve From barry at barrys-emacs.org Wed Feb 15 06:55:24 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 15 Feb 2017 11:55:24 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <20170215111831.GB5689@ando.pearwood.info> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> Message-ID: <1756620.cYo0RJySdd@varric.chelsea.private> On Wednesday, 15 February 2017 22:18:31 GMT Steven D'Aprano wrote: > On Tue, Feb 14, 2017 at 03:51:05PM +0000, Barry Scott wrote: > > A common pattern I use is to have logging calls for debug and information > > with my applications. The logging calls can be separately enabled and > > disabled. > > [...] > > > What would be nice is to be able to avoid evaluation the tuple of > > arguments if debug is> > > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % (expensiveFunction(),) ) > > > > But that is a more code then I would like to write. And if the debug code > > is a performance problem cannot be left in the production code. > > Occasionally people (often me... :-) will suggest that it would be good > if Python had some sort of lightweight "thunk" for delayed evaluation. I > don't have good syntax for that, but let's pretend it was "<# ... #>" > just for the sake of having a way to talk about it. > > We can think of a thunk as being something like syntax for lazy > evaluation, so you could write: > > debugLog( ?info is %r? % (<#expensiveFunction()#>, ) ) > > and the expensive function call would only actually be made if it was > required. The lazy eval sound like a generator. > > Python doesn't have thunks, but there is a relatively heavyweight > solution for delayed evaluation: wrap the code in a function. > > debugLog( ?info is %r?, lambda: expensiveFunction() ) > > > and then adjust debugLog so that if the argument is a function, it will > call the function only when needed: > > def debugLog(message, value): > if debug_log_enabled: > if isinstance(value, types.FunctionType): > value = value() > log(message % value) And reply suggested this and it works well for the obviously expensive calls in benchmarks. > > But I?d like to only write: > > dbg_log( ?a debug message? ) > > > > And have the evaluation of the argument skipped unless its dbg_log is > > enabled. > Indeed. That's what thunks could give us, if only we had a clear picture > of how they would work, when they would be evaluated, and what syntax > they should use. > > In the meantime, there are other solutions, not as (hypothetically) > lightweight, but they'll work *now* rather than needing to wait for > Python 3.7 or 3.9 or 4.5 :-) At the moment I just put up with the cost of debug calls. The typical of debug messages did come up in a PYCON UK talk and the suggestion was to remove the debug logging after the code works. I prefer to leave the debug logging in production code. > But if you have some good, concrete suggestions for how to do thunks in > Python, please tell us! generators was the best thought so far. Barry From george at fischhof.hu Wed Feb 15 07:45:35 2017 From: george at fischhof.hu (George Fischhof) Date: Wed, 15 Feb 2017 13:45:35 +0100 Subject: [Python-ideas] Python package index: query and rating Message-ID: 2017-02-15 12:35 GMT+01:00 Paul Moore : > On 15 February 2017 at 11:07, Steven D'Aprano wrote: > >> I have two ides to improve the package index: > > > > Thank you for your ideas, but I don't think this is the right place for > > suggesting improvements to PyPI. You could try looking through the list > > of other Python mailing lists: > > > > https://mail.python.org/mailman/listinfo > > > > Or perhaps somebody else will suggest a better place to raise these > > issues. > > Hi George, > The PyPI codebase is pretty old and difficult to maintain, so there is > currently a project under way to replace it with a new implementation. > The project is hosted at https://github.com/pypa/warehouse and if > you're interested in getting involved that's probably the best > starting point for you. With regard to your two specific points: > > > 1. Query like in a webshop: checkboxes should use to select from > attributes > > The PyPI search interface is known to be pretty limited. I don't know > what changes have been put into the warehouse interface so far, but > I'd suggest checking out the state of their search, and if you still > feel there's scope for improvements, I'm sure they would be interested > in suggestions/contributions. > > > 2. It would would be good to implement some user rating mechanism like > in the webshops or software stores. > > This has been debated, both here and (I believe) on other lists in the > past. The general consensus is that it's not immediately obvious that > this would be beneficial. I'd suggest you do some searches to dig up > the history of this suggestion (sorry, I don't have any direct links) > and see if you feel there's something that has been missed - if so, > again the best place to propose this would be with the Warehouse guys. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Hi Paul, Steven, Thank You for your answer, i will check the warehouse :-) BR, George -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Feb 15 08:09:52 2017 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 15 Feb 2017 07:09:52 -0600 Subject: [Python-ideas] Python package index: query and rating In-Reply-To: References: <20170215110719.GA5689@ando.pearwood.info> Message-ID: On Wed, Feb 15, 2017 at 5:35 AM, Paul Moore wrote: > On 15 February 2017 at 11:07, Steven D'Aprano wrote: > >> I have two ides to improve the package index: > > > > Thank you for your ideas, but I don't think this is the right place for > > suggesting improvements to PyPI. You could try looking through the list > > of other Python mailing lists: > > > > https://mail.python.org/mailman/listinfo > > > > Or perhaps somebody else will suggest a better place to raise these > > issues. > > Hi George, > The PyPI codebase is pretty old and difficult to maintain, so there is > currently a project under way to replace it with a new implementation. > The project is hosted at https://github.com/pypa/warehouse and if > you're interested in getting involved that's probably the best > starting point for you. With regard to your two specific points: > > > 1. Query like in a webshop: checkboxes should use to select from > attributes > > The PyPI search interface is known to be pretty limited. I don't know > what changes have been put into the warehouse interface so far, but > I'd suggest checking out the state of their search, and if you still > feel there's scope for improvements, I'm sure they would be interested > in suggestions/contributions. > - https://github.com/pypa/warehouse/blob/master/warehouse/search.py - https://github.com/pypa/warehouse/blob/master/warehouse/packaging/search.py - https://github.com/pypa/warehouse/blob/master/warehouse/views.py def search() > > > 2. It would would be good to implement some user rating mechanism like > in the webshops or software stores. > > This has been debated, both here and (I believe) on other lists in the > past. The general consensus is that it's not immediately obvious that > this would be beneficial. I'd suggest you do some searches to dig up > the history of this suggestion (sorry, I don't have any direct links) > and see if you feel there's something that has been missed - if so, > again the best place to propose this would be with the Warehouse guys. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anthony at xtfx.me Wed Feb 15 11:45:10 2017 From: anthony at xtfx.me (C Anthony Risinger) Date: Wed, 15 Feb 2017 10:45:10 -0600 Subject: [Python-ideas] Passive tracing of function objects (was: Efficient debug logging) Message-ID: On Tue, Feb 14, 2017 at 3:55 PM, Barry wrote: > > > The point is that the cost of creating the msg argument can be very high. > > At the point that logging decides to skip output it is to late to save the > cost of creating the arg tuple. > Awhile back I was playing with some code for the purpose of adding debug logs "after the fact": https://github.com/anthonyrisinger/retrospect/blob/master/README.rst I'm not trying to plug it (It's POC, broken, and Python 2) but maybe there is some value because I was trying to solve a similar problem of fast and easy debugging and logs. The idea here is we target a function for "retrospection" by replacing it's code object with a modified variant that implements call outs at different points. Such interesting points might be "on symbol table changes" (new vars) or "on identifier rebinding" (updating vars) or "on line changes" or even "every single tick" to inspect TOS. I like this approach of latching onto the function/code object directly because it does not require a wrapper, but it can have be tricky with decorators depending on what the decorator does with the function it receives (we need access to the function object to replace it's __code__). I don't much like writing logs. They are verbose in code and almost always either much more or less than I need to deal with the problem at hand, and it's not always clear what's important ahead-of-time. With all the implementation details in play, you might end up wrapping code in context managers and whatnot for the sole purpose of logging. Or you have to jump thru hoops to get the exact info you need using many log levels or flags. I think having a way to target functions for real time introspection, then perfectly disable it, could allow for interesting solution to large, cross-cutting concerns like application logging. Is there any interest in this? Say a new function method like fun.trace(vars=[a,b,c])? Is there already a solution or past discussion on this topic? Thanks, -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Feb 15 11:46:03 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 15 Feb 2017 17:46:03 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <1756620.cYo0RJySdd@varric.chelsea.private> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> Message-ID: <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> On 15.02.2017 12:55, Barry Scott wrote: > The lazy eval sound like a generator. > Exactly my thought. Sven From anthony at xtfx.me Wed Feb 15 11:48:22 2017 From: anthony at xtfx.me (C Anthony Risinger) Date: Wed, 15 Feb 2017 10:48:22 -0600 Subject: [Python-ideas] Passive tracing of function objects (was: Efficient debug logging) In-Reply-To: References: Message-ID: On Wed, Feb 15, 2017 at 10:45 AM, C Anthony Risinger wrote: > On Tue, Feb 14, 2017 at 3:55 PM, Barry wrote: >> >> >> The point is that the cost of creating the msg argument can be very high. >> >> At the point that logging decides to skip output it is to late to save >> the cost of creating the arg tuple. >> > > Awhile back I was playing with some code for the purpose of adding debug > logs "after the fact": > > https://github.com/anthonyrisinger/retrospect/blob/master/README.rst > > I'm not trying to plug it (It's POC, broken, and Python 2) but maybe there > is some value because I was trying to solve a similar problem of fast and > easy debugging and logs. > I meant to note that I did in fact use this (plus a wrapper that would take an ENV var with the function to target) a few times to inspect code running deep in Django without ever editing any Django code. It also works for inspecting code you can't edit, say modules installed globally into /usr. -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Wed Feb 15 16:06:14 2017 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 15 Feb 2017 15:06:14 -0600 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> Message-ID: On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > Log "levels" never made sense to me; how can a single dimension be useful > substitute for a number of binary switches? With log "levels", you either > don't have enough logging, or you drown in too much logging (or you manage > a number of loggers, which is worse than logging switches). Again, isn't that what Filters are for? I mean the documentation says: > Filters can be used by Handlers and Loggers for more sophisticated > filtering than is provided by levels. On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > But to stick to the theme of language features: I believe there is a > previous thread that touches on a solution to the original posters > problem: "Alternative to PEP 532: delayed evaluation of expressions" That sounds interesting. It sounds a lot like a future. I like that it's a generalization of the problem (lazy expression evaluation) instead of a narrow problem (conditionally logging expensive messages) that is otherwise easily addressed by custom logic. On 16.02.2017, 5:47 Barry Scott wrote: > __repr__ is interesting however. > > Typically I describe in a string why the value is being logged. That means > that the object is always a string. I cannot recall using debugLog( obj ) > in production. > Loggers can handle that, though. You can easily write: >>> log.debug("log description: val_1=%r, val_2=%s", 3.14, 100) Instead of >>> log.debug("log description: val_1=%r, val_2=%s" % (3.14, 100)) The former won't construct the full message if the log level is higher than debug. It won't repr 3.14 or stringify 100. you can wrap functions in an object that only evaluates them when __repr__ and/or __str__ is called. It doesn't save you much time, because now you're writing a custom class instead of a custom function, I suppose. On Wed, Feb 15, 2017 at 10:46 AM, Sven R. Kunze wrote: > On 15.02.2017 12:55, Barry Scott wrote: > >> The lazy eval sound like a generator. >> >> > Exactly my thought. > > > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Thu Feb 16 05:33:22 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 16 Feb 2017 10:33:22 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> Message-ID: <97F25B50-C7D8-40B2-9B5E-3BF718D77630@barrys-emacs.org> So if python converted: debugLog( <# ?format string %r? % (expensive(),) #> ) Into: def __tmp__(): yield ?format string %r? % (expensive(),) debugLog( __tmp__ ) Then debugLog can detect the generator and call __next__ only if logging is enabled. I guess at that point its the same as using the lambda but you have the syntax sugar of Steven?s <# ? #>. Barry > On 15 Feb 2017, at 16:46, Sven R. Kunze wrote: > > On 15.02.2017 12:55, Barry Scott wrote: >> The lazy eval sound like a generator. >> > > Exactly my thought. > > > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From barry at barrys-emacs.org Thu Feb 16 05:27:47 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Thu, 16 Feb 2017 10:27:47 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> Message-ID: > On 15 Feb 2017, at 21:06, Abe Dillon wrote: > > On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > Log "levels" never made sense to me; how can a single dimension be useful substitute for a number of binary switches? With log "levels", you either don't have enough logging, or you drown in too much logging (or you manage a number of loggers, which is worse than logging switches). > > Again, isn't that what Filters are for? I mean the documentation says: > Filters can be used by Handlers and Loggers for more sophisticated filtering than is provided by levels. Filters could be used instead of a wrapper around the outside of logger.debug() as far as I can see. Buts it not clear that this is better then what I and others seem to do that is wrap the call to logger.debug(). However the use case I?m looking to improve is to avoid the cost of creating the arguments to the logger call. > > On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > But to stick to the theme of language features: I believe there is a previous thread that touches on a solution to the original posters problem: "Alternative to PEP 532: delayed evaluation of expressions" > > That sounds interesting. It sounds a lot like a future. I like that it's a generalization of the problem (lazy expression evaluation) instead of a narrow problem (conditionally logging expensive messages) that is otherwise easily addressed by custom logic. Agreed if this specific use case is solved by a general mechanism that is fantastic. > > On 16.02.2017, 5:47 Barry Scott wrote: > __repr__ is interesting however. > > Typically I describe in a string why the value is being logged. That means > that the object is always a string. I cannot recall using debugLog( obj ) > in production. > > Loggers can handle that, though. You can easily write: > > >>> log.debug("log description: val_1=%r, val_2=%s", 3.14, 100) > > Instead of > > >>> log.debug("log description: val_1=%r, val_2=%s" % (3.14, 100)) > > The former won't construct the full message if the log level is higher than debug. It won't repr 3.14 or stringify 100. Indeed, but it did not stop them all being passed in. And if 3.14 is replaced with calculatePi( places=10000 ) there is no saving. The lambda trick is the best short circuit so far. > you can wrap functions in an object that only evaluates them when __repr__ and/or __str__ is called. It doesn't save you much time, because now you're writing a custom class instead of a custom function, I suppose. > Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Feb 16 07:26:00 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 16 Feb 2017 13:26:00 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: Yeah, I had a similar issue in a previous company. A colleague wrote a script using a regex to remove these debug logs in the .py code. IHMO the clean design for that would be to support officially preprocessors in Python. My PEP opens the gate for that: https://www.python.org/dev/peps/pep-0511/ It would allow to write easily your own light preprocessor just to remove debug logs. Victor Le 14 f?vr. 2017 5:24 PM, "Barry Scott" a ?crit : > A common pattern I use is to have logging calls for debug and information > with my applications. > The logging calls can be separately enabled and disabled. > > For example: > > debug_log_enabled = False > def debugLog( msg ): > If debug_log_enabled: > print( ?Debug: %s? % (msg,) ) > > Then the caller can simple write: > > def main(): > debugLog( ?Start of main? ) > > This is fine until the evaluation of the msg becomes expensive. > > debugLog( ?info is %r? % (expensiveFunction(),) ) > > What would be nice is to be able to avoid evaluation the tuple of > arguments if debug is > disabled as this can be expensive. I can write this: > > if debug_log_enabled: debugLog( ?info is %r? % > (expensiveFunction(),) ) > > But that is a more code then I would like to write. And if the debug code > is a performance problem cannot > be left in the production code. > > I could combine the boolean and the log function by using a class to tidy > up the implementation. > > class DebugLog: > def __init__( self, enabled = False ): > self.enabled = enabled > > def __bool__( self ): > return self.enabled > > def __call__( self, msg ): > if self.enabled: print( ?Debug: %s? % (msg,) ) > > And call like this: > > dbg_log = DebugLog() > > If dbg_log: dbg_log( ?a debug message? ) > > But I?d like to only write: > > dbg_log( ?a debug message? ) > > And have the evaluation of the argument skipped unless its dbg_log is > enabled. > > I cannot see how to do this with python as it stands. > > Something would have to be added to allow python to short circuit the > argument tuple evaluation. > > Maybe python can check for a special dunder on the class that know how to > do this idiom, __if_true_call__? > > Thoughts? > > Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Thu Feb 16 07:55:47 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 16 Feb 2017 13:55:47 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> Message-ID: <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> Some comments: 1. you don't need a preprocessor for this: simply put your logging code into an "if __debug__:" block: https://docs.python.org/3.6/reference/simple_stmts.html?the-assert-statement#grammar-token-assert_stmt and then run your production code with "python -O" (the trick here is that the Python byte code compiler will not even generate code for such ifs) 2. from experience, keeping logging active in production often outweighs the benefits of the little better performance you gain by disabling it - being able to fix a failed app run by replaying transactions based on log file content can be priceless 3. preprocessors are evil, let's please not have them in Python :-) Code example for 1: app.py ------ def test(): assert 1 == 0, "assert raises" if __debug__: raise ValueError('__debug__ run branch') print ('test complete') test() > python3 -O Python 3.4.5 (default, Nov 17 2016, 20:58:13) [GCC 4.8.1 20130909 [gcc-4_8-branch revision 202388]] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import dis >>> import app test complete >>> dis.dis(app) Disassembly of test: 5 0 LOAD_GLOBAL 0 (print) 3 LOAD_CONST 1 ('test complete') 6 CALL_FUNCTION 1 (1 positional, 0 keyword pair) 9 POP_TOP 10 LOAD_CONST 0 (None) 13 RETURN_VALUE >>> On 16.02.2017 13:26, Victor Stinner wrote: > Yeah, I had a similar issue in a previous company. A colleague wrote a > script using a regex to remove these debug logs in the .py code. > > IHMO the clean design for that would be to support officially preprocessors > in Python. My PEP opens the gate for that: > https://www.python.org/dev/peps/pep-0511/ > > It would allow to write easily your own light preprocessor just to remove > debug logs. > > Victor > > Le 14 f?vr. 2017 5:24 PM, "Barry Scott" a ?crit : > >> A common pattern I use is to have logging calls for debug and information >> with my applications. >> The logging calls can be separately enabled and disabled. >> >> For example: >> >> debug_log_enabled = False >> def debugLog( msg ): >> If debug_log_enabled: >> print( ?Debug: %s? % (msg,) ) >> >> Then the caller can simple write: >> >> def main(): >> debugLog( ?Start of main? ) >> >> This is fine until the evaluation of the msg becomes expensive. >> >> debugLog( ?info is %r? % (expensiveFunction(),) ) >> >> What would be nice is to be able to avoid evaluation the tuple of >> arguments if debug is >> disabled as this can be expensive. I can write this: >> >> if debug_log_enabled: debugLog( ?info is %r? % >> (expensiveFunction(),) ) >> >> But that is a more code then I would like to write. And if the debug code >> is a performance problem cannot >> be left in the production code. >> >> I could combine the boolean and the log function by using a class to tidy >> up the implementation. >> >> class DebugLog: >> def __init__( self, enabled = False ): >> self.enabled = enabled >> >> def __bool__( self ): >> return self.enabled >> >> def __call__( self, msg ): >> if self.enabled: print( ?Debug: %s? % (msg,) ) >> >> And call like this: >> >> dbg_log = DebugLog() >> >> If dbg_log: dbg_log( ?a debug message? ) >> >> But I?d like to only write: >> >> dbg_log( ?a debug message? ) >> >> And have the evaluation of the argument skipped unless its dbg_log is >> enabled. >> >> I cannot see how to do this with python as it stands. >> >> Something would have to be added to allow python to short circuit the >> argument tuple evaluation. >> >> Maybe python can check for a special dunder on the class that know how to >> do this idiom, __if_true_call__? >> >> Thoughts? >> >> Barry >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 16 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From victor.stinner at gmail.com Thu Feb 16 08:23:13 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 16 Feb 2017 14:23:13 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> Message-ID: 2017-02-16 13:55 GMT+01:00 M.-A. Lemburg : > 1. you don't need a preprocessor for this: simply put your > logging code into an "if __debug__:" block: The problem with -O is that it also disables assertions, whereas you may want to keep them at runtime on production for good reasons. Victor From mal at egenix.com Thu Feb 16 09:20:51 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 16 Feb 2017 15:20:51 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> Message-ID: On 16.02.2017 14:23, Victor Stinner wrote: > 2017-02-16 13:55 GMT+01:00 M.-A. Lemburg : >> 1. you don't need a preprocessor for this: simply put your >> logging code into an "if __debug__:" block: > > The problem with -O is that it also disables assertions, whereas you > may want to keep them at runtime on production for good reasons. Assertions that you need in production should be written as proper if-statements. I know some people will disagree, but IMO using "assert" is the wrong approach in such situations - it's meant for development and testing only, not as short-cut to avoid having to write a proper error handler :-) If they were, Python would not drop creating code for them when using Python in production -O mode. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 16 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From g.rodola at gmail.com Thu Feb 16 09:24:54 2017 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 16 Feb 2017 15:24:54 +0100 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> Message-ID: On Thu, Feb 16, 2017 at 1:55 PM, M.-A. Lemburg wrote: > Some comments: > > 1. you don't need a preprocessor for this: simply put your > logging code into an "if __debug__:" block: > > https://docs.python.org/3.6/reference/simple_stmts.html? > the-assert-statement#grammar-token-assert_stmt > and then run your production code with "python -O" > (the trick here is that the Python byte code compiler will > not even generate code for such ifs) I didn't know about if __debug__ + -O:. Unbelievable after so many years. -------------- next part -------------- An HTML attachment was scrubbed... URL: From klahnakoski at mozilla.com Thu Feb 16 13:54:45 2017 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Thu, 16 Feb 2017 13:54:45 -0500 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> Message-ID: <7a2fc6a7-9aa5-f83e-ee74-86f96d341838@mozilla.com> On 2017-02-15 16:06, Abe Dillon wrote: > On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > > Log "levels" never made sense to me; how can a single dimension be > useful substitute for a number of binary switches? With log > "levels", you either don't have enough logging, or you drown in > too much logging (or you manage a number of loggers, which is > worse than logging switches). > > > Again, isn't that what Filters are for? I mean the documentation says: > > |Filters| can be used by |Handlers| and |Loggers| for more > sophisticated filtering than is provided by levels. > > You are right that Python logging infrastructure can deal with logs at a finer grain, but the code required to do it is more verbose than logical switches, and may also be non-local, and still does not solve the expensiveFunction() problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu Feb 16 14:39:15 2017 From: barry at python.org (Barry Warsaw) Date: Thu, 16 Feb 2017 14:39:15 -0500 Subject: [Python-ideas] Efficient debug logging References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> Message-ID: <20170216143915.26a7fcf2@subdivisions.wooz.org> On Feb 16, 2017, at 03:20 PM, M.-A. Lemburg wrote: >I know some people will disagree, but IMO using "assert" is the wrong >approach in such situations - it's meant for development and testing >only, not as short-cut to avoid having to write a proper error >handler :-) I use assertions for "things that can never happen", although sometimes they do due to an incomplete understanding of the code, the problem, or the environment. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 801 bytes Desc: OpenPGP digital signature URL: From abedillon at gmail.com Thu Feb 16 16:03:28 2017 From: abedillon at gmail.com (Abe Dillon) Date: Thu, 16 Feb 2017 15:03:28 -0600 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <7a2fc6a7-9aa5-f83e-ee74-86f96d341838@mozilla.com> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> <7a2fc6a7-9aa5-f83e-ee74-86f96d341838@mozilla.com> Message-ID: > > You are right that Python logging infrastructure can deal with logs at a > finer grain, but the code required to do it is more verbose than logical > switches The default behavior of filters seems pretty useful to me (filtering by hierarchy), but, then again, so do the default log levels. I have trouble understanding when you would need much more, which leads me to believe those cases are probably rare and complex (though I might very well be wrong). If you have a complex problem you typically should expect a complex solution like writing a custom log filter. You might think "it would be nice if the solution to this problem was in the standard lib", but we have to be a little picky about what goes into the standard lib least it inflate to a monstrous mess. If it's a niche problem, maybe a custom solution or third party lib *is* the way to go. and may also be non-local There are design patterns to help with that. and still does not solve the expensiveFunction() problem. I wasn't trying to address that problem in that quote. I was addressing the tangent on the insufficient expressiveness of log levels. I personally don't see why you can't use floats for log levels, but I'm interested to know how people are using logs such that they need dozens of levels. That, however; is tangential to the discussion about conditional execution of an expensive function. On Thu, Feb 16, 2017 at 12:54 PM, Kyle Lahnakoski wrote: > > On 2017-02-15 16:06, Abe Dillon wrote: > > On 15.02.2017, 20:39 Kyle Lahnakoski wrote: > >> Log "levels" never made sense to me; how can a single dimension be useful >> substitute for a number of binary switches? With log "levels", you either >> don't have enough logging, or you drown in too much logging (or you manage >> a number of loggers, which is worse than logging switches). > > > Again, isn't that what Filters are for? I mean the documentation says: > >> Filters can be used by Handlers and Loggers for more sophisticated >> filtering than is provided by levels. > > > > You are right that Python logging infrastructure can deal with logs at a > finer grain, but the code required to do it is more verbose than logical > switches, and may also be non-local, and still does not solve the > expensiveFunction() problem. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Thu Feb 16 22:13:19 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 17 Feb 2017 04:13:19 +0100 Subject: [Python-ideas] More classical for-loop Message-ID: Here is a summary of my idea about for-loop. It focuses on readability and does not take in account possible technical nuances. This is my first attempt to write a full proposal and I suppose it is ok to post it here. Many things (readability) can raise opinion based dispute, so it is sort of my opinion. I am not sure how topical/realistic is it, or contains technical flaws, so your note and recommendations are welcome. Proposal -------- Introduce a more classical, "Fortran-ish" loop syntax. The simplest syntax could be: for i somekeyword 10 : which logically will be the same as for i in range(10) : Keyword supposed to have sense only inside the statement, thus not interfere with variables and other keywords. At this moment the proposal will be to use the word "over": for i over 10 : The choice of the word is based only on its look - middle sized, has related meaning(?), and easy to type. Of course, if a better word can be found, it can be used. Note: the meaning of the word is of much less importance than optical qualities. Other good looking options: for i count 10 : for i range 10 : for i steps 10 : for i sieve 10 : Parameters same as in range(): for i over a, b, step : is the same as for i in range(a, b, step) : Rationale ----------- Removing the brackets will help concentrate on other parts of code, though it has some price, since if more than one range parameter is used, it loses some emphasis. This is easiliy compensated by IDE's syntax highlighting, namely by slightly graying out the keyword. Currently the loop statement uses the word "in". This word creates rather unpleasant visual 'stick and hole' effect on eyes due to its short size. Is quite noticable, especially in case of one-letter varaibles (e.g. for i in ...). And the 'bulky' range() part is bit too heavy. More problems to 'holes' effect, especially in monospaced editors, adds the form of glyphs "i" and "n" which both have vertical stick structure and also interfere with the very common variable name "i". Also new sytax needs less typing. Examples -------- Common use case: L = [1,3,5,7] for i over len(L): e = L[i] or: length = len(L) for i over length: e = L[i] Nested: for y over 0,480 : for x over 0,640 : putpixel (x, y) or: width = 640 height = 480 for y over 0, height : for x over 0, width : putpixel (x, y) Surprisingly good result is achieved if simply remove the keyword and put more space: for y 0, height : for x 0, width : putpixel (x, y) But similar effect can be achieved by graying out the word with syntax highlighting. Compare with existing syntax: for y in range(0,480) : for x in range(0,640) : putpixel (x, y) for y in range(0, height) : for x in range(0, width) : putpixel (x, y) Notes ------- Although it is true that the whole idea is more about cosmetics, still the for-loop is important structural component and slight improvement could count. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Feb 16 22:59:03 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 17 Feb 2017 14:59:03 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V wrote: > Common use case: > > L = [1,3,5,7] > > for i over len(L): > e = L[i] > > or: > > length = len(L) > for i over length: > e = L[i] Better use case: for i, e in enumerate(L): ChrisA From josephhackman at gmail.com Fri Feb 17 00:24:53 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 00:24:53 -0500 Subject: [Python-ideas] Delayed Execution via Keyword Message-ID: Howdy All! This suggestion is inspired by the question on "Efficient debug logging". I propose a keyword to mark an expression for delayed/lazy execution, for the purposes of standardizing such behavior across the language. The proposed format is: delayed: i.e. log.info("info is %s", delayed: expensiveFunction()) Unlike 'lambda' which returns a function (so the receiver must be lambda-aware), delayed execution blocks are for all purposes values. The first time the value (rather than location) is read, or any method on the delayed object is called, the expression is executed and the delayed expression is replaced with the result. (Thus, the delayed expression is only every evaluated once). Ideally: a = delayed: 1+2 b = a print(a) #adds 1 and 2, prints 3 # a and b are now both just 3 print(b) #just prints 3 Mechanically, this would be similar to the following: class Delayed(): def __init__(self, func): self.__func = func self.__executed = False self.__value = None def __str__(self): if self.__executed: return self.__value.__str__() self.__value = self.__func() self.__executed = True return self.__value.__str__() def function_print(value): print('function_print') print(value) def function_return_stuff(value): print('function_return_stuff') return value function_print(function_return_stuff('no_delay')) function_print(Delayed(lambda: function_return_stuff('delayed'))) delayed = Delayed(lambda: function_return_stuff('delayed_object')) function_print(delayed) function_print(delayed) Unfortunately, due to https://docs.python.org/3/reference/datamodel.html#special-lookup , this magic delayed class would need to implement many magic methods, as __getattribute__ is not _always_ called. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 00:51:23 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Feb 2017 21:51:23 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: I rather like this at first brush! On Feb 16, 2017 9:25 PM, "Joseph Hackman" wrote: > Howdy All! > > This suggestion is inspired by the question on "Efficient debug logging". > > I propose a keyword to mark an expression for delayed/lazy execution, for > the purposes of standardizing such behavior across the language. > > The proposed format is: > delayed: > i.e. log.info("info is %s", delayed: expensiveFunction()) > > Unlike 'lambda' which returns a function (so the receiver must be > lambda-aware), delayed execution blocks are for all purposes values. The > first time the value (rather than location) is read, or any method on the > delayed object is called, the expression is executed and the delayed > expression is replaced with the result. (Thus, the delayed expression is > only every evaluated once). > > Ideally: > a = delayed: 1+2 > b = a > print(a) #adds 1 and 2, prints 3 > # a and b are now both just 3 > print(b) #just prints 3 > > Mechanically, this would be similar to the following: > > class Delayed(): > def __init__(self, func): > self.__func = func > self.__executed = False > self.__value = None > > def __str__(self): > if self.__executed: > return self.__value.__str__() > self.__value = self.__func() > self.__executed = True > return self.__value.__str__() > > > def function_print(value): > print('function_print') > print(value) > > def function_return_stuff(value): > print('function_return_stuff') > return value > > function_print(function_return_stuff('no_delay')) > > function_print(Delayed(lambda: function_return_stuff('delayed'))) > > delayed = Delayed(lambda: function_return_stuff('delayed_object')) > function_print(delayed) > function_print(delayed) > > Unfortunately, due to https://docs.python.org/3/reference/datamodel.html# > special-lookup , this magic delayed class would need to implement many > magic methods, as __getattribute__ is not _always_ called. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 00:52:20 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 00:52:20 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: You might be interested in https://github.com/llllllllll/lazy_python, which implements the features you describe but instead of a keyword it uses a decorator. On Fri, Feb 17, 2017 at 12:24 AM, Joseph Hackman wrote: > Howdy All! > > This suggestion is inspired by the question on "Efficient debug logging". > > I propose a keyword to mark an expression for delayed/lazy execution, for > the purposes of standardizing such behavior across the language. > > The proposed format is: > delayed: > i.e. log.info("info is %s", delayed: expensiveFunction()) > > Unlike 'lambda' which returns a function (so the receiver must be > lambda-aware), delayed execution blocks are for all purposes values. The > first time the value (rather than location) is read, or any method on the > delayed object is called, the expression is executed and the delayed > expression is replaced with the result. (Thus, the delayed expression is > only every evaluated once). > > Ideally: > a = delayed: 1+2 > b = a > print(a) #adds 1 and 2, prints 3 > # a and b are now both just 3 > print(b) #just prints 3 > > Mechanically, this would be similar to the following: > > class Delayed(): > def __init__(self, func): > self.__func = func > self.__executed = False > self.__value = None > > def __str__(self): > if self.__executed: > return self.__value.__str__() > self.__value = self.__func() > self.__executed = True > return self.__value.__str__() > > > def function_print(value): > print('function_print') > print(value) > > def function_return_stuff(value): > print('function_return_stuff') > return value > > function_print(function_return_stuff('no_delay')) > > function_print(Delayed(lambda: function_return_stuff('delayed'))) > > delayed = Delayed(lambda: function_return_stuff('delayed_object')) > function_print(delayed) > function_print(delayed) > > Unfortunately, due to https://docs.python.org/3/reference/datamodel.html# > special-lookup , this magic delayed class would need to implement many > magic methods, as __getattribute__ is not _always_ called. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 01:23:13 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Feb 2017 22:23:13 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: Dask also has a function delayed() that may be used as an decorator and in other ways like: >>> from dask import delayed >>> from operator import add, mul >>> a = delayed(add)(1, 2) >>> b = delayed(mul)(a, 3) >>> b Delayed('mul-1907f29b-60a4-48af-ba2a-938556555f9b') >>> c = b.compute() >>> c 9 >>> b.dask {'add-d49ba000-dd5d-4031-8c37-6514626a3d81': (, 1, 2), 'mul-1907f29b-60a4-48af-ba2a-938556555f9b': (, 'add-d49ba000-dd5d-4031-8c37-6514626a3d81', 3)} You *can* do pretty much anything you'd want to using this approach... including the real job of Dask, to identify latent parallelism and execute computations on many cores or many machines in a cluster. But actual syntax to do all of this would be really elegant. I think for this to be as useful as I'd want, you'd sometimes need to be to continue delaying computation rather than doing so on every access I guess as syntax, the `delayed:` construct would work (I think having no colon would more closely model `yield` and `yield from` and `await` and `async` which this is kinda-sorta akin to). So for example, in a hypothetical Python 3.7+: >>> a = delayed 1 + 2 >>> b = delayed b * 3 >>> c = delayed 12/3 >>> my_lazy_func(b, delayed c) # evaluates b but not yet c >>> b 9 >>> delayed c If you want to do something like Dask... or for Dask itself to be able to use it in some eventual version, you'd need to be able to keep objects from evaluating even while you passed them around. The obvious use is for finding parallelism, but other things like passing callbacks might find this useful too. Dask delayed objects stay lazy until you explicitly `.compute()` on them (which does so recursively on every lazy object that might go into the computation). This hypothetical new keyword would have object evaluate eagerly *unless* you explicitly kept them lazy. But the idea is that the programmer would still get to decide in their code. On Feb 16, 2017 9:53 PM, "Joseph Jevnik" wrote: > You might be interested in https://github.com/llllllllll/lazy_python, > which implements the features you describe but instead of a keyword it uses > a decorator. > > On Fri, Feb 17, 2017 at 12:24 AM, Joseph Hackman > wrote: > >> Howdy All! >> >> This suggestion is inspired by the question on "Efficient debug logging". >> >> > I propose a keyword to mark an expression for delayed/lazy execution, for >> the purposes of standardizing such behavior across the language. >> >> The proposed format is: >> delayed: >> i.e. log.info("info is %s", delayed: expensiveFunction()) >> >> Unlike 'lambda' which returns a function (so the receiver must be >> lambda-aware), delayed execution blocks are for all purposes values. The >> first time the value (rather than location) is read, or any method on the >> delayed object is called, the expression is executed and the delayed >> expression is replaced with the result. (Thus, the delayed expression is >> only every evaluated once). >> >> Ideally: >> a = delayed: 1+2 >> b = a >> print(a) #adds 1 and 2, prints 3 >> # a and b are now both just 3 >> print(b) #just prints 3 >> >> Mechanically, this would be similar to the following: >> >> class Delayed(): >> def __init__(self, func): >> self.__func = func >> self.__executed = False >> self.__value = None >> >> def __str__(self): >> if self.__executed: >> return self.__value.__str__() >> self.__value = self.__func() >> self.__executed = True >> return self.__value.__str__() >> >> >> def function_print(value): >> print('function_print') >> print(value) >> >> def function_return_stuff(value): >> print('function_return_stuff') >> return value >> >> function_print(function_return_stuff('no_delay')) >> >> function_print(Delayed(lambda: function_return_stuff('delayed'))) >> >> delayed = Delayed(lambda: function_return_stuff('delayed_object')) >> function_print(delayed) >> function_print(delayed) >> >> Unfortunately, due to https://docs.python.org/3/r >> eference/datamodel.html#special-lookup , this magic delayed class would >> need to implement many magic methods, as __getattribute__ is not _always_ >> called. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 17 01:33:51 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 17 Feb 2017 06:33:51 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: David, can you elaborate on your example? if we replaced line four with >>> x = my_lazy_func(b, delayed c) what would the value of `x` be, and how would this differ from either >>> x = delayed my_lazy_func(b, delayed c) or >>> x = delayed my_lazy_func(b, c) To put it another way, why does my_lazy_func being called not evaluate c, or are you implying that it too is some kind of special delayed function that needs to be explicitly computed, like in dask? --Josh On Fri, Feb 17, 2017 at 1:23 AM David Mertz wrote: > Dask also has a function delayed() that may be used as an decorator and in > other ways like: > > >>> from dask import delayed > > >>> from operator import add, mul > >>> a = delayed(add)(1, 2) > >>> b = delayed(mul)(a, 3) > > >>> b > Delayed('mul-1907f29b-60a4-48af-ba2a-938556555f9b') > > >>> c = b.compute() > > >>> c > 9 > > >>> b.dask > {'add-d49ba000-dd5d-4031-8c37-6514626a3d81': (, 1, > 2), > 'mul-1907f29b-60a4-48af-ba2a-938556555f9b': (, > 'add-d49ba000-dd5d-4031-8c37-6514626a3d81', > 3)} > > > You *can* do pretty much anything you'd want to using this approach... > including the real job of Dask, to identify latent parallelism and execute > computations on many cores or many machines in a cluster. > > But actual syntax to do all of this would be really elegant. I think for > this to be as useful as I'd want, you'd sometimes need to be to continue > delaying computation rather than doing so on every access I guess as > syntax, the `delayed:` construct would work (I think having no colon would > more closely model `yield` and `yield from` and `await` and `async` which > this is kinda-sorta akin to). > > So for example, in a hypothetical Python 3.7+: > > >>> a = delayed 1 + 2 > > >>> b = delayed b * 3 > > >>> c = delayed 12/3 > > >>> my_lazy_func(b, delayed c) # evaluates b but not yet c > >>> b > 9 > >>> delayed c > > > > If you want to do something like Dask... or for Dask itself to be able to > use it in some eventual version, you'd need to be able to keep objects from > evaluating even while you passed them around. The obvious use is for > finding parallelism, but other things like passing callbacks might find > this useful too. > > Dask delayed objects stay lazy until you explicitly `.compute()` on them > (which does so recursively on every lazy object that might go into the > computation). This hypothetical new keyword would have object evaluate > eagerly *unless* you explicitly kept them lazy. But the idea is that the > programmer would still get to decide in their code. > > > On Feb 16, 2017 9:53 PM, "Joseph Jevnik" wrote: > > You might be interested in https://github.com/llllllllll/lazy_python, > which implements the features you describe but instead of a keyword it uses > a decorator. > > On Fri, Feb 17, 2017 at 12:24 AM, Joseph Hackman > wrote: > > Howdy All! > > This suggestion is inspired by the question on "Efficient debug logging". > > > I propose a keyword to mark an expression for delayed/lazy execution, for > the purposes of standardizing such behavior across the language. > > The proposed format is: > delayed: > i.e. log.info("info is %s", delayed: expensiveFunction()) > > Unlike 'lambda' which returns a function (so the receiver must be > lambda-aware), delayed execution blocks are for all purposes values. The > first time the value (rather than location) is read, or any method on the > delayed object is called, the expression is executed and the delayed > expression is replaced with the result. (Thus, the delayed expression is > only every evaluated once). > > Ideally: > a = delayed: 1+2 > b = a > print(a) #adds 1 and 2, prints 3 > # a and b are now both just 3 > print(b) #just prints 3 > > Mechanically, this would be similar to the following: > > class Delayed(): > def __init__(self, func): > self.__func = func > self.__executed = False > self.__value = None > > def __str__(self): > if self.__executed: > return self.__value.__str__() > self.__value = self.__func() > self.__executed = True > return self.__value.__str__() > > > def function_print(value): > print('function_print') > print(value) > > def function_return_stuff(value): > print('function_return_stuff') > return value > > function_print(function_return_stuff('no_delay')) > > function_print(Delayed(lambda: function_return_stuff('delayed'))) > > delayed = Delayed(lambda: function_return_stuff('delayed_object')) > function_print(delayed) > function_print(delayed) > > Unfortunately, due to > https://docs.python.org/3/reference/datamodel.html#special-lookup , this > magic delayed class would need to implement many magic methods, as > __getattribute__ is not _always_ called. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 01:55:40 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Feb 2017 22:55:40 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: On Thu, Feb 16, 2017 at 10:33 PM, Joshua Morton wrote: > David, can you elaborate on your example? > > if we replaced line four with > > >>> x = my_lazy_func(b, delayed c) > > what would the value of `x` be, and how would this differ from either > The value of the function would be whatever you want, including another delayed object, potentially. Here's a toy function just to show the pattern I have in mind: def my_lazy_func(x, y): if x == 0: # evaluate y if it was lazy, or just return plain value return y elif x > 0: # create a lazy object whether y was lazy or concrete # if y *was* lazy, it remains so in delayed expression return delayed y + 17 elif x < -10: # evaluate y, if delayed, by virtue of being part of # plain non-delayed RHS expression z = y * 6 return z else: # y never mentioned, so if it was lazy nothing changes return x - 22 Whether or not 'c' (called 'y' within the function scope) gets evaluated/computed depends on which branch was taken within the function. Once the delayed computation is done, the object at the delayed address becomes concrete thereafter to avoid repeated (and potentially stateful or side-effect causing evaluation). One question is whether you'd need to write: m = delayed 1 + 2 n = delayed m * 3 # This seems ugly and unnecessary (but should work) q = delayed m / delayed n # This seems better (does not evaluate m or n) # everything lazy within delayed expression stays unevaluated q2 = delayed m/n If you ever need to explicitly evaluate a delayed object, it is as simple as putting a name pointing to it on a line by itself: f = delayed 1 + 2 # We want to evaluate f before next op for some reason f # f is already a concrete value now, before calculating g g = f * 7 So for example, in a hypothetical Python 3.7+: >> >> >>> a = delayed 1 + 2 >> >> >>> b = delayed b * 3 >> >> >>> c = delayed 12/3 >> >> >>> my_lazy_func(b, delayed c) # evaluates b but not yet c >> >>> b >> 9 >> >>> delayed c >> >> >> >> If you want to do something like Dask... or for Dask itself to be able to >> use it in some eventual version, you'd need to be able to keep objects from >> evaluating even while you passed them around. The obvious use is for >> finding parallelism, but other things like passing callbacks might find >> this useful too. >> >> Dask delayed objects stay lazy until you explicitly `.compute()` on them >> (which does so recursively on every lazy object that might go into the >> computation). This hypothetical new keyword would have object evaluate >> eagerly *unless* you explicitly kept them lazy. But the idea is that >> the programmer would still get to decide in their code. >> >> >> On Feb 16, 2017 9:53 PM, "Joseph Jevnik" wrote: >> >> You might be interested in https://github.com/llllllllll/lazy_python, >> which implements the features you describe but instead of a keyword it uses >> a decorator. >> >> On Fri, Feb 17, 2017 at 12:24 AM, Joseph Hackman > > wrote: >> >> Howdy All! >> >> This suggestion is inspired by the question on "Efficient debug logging". >> >> >> I propose a keyword to mark an expression for delayed/lazy execution, for >> the purposes of standardizing such behavior across the language. >> >> The proposed format is: >> delayed: >> i.e. log.info("info is %s", delayed: expensiveFunction()) >> >> Unlike 'lambda' which returns a function (so the receiver must be >> lambda-aware), delayed execution blocks are for all purposes values. The >> first time the value (rather than location) is read, or any method on the >> delayed object is called, the expression is executed and the delayed >> expression is replaced with the result. (Thus, the delayed expression is >> only every evaluated once). >> >> Ideally: >> a = delayed: 1+2 >> b = a >> print(a) #adds 1 and 2, prints 3 >> # a and b are now both just 3 >> print(b) #just prints 3 >> >> Mechanically, this would be similar to the following: >> >> class Delayed(): >> def __init__(self, func): >> self.__func = func >> self.__executed = False >> self.__value = None >> >> def __str__(self): >> if self.__executed: >> return self.__value.__str__() >> self.__value = self.__func() >> self.__executed = True >> return self.__value.__str__() >> >> >> def function_print(value): >> print('function_print') >> print(value) >> >> def function_return_stuff(value): >> print('function_return_stuff') >> return value >> >> function_print(function_return_stuff('no_delay')) >> >> function_print(Delayed(lambda: function_return_stuff('delayed'))) >> >> delayed = Delayed(lambda: function_return_stuff('delayed_object')) >> function_print(delayed) >> function_print(delayed) >> >> Unfortunately, due to https://docs.python.org/3/reference/datamodel.html# >> special-lookup , this magic delayed class would need to implement many >> magic methods, as __getattribute__ is not _always_ called. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 02:15:24 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Feb 2017 23:15:24 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: I think maybe the idiomatic pattern should be assignment rather than just bare name. E.g. f = delayed 1 + 2 # We want to evaluate f before next op for some reason *f = f* # f is already a concrete value now, before calculating g g = f * 7 I think if we follow my rule that "everything lazy within delayed expression stays unevaluated" that needs to apply to function calls too. So I think in: x = delayed my_lazy_func(b, c) This would not affect the laziness of 'b' or 'c' with that line. It also would not run any flow control within the function, etc. Basically, it's almost like wrapping it in a lambda: x = lambda: my_lazy_func(b, c) Except that when you finally *do* want the value out of 'x' you don't spell that as 'x() + 7' but simply as 'x + 7'. This also means that the above delayed function call would be functionally identical if you spelled it: x = delayed my_lazy_func(delayed b, delayed c) This also means that a 'delayed' object needs to be idempotent. So x = delayed 2+2 y = delayed x z = delayed delayed delayed y Wrapping more delays around an existing delayed object should probably just keep the same object rather than "doubly delaying" it. If there is some reason to create separate delayed objects that isn't occurring to me, evaluating 'z' would still go through the multiple evaluation levels until it got to a non-delayed value. On Thu, Feb 16, 2017 at 10:55 PM, David Mertz wrote: > On Thu, Feb 16, 2017 at 10:33 PM, Joshua Morton > wrote: > >> David, can you elaborate on your example? >> >> if we replaced line four with >> >> >>> x = my_lazy_func(b, delayed c) >> >> what would the value of `x` be, and how would this differ from either >> > > The value of the function would be whatever you want, including another > delayed object, potentially. > > Here's a toy function just to show the pattern I have in mind: > > def my_lazy_func(x, y): > > if x == 0: > # evaluate y if it was lazy, or just return plain value > > return y > > elif x > 0: > > # create a lazy object whether y was lazy or concrete > > # if y *was* lazy, it remains so in delayed expression > > return delayed y + 17 > > elif x < -10: > > # evaluate y, if delayed, by virtue of being part of > > # plain non-delayed RHS expression > > z = y * 6 > > return z > > else: > > # y never mentioned, so if it was lazy nothing changes > > return x - 22 > > > Whether or not 'c' (called 'y' within the function scope) gets > evaluated/computed depends on which branch was taken within the function. > Once the delayed computation is done, the object at the delayed address > becomes concrete thereafter to avoid repeated (and potentially stateful or > side-effect causing evaluation). > > One question is whether you'd need to write: > > m = delayed 1 + 2 > n = delayed m * 3 > # This seems ugly and unnecessary (but should work) > q = delayed m / delayed n > # This seems better (does not evaluate m or n) > # everything lazy within delayed expression stays unevaluated > > q2 = delayed m/n > > If you ever need to explicitly evaluate a delayed object, it is as simple > as putting a name pointing to it on a line by itself: > > f = delayed 1 + 2 > > # We want to evaluate f before next op for some reason > > f > > # f is already a concrete value now, before calculating g > > g = f * 7 > > > > So for example, in a hypothetical Python 3.7+: >>> >>> >>> a = delayed 1 + 2 >>> >>> >>> b = delayed b * 3 >>> >>> >>> c = delayed 12/3 >>> >>> >>> my_lazy_func(b, delayed c) # evaluates b but not yet c >>> >>> b >>> 9 >>> >>> delayed c >>> >>> >>> >>> If you want to do something like Dask... or for Dask itself to be able >>> to use it in some eventual version, you'd need to be able to keep objects >>> from evaluating even while you passed them around. The obvious use is for >>> finding parallelism, but other things like passing callbacks might find >>> this useful too. >>> >>> Dask delayed objects stay lazy until you explicitly `.compute()` on them >>> (which does so recursively on every lazy object that might go into the >>> computation). This hypothetical new keyword would have object evaluate >>> eagerly *unless* you explicitly kept them lazy. But the idea is that >>> the programmer would still get to decide in their code. >>> >>> >>> On Feb 16, 2017 9:53 PM, "Joseph Jevnik" wrote: >>> >>> You might be interested in https://github.com/llllllllll/lazy_python, >>> which implements the features you describe but instead of a keyword it uses >>> a decorator. >>> >>> On Fri, Feb 17, 2017 at 12:24 AM, Joseph Hackman < >>> josephhackman at gmail.com> wrote: >>> >>> Howdy All! >>> >>> This suggestion is inspired by the question on "Efficient debug logging". >>> >>> >>> I propose a keyword to mark an expression for delayed/lazy execution, >>> for the purposes of standardizing such behavior across the language. >>> >>> The proposed format is: >>> delayed: >>> i.e. log.info("info is %s", delayed: expensiveFunction()) >>> >>> Unlike 'lambda' which returns a function (so the receiver must be >>> lambda-aware), delayed execution blocks are for all purposes values. The >>> first time the value (rather than location) is read, or any method on the >>> delayed object is called, the expression is executed and the delayed >>> expression is replaced with the result. (Thus, the delayed expression is >>> only every evaluated once). >>> >>> Ideally: >>> a = delayed: 1+2 >>> b = a >>> print(a) #adds 1 and 2, prints 3 >>> # a and b are now both just 3 >>> print(b) #just prints 3 >>> >>> Mechanically, this would be similar to the following: >>> >>> class Delayed(): >>> def __init__(self, func): >>> self.__func = func >>> self.__executed = False >>> self.__value = None >>> >>> def __str__(self): >>> if self.__executed: >>> return self.__value.__str__() >>> self.__value = self.__func() >>> self.__executed = True >>> return self.__value.__str__() >>> >>> >>> def function_print(value): >>> print('function_print') >>> print(value) >>> >>> def function_return_stuff(value): >>> print('function_return_stuff') >>> return value >>> >>> function_print(function_return_stuff('no_delay')) >>> >>> function_print(Delayed(lambda: function_return_stuff('delayed'))) >>> >>> delayed = Delayed(lambda: function_return_stuff('delayed_object')) >>> function_print(delayed) >>> function_print(delayed) >>> >>> Unfortunately, due to https://docs.python.org/3/r >>> eference/datamodel.html#special-lookup , this magic delayed class would >>> need to implement many magic methods, as __getattribute__ is not _always_ >>> called. >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 02:26:28 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Feb 2017 23:26:28 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: > > This also means that a 'delayed' object needs to be idempotent. So > > x = delayed 2+2 > > y = delayed x > > z = delayed delayed delayed y > > > Wrapping more delays around an existing delayed object should probably > just keep the same object rather than "doubly delaying" it. If there is > some reason to create separate delayed objects that isn't occurring to me, > evaluating 'z' would still go through the multiple evaluation levels until > it got to a non-delayed value. > This is sort of like how iterators "return self" and 'it = iter(it)'. In the case of Dask, wrapping more delayed objects creates layers of these lazy objects. But I think it has to because it's not part of the syntax. Actually, I guess Dask could do graph reduction without actual computation if it wanted to. But this is the current behavior: >>> def unchanged(x): ... return x >>> a = delayed(unchanged)(42) >>> a Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') >>> b = delayed(unchanged)(a) >>> c = delayed(unchanged)(b) >>> c Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') >>> c.dask {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': (, 42), 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': (, 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': (, 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} >>> c.compute() 42 Actually Dask *cannot* know that "unchanged()" is the function that makes no transformation on its one parameter. From what it can see, it's just a function that does *something*. And I guess similarly in the proposed syntax, anything other than a plain name after the 'delayed' would still need to create a new delayed object. So it's all an edge case that doesn't make much difference. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 03:02:33 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 03:02:33 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: You can let dask "see" into the function by entering it and wrapping all of the operations in `delayed`; this is how daisy[0] builds up large compute graphs. In this case, you could "inline" the identity function and the delayed object would flow through the function and the call to identity never makes it into the task graph. [0] http://daisy-python.readthedocs.io/en/latest/appendix.html#daisy.autodask On Fri, Feb 17, 2017 at 2:26 AM, David Mertz wrote: > On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: >> >> This also means that a 'delayed' object needs to be idempotent. So >> >> x = delayed 2+2 >> >> y = delayed x >> >> z = delayed delayed delayed y >> >> >> Wrapping more delays around an existing delayed object should probably >> just keep the same object rather than "doubly delaying" it. If there is >> some reason to create separate delayed objects that isn't occurring to me, >> evaluating 'z' would still go through the multiple evaluation levels until >> it got to a non-delayed value. >> > > This is sort of like how iterators "return self" and 'it = iter(it)'. > > In the case of Dask, wrapping more delayed objects creates layers of these > lazy objects. But I think it has to because it's not part of the syntax. > Actually, I guess Dask could do graph reduction without actual computation > if it wanted to. But this is the current behavior: > > >>> def unchanged(x): > ... return x > >>> a = delayed(unchanged)(42) > >>> a > Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') > >>> b = delayed(unchanged)(a) > >>> c = delayed(unchanged)(b) > >>> c > Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') > >>> c.dask > {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': ( __main__.unchanged>, > 42), > 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': ( __main__.unchanged>, > 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), > 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': ( __main__.unchanged>, > 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} > > >>> c.compute() > 42 > > > Actually Dask *cannot* know that "unchanged()" is the function that makes > no transformation on its one parameter. From what it can see, it's just a > function that does *something*. And I guess similarly in the proposed > syntax, anything other than a plain name after the 'delayed' would still > need to create a new delayed object. So it's all an edge case that doesn't > make much difference. > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 03:31:19 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 00:31:19 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: I had forgotten about Daisy! It's an interesting project too. The behavior of 'autodask()' is closer to what I'd want in new syntax than is plain dask.delayed(). I'm not sure of all the corners. But is definitely love to have it for expressions generally, not only pure functions. On Feb 17, 2017 12:03 AM, "Joseph Jevnik" wrote: > You can let dask "see" into the function by entering it and wrapping all > of the operations in `delayed`; this is how daisy[0] builds up large > compute graphs. In this case, you could "inline" the identity function and > the delayed object would flow through the function and the call to identity > never makes it into the task graph. > > [0] http://daisy-python.readthedocs.io/en/latest/ > appendix.html#daisy.autodask > > On Fri, Feb 17, 2017 at 2:26 AM, David Mertz wrote: > >> On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: >>> >>> This also means that a 'delayed' object needs to be idempotent. So >>> >>> x = delayed 2+2 >>> >>> y = delayed x >>> >>> z = delayed delayed delayed y >>> >>> >>> Wrapping more delays around an existing delayed object should probably >>> just keep the same object rather than "doubly delaying" it. If there is >>> some reason to create separate delayed objects that isn't occurring to me, >>> evaluating 'z' would still go through the multiple evaluation levels until >>> it got to a non-delayed value. >>> >> >> This is sort of like how iterators "return self" and 'it = iter(it)'. >> >> In the case of Dask, wrapping more delayed objects creates layers of >> these lazy objects. But I think it has to because it's not part of the >> syntax. Actually, I guess Dask could do graph reduction without actual >> computation if it wanted to. But this is the current behavior: >> >> >>> def unchanged(x): >> ... return x >> >>> a = delayed(unchanged)(42) >> >>> a >> Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') >> >>> b = delayed(unchanged)(a) >> >>> c = delayed(unchanged)(b) >> >>> c >> Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') >> >>> c.dask >> {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': (> __main__.unchanged>, >> 42), >> 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': (> __main__.unchanged>, >> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), >> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': (> __main__.unchanged>, >> 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} >> >> >>> c.compute() >> 42 >> >> >> Actually Dask *cannot* know that "unchanged()" is the function that >> makes no transformation on its one parameter. From what it can see, it's >> just a function that does *something*. And I guess similarly in the >> proposed syntax, anything other than a plain name after the 'delayed' would >> still need to create a new delayed object. So it's all an edge case that >> doesn't make much difference. >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. Intellectual property is >> to the 21st century what the slave trade was to the 16th. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavol.lisy at gmail.com Fri Feb 17 03:34:51 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Fri, 17 Feb 2017 09:34:51 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 2/17/17, Mikhail V wrote: > Rationale > ----------- > > Removing the brackets will help concentrate on other > parts of code [...] Do you think that for i in steps * 10: for i in steps * 1-10: for i in steps * 1-10 ^ 2: are better than for i in range(10): for i in range(1, 10): for i in range(1, 10, 2): because brackets are gone? I am asking because you could simply implement it with current syntax. And I am not sure if removing bracket is really good enough. We could use though experiment: Is it good for you? Do you want to implement package and use "from my_package import steps" and "for in steps" in your (not performance critical) code? BTW proposing new keyword is often bad idea because it break backward compatibility. Sometimes existing keywords or operators could be used. From joejev at gmail.com Fri Feb 17 04:06:27 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 04:06:27 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: Even with the new syntax I would highly discourage delaying a function with observable side effects. It would make reasoning about the behavior of the program very difficult and debugging becomes much harder. On Fri, Feb 17, 2017 at 3:31 AM, David Mertz wrote: > I had forgotten about Daisy! It's an interesting project too. The behavior > of 'autodask()' is closer to what I'd want in new syntax than is plain > dask.delayed(). I'm not sure of all the corners. But is definitely love to > have it for expressions generally, not only pure functions. > > On Feb 17, 2017 12:03 AM, "Joseph Jevnik" wrote: > >> You can let dask "see" into the function by entering it and wrapping all >> of the operations in `delayed`; this is how daisy[0] builds up large >> compute graphs. In this case, you could "inline" the identity function and >> the delayed object would flow through the function and the call to identity >> never makes it into the task graph. >> >> [0] http://daisy-python.readthedocs.io/en/latest/appendix.html# >> daisy.autodask >> >> On Fri, Feb 17, 2017 at 2:26 AM, David Mertz wrote: >> >>> On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: >>>> >>>> This also means that a 'delayed' object needs to be idempotent. So >>>> >>>> x = delayed 2+2 >>>> >>>> y = delayed x >>>> >>>> z = delayed delayed delayed y >>>> >>>> >>>> Wrapping more delays around an existing delayed object should probably >>>> just keep the same object rather than "doubly delaying" it. If there is >>>> some reason to create separate delayed objects that isn't occurring to me, >>>> evaluating 'z' would still go through the multiple evaluation levels until >>>> it got to a non-delayed value. >>>> >>> >>> This is sort of like how iterators "return self" and 'it = iter(it)'. >>> >>> In the case of Dask, wrapping more delayed objects creates layers of >>> these lazy objects. But I think it has to because it's not part of the >>> syntax. Actually, I guess Dask could do graph reduction without actual >>> computation if it wanted to. But this is the current behavior: >>> >>> >>> def unchanged(x): >>> ... return x >>> >>> a = delayed(unchanged)(42) >>> >>> a >>> Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') >>> >>> b = delayed(unchanged)(a) >>> >>> c = delayed(unchanged)(b) >>> >>> c >>> Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') >>> >>> c.dask >>> {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': (>> __main__.unchanged>, >>> 42), >>> 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': (>> __main__.unchanged>, >>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), >>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': (>> __main__.unchanged>, >>> 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} >>> >>> >>> c.compute() >>> 42 >>> >>> >>> Actually Dask *cannot* know that "unchanged()" is the function that >>> makes no transformation on its one parameter. From what it can see, it's >>> just a function that does *something*. And I guess similarly in the >>> proposed syntax, anything other than a plain name after the 'delayed' would >>> still need to create a new delayed object. So it's all an edge case that >>> doesn't make much difference. >>> >>> -- >>> Keeping medicines from the bloodstreams of the sick; food >>> from the bellies of the hungry; books from the hands of the >>> uneducated; technology from the underdeveloped; and putting >>> advocates of freedom in prisons. Intellectual property is >>> to the 21st century what the slave trade was to the 16th. >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.mertz at gmail.com Fri Feb 17 04:14:47 2017 From: david.mertz at gmail.com (David Mertz) Date: Fri, 17 Feb 2017 01:14:47 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: Agreed. But there might be cases where something occurring at most one?at some unspecified time?is desirable behavior. In general though, I think avoiding side effects should be programming recommendations, not anything enforced. This model isn't really so different from what we do with asyncio and its "call soon" indeterminate order. On Feb 17, 2017 1:07 AM, "Joseph Jevnik" wrote: > Even with the new syntax I would highly discourage delaying a function > with observable side effects. It would make reasoning about the behavior of > the program very difficult and debugging becomes much harder. > > On Fri, Feb 17, 2017 at 3:31 AM, David Mertz wrote: > >> I had forgotten about Daisy! It's an interesting project too. The >> behavior of 'autodask()' is closer to what I'd want in new syntax than is >> plain dask.delayed(). I'm not sure of all the corners. But is definitely >> love to have it for expressions generally, not only pure functions. >> >> On Feb 17, 2017 12:03 AM, "Joseph Jevnik" wrote: >> >>> You can let dask "see" into the function by entering it and wrapping all >>> of the operations in `delayed`; this is how daisy[0] builds up large >>> compute graphs. In this case, you could "inline" the identity function and >>> the delayed object would flow through the function and the call to identity >>> never makes it into the task graph. >>> >>> [0] http://daisy-python.readthedocs.io/en/latest/appendix.html#d >>> aisy.autodask >>> >>> On Fri, Feb 17, 2017 at 2:26 AM, David Mertz wrote: >>> >>>> On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: >>>>> >>>>> This also means that a 'delayed' object needs to be idempotent. So >>>>> >>>>> x = delayed 2+2 >>>>> >>>>> y = delayed x >>>>> >>>>> z = delayed delayed delayed y >>>>> >>>>> >>>>> Wrapping more delays around an existing delayed object should probably >>>>> just keep the same object rather than "doubly delaying" it. If there is >>>>> some reason to create separate delayed objects that isn't occurring to me, >>>>> evaluating 'z' would still go through the multiple evaluation levels until >>>>> it got to a non-delayed value. >>>>> >>>> >>>> This is sort of like how iterators "return self" and 'it = iter(it)'. >>>> >>>> In the case of Dask, wrapping more delayed objects creates layers of >>>> these lazy objects. But I think it has to because it's not part of the >>>> syntax. Actually, I guess Dask could do graph reduction without actual >>>> computation if it wanted to. But this is the current behavior: >>>> >>>> >>> def unchanged(x): >>>> ... return x >>>> >>> a = delayed(unchanged)(42) >>>> >>> a >>>> Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') >>>> >>> b = delayed(unchanged)(a) >>>> >>> c = delayed(unchanged)(b) >>>> >>> c >>>> Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') >>>> >>> c.dask >>>> {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': (>>> __main__.unchanged>, >>>> 42), >>>> 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': (>>> __main__.unchanged>, >>>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), >>>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': (>>> __main__.unchanged>, >>>> 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} >>>> >>>> >>> c.compute() >>>> 42 >>>> >>>> >>>> Actually Dask *cannot* know that "unchanged()" is the function that >>>> makes no transformation on its one parameter. From what it can see, it's >>>> just a function that does *something*. And I guess similarly in the >>>> proposed syntax, anything other than a plain name after the 'delayed' would >>>> still need to create a new delayed object. So it's all an edge case that >>>> doesn't make much difference. >>>> >>>> -- >>>> Keeping medicines from the bloodstreams of the sick; food >>>> from the bellies of the hungry; books from the hands of the >>>> uneducated; technology from the underdeveloped; and putting >>>> advocates of freedom in prisons. Intellectual property is >>>> to the 21st century what the slave trade was to the 16th. >>>> >>> >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Fri Feb 17 04:31:29 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 17 Feb 2017 10:31:29 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: <3ed5edb4-2a6a-ccc4-7b8a-1489e0683e65@mail.de> On 17.02.2017 04:59, Chris Angelico wrote: > On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V wrote: >> Common use case: >> >> L = [1,3,5,7] >> >> for i over len(L): >> e = L[i] >> >> or: >> >> length = len(L) >> for i over length: >> e = L[i] > Better use case: > > for i, e in enumerate(L): I totally agree with Chris here. For me, there's no need for another for loop syntax. It would be better to learn Python idioms instead. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Fri Feb 17 04:32:48 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 17 Feb 2017 10:32:48 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: Hi all, If we want this it might be interesting to investigate what the Scheme community has been doing, since they have had this (under the name "promises") for many years. Basically: Scheme: (delay expr) <=> proposed Python: delayed: expr The Scheme community has experimented with what they call "auto-forcing", i.e. a promise can be given to any primitive operation and is then forced. However this has not caught on. Possibly for a good reason ;-) (My gut feeling: too much magic. Explicit is better than implicit.) Note that Racket/PLT Scheme has also "lazy" in addition to "delay". The rationale for this is given in: "How to add laziness to a strict language without even being odd", Philip Wadler, Walid Taha, David MacQueen https://www.researchgate.net/publication/2646969_How_to_Add_Laziness_to_a_Strict_Language_Without_Even_Being_Odd It would be good to read and consider this before we reinvent the square wheel ;-) Stephan 2017-02-17 10:14 GMT+01:00 David Mertz : > Agreed. But there might be cases where something occurring at most one?at > some unspecified time?is desirable behavior. In general though, I think > avoiding side effects should be programming recommendations, not anything > enforced. > > This model isn't really so different from what we do with asyncio and its > "call soon" indeterminate order. > > On Feb 17, 2017 1:07 AM, "Joseph Jevnik" wrote: > >> Even with the new syntax I would highly discourage delaying a function >> with observable side effects. It would make reasoning about the behavior of >> the program very difficult and debugging becomes much harder. >> >> On Fri, Feb 17, 2017 at 3:31 AM, David Mertz wrote: >> >>> I had forgotten about Daisy! It's an interesting project too. The >>> behavior of 'autodask()' is closer to what I'd want in new syntax than is >>> plain dask.delayed(). I'm not sure of all the corners. But is definitely >>> love to have it for expressions generally, not only pure functions. >>> >>> On Feb 17, 2017 12:03 AM, "Joseph Jevnik" wrote: >>> >>>> You can let dask "see" into the function by entering it and wrapping >>>> all of the operations in `delayed`; this is how daisy[0] builds up large >>>> compute graphs. In this case, you could "inline" the identity function and >>>> the delayed object would flow through the function and the call to identity >>>> never makes it into the task graph. >>>> >>>> [0] http://daisy-python.readthedocs.io/en/latest/appendix.html#d >>>> aisy.autodask >>>> >>>> On Fri, Feb 17, 2017 at 2:26 AM, David Mertz wrote: >>>> >>>>> On Thu, Feb 16, 2017 at 11:15 PM, David Mertz wrote: >>>>>> >>>>>> This also means that a 'delayed' object needs to be idempotent. So >>>>>> >>>>>> x = delayed 2+2 >>>>>> >>>>>> y = delayed x >>>>>> >>>>>> z = delayed delayed delayed y >>>>>> >>>>>> >>>>>> Wrapping more delays around an existing delayed object should >>>>>> probably just keep the same object rather than "doubly delaying" it. If >>>>>> there is some reason to create separate delayed objects that isn't >>>>>> occurring to me, evaluating 'z' would still go through the multiple >>>>>> evaluation levels until it got to a non-delayed value. >>>>>> >>>>> >>>>> This is sort of like how iterators "return self" and 'it = iter(it)'. >>>>> >>>>> In the case of Dask, wrapping more delayed objects creates layers of >>>>> these lazy objects. But I think it has to because it's not part of the >>>>> syntax. Actually, I guess Dask could do graph reduction without actual >>>>> computation if it wanted to. But this is the current behavior: >>>>> >>>>> >>> def unchanged(x): >>>>> ... return x >>>>> >>> a = delayed(unchanged)(42) >>>>> >>> a >>>>> Delayed('unchanged-1780fed6-f835-4c31-a86d-50015ae1449a') >>>>> >>> b = delayed(unchanged)(a) >>>>> >>> c = delayed(unchanged)(b) >>>>> >>> c >>>>> Delayed('unchanged-adc5e307-6e33-45bf-ad73-150b906e921d') >>>>> >>> c.dask >>>>> {'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a': (>>>> __main__.unchanged>, >>>>> 42), >>>>> 'unchanged-adc5e307-6e33-45bf-ad73-150b906e921d': (>>>> __main__.unchanged>, >>>>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0'), >>>>> 'unchanged-c3282bc4-bdaa-4148-8509-9155cac83ef0': (>>>> __main__.unchanged>, >>>>> 'unchanged-1780fed6-f835-4c31-a86d-50015ae1449a')} >>>>> >>>>> >>> c.compute() >>>>> 42 >>>>> >>>>> >>>>> Actually Dask *cannot* know that "unchanged()" is the function that >>>>> makes no transformation on its one parameter. From what it can see, it's >>>>> just a function that does *something*. And I guess similarly in the >>>>> proposed syntax, anything other than a plain name after the 'delayed' would >>>>> still need to create a new delayed object. So it's all an edge case that >>>>> doesn't make much difference. >>>>> >>>>> -- >>>>> Keeping medicines from the bloodstreams of the sick; food >>>>> from the bellies of the hungry; books from the hands of the >>>>> uneducated; technology from the underdeveloped; and putting >>>>> advocates of freedom in prisons. Intellectual property is >>>>> to the 21st century what the slave trade was to the 16th. >>>>> >>>> >>>> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Fri Feb 17 05:22:25 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 17 Feb 2017 11:22:25 +0100 Subject: [Python-ideas] Light-weight call-by-name syntax in Python Message-ID: Proposal: Light-weight call-by-name syntax in Python The following syntax a : b is to be interpreted as: a(lambda: b) Effectively, this gives a "light-weight macro system" to Python, since it allows with little syntax to indicate that the argument to a function is not to be immediately invoked. It is a generalization of special-case syntax proposals like delayed: In this proposal, `delayed' can be a normal callable. Motivating examples follow: # Logging # The following assumes the logging library has been extended to support # a callable argument in addition to a string. import logging logging.debug: "I have a %s" % expensive_function() # Spawn parallel tasks # This would work with the existing concurrent.futures unmodified. from concurrent.futures import ThreadPoolExecutor executor = ThreadPoolExecutor(max_workers=4) future = executor.submit: some_long_running_function(another_long_one()) # Asyncio loop.call_soon_threadsafe: do_something(42) # Custom asserts from some_module_implementing_contracts import precondition def foo(x): precondition: x >= 0 return math.sqrt(x) # Lazy evaluation class delayed: def __init__(self, callback): self.callback = callback self.value = None def __call__(self): if self.callback is not None: self.value = self.callback() self.callback = None return self.value promise = delayed: a() + b() print(promise()) # default dictionary dd = collections.defaultdict: MyDefaultObject(42) print(dd["foo"]) # => prints MyDefaultObject(42) Notes on syntax: a : b is my preferred syntax. It would conflict with existing use of : in a few places, e.g. in if and while. My preferred approach is that we would need to write (a : b) there. An alternative is a as-yet-unused token such as :: . Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Feb 17 05:28:36 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 17 Feb 2017 02:28:36 -0800 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 2:22 AM, Stephan Houben wrote: > Proposal: Light-weight call-by-name syntax in Python > > The following syntax > a : b > is to be interpreted as: > a(lambda: b) > > Effectively, this gives a "light-weight macro system" to Python, > since it allows with little syntax to indicate that the argument to > a function is not to be immediately invoked. > > It is a generalization of special-case syntax proposals like > delayed: > In this proposal, `delayed' can be a normal callable. Note that this is definitely a different proposal from the original, since the original proposer's goal was to be able to use this with existing, unmodified functions that expect a regular value, not a lambda. I don't really see how that goal can be accomplished without massively revising Python's runtime model, so this doesn't really bother me, but it should be noted :-). Anyway, you might also be interested in: https://mail.python.org/pipermail/python-ideas/2016-November/043590.html which is a similar idea and some more motivating examples, except that it allows for the full richness of Python's call syntax, and passes ASTs rather than lambdas to allow for non-standard evaluation rules. -n -- Nathaniel J. Smith -- https://vorpus.org From stephanh42 at gmail.com Fri Feb 17 05:46:00 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 17 Feb 2017 11:46:00 +0100 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: Hi Nathaniel, 2017-02-17 11:28 GMT+01:00 Nathaniel Smith : > > Note that this is definitely a different proposal from the original, > since the original proposer's goal was to be able to use this with > existing, unmodified functions that expect a regular value, not a > lambda. > > I don't really see how that goal can be accomplished without massively > revising Python's runtime model, so this doesn't really bother me, but > it should be noted :-). > Yep ;-) > > Anyway, you might also be interested in: > > https://mail.python.org/pipermail/python-ideas/2016- > November/043590.html > > which is a similar idea and some more motivating examples, except that > it allows for the full richness of Python's call syntax, and passes > ASTs rather than lambdas to allow for non-standard evaluation rules. > > Frankly, I think that proposal is too baroque. The simplicity of my proposal is intentional. Just passing in a callable makes it simpler and makes it also more likely that existing functions could be used with it. I have considered adding all kind of syntax for adding extra arguments to the wrapped callable, extra arguments for the invoked function, ... basically that just makes it a mess. If you want that, use the existing syntax. By the way, got a private e-mail mentioning that the proposed syntax is already claimed by type annotations. So therefore a :: b is now my proposed syntax. Stephan > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 17 06:10:05 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Feb 2017 22:10:05 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: <20170217111005.GK5689@ando.pearwood.info> On Fri, Feb 17, 2017 at 12:24:53AM -0500, Joseph Hackman wrote: > I propose a keyword to mark an expression for delayed/lazy execution, for > the purposes of standardizing such behavior across the language. > > The proposed format is: > delayed: > i.e. log.info("info is %s", delayed: expensiveFunction()) Keywords are difficult: since by definition they are not backwards compatible, they make it hard for people to write version independent code, and will break people's code. Especially something like "delayed", I expect that there is lots of code that used "delayed" as a regular name. if status.delayed: ... A new keyword means it can't be back-ported to older versions, and will break code. > Unlike 'lambda' which returns a function (so the receiver must be > lambda-aware), delayed execution blocks are for all purposes values. The > first time the value (rather than location) is read, What counts as "reading" a value? Based on your example below, I can't tell if passing the object to *any* function is enough to trigger evaluation, or specifically print is the magic that makes it happen. > or any method on the delayed object is called, I don't think that can work -- it would have to be any attribute access, surely, because Python couldn't tell if the attribute was a method or not until it evaluated the lazy object. Consider: spam = delayed: complex_calculation() a = spam.thingy What's `a` at this point? Is is still some sort of lazy object, waiting to be evaluated? If so, how is Python supposed to know if its a method? result = a() > the expression is executed and the delayed > expression is replaced with the result. (Thus, the delayed expression is > only every evaluated once). That's easily done by having the "delayed" keyword cache each expression it sees, but that seems like a bad idea to me: spam = delayed: get_random_string() eggs = delayed: get_random_string() # the same expression spam.upper() # convert to a real value assert spam == eggs # always true, as they are the same expression Worse, suppose module a.py has: spam = delayed: calculate(1) and module b.py has: eggs = delayed: calculate(1) where a.calculate and b.calculate do completely different things. The result you get will depend on which happens to be evaluated first and cached, and would be a nightmare to debug. Truely spooky action-at-a- distance code. I think it is better to stick to a more straight-forward, easily understood and debugged system based on object identity rather than expressions. > Ideally: > a = delayed: 1+2 > b = a > print(a) #adds 1 and 2, prints 3 > # a and b are now both just 3 > print(b) #just prints 3 That would work based on object identity too. By the way, that's probably not the best example, because the keyhole optimizer will likely have compiled 1+2 as just 3, so you're effectively writing: a = delayed: 3 At least, that's what I would want: I would argue strongly against lazy objects somehow defeating the keyhole optimizer. If I write: a = delayed: complex_calculation(1+2+3, 4.5/3, 'abcd'*3) what I hope will be compiled is: a = delayed: complex_calculation(6, 0.6428571428571429, 'abcdabcdabcd') same as it would be now (at least in CPython), apart from the "delayed:" keyword. a = delayed: complex_calculation(1+2+3, 4.5/3, 'abcd'*3) > Mechanically, this would be similar to the following: > > class Delayed(): > def __init__(self, func): > self.__func = func > self.__executed = False > self.__value = None > > def __str__(self): > if self.__executed: > return self.__value.__str__() > self.__value = self.__func() > self.__executed = True > return self.__value.__str__() So you're suggesting that calling str(delayed_object) is the one way to force evaluation? I have no idea what the following code is supposed to mean. > def function_print(value): > print('function_print') > print(value) > > def function_return_stuff(value): > print('function_return_stuff') > return value > > function_print(function_return_stuff('no_delay')) > > function_print(Delayed(lambda: function_return_stuff('delayed'))) > > delayed = Delayed(lambda: function_return_stuff('delayed_object')) > function_print(delayed) > function_print(delayed) -- Steve From rosuav at gmail.com Fri Feb 17 06:21:07 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 17 Feb 2017 22:21:07 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <20170217111005.GK5689@ando.pearwood.info> References: <20170217111005.GK5689@ando.pearwood.info> Message-ID: On Fri, Feb 17, 2017 at 10:10 PM, Steven D'Aprano wrote: >> the expression is executed and the delayed >> expression is replaced with the result. (Thus, the delayed expression is >> only every evaluated once). > > That's easily done by having the "delayed" keyword cache each expression > it sees, but that seems like a bad idea to me: > > spam = delayed: get_random_string() > eggs = delayed: get_random_string() # the same expression > > spam.upper() # convert to a real value > > assert spam == eggs # always true, as they are the same expression > > Worse, suppose module a.py has: > > spam = delayed: calculate(1) > > and module b.py has: > > eggs = delayed: calculate(1) > > where a.calculate and b.calculate do completely different things. The > result you get will depend on which happens to be evaluated first and > cached, and would be a nightmare to debug. Truely spooky action-at-a- > distance code. My understanding is that a single 'delayed expression' will be evaluated at most once. It's more like this: spam = delayed: get_random_string() eggs = spam spam.upper() # At this point, the object becomes a real value assert spam is eggs # always true, as they are the same object Two instances of "delayed:" will create two distinct delayed expressions. The big question, though, is what triggers evaluation. AIUI, merely referencing the object doesn't (so "eggs = spam" won't force evaluation), but I'm not sure what does. Do delayed-expressions have identities or only values? For example: rand = delayed: random.randrange(10) otherrand = rand assert rand is otherrand # legal? randid = id(rand) # legal? print(rand) # force to concrete value assert any(rand is x for x in range(10)) # CPython int caching assert randid == id(rand) # now what? Alternatively, once the value becomes concrete, the delayed-expression becomes a trampoline/proxy to the actual value. Its identity remains unchanged, but all attribute lookups would be passed on to the other object. That does mean a permanent performance penalty though - particularly if it's doing it all in Python code rather than some sort of quick C bouncer. ChrisA From josephhackman at gmail.com Fri Feb 17 10:12:12 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 10:12:12 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> Message-ID: <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> A few points for clarity: Yes, I would expect each instance of delayed to result in a new delayed expression, without caching, except for multiple calls to that same delayed expression instance. Also, I suggested the colon : because unlike async/await, the following expression is NOT executed at the time delayed is executed. The only other ways to do this on Python I know of are def and lambda, both of which use colons to highlight this. As for what triggers execution? I think everything except being on the right side of an assignment. Even identity. So if a delayed expression would evaluate to None, then code that checks is None should return true. I think this is important to ensure that no code needs to be changed to support this feature. So for Chris: yes, rand is otherrand not because the delayed instances are the same, but because the value under them is the same, and the is should trigger evaluation. Otherwise code would need to be delayed aware. Finally as for the word delayed. Yes, it could be a symbol, or existing keyword, but I have no suggestions on that front. -Joseph > On Feb 17, 2017, at 6:21 AM, Chris Angelico wrote: > > On Fri, Feb 17, 2017 at 10:10 PM, Steven D'Aprano wrote: >>> the expression is executed and the delayed >>> expression is replaced with the result. (Thus, the delayed expression is >>> only every evaluated once). >> >> That's easily done by having the "delayed" keyword cache each expression >> it sees, but that seems like a bad idea to me: >> >> spam = delayed: get_random_string() >> eggs = delayed: get_random_string() # the same expression >> >> spam.upper() # convert to a real value >> >> assert spam == eggs # always true, as they are the same expression >> >> Worse, suppose module a.py has: >> >> spam = delayed: calculate(1) >> >> and module b.py has: >> >> eggs = delayed: calculate(1) >> >> where a.calculate and b.calculate do completely different things. The >> result you get will depend on which happens to be evaluated first and >> cached, and would be a nightmare to debug. Truely spooky action-at-a- >> distance code. > > My understanding is that a single 'delayed expression' will be > evaluated at most once. It's more like this: > > spam = delayed: get_random_string() > eggs = spam > > spam.upper() # At this point, the object becomes a real value > assert spam is eggs # always true, as they are the same object > > Two instances of "delayed:" will create two distinct delayed expressions. > > The big question, though, is what triggers evaluation. AIUI, merely > referencing the object doesn't (so "eggs = spam" won't force > evaluation), but I'm not sure what does. > > Do delayed-expressions have identities or only values? For example: > > rand = delayed: random.randrange(10) > otherrand = rand > assert rand is otherrand # legal? > randid = id(rand) # legal? > print(rand) # force to concrete value > assert any(rand is x for x in range(10)) # CPython int caching > assert randid == id(rand) # now what? > > Alternatively, once the value becomes concrete, the delayed-expression > becomes a trampoline/proxy to the actual value. Its identity remains > unchanged, but all attribute lookups would be passed on to the other > object. That does mean a permanent performance penalty though - > particularly if it's doing it all in Python code rather than some sort > of quick C bouncer. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From pavol.lisy at gmail.com Fri Feb 17 10:02:28 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Fri, 17 Feb 2017 16:02:28 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> Message-ID: On 2/17/17, Chris Angelico wrote: > Do delayed-expressions have identities or only values? For example: > > rand = delayed: random.randrange(10) > otherrand = rand > assert rand is otherrand # legal? > randid = id(rand) # legal? > print(rand) # force to concrete value > assert any(rand is x for x in range(10)) # CPython int caching > assert randid == id(rand) # now what? > > Alternatively, once the value becomes concrete, the delayed-expression > becomes a trampoline/proxy to the actual value. Its identity remains > unchanged, but all attribute lookups would be passed on to the other > object. That does mean a permanent performance penalty though - > particularly if it's doing it all in Python code rather than some sort > of quick C bouncer. what about: lazy_string = delayed: f"{fnc()}" could it be possible? BTW I was also thinking about something like l-strings (aka lazy f-string) to solve this problem: logger.debug("format {expensive()}") Still not sure if it is a good idea but with something like this: class Lazy_String: ''' this is only quick and dirty test implementation! ''' def __init__(self, string=None): if string is None: self.body = "'None'" else: self.body = compile('f'+repr(string), "", 'eval') def __sub__(self, string): self.__init__(string) return self def __str__(self): return eval(self.body) def __repr__(self): return self.__str__() lazy = Lazy_String() we could use this: from lazy_string import lazy as l logger.debug(l-"format {expensive()}") # kind of l-string without new syntax From rosuav at gmail.com Fri Feb 17 10:45:16 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Feb 2017 02:45:16 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> Message-ID: On Sat, Feb 18, 2017 at 2:12 AM, Joseph Hackman wrote: > As for what triggers execution? I think everything except being on the right side of an assignment. Even identity. So if a delayed expression would evaluate to None, then code that checks is None should return true. I think this is important to ensure that no code needs to be changed to support this feature. > > So for Chris: yes, rand is otherrand not because the delayed instances are the same, but because the value under them is the same, and the is should trigger evaluation. Otherwise code would need to be delayed aware. > Interesting. Okay. So in effect, these things aren't objects, they're magic constructs that turn into objects the moment you do anything with them, even an identity check. That makes sense. Can you put deferred objects into collections, or will they instantly collapse into concrete ones? ChrisA From josephhackman at gmail.com Fri Feb 17 11:29:58 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 11:29:58 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> Message-ID: <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Pavol: I think that some sort of magic string that is not a string and is actually containing Python code could function, but is less elegant. ChrisA: I am not sure about collections. I think it may be fine to not special case it: if the act of putting it in the collection reads anything, then it is evaluated, and if it doesn't it isn't. The ideal design goal for this would be that all existing code continues to function as if the change wasn't made at all, except that the value is evaluated at a different time. -Joseph > On Feb 17, 2017, at 10:45 AM, Chris Angelico wrote: > >> On Sat, Feb 18, 2017 at 2:12 AM, Joseph Hackman wrote: >> As for what triggers execution? I think everything except being on the right side of an assignment. Even identity. So if a delayed expression would evaluate to None, then code that checks is None should return true. I think this is important to ensure that no code needs to be changed to support this feature. >> >> So for Chris: yes, rand is otherrand not because the delayed instances are the same, but because the value under them is the same, and the is should trigger evaluation. Otherwise code would need to be delayed aware. >> > > Interesting. Okay. So in effect, these things aren't objects, they're > magic constructs that turn into objects the moment you do anything > with them, even an identity check. That makes sense. > > Can you put deferred objects into collections, or will they instantly > collapse into concrete ones? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mikhailwas at gmail.com Fri Feb 17 11:30:53 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 17 Feb 2017 17:30:53 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 17 February 2017 at 04:59, Chris Angelico wrote: > On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V wrote: > > Common use case: > > > > L = [1,3,5,7] > > > > for i over len(L): > > e = L[i] > > > > or: > > > > length = len(L) > > for i over length: > > e = L[i] > > Better use case: > > for i, e in enumerate(L): > > This would be more compact, yet less readable, more error prone variant. I'd avoid it it all costs and even if I don't need the index further in loop body, (which happens rarely in my experience) I write e=L[i] in second line to make the code more verbose and keep the flow order. So your variant (and those proposed in PEP-212) could serve in list comprehensions for example, but for common 'expanded' code I find it decline of readability, and creating totally different variants for same iteration idea. But partially that could be simply matter of habit and love to contractions. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 17 11:34:15 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Feb 2017 03:34:15 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman wrote: > ChrisA: I am not sure about collections. I think it may be fine to not special case it: if the act of putting it in the collection reads anything, then it is evaluated, and if it doesn't it isn't. The ideal design goal for this would be that all existing code continues to function as if the change wasn't made at all, except that the value is evaluated at a different time. > Yeah, I'm just worried that it'll become useless without that. For instance, passing arguments to a function that uses *a,**kw is going to package your thunk into a collection, and that's how (eg) the logging module will process it. It's not going to be easy to have a simple AND useful definition of "this collapses the waveform, that keeps it in a quantum state", but sorting that out is fairly key to the proposal. ChrisA From rosuav at gmail.com Fri Feb 17 11:37:12 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Feb 2017 03:37:12 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On Sat, Feb 18, 2017 at 3:30 AM, Mikhail V wrote: > On 17 February 2017 at 04:59, Chris Angelico wrote: >> >> On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V wrote: >> > Common use case: >> > >> > L = [1,3,5,7] >> > >> > for i over len(L): >> > e = L[i] >> > >> > or: >> > >> > length = len(L) >> > for i over length: >> > e = L[i] >> >> Better use case: >> >> for i, e in enumerate(L): >> > > This would be more compact, yet less readable, more error prone variant. > I'd avoid it it all costs and even if I don't need the index further in loop > body, > (which happens rarely in my experience) I write e=L[i] in second line > to make the code more verbose and keep the flow order. > So your variant (and those proposed in PEP-212) could serve in list > comprehensions for example, but for common 'expanded' code I find it > decline of readability, and creating totally different variants for same > iteration idea. > But partially that could be simply matter of habit and love to contractions. If you don't need the index, why not just iterate over the list directly? for e in L: That's the single most obvious way to step through a collection in Python. What do you need to count up to the length for? ChrisA From mehaase at gmail.com Fri Feb 17 11:37:04 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Fri, 17 Feb 2017 11:37:04 -0500 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) Message-ID: In the debug logging thread, somebody suggested the "%r" format specifier and a custom __repr__. This is a neat solution because Python logging already includes a "delayed" evaluation of sorts: it formats the logging string *after* it determines that the message's log level is greater than or equal to the logger's level. However, wrapping up every expensive debugging computation in a new class might get tedious, so here's a strawman proposal for something lighter weight. Python has two string formatting mini-languages. Both allow formatting flags, for example in "%03d", the "0" (zero) is a flag that means to pad with leading zeroes. I propose to add a string format flag to both mini-languages that explicitly says, "this argument is callable, and its *return value* should be formatted." In current Python: >>> expensive = lambda: 2 >>> logging.debug('%03d %03d', 1, expensive()) In this case, the "expensive" code is executed even though nothing is logged. >>> expensive = lambda: 2 >>> logging.root.setLevel(logging.DEBUG) >>> logging.debug('%03d %03d', 1, expensive()) DEBUG:root:001 002 With a different log level, the expensive code is executed and the message is logged. The suggested change is to add a "C" flag to the formatting language that indicates an argument is callable. When formatting the string, the argument will be called and its result will be used for the formatting operation. The C flag must be handled before any other flags, regardless of the order in which they are specified. The callable's return value is then formatted according to the rest of the specifier. With this change, the above example can now be written as: >>> expensive = lambda: 2 >>> logging.debug('%03d %C03d', 1, expensive) The "expensive" code is not executed at this log level. >>> expensive = lambda: 2 >>> logging.root.setLevel(logging.DEBUG) >>> logging.warn('%03d %C03d', 1, expensive) WARNING:root:001 002 The expensive code is *only* executed when the message will be logged. The callable's return value is formatted with the specifier '%03d', i.e. the same specifier as the callable but without the "C" flag. Pros: 1) Much smaller change to the language. 2) Simplifies a common (?) problem with logging. 3) Applies to all string formatting, not just logging. (But I'm not sure what the other use cases are.) Cons: 1) Doesn't generalize as well as the various "delayed" proposals. (But I'm not sure what other use cases "delayed" has.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 11:43:31 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 11:43:31 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Agreed. I think this may require some TLC to get right, but posting here for feedback on the idea overall seemed like a good start. As far as I know, the basic list and dict do not inspect what they contain. I.e. d = {} d['a']= delayed: stuff() b=d['a'] b would end up as still the thunk, and stuff wouldn't be executed until either d['a'] or b actually is read from. -Joseph > On Feb 17, 2017, at 11:34 AM, Chris Angelico wrote: > >> On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman wrote: >> ChrisA: I am not sure about collections. I think it may be fine to not special case it: if the act of putting it in the collection reads anything, then it is evaluated, and if it doesn't it isn't. The ideal design goal for this would be that all existing code continues to function as if the change wasn't made at all, except that the value is evaluated at a different time. >> > > Yeah, I'm just worried that it'll become useless without that. For > instance, passing arguments to a function that uses *a,**kw is going > to package your thunk into a collection, and that's how (eg) the > logging module will process it. > > It's not going to be easy to have a simple AND useful definition of > "this collapses the waveform, that keeps it in a quantum state", but > sorting that out is fairly key to the proposal. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abedillon at gmail.com Fri Feb 17 11:57:14 2017 From: abedillon at gmail.com (Abe Dillon) Date: Fri, 17 Feb 2017 10:57:14 -0600 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I'd like to suggest a shorter keyword: `lazy` This isn't an endorsement. I haven't had time to digest how big this change would be. If this is implemented, I'd also like to suggest that perhaps packing and unpacking should be delayed by default and not evaluated until the contents are used. It might save on many pesky edge cases that would evaluate your expression unnecessarily. On Fri, Feb 17, 2017 at 10:43 AM, Joseph Hackman wrote: > Agreed. I think this may require some TLC to get right, but posting here > for feedback on the idea overall seemed like a good start. As far as I > know, the basic list and dict do not inspect what they contain. I.e. > > d = {} > d['a']= delayed: stuff() > b=d['a'] > > b would end up as still the thunk, and stuff wouldn't be executed until > either d['a'] or b actually is read from. > > -Joseph > > > On Feb 17, 2017, at 11:34 AM, Chris Angelico wrote: > > > >> On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman < > josephhackman at gmail.com> wrote: > >> ChrisA: I am not sure about collections. I think it may be fine to not > special case it: if the act of putting it in the collection reads anything, > then it is evaluated, and if it doesn't it isn't. The ideal design goal for > this would be that all existing code continues to function as if the change > wasn't made at all, except that the value is evaluated at a different time. > >> > > > > Yeah, I'm just worried that it'll become useless without that. For > > instance, passing arguments to a function that uses *a,**kw is going > > to package your thunk into a collection, and that's how (eg) the > > logging module will process it. > > > > It's not going to be easy to have a simple AND useful definition of > > "this collapses the waveform, that keeps it in a quantum state", but > > sorting that out is fairly key to the proposal. > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Fri Feb 17 12:12:08 2017 From: abedillon at gmail.com (Abe Dillon) Date: Fri, 17 Feb 2017 11:12:08 -0600 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Actually, following from the idea that packing and unpacking variables should be delayed by default, it might make sense to use syntax like: >>> a = *(2+2) >>> b = a + 1 Instead of >>> a = lazy 2+2 # or whatever you want the keyword to be >>> b = a + 1 That syntax sort-of resembles generator expressions, however; I usually like how python favors actual words over obscure symbol combinations for readability's sake. On Fri, Feb 17, 2017 at 10:57 AM, Abe Dillon wrote: > I'd like to suggest a shorter keyword: `lazy` > > This isn't an endorsement. I haven't had time to digest how big this > change would be. > > If this is implemented, I'd also like to suggest that perhaps packing and > unpacking should be delayed by default and not evaluated until the contents > are used. It might save on many pesky edge cases that would evaluate your > expression unnecessarily. > > On Fri, Feb 17, 2017 at 10:43 AM, Joseph Hackman > wrote: > >> Agreed. I think this may require some TLC to get right, but posting here >> for feedback on the idea overall seemed like a good start. As far as I >> know, the basic list and dict do not inspect what they contain. I.e. >> >> d = {} >> d['a']= delayed: stuff() >> b=d['a'] >> >> b would end up as still the thunk, and stuff wouldn't be executed until >> either d['a'] or b actually is read from. >> >> -Joseph >> >> > On Feb 17, 2017, at 11:34 AM, Chris Angelico wrote: >> > >> >> On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman < >> josephhackman at gmail.com> wrote: >> >> ChrisA: I am not sure about collections. I think it may be fine to not >> special case it: if the act of putting it in the collection reads anything, >> then it is evaluated, and if it doesn't it isn't. The ideal design goal for >> this would be that all existing code continues to function as if the change >> wasn't made at all, except that the value is evaluated at a different time. >> >> >> > >> > Yeah, I'm just worried that it'll become useless without that. For >> > instance, passing arguments to a function that uses *a,**kw is going >> > to package your thunk into a collection, and that's how (eg) the >> > logging module will process it. >> > >> > It's not going to be easy to have a simple AND useful definition of >> > "this collapses the waveform, that keeps it in a quantum state", but >> > sorting that out is fairly key to the proposal. >> > >> > ChrisA >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Fri Feb 17 12:31:19 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 17 Feb 2017 18:31:19 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 17 February 2017 at 17:37, Chris Angelico wrote: > On Sat, Feb 18, 2017 at 3:30 AM, Mikhail V wrote: > > On 17 February 2017 at 04:59, Chris Angelico wrote: > >> > >> On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V > wrote: > >> > Common use case: > >> > > >> > L = [1,3,5,7] > >> > > >> > for i over len(L): > >> > e = L[i] > >> > > >> > or: > >> > > >> > length = len(L) > >> > for i over length: > >> > e = L[i] > >> > >> Better use case: > >> > >> for i, e in enumerate(L): > >> > > > > This would be more compact, yet less readable, more error prone variant. > > I'd avoid it it all costs and even if I don't need the index further in > loop > > body, > > (which happens rarely in my experience) I write e=L[i] in second line > > to make the code more verbose and keep the flow order. > > So your variant (and those proposed in PEP-212) could serve in list > > comprehensions for example, but for common 'expanded' code I find it > > decline of readability, and creating totally different variants for same > > iteration idea. > > But partially that could be simply matter of habit and love to > contractions. > > If you don't need the index, why not just iterate over the list directly? > > for e in L: > > That's the single most obvious way to step through a collection in > Python. What do you need to count up to the length for? > > I have said I need the index, probably you've misread my last comment. Further more I explained why I think iteration over index should be the preferred way, it help with readability a lot. All my learning years ended up with rewriting most code to "for i in range()" and I slap myself when I start to write "for e in L". It is exactly where TOOWTDI applies perfectly and it is integer iteration for me. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 17 12:40:07 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Feb 2017 04:40:07 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On Sat, Feb 18, 2017 at 4:31 AM, Mikhail V wrote: > I have said I need the index, probably you've misread my last comment. > Further more I explained why I think iteration over index should be the > preferred way, it help with readability a lot. Further discussion probably should be redirected to python-list, but I'll elaborate here to explain why I do not support your proposal. You have a shelf of books. I ask you to go through the books and read their titles out. How do you do it? * Point your finger to the first book - probably the leftmost one, but you might use some other order. * Read out the title of that book. * Point your finger to the next book. * Read out that book's title. * Continue until you reach the end of the shelf. This corresponds to this code: for book in books: print(book.title) Your style of iteration would be more like this: * See how many books there are * Locate the first book on the shelf * Read out the title of book #1 * Locate the second book on the shelf * Read out book title #2 * Continue until you've read as many titles as you originally counted books. This is the code: for i in range(len(books)): print(books[i].title) Which of these real-world algorithms is more natural? The only reason the iteration counter feels more readable to you is that you're accustomed to it. It's not an objective measure, it's an entirely subjective one. That's not to say that it's unimportant for that reason, but your line of argument must be "this feels more comfortable TO ME", rather than "this is more readable". Further elaboration on python-list, as mentioned above. ChrisA From abedillon at gmail.com Fri Feb 17 13:19:23 2017 From: abedillon at gmail.com (Abe Dillon) Date: Fri, 17 Feb 2017 12:19:23 -0600 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: > > Further more I explained why I think iteration over index should be the > preferred way, it help with readability a lot. I think you'll find this statement at odds with most of the Python community, especially Guido. I find looping over objects in a collection is what you want to do 90% of the time so adding an extra step just adds noise. I find that most people who prefer looping over indexes do so because they are more familiar with languages like C, not because it is inherently more readable. Being more comfortable with a worse way of doing something isn't a good justification for changing the language. On Fri, Feb 17, 2017 at 11:31 AM, Mikhail V wrote: > > On 17 February 2017 at 17:37, Chris Angelico wrote: > >> On Sat, Feb 18, 2017 at 3:30 AM, Mikhail V wrote: >> > On 17 February 2017 at 04:59, Chris Angelico wrote: >> >> >> >> On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V >> wrote: >> >> > Common use case: >> >> > >> >> > L = [1,3,5,7] >> >> > >> >> > for i over len(L): >> >> > e = L[i] >> >> > >> >> > or: >> >> > >> >> > length = len(L) >> >> > for i over length: >> >> > e = L[i] >> >> >> >> Better use case: >> >> >> >> for i, e in enumerate(L): >> >> >> > >> > This would be more compact, yet less readable, more error prone variant. >> > I'd avoid it it all costs and even if I don't need the index further in >> loop >> > body, >> > (which happens rarely in my experience) I write e=L[i] in second line >> > to make the code more verbose and keep the flow order. >> > So your variant (and those proposed in PEP-212) could serve in list >> > comprehensions for example, but for common 'expanded' code I find it >> > decline of readability, and creating totally different variants for same >> > iteration idea. >> > But partially that could be simply matter of habit and love to >> contractions. >> >> If you don't need the index, why not just iterate over the list directly? >> >> for e in L: >> >> That's the single most obvious way to step through a collection in >> Python. What do you need to count up to the length for? >> >> > I have said I need the index, probably you've misread my last comment. > Further more I explained why I think iteration over index should be the > preferred way, it help with readability a lot. > All my learning years ended up with rewriting most code to "for i in > range()" > and I slap myself when I start to write "for e in L". > It is exactly where TOOWTDI applies perfectly and it is integer iteration > for me. > > Mikhail > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Feb 17 13:41:58 2017 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 17 Feb 2017 18:41:58 +0000 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) In-Reply-To: References: Message-ID: <39ff2760-3bd2-d1a0-fa83-7e7a2b89d9da@mrabarnett.plus.com> On 2017-02-17 16:37, Mark E. Haase wrote: > In the debug logging thread, somebody suggested the "%r" format > specifier and a custom __repr__. This is a neat solution because Python > logging already includes a "delayed" evaluation of sorts: it formats the > logging string *after* it determines that the message's log level is > greater than or equal to the logger's level. However, wrapping up every > expensive debugging computation in a new class might get tedious, so > here's a strawman proposal for something lighter weight. > > Python has two string formatting mini-languages. Both allow formatting > flags, for example in "%03d", the "0" (zero) is a flag that means to pad > with leading zeroes. I propose to add a string format flag to both > mini-languages that explicitly says, "this argument is callable, and its > *return value* should be formatted." > > In current Python: > > >>> expensive = lambda: 2 > >>> logging.debug('%03d %03d', 1, expensive()) > > In this case, the "expensive" code is executed even though nothing is > logged. > > >>> expensive = lambda: 2 > >>> logging.root.setLevel(logging.DEBUG) > >>> logging.debug('%03d %03d', 1, expensive()) > DEBUG:root:001 002 > > With a different log level, the expensive code is executed and the > message is logged. > > The suggested change is to add a "C" flag to the formatting language > that indicates an argument is callable. When formatting the string, the > argument will be called and its result will be used for the formatting > operation. The C flag must be handled before any other flags, regardless > of the order in which they are specified. The callable's return value is > then formatted according to the rest of the specifier. > A letter usually marks the end of the format, e.g. '%03d', so '%C03d' could be read as a format '%C' (currently undefined) followed by '03d'. I'd suggest using some punctuation character instead. It would also be a good idea for the 'new-style' format strings to use the same character. > With this change, the above example can now be written as: > > >>> expensive = lambda: 2 > >>> logging.debug('%03d %C03d', 1, expensive) > > The "expensive" code is not executed at this log level. > > >>> expensive = lambda: 2 > >>> logging.root.setLevel(logging.DEBUG) > >>> logging.warn('%03d %C03d', 1, expensive) > WARNING:root:001 002 > > The expensive code is *only* executed when the message will be logged. > The callable's return value is formatted with the specifier '%03d', i.e. > the same specifier as the callable but without the "C" flag. > > > > Pros: > > 1) Much smaller change to the language. > 2) Simplifies a common (?) problem with logging. > 3) Applies to all string formatting, not just logging. (But I'm not sure > what the other use cases are.) > > Cons: > > 1) Doesn't generalize as well as the various "delayed" proposals. (But > I'm not sure what other use cases "delayed" has.) > From gugurehermann at gmail.com Fri Feb 17 13:43:13 2017 From: gugurehermann at gmail.com (Gustavo Rehermann) Date: Fri, 17 Feb 2017 16:43:13 -0200 Subject: [Python-ideas] Fwd: Detecting Attribute Updates on Objects In-Reply-To: References: Message-ID: Whoops, I forgot to subscribe before. Seems like now I'm able to send this idea, so let's try again :) ================ According to this idea, the interpreter calls a function on an object O everytime an attribute A of it changes, recursing through objects that contain O as attribute as well. Avoiding loops too (refer to P3 below). Explanation: P1. When setting any property of an object inside a property, the interpreter should also call "__update_attr__(self, attr_name, old_item, new_item)" in the object that contains it. This automatically calls the same thing on every object that contains this one too, until a loop is hit (see P3), which makes it possible for a perfect dict synchronization with a file or stuff like that. This should also work for lists and any kind of object that contains attributes; so for every object O that contains A as attribute, whenever updating any of A's attributse call __update_attr__() or something on O. P2. From here on, refer as parents to any objects which contain the current object as attribute. P3. Python internally stores a cache - a tree of references - to speed up the process. if __update_attr__ finds a loop then it stops at the top container and instead sends it a call __update_loop__(). Whenever either or both __update_attr__ or __update_loop__() functions are overwritten by a non-function, the interpreter skips that call in the tree (see P5). P4. To save on performance, the interpreter first checks the __detect_update__ attribute of the class (it is a single-byte int/ctypes char, and is used as a bit mask): if bit 1 is OFF, then proceed the calls; otherwise, skip this and go to the parents in the tree (or don't call __update_loop__ at all if the call was about an update loop); if bit 2 is ON, then don't iterate on the parent; otherwise proceed to every parents as normal. P5. Bit 3 is only accessible from an interpreter (any attempt to change it will have the interpreter change it back to what it was before), and when ON forces the interpreter to skip this object or stop iterating, depending on Bit 2. This bit has precedence over Bit 1 and is ON always if __update_attr__ is a non-function. Other bits are ignored. P6. By default __detect_update__ is b10 on both bytes, or b00 if the function was overwritten by another function. Thank you, Gustavo Ramos "Gustavo6046" Rehermann -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 17 13:55:22 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 17 Feb 2017 18:55:22 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I did some quick thinking and a bit of research about some aspects of this proposal: There are a number of keyword options (delay, defer, lazy, delayed, deferred, etc.), a quick look through github says that of these, "deferred" seems to be the least used, but it still comes up quite a lot (350K times, lazy appears over 2 million). That's unfortunate, but I'm wondering how common async and await were when that was proposed and accepted? Another potential pitfall I'm realizing now (and apologies if this is unnecessary bikeshedding) is that this may require another keyword. If I understand the proposal correctly, at this point the suggestion is that `delayed ` delays the evaluation of EXPR until it is requested in a non-delayed context. That is >>> x = delayed 1 + 2 >>> y = delayed 3 + 4 >>> z = delayed x + y # neither x nor y has been evaluated yet >>> z 10 This works fine for simple cases, but there might be situations where someone wants to control evaluation a bit more. As soon as we add any kind of state things get icky: # assuming Foo is a class an attribute `value` and a method # `inc` that increments the value and returns the current value >>> foo = Foo(value=0) >>> x = delayed sum([foo.inc(), foo.inc()]) >>> foo.inc() 1 >>> x 5 However, assuming this is a real, more complex system, we might want a way to have one or both of those inc calls be eagerly evaluated, which requires a way of signalling that in some way: >>> foo = Foo(value=0) >>> x = delayed sum([foo.inc(), eager foo.inc()]) >>> foo.inc() 2 >>> foo.inc() 4 Perhaps luckily, `eager` is less commonly used than even `delayed`, but still 2 keywords is an even higher bar. I guess another alternative would be to require annotating all subexpressions with `delayed`, but then that turns the above into >>> foo = Foo(value=0) >>> x = delayed sum([delayed foo.inc(), delayed foo.inc()]) >>> foo.inc() 1 >>> x 5 At which point `delayed` would need to be a much shorter keyword (the heathen in me says overload `del`). --Josh On Fri, Feb 17, 2017 at 12:13 PM Abe Dillon wrote: > Actually, following from the idea that packing and unpacking variables > should be delayed by default, it might make sense to use syntax like: > > >>> a = *(2+2) > >>> b = a + 1 > > Instead of > > >>> a = lazy 2+2 # or whatever you want the keyword to be > >>> b = a + 1 > > That syntax sort-of resembles generator expressions, however; I usually > like how python favors actual words over obscure symbol combinations for > readability's sake. > > On Fri, Feb 17, 2017 at 10:57 AM, Abe Dillon wrote: > > I'd like to suggest a shorter keyword: `lazy` > > This isn't an endorsement. I haven't had time to digest how big this > change would be. > > If this is implemented, I'd also like to suggest that perhaps packing and > unpacking should be delayed by default and not evaluated until the contents > are used. It might save on many pesky edge cases that would evaluate your > expression unnecessarily. > > On Fri, Feb 17, 2017 at 10:43 AM, Joseph Hackman > wrote: > > Agreed. I think this may require some TLC to get right, but posting here > for feedback on the idea overall seemed like a good start. As far as I > know, the basic list and dict do not inspect what they contain. I.e. > > d = {} > d['a']= delayed: stuff() > b=d['a'] > > b would end up as still the thunk, and stuff wouldn't be executed until > either d['a'] or b actually is read from. > > -Joseph > > > On Feb 17, 2017, at 11:34 AM, Chris Angelico wrote: > > > >> On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman < > josephhackman at gmail.com> wrote: > >> ChrisA: I am not sure about collections. I think it may be fine to not > special case it: if the act of putting it in the collection reads anything, > then it is evaluated, and if it doesn't it isn't. The ideal design goal for > this would be that all existing code continues to function as if the change > wasn't made at all, except that the value is evaluated at a different time. > >> > > > > Yeah, I'm just worried that it'll become useless without that. For > > instance, passing arguments to a function that uses *a,**kw is going > > to package your thunk into a collection, and that's how (eg) the > > logging module will process it. > > > > It's not going to be easy to have a simple AND useful definition of > > "this collapses the waveform, that keeps it in a quantum state", but > > sorting that out is fairly key to the proposal. > > > > ChrisA > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 14:13:04 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 14:13:04 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Hey! Excellent feedback! In my mind, which word is selected doesn't matter much to me. I think the technical term is 'thunk'? I think delayed is most clear. I'm not sure if eager execution is so common in this framework it needs its own keyword. Notably, default Python will handle that case x= Delayed: sum((foo.inc(), foo.inc())) Can become y=foo.inc() x= delayed: sum((foo.inc(), y)) It's less sugary, but seems ok to me. How does that seem? -Joseph > On Feb 17, 2017, at 1:55 PM, Joshua Morton wrote: > > I did some quick thinking and a bit of research about some aspects of this proposal: > > There are a number of keyword options (delay, defer, lazy, delayed, deferred, etc.), a quick look through github says that of these, "deferred" seems to be the least used, but it still comes up quite a lot (350K times, lazy appears over 2 million). That's unfortunate, but I'm wondering how common async and await were when that was proposed and accepted? > > Another potential pitfall I'm realizing now (and apologies if this is unnecessary bikeshedding) is that this may require another keyword. If I understand the proposal correctly, at this point the suggestion is that `delayed ` delays the evaluation of EXPR until it is requested in a non-delayed context. That is > > >>> x = delayed 1 + 2 > >>> y = delayed 3 + 4 > >>> z = delayed x + y # neither x nor y has been evaluated yet > >>> z > 10 > > This works fine for simple cases, but there might be situations where someone wants to control evaluation a bit more. As soon as we add any kind of state things get icky: > > # assuming Foo is a class an attribute `value` and a method > # `inc` that increments the value and returns the current value > >>> foo = Foo(value=0) > >>> x = delayed sum([foo.inc(), foo.inc()]) > >>> foo.inc() > 1 > >>> x > 5 > > However, assuming this is a real, more complex system, we might want a way to have one or both of those inc calls be eagerly evaluated, which requires a way of signalling that in some way: > > >>> foo = Foo(value=0) > >>> x = delayed sum([foo.inc(), eager foo.inc()]) > >>> foo.inc() > 2 > >>> foo.inc() > 4 > > Perhaps luckily, `eager` is less commonly used than even `delayed`, but still 2 keywords is an even higher bar. I guess another alternative would be to require annotating all subexpressions with `delayed`, but then that turns the above into > > >>> foo = Foo(value=0) > >>> x = delayed sum([delayed foo.inc(), delayed foo.inc()]) > >>> foo.inc() > 1 > >>> x > 5 > > At which point `delayed` would need to be a much shorter keyword (the heathen in me says overload `del`). > > --Josh > >> On Fri, Feb 17, 2017 at 12:13 PM Abe Dillon wrote: >> Actually, following from the idea that packing and unpacking variables should be delayed by default, it might make sense to use syntax like: >> >> >>> a = *(2+2) >> >>> b = a + 1 >> >> Instead of >> >> >>> a = lazy 2+2 # or whatever you want the keyword to be >> >>> b = a + 1 >> >> That syntax sort-of resembles generator expressions, however; I usually like how python favors actual words over obscure symbol combinations for readability's sake. >> >> On Fri, Feb 17, 2017 at 10:57 AM, Abe Dillon wrote: >> I'd like to suggest a shorter keyword: `lazy` >> >> This isn't an endorsement. I haven't had time to digest how big this change would be. >> >> If this is implemented, I'd also like to suggest that perhaps packing and unpacking should be delayed by default and not evaluated until the contents are used. It might save on many pesky edge cases that would evaluate your expression unnecessarily. >> >> On Fri, Feb 17, 2017 at 10:43 AM, Joseph Hackman wrote: >> Agreed. I think this may require some TLC to get right, but posting here for feedback on the idea overall seemed like a good start. As far as I know, the basic list and dict do not inspect what they contain. I.e. >> >> d = {} >> d['a']= delayed: stuff() >> b=d['a'] >> >> b would end up as still the thunk, and stuff wouldn't be executed until either d['a'] or b actually is read from. >> >> -Joseph >> >> > On Feb 17, 2017, at 11:34 AM, Chris Angelico wrote: >> > >> >> On Sat, Feb 18, 2017 at 3:29 AM, Joseph Hackman wrote: >> >> ChrisA: I am not sure about collections. I think it may be fine to not special case it: if the act of putting it in the collection reads anything, then it is evaluated, and if it doesn't it isn't. The ideal design goal for this would be that all existing code continues to function as if the change wasn't made at all, except that the value is evaluated at a different time. >> >> >> > >> > Yeah, I'm just worried that it'll become useless without that. For >> > instance, passing arguments to a function that uses *a,**kw is going >> > to package your thunk into a collection, and that's how (eg) the >> > logging module will process it. >> > >> > It's not going to be easy to have a simple AND useful definition of >> > "this collapses the waveform, that keeps it in a quantum state", but >> > sorting that out is fairly key to the proposal. >> > >> > ChrisA >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 14:26:00 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 19:26:00 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I think trying to eager-ify subexpressions is absurdly difficult to do right, and also a problem that occurs in other places in Python already, so solving it only for this new thing that might very well go no further is a bit odd. I don't think versions that aren't transparent are much use. > Interesting. Okay. So in effect, these things aren't objects, they're magic constructs that turn into objects the moment you do anything with them, even an identity check. That makes sense. This seems unfortunate. Why not make these things objects that replace themselves with the evaluated-to object when they're used? > "this collapses the waveform, that keeps it in a quantum state" That's a bit of a false dichotomy ;) I suggest that operators on delayed-objects defer evaluation iff all of their operands are delayed, with some hopefully-obvious exceptions: - Function call: delayed_thing() should evaluate delayed_thing - Attribute and item access should evaluate the container and key: even if both operands are delayed, in Python we have to assume things are mutable (especially if we don't know what they are yet), so we can't guarantee that delaying the lookup is valid. Just passing something to a function shouldn't collapse it. That'd make this completely useless. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 14:38:16 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 14:38:16 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Delayed execution and respecting mutable semantics seems like a nightmare. For most indexers we assume hashability which implies immutability, why can't we also do that here? Also, why do we need to evaluate callables eagerly? re the thunk replacing itself with the result instead of memoizing the result and living as an indirection: This is most likely impossible with the current memory model in CPython. Not all objects occupy the same space in memory so you wouldn't know how much space to allocate for the thunk. The interpreter has no way to find all the pointers in use so it cannot just do pointer cleanups to make everyone point to the newly allocated result. On Fri, Feb 17, 2017 at 2:26 PM, Ed Kellett wrote: > I think trying to eager-ify subexpressions is absurdly difficult to do > right, and also a problem that occurs in other places in Python already, so > solving it only for this new thing that might very well go no further is a > bit odd. > > I don't think versions that aren't transparent are much use. > > > Interesting. Okay. So in effect, these things aren't objects, they're > magic constructs that turn into objects the moment you do anything > with them, even an identity check. That makes sense. > > This seems unfortunate. Why not make these things objects that replace > themselves with the evaluated-to object when they're used? > > > "this collapses the waveform, that keeps it in a quantum state" > > That's a bit of a false dichotomy ;) > > I suggest that operators on delayed-objects defer evaluation iff all of > their operands are delayed, with some hopefully-obvious exceptions: > - Function call: delayed_thing() should evaluate delayed_thing > - Attribute and item access should evaluate the container and key: even if > both operands are delayed, in Python we have to assume things are mutable > (especially if we don't know what they are yet), so we can't guarantee that > delaying the lookup is valid. > > Just passing something to a function shouldn't collapse it. That'd make > this completely useless. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mehaase at gmail.com Fri Feb 17 14:39:59 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Fri, 17 Feb 2017 14:39:59 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton wrote: > but I'm wondering how common async and await were when that was proposed > and accepted? Actually, "async" and "await" are backwards compatible due to a clever tokenizer hack. The "async" keyword may only appear in a few places (e.g. async def), and it is treated as a name anywhere else.The "await" keyword may only appear inside an "async def" and is treated as a name everywhere else. Therefore... >>> async = 1 >>> await = 1 ...these are both valid in Python 3.5. This example is helpful when proposing new keywords. More info: https://www.python.org/dev/peps/pep-0492/#transition-plan -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 15:31:16 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 15:31:16 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Couldn't the same thing be true of delayed if it is always followed by a colon? I.e. delayed=1 x= delayed: slow_function() print(delayed) # prints 1 -Joseph > On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: > >> On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton wrote: >> but I'm wondering how common async and await were when that was proposed and accepted? > > > Actually, "async" and "await" are backwards compatible due to a clever tokenizer hack. The "async" keyword may only appear in a few places (e.g. async def), and it is treated as a name anywhere else.The "await" keyword may only appear inside an "async def" and is treated as a name everywhere else. Therefore... > > >>> async = 1 > >>> await = 1 > > ...these are both valid in Python 3.5. This example is helpful when proposing new keywords. > > More info: https://www.python.org/dev/peps/pep-0492/#transition-plan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 17 15:40:03 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 17 Feb 2017 20:40:03 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I think it could even be true without, but the colon may cause ambiguity problems with function annotations. def foo(delayed: delayed: 1 + 2) is a bit odd, especially if `delayed` is chainable. --Josh On Fri, Feb 17, 2017 at 3:32 PM Joseph Hackman wrote: > Couldn't the same thing be true of delayed if it is always followed by a > colon? > > I.e. > delayed=1 > x= delayed: slow_function() > print(delayed) # prints 1 > > -Joseph > > On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: > > On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton > wrote: > > but I'm wondering how common async and await were when that was proposed > and accepted? > > > Actually, "async" and "await" are backwards compatible due to a clever > tokenizer hack. The "async" keyword may only appear in a few places (e.g. > async def), and it is treated as a name anywhere else.The "await" keyword > may only appear inside an "async def" and is treated as a name everywhere > else. Therefore... > > >>> async = 1 > >>> await = 1 > > ...these are both valid in Python 3.5. This example is helpful when > proposing new keywords. > > More info: https://www.python.org/dev/peps/pep-0492/#transition-plan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Fri Feb 17 15:33:36 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Fri, 17 Feb 2017 21:33:36 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 17 February 2017 at 18:40, Chris Angelico wrote: > Further discussion probably should be redirected to python-list, but > I'll elaborate here to explain why I do not support your proposal. > I don't see why you want redirect me to python-list, and how exactly do you see it, start a related discussion there? > You have a shelf of books. I ask you to go through the books and read > their titles out. How do you do it? > I think first I would suppose that probably I'll need not all of the books, but say 20, or want to count something. And that is life - if I want to batch process 100 movie files obviously I'll first try to process one file. And then, if all is ok, I'll uncomment the full loop line and comment out the partial loop line. It is not known how much percent of processing algorithms does not require index *at all*. For me it is probably 20% or so. I do math, all sorts of batch processings and even in unexpected cases "for in range()" comes in handy, e.g: D = {"a":5, "b":10, "c":15} keys = D.keys() d = len(D) for i in range(0, d) : key = keys[i] print "progress:", i So I'd say there is a big amount of cases where it is natural and the proposal should not contradict with current usage, but rather is a syntactic sugar for 'for in range()" which, IMO has some purpose due to frequent usage, hence there are some PEPs related to this. Mikhail -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Fri Feb 17 15:41:16 2017 From: abedillon at gmail.com (Abe Dillon) Date: Fri, 17 Feb 2017 14:41:16 -0600 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: > > Couldn't the same thing be true of delayed if it is always followed by a > colon? No. Because there are other reasons you'd follow the variable `delayed` with a colon: >>> delayed = 1 >>> d = {delayed: "oops!"} My earlier proposal (using unpacking syntax) doesn't work for the same reason. On Fri, Feb 17, 2017 at 2:31 PM, Joseph Hackman wrote: > Couldn't the same thing be true of delayed if it is always followed by a > colon? > > I.e. > delayed=1 > x= delayed: slow_function() > print(delayed) # prints 1 > > -Joseph > > On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: > > On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton > wrote: > >> but I'm wondering how common async and await were when that was proposed >> and accepted? > > > Actually, "async" and "await" are backwards compatible due to a clever > tokenizer hack. The "async" keyword may only appear in a few places (e.g. > async def), and it is treated as a name anywhere else.The "await" keyword > may only appear inside an "async def" and is treated as a name everywhere > else. Therefore... > > >>> async = 1 > >>> await = 1 > > ...these are both valid in Python 3.5. This example is helpful when > proposing new keywords. > > More info: https://www.python.org/dev/peps/pep-0492/#transition-plan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 16:14:29 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 21:14:29 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 19:38 Joseph Jevnik wrote: > Delayed execution and respecting mutable semantics seems like a nightmare. > For most indexers we assume hashability which implies immutability, why > can't we also do that here? Also, why do we need to evaluate callables > eagerly? > Respecting mutability: we just have to always, we don't know if a delayed thing is hashable until we evaluate it. This thing has implications for existing code (since delayed objects can get anywhere) so it should be careful not to do anything too unpredictable, and I think d[k] meaning "whatever is in d[k] in five minutes' time" is unpredictable. One can always delay: d[k] if it's wanted. Evaluate calls: because if you don't, there's no way to say "strictly evaluate x() for its side effects". -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 16:17:40 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 16:17:40 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: There is no existing code that uses delayed execution so we don't need to worry about breaking it. I think it would be much easier to reason about if forcing an expression was always explicit. I am not sure what you mean with the second case; why are you delaying a function if you care about the observable side-effect? On Fri, Feb 17, 2017 at 4:14 PM, Ed Kellett wrote: > On Fri, 17 Feb 2017 at 19:38 Joseph Jevnik wrote: > >> Delayed execution and respecting mutable semantics seems like a >> nightmare. For most indexers we assume hashability which implies >> immutability, why can't we also do that here? Also, why do we need to >> evaluate callables eagerly? >> > > Respecting mutability: we just have to always, we don't know if a delayed > thing is hashable until we evaluate it. This thing has implications for > existing code (since delayed objects can get anywhere) so it should be > careful not to do anything too unpredictable, and I think d[k] meaning > "whatever is in d[k] in five minutes' time" is unpredictable. One can > always delay: d[k] if it's wanted. > > Evaluate calls: because if you don't, there's no way to say "strictly > evaluate x() for its side effects". > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 16:20:23 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 16:20:23 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: About the "whatever is d[k]" in five minutes comment: If I created an explict closure like: `thunk = lambda: d[k]` and then mutated `d` before evaluating the closure you would have the same issue. I don't think it is that confusing. If you need to know what `d[k]` evaluates to right now then the order of evaluation is part of the correctness of your program and you need to sequence execution such that `d` is evaluated before creating that closure. On Fri, Feb 17, 2017 at 4:17 PM, Joseph Jevnik wrote: > There is no existing code that uses delayed execution so we don't need to > worry about breaking it. I think it would be much easier to reason about if > forcing an expression was always explicit. I am not sure what you mean with > the second case; why are you delaying a function if you care about the > observable side-effect? > > On Fri, Feb 17, 2017 at 4:14 PM, Ed Kellett wrote: > >> On Fri, 17 Feb 2017 at 19:38 Joseph Jevnik wrote: >> >>> Delayed execution and respecting mutable semantics seems like a >>> nightmare. For most indexers we assume hashability which implies >>> immutability, why can't we also do that here? Also, why do we need to >>> evaluate callables eagerly? >>> >> >> Respecting mutability: we just have to always, we don't know if a delayed >> thing is hashable until we evaluate it. This thing has implications for >> existing code (since delayed objects can get anywhere) so it should be >> careful not to do anything too unpredictable, and I think d[k] meaning >> "whatever is in d[k] in five minutes' time" is unpredictable. One can >> always delay: d[k] if it's wanted. >> >> Evaluate calls: because if you don't, there's no way to say "strictly >> evaluate x() for its side effects". >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 16:30:25 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 16:30:25 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: <55808B46-8F10-4BBC-9DD9-D359AB388592@gmail.com> Abe- You are correct. However I think it may still be salvageable. In your code example, you could be either making a dict with a key of 1, or a set of a delayed object. But there's no reason to build a set of a delayed object because hashing it would immediately un-delay. Similarly, I am not sure delayed should be allowed inside of function headers. So we could say that dictionary keys and sets shouldn't be allowed to use the delayed keyword. Same with function headers. Are there any other collisions? -Joseph On Feb 17, 2017, at 3:41 PM, Abe Dillon wrote: >> Couldn't the same thing be true of delayed if it is always followed by a colon? > > No. Because there are other reasons you'd follow the variable `delayed` with a colon: > > >>> delayed = 1 > >>> d = {delayed: "oops!"} > > My earlier proposal (using unpacking syntax) doesn't work for the same reason. > >> On Fri, Feb 17, 2017 at 2:31 PM, Joseph Hackman wrote: >> Couldn't the same thing be true of delayed if it is always followed by a colon? >> >> I.e. >> delayed=1 >> x= delayed: slow_function() >> print(delayed) # prints 1 >> >> -Joseph >> >>> On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: >>> >>>> On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton wrote: >>>> but I'm wondering how common async and await were when that was proposed and accepted? >>> >>> >>> Actually, "async" and "await" are backwards compatible due to a clever tokenizer hack. The "async" keyword may only appear in a few places (e.g. async def), and it is treated as a name anywhere else.The "await" keyword may only appear inside an "async def" and is treated as a name everywhere else. Therefore... >>> >>> >>> async = 1 >>> >>> await = 1 >>> >>> ...these are both valid in Python 3.5. This example is helpful when proposing new keywords. >>> >>> More info: https://www.python.org/dev/peps/pep-0492/#transition-plan >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at lucidity.plus.com Fri Feb 17 16:35:52 2017 From: python at lucidity.plus.com (Erik) Date: Fri, 17 Feb 2017 21:35:52 +0000 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: <06b34e3c-838f-02b5-0788-171ecd0096d9@lucidity.plus.com> On 17/02/17 10:22, Stephan Houben wrote: > Proposal: Light-weight call-by-name syntax in Python > > The following syntax > a : b > is to be interpreted as: > a(lambda: b) Isn't this too general a syntax? Doesn't it lead to something like: if a: b: c: d: e: pass E. From edk141 at gmail.com Fri Feb 17 16:49:57 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 21:49:57 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 21:18 Joseph Jevnik wrote: > There is no existing code that uses delayed execution so we don't need to > worry about breaking it. > I think you're missing the point here. This thing is transparent?that's sort of the entire point?so you can pass delayed expressions to other things, and it would be better if they didn't have insane behaviour. > I think it would be much easier to reason about if forcing an expression > was always explicit. I am not sure what you mean with the second case; why > are you delaying a function if you care about the observable side-effect? > You don't delay the function, you delay an expression that evaluates to it. You should be able to pass the result to *any* existing code that expects a function and sometimes calls it, and the function should be called when that happens, rather than evaluated to a delayed object and then discarded. -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 16:52:38 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 21:52:38 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 21:21 Joseph Jevnik wrote: > About the "whatever is d[k]" in five minutes comment: If I created an > explict closure like: `thunk = lambda: d[k]` and then mutated `d` before > evaluating the closure you would have the same issue. I don't think it is > that confusing. If you need to know what `d[k]` evaluates to right now then > the order of evaluation is part of the correctness of your program and you > need to sequence execution such that `d` is evaluated before creating that > closure. > If you create an explicit closure, sure. With delayed expressions, you could explicitly delay d[k], too. If you have an existing d and k, potentially passed to you by somebody else's code, the delayedness of d and k should not inflict arbitrarily-delayed sequencing on your attempt to find out what d[k] is now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Fri Feb 17 16:56:47 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Fri, 17 Feb 2017 16:56:47 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: > You should be able to pass the result to *any* existing code that expects a function and sometimes calls it, and the function should be called when that happens, rather than evaluated to a delayed object and then discarded. I disagree with this claim because I do not think that you should have side effects and delayed execution anywhere near each other. You only open youself up to a long list of special cases for when and where things get evaluated. On Fri, Feb 17, 2017 at 4:49 PM, Ed Kellett wrote: > On Fri, 17 Feb 2017 at 21:18 Joseph Jevnik wrote: > >> There is no existing code that uses delayed execution so we don't need to >> worry about breaking it. >> > > I think you're missing the point here. This thing is transparent?that's > sort of the entire point?so you can pass delayed expressions to other > things, and it would be better if they didn't have insane behaviour. > > >> I think it would be much easier to reason about if forcing an expression >> was always explicit. I am not sure what you mean with the second case; why >> are you delaying a function if you care about the observable side-effect? >> > > You don't delay the function, you delay an expression that evaluates to > it. You should be able to pass the result to *any* existing code that > expects a function and sometimes calls it, and the function should be > called when that happens, rather than evaluated to a delayed object and > then discarded. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 17 16:57:56 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 17 Feb 2017 21:57:56 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: Ed, I'm not seeing this perceived problem either. if we have >>> d = delayed {'a': 1, 'b': 2} # I'm not sure how this is delayed exactly, but sure >>> k = delayed string.ascii_lowercase[0] >>> d[k] 1 I'm not sure how the delayedness of any of the subexpressions matter, since evaluating the parent expression will evaluate all the way down. --Josh On Fri, Feb 17, 2017 at 4:53 PM Ed Kellett wrote: > On Fri, 17 Feb 2017 at 21:21 Joseph Jevnik wrote: > > About the "whatever is d[k]" in five minutes comment: If I created an > explict closure like: `thunk = lambda: d[k]` and then mutated `d` before > evaluating the closure you would have the same issue. I don't think it is > that confusing. If you need to know what `d[k]` evaluates to right now then > the order of evaluation is part of the correctness of your program and you > need to sequence execution such that `d` is evaluated before creating that > closure. > > > If you create an explicit closure, sure. With delayed expressions, you > could explicitly delay d[k], too. If you have an existing d and k, > potentially passed to you by somebody else's code, the delayedness of d and > k should not inflict arbitrarily-delayed sequencing on your attempt to find > out what d[k] is now. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 17:01:24 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 22:01:24 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 21:57 Joseph Jevnik wrote: > > You should be able to pass the result to *any* existing code that > expects a function and sometimes calls it, and the function should be > called when that happens, rather than evaluated to a delayed object and > then discarded. > > I disagree with this claim because I do not think that you should have > side effects and delayed execution anywhere near each other. > If Python gets delayed execution, it's going to be near side effects. That's just the reality we live in. > You only open youself up to a long list of special cases for when and > where things get evaluated. > Not really. With the function call example, as long as x() always evaluates x (rather than becoming a delayed call to x), we're all good. Remember that this has nothing to do with the contents of x, which indeed shouldn't use delays if it cares about side effects?what might be delayed here is the expression that finds x. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Fri Feb 17 17:02:01 2017 From: abedillon at gmail.com (Abe Dillon) Date: Fri, 17 Feb 2017 16:02:01 -0600 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <55808B46-8F10-4BBC-9DD9-D359AB388592@gmail.com> References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> <55808B46-8F10-4BBC-9DD9-D359AB388592@gmail.com> Message-ID: I'm fairly novice, so I could be way off base here, but it seems like the inevitable conclusion to this problem is something like JIT compilation, right? (admittedly, I know very little about JIT compilation) Python seems to be accumulating a lot of different approaches to achieving very similar things: asynchronous and/or lazy execution. We have generators, futures, asyncio, async/await, and probably more that I'm not thinking of. It seems like it should be possible for the interpreter to determine when an expression absolutely *must* be evaluated in many cases. If I write: >>> log.debug("data = %s", some_expensive_function()) Then it should be possible for python to put off evaluating that function or even building a full argument tuple if the log level is higher than debug. I know code with side-effects, especially I/O related side-effects would be difficult or impossible to manage within that context (the interpreter wouldn't be able to know that a write to a file has to occur before a read from that file for instance. Maybe it would make more sense to mark such side-effects to tell the interpreter "you must evaluate all deferred expressions before executing this because it changes stuff that Python can't keep track of". instead of marking code that the compiler can optimize. My gut reaction says this is not the answer because it emphasizes optimization over correct behavior, but I wanted to share the thought. On Fri, Feb 17, 2017 at 3:30 PM, Joseph Hackman wrote: > Abe- > > You are correct. However I think it may still be salvageable. > > In your code example, you could be either making a dict with a key of 1, > or a set of a delayed object. But there's no reason to build a set of a > delayed object because hashing it would immediately un-delay. > > Similarly, I am not sure delayed should be allowed inside of function > headers. > > So we could say that dictionary keys and sets shouldn't be allowed to use > the delayed keyword. Same with function headers. Are there any other > collisions? > > -Joseph > > On Feb 17, 2017, at 3:41 PM, Abe Dillon wrote: > > Couldn't the same thing be true of delayed if it is always followed by a >> colon? > > > No. Because there are other reasons you'd follow the variable `delayed` > with a colon: > > >>> delayed = 1 > >>> d = {delayed: "oops!"} > > My earlier proposal (using unpacking syntax) doesn't work for the same > reason. > > On Fri, Feb 17, 2017 at 2:31 PM, Joseph Hackman > wrote: > >> Couldn't the same thing be true of delayed if it is always followed by a >> colon? >> >> I.e. >> delayed=1 >> x= delayed: slow_function() >> print(delayed) # prints 1 >> >> -Joseph >> >> On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: >> >> On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton > > wrote: >> >>> but I'm wondering how common async and await were when that was proposed >>> and accepted? >> >> >> Actually, "async" and "await" are backwards compatible due to a clever >> tokenizer hack. The "async" keyword may only appear in a few places (e.g. >> async def), and it is treated as a name anywhere else.The "await" keyword >> may only appear inside an "async def" and is treated as a name everywhere >> else. Therefore... >> >> >>> async = 1 >> >>> await = 1 >> >> ...these are both valid in Python 3.5. This example is helpful when >> proposing new keywords. >> >> More info: https://www.python.org/dev/peps/pep-0492/#transition-plan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 17:02:57 2017 From: edk141 at gmail.com (Ed Kellett) Date: Fri, 17 Feb 2017 22:02:57 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 21:58 Joshua Morton wrote: > Ed, I'm not seeing this perceived problem either. > > if we have > > >>> d = delayed {'a': 1, 'b': 2} # I'm not sure how this is delayed > exactly, but sure > >>> k = delayed string.ascii_lowercase[0] > >>> d[k] > 1 > My problem with this doesn't have to do with subexpressions. If d[k] for delayed d and k yields delayed d[k], then someone mutates d, you get an unexpected result. So I'm suggesting that d[k] for delayed d and k should evaluate d and k, instead. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 17:09:26 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 14:09:26 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: That was a problem with the colon that occurred to me. I think it can't be tokenized in function annotations. Plus I still think the no-colon looks better. But that's bikeshedding. Also other words are plausible. I like lazy even more than delayed, I think. Still, I'd love the construct whatever the exact spelling. On Feb 17, 2017 12:41 PM, "Joshua Morton" wrote: > I think it could even be true without, but the colon may cause ambiguity > problems with function annotations. > > def foo(delayed: delayed: 1 + 2) > > is a bit odd, especially if `delayed` is chainable. > > --Josh > > On Fri, Feb 17, 2017 at 3:32 PM Joseph Hackman > wrote: > >> Couldn't the same thing be true of delayed if it is always followed by a >> colon? >> >> I.e. >> delayed=1 >> x= delayed: slow_function() >> print(delayed) # prints 1 >> >> -Joseph >> >> On Feb 17, 2017, at 2:39 PM, Mark E. Haase wrote: >> >> On Fri, Feb 17, 2017 at 1:55 PM, Joshua Morton > > wrote: >> >> but I'm wondering how common async and await were when that was proposed >> and accepted? >> >> >> Actually, "async" and "await" are backwards compatible due to a clever >> tokenizer hack. The "async" keyword may only appear in a few places (e.g. >> async def), and it is treated as a name anywhere else.The "await" keyword >> may only appear inside an "async def" and is treated as a name everywhere >> else. Therefore... >> >> >>> async = 1 >> >>> await = 1 >> >> ...these are both valid in Python 3.5. This example is helpful when >> proposing new keywords. >> >> More info: https://www.python.org/dev/peps/pep-0492/#transition-plan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 17 17:31:00 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 14:31:00 -0800 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: Iterating over range(len(collection)) is one of the worst anti-patterns in Python. I take great pains to slap my students who do that. On Feb 17, 2017 9:32 AM, "Mikhail V" wrote: > > On 17 February 2017 at 17:37, Chris Angelico wrote: > >> On Sat, Feb 18, 2017 at 3:30 AM, Mikhail V wrote: >> > On 17 February 2017 at 04:59, Chris Angelico wrote: >> >> >> >> On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V >> wrote: >> >> > Common use case: >> >> > >> >> > L = [1,3,5,7] >> >> > >> >> > for i over len(L): >> >> > e = L[i] >> >> > >> >> > or: >> >> > >> >> > length = len(L) >> >> > for i over length: >> >> > e = L[i] >> >> >> >> Better use case: >> >> >> >> for i, e in enumerate(L): >> >> >> > >> > This would be more compact, yet less readable, more error prone variant. >> > I'd avoid it it all costs and even if I don't need the index further in >> loop >> > body, >> > (which happens rarely in my experience) I write e=L[i] in second line >> > to make the code more verbose and keep the flow order. >> > So your variant (and those proposed in PEP-212) could serve in list >> > comprehensions for example, but for common 'expanded' code I find it >> > decline of readability, and creating totally different variants for same >> > iteration idea. >> > But partially that could be simply matter of habit and love to >> contractions. >> >> If you don't need the index, why not just iterate over the list directly? >> >> for e in L: >> >> That's the single most obvious way to step through a collection in >> Python. What do you need to count up to the length for? >> >> > I have said I need the index, probably you've misread my last comment. > Further more I explained why I think iteration over index should be the > preferred way, it help with readability a lot. > All my learning years ended up with rewriting most code to "for i in > range()" > and I slap myself when I start to write "for e in L". > It is exactly where TOOWTDI applies perfectly and it is integer iteration > for me. > > Mikhail > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 17:35:27 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 17:35:27 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I think we should use the colon to make the delayed word (or whatever word is selected), unambiguously used in this way (and to prevent any existing code from breaking). On 17 February 2017 at 17:09, David Mertz wrote: > That was a problem with the colon that occurred to me. I think it can't be > tokenized in function annotations. > I don't see any reason for delayed execution and function annotations to mix. i.e. def foo(delayed: bar): pass would define a function that takes one argument, named delayed, of type bar. > > Plus I still think the no-colon looks better. But that's bikeshedding. > Also other words are plausible. I like lazy even more than delayed, I > think. Still, I'd love the construct whatever the exact spelling. > > I'm not particularly married to delayed, but I don't know how to properly vet this inside the community. I'm glad you like the proposal! -Joseph -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Fri Feb 17 17:51:39 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Fri, 17 Feb 2017 16:51:39 -0600 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: I think fundamentally by special-casing a for-loop variant, you have a construct with limited/no generality that's simply an additional burden to learn. You're kind of doing the opposite of converting print from a statement into a function. I far prefer the print function because it's a function like every other and doesn't have it's own curious syntax (if only assert was also fixed..., assert(cond, reason) has bit me a few times) It's nice that you can put *anything* that supports the iterator protocol after "for x in" and Python will "iterate" over it, however that is defined for the object. The thing might not have a useful integer index at all (dictionaries, file systems...) and the loop doesn't care. If you *do* want that index, you can enumerate() or range(len()), which I sometimes do if modifying a list in-place, but it's kinda icky as David suggests. Nick On Fri, Feb 17, 2017 at 4:31 PM, David Mertz wrote: > Iterating over range(len(collection)) is one of the worst anti-patterns in > Python. I take great pains to slap my students who do that. > > On Feb 17, 2017 9:32 AM, "Mikhail V" wrote: > >> >> On 17 February 2017 at 17:37, Chris Angelico wrote: >> >>> On Sat, Feb 18, 2017 at 3:30 AM, Mikhail V wrote: >>> > On 17 February 2017 at 04:59, Chris Angelico wrote: >>> >> >>> >> On Fri, Feb 17, 2017 at 2:13 PM, Mikhail V >>> wrote: >>> >> > Common use case: >>> >> > >>> >> > L = [1,3,5,7] >>> >> > >>> >> > for i over len(L): >>> >> > e = L[i] >>> >> > >>> >> > or: >>> >> > >>> >> > length = len(L) >>> >> > for i over length: >>> >> > e = L[i] >>> >> >>> >> Better use case: >>> >> >>> >> for i, e in enumerate(L): >>> >> >>> > >>> > This would be more compact, yet less readable, more error prone >>> variant. >>> > I'd avoid it it all costs and even if I don't need the index further >>> in loop >>> > body, >>> > (which happens rarely in my experience) I write e=L[i] in second line >>> > to make the code more verbose and keep the flow order. >>> > So your variant (and those proposed in PEP-212) could serve in list >>> > comprehensions for example, but for common 'expanded' code I find it >>> > decline of readability, and creating totally different variants for >>> same >>> > iteration idea. >>> > But partially that could be simply matter of habit and love to >>> contractions. >>> >>> If you don't need the index, why not just iterate over the list directly? >>> >>> for e in L: >>> >>> That's the single most obvious way to step through a collection in >>> Python. What do you need to count up to the length for? >>> >>> >> I have said I need the index, probably you've misread my last comment. >> Further more I explained why I think iteration over index should be the >> preferred way, it help with readability a lot. >> All my learning years ended up with rewriting most code to "for i in >> range()" >> and I slap myself when I start to write "for e in L". >> It is exactly where TOOWTDI applies perfectly and it is integer iteration >> for me. >> >> Mikhail >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Fri Feb 17 18:05:08 2017 From: barry at barrys-emacs.org (Barry) Date: Fri, 17 Feb 2017 23:05:08 +0000 Subject: [Python-ideas] Efficient debug logging In-Reply-To: References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <20170215111831.GB5689@ando.pearwood.info> <1756620.cYo0RJySdd@varric.chelsea.private> <3dd7f7d0-2d86-bcf9-b1d4-fc8d02780e3e@mail.de> <7a2fc6a7-9aa5-f83e-ee74-86f96d341838@mozilla.com> Message-ID: > On 16 Feb 2017, at 21:03, Abe Dillon wrote: > > I personally don't see why you can't use floats for log levels, but I'm interested to know how people are using logs such that they need dozens of levels. That, however; is tangential to the discussion about conditional execution of an expensive function. Its not lots of levels its lots of categories. I did add one level, INFOHEADER, but not for debug. It you are interested in what I did you could look at this project: https://github.com/barry-scott/scm-workbench Have a look at: https://github.com/barry-scott/scm-workbench/blob/master/Source/Scm/wb_scm_debug.py https://github.com/barry-scott/scm-workbench/blob/master/Source/Common/wb_debug.py Then look at almost any module and see the use of the _debug calls. Barry -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 18:06:26 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 18:06:26 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <20170217111005.GK5689@ando.pearwood.info> References: <20170217111005.GK5689@ando.pearwood.info> Message-ID: On 17 February 2017 at 06:10, Steven D'Aprano wrote: > On Fri, Feb 17, 2017 at 12:24:53AM -0500, Joseph Hackman wrote: > > > I propose a keyword to mark an expression for delayed/lazy execution, for > > the purposes of standardizing such behavior across the language. > > > > The proposed format is: > > delayed: > > i.e. log.info("info is %s", delayed: expensiveFunction()) > > Keywords are difficult: since by definition they are not backwards > compatible, they make it hard for people to write version independent > code, and will break people's code. Especially something like "delayed", > I expect that there is lots of code that used "delayed" as a regular > name. > > if status.delayed: > ... > I think it would be key, like async/await, to narrowly define the scope in which the word delayed functions as a keyword. There is enough information for the compiler to know that you don't mean the delayed keyword there because: 1. It's immediately after a dot, but more importantly 2. It's in a bare 'if'. There's no way the execution could be delayed. delayed=False if delayed: is still protected by #2. In a case where delayed would make sense, it also is unambiguous if either_or(True, delayed:expensive_function()): is clearly using the delayed keyword, rather than the delayed defined as False above. (Notably, the built-in 'and' and 'or' shouldn't use delayed:, as the short-circuiting logic is already well defined. So, in short, in an if, for or while, the delayed keyword is only used if it is inside a function call (or something like that). > A new keyword means it can't be back-ported to older versions, and will > break code. > > async and await both work fine, for the reasons listed above. I'll admit there may be more nuance required here, but it should be both possible, and fairly intuitive based on when people would be using delayed execution. > > > Unlike 'lambda' which returns a function (so the receiver must be > > lambda-aware), delayed execution blocks are for all purposes values. The > > first time the value (rather than location) is read, > > What counts as "reading" a value? Based on your example below, I can't > tell if passing the object to *any* function is enough to trigger > evaluation, or specifically print is the magic that makes it happen. > So far I'm going with pretty much anything that isn't being the right-hand of an assignment. So coercion to different types, hashing (for use as a key in a dict or set), __repr__, etc would all be covered, as well as identity and comparisons. i.e.: def expensive_function(x,y): if x and y is not None: print('yippie skippy') expensive_function(True, delayed: evaluates_to_none()) The idea put forth here would cover this, by evaluating to perform the is. > > or any method on the delayed object is called, > > I don't think that can work -- it would have to be any attribute access, > surely, because Python couldn't tell if the attribute was a method or > not until it evaluated the lazy object. Consider: > > spam = delayed: complex_calculation() > a = spam.thingy > Since spam.thingy is an access on spam, it would have been evaluated before 'thingy' was read. > What's `a` at this point? Is is still some sort of lazy object, waiting > to be evaluated? If so, how is Python supposed to know if its a method? > > result = a() > > > > the expression is executed and the delayed > > expression is replaced with the result. (Thus, the delayed expression is > > only every evaluated once). > > That's easily done by having the "delayed" keyword cache each expression > it sees, but that seems like a bad idea to me: > > spam = delayed: get_random_string() > eggs = delayed: get_random_string() # the same expression > spam.upper() # convert to a real value > > assert spam == eggs # always true, as they are the same expression > Since spam and eggs are two different instances of delayed expression, each one would be evaluated separately when they are read from (as operands for the equals operator). So no, even without the spam.upper(), they would not match. > > Worse, suppose module a.py has: > > spam = delayed: calculate(1) > > and module b.py has: > > eggs = delayed: calculate(1) > > where a.calculate and b.calculate do completely different things. The > result you get will depend on which happens to be evaluated first and > cached, and would be a nightmare to debug. Truely spooky action-at-a- > distance code. > > I think it is better to stick to a more straight-forward, easily > understood and debugged system based on object identity rather than > expressions. > > The caching means that: spam = delayed: calculate(1) eggs = spam eggs == spam would be true, and calculate would have only been called once, not twice. > > Ideally: > > a = delayed: 1+2 > > b = a > > print(a) #adds 1 and 2, prints 3 > > # a and b are now both just 3 > > print(b) #just prints 3 > > That would work based on object identity too. > > By the way, that's probably not the best example, because the keyhole > optimizer will likely have compiled 1+2 as just 3, so you're effectively > writing: > > a = delayed: 3 > > At least, that's what I would want: I would argue strongly against lazy > objects somehow defeating the keyhole optimizer. If I write: > > a = delayed: complex_calculation(1+2+3, 4.5/3, 'abcd'*3) > > what I hope will be compiled is: > > a = delayed: complex_calculation(6, 0.6428571428571429, 'abcdabcdabcd') > > same as it would be now (at least in CPython), apart from the "delayed:" > keyword. > a = delayed: complex_calculation(1+2+3, 4.5/3, 'abcd'*3) > I'm not sure what this is trying to do, so it's hard for me to weigh in. It's totally fine if delayed:1+2 is exactly the same as delayed:3 and/or 3 itself. > > Mechanically, this would be similar to the following: > > > > class Delayed(): > > def __init__(self, func): > > self.__func = func > > self.__executed = False > > self.__value = None > > > > def __str__(self): > > if self.__executed: > > return self.__value.__str__() > > self.__value = self.__func() > > self.__executed = True > > return self.__value.__str__() > > So you're suggesting that calling str(delayed_object) is the one way to > force evaluation? > Not at all, pleas see above. > I have no idea what the following code is supposed to mean. > > > > def function_print(value): > > print('function_print') > > print(value) > > > > def function_return_stuff(value): > > print('function_return_stuff') > > return value > > > > function_print(function_return_stuff('no_delay')) > > > > function_print(Delayed(lambda: function_return_stuff('delayed'))) > > > > delayed = Delayed(lambda: function_return_stuff('delayed_object')) > > function_print(delayed) > > function_print(delayed) > If you run this code block, it will demonstrate a number of different orders of execution, indicating that the delayed execution does function as expected. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Fri Feb 17 18:13:01 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Fri, 17 Feb 2017 23:13:01 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: @ Joseph Function annotations can be arbitrary python expressions, it is completely legal to have something like >>> def foo(bar: lambda x: x + 1): ... pass Why you would want that I can't say, but it is legal. In the same way, `def foo(bar: delayed 1 + 1)` should probably be legal syntax, even if the use is inexplicable. (also note that the `:` works with lambda because lambda cannot be used as an identifier). In any case, as David said, bikeshedding. @ Ed Its my understanding that d[k] is always d[k], even if d or k or both are delayed. On the other hand, `delayed d[k]` would not be, but you would need to explicitly state that. I think its worth expanding on the example Joseph made. I think it makes sense to consider this proposal to be `x = delayed ` is essentially equivalent to `x = lambda: `, except that there will be no need to explicitly call `x()` to get the delayed value, instead it will be evaluated the first time its needed, transparently. This, for the moment, assumes that this doesn't cause enormous interpreter issues, but I don't think it will. That is, there is no "delayed" object that is created and called, and so as a user you really won't care if an object is "delayed" or not, you'll just use it and it will be there. Do you understand this proposal differently? --Josh On Fri, Feb 17, 2017 at 5:35 PM Joseph Hackman wrote: > I think we should use the colon to make the delayed word (or whatever word > is selected), unambiguously used in this way (and to prevent any existing > code from breaking). > > On 17 February 2017 at 17:09, David Mertz wrote: > > That was a problem with the colon that occurred to me. I think it can't be > tokenized in function annotations. > > > I don't see any reason for delayed execution and function annotations to > mix. i.e. > def foo(delayed: bar): > pass > would define a function that takes one argument, named delayed, of type > bar. > > > > Plus I still think the no-colon looks better. But that's bikeshedding. > Also other words are plausible. I like lazy even more than delayed, I > think. Still, I'd love the construct whatever the exact spelling. > > > I'm not particularly married to delayed, but I don't know how to properly > vet this inside the community. I'm glad you like the proposal! > > -Joseph > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Fri Feb 17 18:20:35 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 18:20:35 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On 17 February 2017 at 18:13, Joshua Morton wrote: > @ Joseph > > Function annotations can be arbitrary python expressions, it is completely > legal to have something like > > >>> def foo(bar: lambda x: x + 1): > ... pass > > Why you would want that I can't say, but it is legal. In the same way, > `def foo(bar: delayed 1 + 1)` should probably be legal syntax, even if the > use is inexplicable. (also note that the `:` works with lambda because > lambda cannot be used as an identifier). In any case, as David said, > bikeshedding. > Sorry for lack of clarity: I see that it is legal for lambda, I suggest that the value of extending this to delayed: to be not worth the cost of potentially being backwards-impompatible. I say that because if delayed: were to work in function definitions, whenever the definition was evaluated, the delayed would be as well. (The same is true for if, for, and while.) This would be different if the delayed was inside a function call inside a function definition. (but in that case there would be no collision). -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Fri Feb 17 19:28:13 2017 From: edk141 at gmail.com (Ed Kellett) Date: Sat, 18 Feb 2017 00:28:13 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, 17 Feb 2017 at 23:14 Joshua Morton wrote: > @ Ed > > Its my understanding that d[k] is always d[k], even if d or k or both are > delayed. On the other hand, `delayed d[k]` would not be, but you would need > to explicitly state that. I think its worth expanding on the example Joseph > made. > > I think it makes sense to consider this proposal to be `x = delayed > ` is essentially equivalent to `x = lambda: `, except that > there will be no need to explicitly call `x()` to get the delayed value, > instead it will be evaluated the first time its needed, transparently. > This, for the moment, assumes that this doesn't cause enormous interpreter > issues, but I don't think it will. That is, there is no "delayed" object > that is created and called, and so as a user you really won't care if an > object is "delayed" or not, you'll just use it and it will be there. > > Do you understand this proposal differently? > > --Josh > Chris mentioned something about it being difficult to decide what evaluates a delayed thing, and what maintains it. This tangent started with my suggesting that operators should maintain delayed-ness iff all their operands are delayed, with ., [] and () as exceptions. That is, I'm suggesting that d[k] always evaluate d and k, but a + b might defer evaluation if a and b are both delayed already. Roughly, I guess my rationale is something like "let operators combine multiple delayed-objects into one, unless that would break things"?and at least by convention, operators that aren't ., [] or () don't have behaviour that would be broken. Ed -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 17 19:41:26 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 11:41:26 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <55808B46-8F10-4BBC-9DD9-D359AB388592@gmail.com> Message-ID: <20170218004126.GL5689@ando.pearwood.info> On Fri, Feb 17, 2017 at 04:02:01PM -0600, Abe Dillon wrote: > I'm fairly novice, so I could be way off base here, but it seems like the > inevitable conclusion to this problem is something like JIT compilation, > right? (admittedly, I know very little about JIT compilation) No. JIT compilation delays *compiling* the code to run-time. This is a proposal for delaying *running* the code until such time as some other piece of code actually needs the result. An example might help. Suppose we want to get the one millionth prime number, a task which is of moderate difficulty and may take a while: print("Start") result = get_nth_prime(10**6) print("Done") print(result) On my computer, using a pure-Python implementation, it takes about 11 seconds to find the millionth prime 15485863, so there'll be a delay of 11 seconds between printing Start and Done, but printing the result is instantaneous. That's true regardless of when and how the code is compiled. (Where a JIT compiler is useful is that it may be possible to use runtime information available to the interpreter to compile all or some of the Python code to efficient machine code, allowing the function to run faster. That's how PyPy works.) If we make the code *delayed* then the situation is different: print("Start") result = delayed: get_nth_prime(10**6) # I dislike this syntax print("Done") print(result) Now Start and Done are printed virtually instantaneously, but there is an 11 second delay *after* Done is printed, when the result is reified (made real; the calculation is actually performed) and printed. > Python seems to be accumulating a lot of different approaches to achieving > very similar things: asynchronous and/or lazy execution. We have > generators, futures, asyncio, async/await, and probably more that I'm not > thinking of. It seems like it should be possible for the interpreter to > determine when an expression absolutely *must* be evaluated in many cases. If the people debating this proposal cannot even agree on when the expression must be evaluated, how could the interpreter do it? > I know code with side-effects, especially I/O related side-effects would be > difficult or impossible to manage within that context (the interpreter > wouldn't be able to know that a write to a file has to occur before a read > from that file for instance. I think side-effects is a red herring. The obvious rule is: side-effects occur when the delayed thunk is reified. If you care about the actual timing of the side-effects, then don't use delayed evaluation. If you don't care, then who cares if the side-effect is delayed? -- Steve From steve at pearwood.info Fri Feb 17 20:23:46 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 12:23:46 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> Message-ID: <20170218012345.GM5689@ando.pearwood.info> On Fri, Feb 17, 2017 at 06:06:26PM -0500, Joseph Hackman wrote: [...] > I think it would be key, like async/await, to narrowly define the scope in > which the word delayed functions as a keyword. The PEP makes it clear that's just a transition phase: they will be turned into proper keywords in Python 3.7. https://www.python.org/dev/peps/pep-0492/#id80 Python has had "pseudo-keywords" in the past, like "as": [steve at ando ~]$ python2.5 -c "import math as as; print as" and it is my understanding that the core developers dislike this sort of thing. As do I. You shouldn't count as getting the same special treament as async/await. Maybe you will, maybe you won't. > > A new keyword means it can't be back-ported to older versions, and will > > break code. > > > > > async and await both work fine, for the reasons listed above. You're missing the point: code that uses async and await, whether as pseduo-keywords or actual keywords, cannot easily be backported to Python 3.4 or older. If Python introduces a new built-in, say Aardvark, then it can be back-ported: try: Aardvark except NameError: from backport import Aardvark No such thing is possible for new syntax. So that counts as a disadvantage of new syntax. Are we positive that there *must* be new syntax to solve this problem? (I think probably so, but it still counts as a disadvantage: that means that the usefulness is reduced.) > > > Unlike 'lambda' which returns a function (so the receiver must be > > > lambda-aware), delayed execution blocks are for all purposes values. The > > > first time the value (rather than location) is read, > > > > What counts as "reading" a value? Based on your example below, I can't > > tell if passing the object to *any* function is enough to trigger > > evaluation, or specifically print is the magic that makes it happen. > > So far I'm going with pretty much anything that isn't being the right-hand > of an assignment. So coercion to different types, hashing (for use as a key > in a dict or set), __repr__, etc would all be covered, as well as identity > and comparisons. i.e.: [...] That will make it pretty much impossible to tell whether something is a delayed "thunk" or not, since *any* attempt to inspect it in any way will cause it to reify. Maybe that's what we want. > > That's easily done by having the "delayed" keyword cache each expression > > it sees, but that seems like a bad idea to me: > > > > spam = delayed: get_random_string() > > eggs = delayed: get_random_string() # the same expression > > > > spam.upper() # convert to a real value > > > > assert spam == eggs # always true, as they are the same expression > > > > Since spam and eggs are two different instances of delayed expression, each > one would be evaluated separately when they are read from (as operands for > the equals operator). So no, even without the spam.upper(), they would not > match. Earlier we talked about delayed *expressions* always generating the same value, now you're talking about *instances* rather than expressions. It makes sense to have keep the standard Python object semantics, rather than have the value of a delayed thunk cached by the textual expression that generated it. > > I think it is better to stick to a more straight-forward, easily > > understood and debugged system based on object identity rather than > > expressions. > > > > > The caching means that: > spam = delayed: calculate(1) > eggs = spam > > eggs == spam would be true, and calculate would have only been called once, > not twice. That's not caching, that's simple identity. That's how assignment works in Python, delayed calculation or not. -- Steve From josephhackman at gmail.com Fri Feb 17 21:14:29 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Fri, 17 Feb 2017 21:14:29 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <20170218012345.GM5689@ando.pearwood.info> References: <20170217111005.GK5689@ando.pearwood.info> <20170218012345.GM5689@ando.pearwood.info> Message-ID: On 17 February 2017 at 20:23, Steven D'Aprano wrote: > > > I think it would be key, like async/await, to narrowly define the scope > in > > which the word delayed functions as a keyword. > > The PEP makes it clear that's just a transition phase: they will be > turned into proper keywords in Python 3.7. > > https://www.python.org/dev/peps/pep-0492/#id80 > > Python has had "pseudo-keywords" in the past, like "as": > > [steve at ando ~]$ python2.5 -c "import math as as; print as" > > > and it is my understanding that the core developers dislike this sort of > thing. As do I. You shouldn't count as getting the same special treament > as async/await. Maybe you will, maybe you won't. > Very well put! Do you have any suggestions for doing something in the same vein? I think there's been a [...] > That will make it pretty much impossible to tell whether something is a > delayed "thunk" or not, since *any* attempt to inspect it in any way > will cause it to reify. > > Maybe that's what we want. > In my mind, this is a plus. The only way to determine if something is delayed would be something that doesn't apply to anything else, so code never needs to be aware of delayed. > Earlier we talked about delayed *expressions* always generating the same > value, now you're talking about *instances* rather than expressions. It > makes sense to have keep the standard Python object semantics, rather > than have the value of a delayed thunk cached by the textual expression > that generated it. You are totally right. I agree that the nomenclature is important, and I think we're on the same page. [...] Steve- I really appreciate the thoughtful feedback! Please let me know if you have suggestions; I don't expect the idea to be acceptable out-of-the-gate. :) -Joseph -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Feb 17 21:20:35 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 17 Feb 2017 18:20:35 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: On Thu, Feb 16, 2017 at 9:24 PM, Joseph Hackman wrote: > Howdy All! > > This suggestion is inspired by the question on "Efficient debug logging". > > I propose a keyword to mark an expression for delayed/lazy execution, for > the purposes of standardizing such behavior across the language. > > The proposed format is: > delayed: > i.e. log.info("info is %s", delayed: expensiveFunction()) People seem very excited about this as an idea, but I don't understand how it can be implemented. For example, how do you propose to handle code like this? value = delayed: some_dict.get("whatever") if value is None: ... I.e., the question is, how does 'is' work on delayed objects? I guess it has to force the promise and walk the proxy chain in each input and then do an 'is' on the base objects? This seems like a really deep and confusing change to Python's object model for a pretty marginal feature. (This is a special case of the general observation that it's just not possible to implement fully-transparent proxy objects in Python.) -n -- Nathaniel J. Smith -- https://vorpus.org From mikhailwas at gmail.com Fri Feb 17 21:27:00 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sat, 18 Feb 2017 03:27:00 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: A short Meta-note: I see most people are bottom-replying and still many do top-reply, namely you Nick always do. I dont know if there is a rule, but it makes quite hard to manage/read post with mixed posting style. On 17 February 2017 at 23:51, Nick Timkovich wrote: > > I think fundamentally by special-casing a for-loop variant, > you have a construct with limited/no generality that's > simply an additional burden to learn. I see it is almost a tradition to give negative comments, and that is ok in many cases. But I am slightly worried how *quick* you make judgements. In what sense iteration over integer is limited? It cannot write a program for you, no doubt. If you look through examples I made, including iterating over dictionary, you will see that you can do everything with simple iteration, including cases where you do not even have any sequence which you can put in, e.g. simple loop with constant parameters, which comes in extremely handy, e.g. in simple batch scripts. Probably you mean that it can come in play with Python's inner mechanics which will lead to performance loss - yes, can be, but I was not going to argue that. "burden to learn" - I hope you are not serious :) From jsbueno at python.org.br Fri Feb 17 22:13:28 2017 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 18 Feb 2017 01:13:28 -0200 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 18 February 2017 at 00:27, Mikhail V wrote: > A short Meta-note: I see most people are bottom-replying > and still many do top-reply, namely you Nick always do. > I dont know if there is a rule, but it makes quite hard to > manage/read post with mixed posting style. > > On 17 February 2017 at 23:51, Nick Timkovich wrote: >> >> I think fundamentally by special-casing a for-loop variant, >> you have a construct with limited/no generality that's >> simply an additional burden to learn. > > I see it is almost a tradition to give negative comments, and > that is ok in many cases. But I am slightly worried how *quick* you This is not quick - it is based on 25+ years of Python history, with iterating over object collections. And still with you being able to do: r = range for i in r(20): pass if you really need. Now, if you haven't noticed, unlike some other proposes with extended threads, there is really no one that supports this idea here. Not a single point raised, or agreed by anyone else than you that would objectively improve the language. (Ok, half it, I thin k I saw someone commenting that such a syntax could have a use in comprehension one liners). On the other hand, yu have being pointed to this being a complete Python anti-pattern. Anyway, if you don't think it is time for you to concede this is not a good idea at all, maybe it is time to use common sense, and send one last e-mail asking for any support for this idea at all. If no one else steps up suppoting at least some aspect of your proposal , I don't see the point in continuing this thread. > make judgements. In what sense iteration over integer > is limited? It cannot write a program for you, no doubt. > If you look through examples I made, including > iterating over dictionary, you will see that you can do everything > with simple iteration, including cases where you do not > even have any sequence which you can put in, e.g. simple > loop with constant parameters, which comes in extremely > handy, e.g. in simple batch scripts. You can still use range. > Probably you mean that it can come in play with Python's > inner mechanics which will lead to performance loss - > yes, can be, but I was not going to argue that. It is not a matter of performance: it leads to readability loss. And no gain at all. > > > "burden to learn" - I hope you are not serious :) No, this is serious. You duplicate the syntax possibilities of one ot he most basics syntactic elements on the language - when most of us have agreed that "There should be one-- and preferably only one --obvious way to do it". One can start coding in Python if after a couple minutes of tutorial, he learns about "for", "if", "def" and a couple data primitives - and maybe "print" and "input" for some UI. So, out of 3 needed statements to start coding, you are doubling the syntax possibilities on one of them. There s no kidding about augmenting the "burden to learn" here. (By the way, one of your arguments is about the keyword "in" being too short and hard to read - I should point that certainly most Python code these days is read in a tool that does syntax highlighting, and "in" shows in a very distinct way of the variables and expressions needed to the "for" loop - and that includes even the interactive environment as provided by ipython). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Fri Feb 17 22:31:40 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Feb 2017 14:31:40 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On Sat, Feb 18, 2017 at 2:13 PM, Joao S. O. Bueno wrote: > One can start coding in Python if after a couple minutes of tutorial, he learns > about "for", "if", "def" and a couple data primitives - and maybe > "print" and "input" > for some UI. So, out of 3 needed statements to start coding, you are doubling > the syntax possibilities on one of them. There s no kidding about > augmenting the "burden to learn" here. > And if you want a point of comparison, look at JavaScript's for loop. What's the difference between "for (x in y)" and "for (x of y)"? When should you use each one? More ways to use a fundamental statement == more confusion. ChrisA From steve at pearwood.info Fri Feb 17 23:43:26 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 15:43:26 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: <20170218044326.GN5689@ando.pearwood.info> On Fri, Feb 17, 2017 at 06:31:19PM +0100, Mikhail V wrote: > I have said I need the index, probably you've misread my last comment. > Further more I explained why I think iteration over index should be the > preferred way, it help with readability a lot. Your concept of readability is clearly radically different from that of the majority of the Python community. > All my learning years ended up with rewriting most code to "for i in > range()" How do you cope with generators and iterators that don't have a length? How do you cope with iterables which are infinite? > and I slap myself when I start to write "for e in L". > It is exactly where TOOWTDI applies perfectly and it is integer iteration > for me. It sounds like Python is not a good match for the way you think. I don't say that as a put-down, but perhaps you would be happier if you found another language that works the way you would like, instead of trying to force Python to be something it isn't? -- Steve From mertz at gnosis.cx Sat Feb 18 00:27:35 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 21:27:35 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: On Fri, Feb 17, 2017 at 2:35 PM, Joseph Hackman wrote: > I think we should use the colon to make the delayed word (or whatever word > is selected), unambiguously used in this way (and to prevent any existing > code from breaking). > > On 17 February 2017 at 17:09, David Mertz wrote: > >> That was a problem with the colon that occurred to me. I think it can't >> be tokenized in function annotations. >> > > I don't see any reason for delayed execution and function annotations to > mix. i.e. > def foo(delayed: bar): > pass > would define a function that takes one argument, named delayed, of type > bar. > I still think the colon is ugly and inconsistent with other Python uses. I know you are trying for analogy with lambda (which is related, yes). But the analogies with yield, yield from, async, and await feel much stronger to me. Also, 'lambda' *requires* the colon since it might take arguments and that is necessary to tell when they end.[*] 'delayed' like those other words I mention has no such need. That said, I think you are right that it makes no sense to declare a function signature with 'delayed' (or 'lazy', 'deferred', whatever word). Calling it definitely! This feels important: x = foo(delayed very_complex_computation()) But in the definition signature it feels nonsensical, I agree. However, that still doesn't answer other uses of the colon: {delayed: 17} # No idea if this is a set of one delayed object or a dictionary lambda delayed: delayed: 17 # Just don't know where to start with this All these problems simply go away if we drop the colon. [*] i.e. what would this colon-free lambda mean: 'lambda a, b, c'? A function of no arguments return a tuple? a function of three arguments? -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Feb 18 00:34:08 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 21:34:08 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 6:20 PM, Nathaniel Smith wrote: > > value = delayed: some_dict.get("whatever") > if value is None: > ... > > I.e., the question is, how does 'is' work on delayed objects? I guess > it has to force the promise and walk the proxy chain in each input and > then do an 'is' on the base objects? You've explained the semantics exactly. That's not confusing at all. If the next line after creating delayed 'value' is to check it against something (whether for equality or identity) then obviously it was pointless to make it delayed. But that's not the only pattern: value = delayed some_dict.get(expensive_key_lookup(), expensive_default_calculation()) ... lots more code ... # OK, there might come a time to concretize if some_unrelated_thing and value is None: # do this, but only concretize 'value' if # the first part of conjunction was truthy -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Sat Feb 18 00:34:28 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Sat, 18 Feb 2017 00:34:28 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: Well, yes. I think the 'is' operator is where other attempts fall short, and why it would require a change to Python. But yes, it would need to force the promise. On 17 February 2017 at 21:20, Nathaniel Smith wrote: > On Thu, Feb 16, 2017 at 9:24 PM, Joseph Hackman > wrote: > > Howdy All! > > > > This suggestion is inspired by the question on "Efficient debug logging". > > > > I propose a keyword to mark an expression for delayed/lazy execution, for > > the purposes of standardizing such behavior across the language. > > > > The proposed format is: > > delayed: > > i.e. log.info("info is %s", delayed: expensiveFunction()) > > People seem very excited about this as an idea, but I don't understand > how it can be implemented. > > For example, how do you propose to handle code like this? > > value = delayed: some_dict.get("whatever") > if value is None: > ... > > I.e., the question is, how does 'is' work on delayed objects? I guess > it has to force the promise and walk the proxy chain in each input and > then do an 'is' on the base objects? This seems like a really deep and > confusing change to Python's object model for a pretty marginal > feature. (This is a special case of the general observation that it's > just not possible to implement fully-transparent proxy objects in > Python.) > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Sat Feb 18 00:35:03 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Sat, 18 Feb 2017 00:35:03 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <7FF1CBC9-0077-4EED-B868-D6CB8F3ABEA4@gmail.com> <06903843-9AF5-4CF4-A6AF-FC393314F4DC@gmail.com> Message-ID: I'm not married to the colon. Does anyone else see any issue with dropping it? On 18 February 2017 at 00:27, David Mertz wrote: > On Fri, Feb 17, 2017 at 2:35 PM, Joseph Hackman > wrote: > >> I think we should use the colon to make the delayed word (or whatever >> word is selected), unambiguously used in this way (and to prevent any >> existing code from breaking). >> >> On 17 February 2017 at 17:09, David Mertz wrote: >> >>> That was a problem with the colon that occurred to me. I think it can't >>> be tokenized in function annotations. >>> >> >> I don't see any reason for delayed execution and function annotations to >> mix. i.e. >> def foo(delayed: bar): >> pass >> would define a function that takes one argument, named delayed, of type >> bar. >> > > I still think the colon is ugly and inconsistent with other Python uses. > I know you are trying for analogy with lambda (which is related, yes). But > the analogies with yield, yield from, async, and await feel much stronger > to me. Also, 'lambda' *requires* the colon since it might take arguments > and that is necessary to tell when they end.[*] 'delayed' like those other > words I mention has no such need. > > That said, I think you are right that it makes no sense to declare a > function signature with 'delayed' (or 'lazy', 'deferred', whatever word). > Calling it definitely! This feels important: > > x = foo(delayed very_complex_computation()) > > But in the definition signature it feels nonsensical, I agree. However, > that still doesn't answer other uses of the colon: > > {delayed: 17} # No idea if this is a set of one delayed object or a > dictionary > lambda delayed: delayed: 17 # Just don't know where to start with this > > All these problems simply go away if we drop the colon. > > > [*] i.e. what would this colon-free lambda mean: 'lambda a, b, c'? A > function of no arguments return a tuple? a function of three arguments? > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Feb 18 00:45:48 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 21:45:48 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <20170218012345.GM5689@ando.pearwood.info> References: <20170217111005.GK5689@ando.pearwood.info> <20170218012345.GM5689@ando.pearwood.info> Message-ID: On Fri, Feb 17, 2017 at 5:23 PM, Steven D'Aprano wrote: > > try: > Aardvark > except NameError: > from backport import Aardvark > > No such thing is possible for new syntax. So that counts as a > disadvantage of new syntax. Are we positive that there *must* be new > syntax to solve this problem? > I agree it counts as a disadvantage. But Dask and lazy, and even just using lambdas as "thunks" push what we can do as far as we can without syntax. Those will always require a `obj.compute()` or `obj()` or `eval(obj)` or something else like that to force the "thunk" to concretize. > I think side-effects is a red herring. The obvious rule is: side-effects > occur when the delayed thunk is reified. If you care about the actual > timing of the side-effects, then don't use delayed evaluation. If you > don't care, then who cares if the side-effect is delayed? > Exactly! The same rule applies when writing any computational functions too. If you worry whether they are pure, don't have side effects. Or if you don't care about the side-effects too much (for example, when or if they happen specifically), that's fine and accept it. > > So far I'm going with pretty much anything that isn't being the > right-hand > > of an assignment. So coercion to different types, hashing (for use as a > key > > in a dict or set), __repr__, etc would all be covered, as well as > identity > > and comparisons. i.e.: > [...] > > That will make it pretty much impossible to tell whether something is a > delayed "thunk" or not, since *any* attempt to inspect it in any way > will cause it to reify. Maybe that's what we want. This feels like a disadvantage, and an important one. Most "normal" programmers should never have to care whether something is delayed or has been concretized already. But people writing debuggers, profilers, etc. really do want to know. There should be some way at poking at an object if you really want to without concretizing it. I wouldn't care if this was some ugly and obscure device like 'inspect._is_delayed(my_obj._co_delayed)' that has different semantics than other function calls. Maybe "the uglier the better" in this case, since it *should* be reserved for special purposes only. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Feb 18 00:53:49 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 17 Feb 2017 21:53:49 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <20170217111005.GK5689@ando.pearwood.info> <20170218012345.GM5689@ando.pearwood.info> Message-ID: On Fri, Feb 17, 2017 at 9:45 PM, David Mertz wrote: > That will make it pretty much impossible to tell whether something is a >> > delayed "thunk" or not, since *any* attempt to inspect it in any way >> will cause it to reify. Maybe that's what we want. > > > This feels like a disadvantage, and an important one. Most "normal" > programmers should never have to care whether something is delayed or has > been concretized already. But people writing debuggers, profilers, etc. > really do want to know. > > There should be some way at poking at an object if you really want to > without concretizing it. I wouldn't care if this was some ugly and obscure > device like 'inspect._is_delayed(my_obj._co_delayed)' that has different > semantics than other function calls. Maybe "the uglier the better" in this > case, since it *should* be reserved for special purposes only. > If we assume idempotency (which I'm not certain whether we can/should) then we could spell the check like this: if delayed delayed my_obj is delayed my_obj: print("Yep, it's delayed and I haven't concretized it") That has the dual advantages of being both ugly and obvious. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Feb 18 02:05:10 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 18:05:10 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: <20170218070509.GO5689@ando.pearwood.info> On Sat, Feb 18, 2017 at 03:27:00AM +0100, Mikhail V wrote: > In what sense iteration over integer is limited? It cannot iterate over something where the length is unknown in advance, or infinite, or not meaningfully indexed by integers. Here are four for-loops. How would you re-write this using indexing? Don't forget the list comprehension! for directory, subdirs, files in os.walk(top): files.sort() for f in files: print(os.path.join(top, directory, f)) # skip .git and .hg directories, and those ending with ~ for d in ('.git', '.hg'): try: subdirs.remove(d) except ValueError: pass subdirs[:] = [d for d in subdirs if not d.endswith('~')] subdirs.sort() -- Steve From steve at pearwood.info Sat Feb 18 02:25:43 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 18:25:43 +1100 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) In-Reply-To: References: Message-ID: <20170218072543.GP5689@ando.pearwood.info> On Fri, Feb 17, 2017 at 11:37:04AM -0500, Mark E. Haase wrote: > Python has two string formatting mini-languages. Four. % string formatting, .format method, f-strings, string.Template strings. But who's counting? :-) > Both allow formatting > flags, for example in "%03d", the "0" (zero) is a flag that means to pad > with leading zeroes. I propose to add a string format flag to both > mini-languages that explicitly says, "this argument is callable, and its > *return value* should be formatted." The % string format codes are intentionally limited to more-or-less the similar codes available in C, and won't be given any newer functionality. It's really only f-strings and .format method that this change could be applied to. [...] > The suggested change is to add a "C" flag to the formatting language that > indicates an argument is callable. When formatting the string, the argument > will be called and its result will be used for the formatting operation. > The C flag must be handled before any other flags, regardless of the order > in which they are specified. The callable's return value is then formatted > according to the rest of the specifier. > > With this change, the above example can now be written as: > > >>> expensive = lambda: 2 > >>> logging.debug('%03d %C03d', 1, expensive) > > The "expensive" code is not executed at this log level. I see three problems: (1) It will be a bug magnet. People will accidently write logging.debug('%03d %C03d', 1, expensive()) and then not only will their code still be slow, but they'll have to debug mysterious TypeError: 'int' object is not callable exceptions, but only *sometimes*. Most insideously, these Heisenbugs will only occur when they turn the log level all the way up to debugging, which will crash their program *before* logging the error! (2) It requires the expensive calculation to be wrapped in a lambda: logging.debug('%03d %C03d', 1, lambda: nth_prime(10**8) + 1) which I guess is kind of Python's way of spelling a thunk, but people don't like using lambda for that. (3) It is useless for delaying evaluation of something that isn't going to be converted into a string. -- Steven From mikhailwas at gmail.com Sat Feb 18 02:29:27 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sat, 18 Feb 2017 08:29:27 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On 18 February 2017 at 04:13, Joao S. O. Bueno wrote: > You can still use range. Yes thats what I do, see my proposal > I don't see the point in continuing this thread. How does this add to the syntax discussion? I was replying to Nicks quite vague comments which were supposed to be critics. >> "burden to learn" - I hope you are not serious :) > No, this is serious. > You duplicate the syntax possibilities of one ot he > most basics syntactic elements How do you count duplicate? And what is the sense to speak about 'burden to learn' before the new syntax get approved? If it will be the new syntax, then you will need to learn the new syntax only. You want throw iterables in it, do it: for e over Sequence : I see this probably could have some ambiguity problems but is it such an unsolvable problem? For integers write e.g. like this: for i over 0, N : or for i over *N : What is the problem? > it leads to readability loss What exactly are you talking about? > most Python code these days is read in > a tool that does syntax highlighting You suppose I don't know about it? Then you are extremely underestimating me. Mikhail From toddrjen at gmail.com Sat Feb 18 03:19:21 2017 From: toddrjen at gmail.com (Todd) Date: Sat, 18 Feb 2017 03:19:21 -0500 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: Message-ID: On Feb 18, 2017 02:30, "Mikhail V" wrote: On 18 February 2017 at 04:13, Joao S. O. Bueno wrote: > I don't see the point in continuing this thread. How does this add to the syntax discussion? I was replying to Nicks quite vague comments which were supposed to be critics. There is no point discussing an idea on this mailing list if there is no chance it is going to be implemented. You can talk about it on the general list, as has been suggested to you before. But once an idea has been largely rejected, as this one has, this mailing list is no longer the right place for it. >> "burden to learn" - I hope you are not serious :) > No, this is serious. > You duplicate the syntax possibilities of one ot he > most basics syntactic elements How do you count duplicate? And what is the sense to speak about 'burden to learn' before the new syntax get approved? If it will be the new syntax, then you will need to learn the new syntax only. The existing syntax will not be removed. It would break all python code ever written in the most fundamental way. That is never going to happen. You want throw iterables in it, do it: for e over Sequence : I see this probably could have some ambiguity problems but is it such an unsolvable problem? For integers write e.g. like this: for i over 0, N : or for i over *N : What is the problem? So anyone learning python would need to know both syntaxes to be able to work with other peoples' code. But we would end up with two syntaxes that do exactly the same thing except with either an integer or two or three integers separated by commas, in which case they behave completely differently. That will be confusing even to experienced developers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Sat Feb 18 05:00:14 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sat, 18 Feb 2017 11:00:14 +0100 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: <06b34e3c-838f-02b5-0788-171ecd0096d9@lucidity.plus.com> Message-ID: Hi Erik, I have changed my proposal to the alternative syntax a:: b (Note my preferred spacing. This is to make it read like some annoation applied to the expression, like delayed:: expensive_function()+1 ) Since :: is a binary operator, we need to think about associativity. My conservative proposal would be to make it non-associative, you would have to write explicitly a:: (b:: c) or (a:: b):: c Stephan Op 17 feb. 2017 22:35 schreef "Erik" : On 17/02/17 10:22, Stephan Houben wrote: > Proposal: Light-weight call-by-name syntax in Python > > The following syntax > a : b > is to be interpreted as: > a(lambda: b) > Isn't this too general a syntax? Doesn't it lead to something like: if a: b: c: d: e: pass E. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Feb 18 05:55:32 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 18 Feb 2017 02:55:32 -0800 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 2:46 AM, Stephan Houben wrote: > Hi Nathaniel, > > > 2017-02-17 11:28 GMT+01:00 Nathaniel Smith : >> >> Note that this is definitely a different proposal from the original, >> since the original proposer's goal was to be able to use this with >> existing, unmodified functions that expect a regular value, not a >> lambda. >> >> I don't really see how that goal can be accomplished without massively >> revising Python's runtime model, so this doesn't really bother me, but >> it should be noted :-). > > > Yep ;-) > >> >> >> Anyway, you might also be interested in: >> >> >> https://mail.python.org/pipermail/python-ideas/2016-November/043590.html >> >> which is a similar idea and some more motivating examples, except that >> it allows for the full richness of Python's call syntax, and passes >> ASTs rather than lambdas to allow for non-standard evaluation rules. >> > > Frankly, I think that proposal is too baroque. > The simplicity of my proposal is intentional. I feel like using Python's existing call syntax is less baroque then inventing a new, redundant call syntax that doesn't look like anything else in the language, but I suppose tastes will differ :-). > Just passing in a callable makes it simpler and makes it > also more likely that existing functions could be used with it. This is also potentially a downside, though... the macro-call proposal not only handles the cases you're worrying about, but also handles a bunch of cases that are basically impossible to handle in Python right now. OTOH all the cases where your proposal is useful are ones that Python can already handle at the cost of adding 6 characters. That's not a terribly compelling argument for violating TOOWTDI, IMO. -n -- Nathaniel J. Smith -- https://vorpus.org From stephanh42 at gmail.com Sat Feb 18 05:59:22 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sat, 18 Feb 2017 11:59:22 +0100 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: 2017-02-18 11:55 GMT+01:00 Nathaniel Smith : > > This is also potentially a downside, though... the macro-call proposal > not only handles the cases you're worrying about, but also handles a > bunch of cases that are basically impossible to handle in Python right > now. OTOH all the cases where your proposal is useful are ones that > Python can already handle at the cost of adding 6 characters. That's not a terribly compelling argument for violating TOOWTDI, IMO. > > You know, you are probably right ;-) It was, in any case, more of a strawman counter-proposal to the "delayed: ..." syntax. Stephan > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Feb 18 06:10:09 2017 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 18 Feb 2017 03:10:09 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 9:34 PM, David Mertz wrote: > On Fri, Feb 17, 2017 at 6:20 PM, Nathaniel Smith wrote: >> >> value = delayed: some_dict.get("whatever") >> if value is None: >> ... >> >> I.e., the question is, how does 'is' work on delayed objects? I guess >> it has to force the promise and walk the proxy chain in each input and >> then do an 'is' on the base objects? > > > You've explained the semantics exactly. That's not confusing at all. Okay... so what if I want to check if two objects refer to the same delayed computation? I guess you can say that's just not supported, but that's *extraordinarily weird* for Python. And at the implementation level... so you just added two type checks and two branches to every 'is' call; this seems concerning. And now 'is' can raise an error, which it never could before. You also AFAICT have to modify every single C extension function to check for and handle these things, which is probably impossible even if the overhead of all the checks is acceptable, which isn't obvious. I'm just not seeing how this could be implemented. -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Sat Feb 18 06:39:38 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Feb 2017 22:39:38 +1100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: <20170218113938.GQ5689@ando.pearwood.info> Using "delayed" in function signatures: On Fri, Feb 17, 2017 at 09:27:35PM -0800, David Mertz wrote: > That said, I think you are right that it makes no sense to declare a > function signature with 'delayed' (or 'lazy', 'deferred', whatever word). It makes perfect sense! That gives us function defaults which are evaluated only the first time they are needed, instead of when the function is defined. def function(spam, eggs=delayed nth_prime(10**9)): ... would be equivalent to: _CACHED_DEFAULT = None def function(spam, eggs=None): if eggs is None: if _CACHED_DEFAULT is None: _CACHED_DEFAULT = nth_prime(10**9) return _CACHED_DEFAULT ... I've done this in real life, e.g. to set up an expensive lookup table. The caller might provide their own, but if not, the function has its own default, where I want to delay generating the default until it is actually needed. -- Steve From pavol.lisy at gmail.com Sat Feb 18 08:59:17 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sat, 18 Feb 2017 14:59:17 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <20170218004126.GL5689@ando.pearwood.info> References: <55808B46-8F10-4BBC-9DD9-D359AB388592@gmail.com> <20170218004126.GL5689@ando.pearwood.info> Message-ID: On 2/18/17, Steven D'Aprano wrote: Sorry Steve that I use your words probably too much out of context! I just want to reuse your examples to analyze if proposed "delayed execution" is really necessary. Thanks for them! :) > print("Start") > result = delayed: get_nth_prime(10**6) # I dislike this syntax > print("Done") > print(result) If we change "delayed" keyword to "lambda" keyword then we just need to explicitly say when to evaluate (which I am not sure is bad thing). On 2/18/17, Steven D'Aprano wrote: > The caching means that: > spam = delayed: calculate(1) > eggs = spam > > eggs == spam would be true, and calculate would have only been called once, > not twice. If we really need something like this then we could use "def" keyword (for delayed execution) and explicitely caching (for example with functools.lru_cache). print("Start") @functools.lru_cache(1) def result(): return get_nth_prime(10**6) # this code is delayed print("Done") print(result()) print("Start 2") print(result()) result2 = result print(result() == result2()) print("Done 2") From mikhailwas at gmail.com Sat Feb 18 14:35:53 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sat, 18 Feb 2017 20:35:53 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: <20170218044326.GN5689@ando.pearwood.info> References: <20170218044326.GN5689@ando.pearwood.info> Message-ID: On 18 February 2017 at 05:43, Steven D'Aprano wrote: > On Fri, Feb 17, 2017 at 06:31:19PM +0100, Mikhail V wrote: > >> I have said I need the index, probably you've misread my last comment. >> Further more I explained why I think iteration over index should be the >> preferred way, it help with readability a lot. > > Your concept of readability is clearly radically different from that of > the majority of the Python community. > > >> All my learning years ended up with rewriting most code to "for i in >> range()" > > How do you cope with generators and iterators that don't have a length? > > How do you cope with iterables which are infinite? > > >> and I slap myself when I start to write "for e in L". >> It is exactly where TOOWTDI applies perfectly and it is integer iteration >> for me. > > It sounds like Python is not a good match for the way you think. I don't > say that as a put-down, but perhaps you would be happier if you found > another language that works the way you would like, instead of trying to > force Python to be something it isn't? > > > -- > Steve You mean what my proposal would bring technically better than e.g.: for i,e in enumerate(Seq) Well, nothing, and I will simply use it, with only difference it could be: for i,e over enumerate(Seq) In this case only space holes will be smoothed out, so pure optical fix. As for my comment about indexing preference- something inside me would push me to write something like: for i index Seq: e = Seq[i] One could call the second line 'noise', but I can't explain why, it helps me to read, and I am not biased to C style or anything. Also it probably has to do with variables names, and here I see it on the second line, note that it is not always 'e' and they often make sense, e.g. 'files', 'dirs'. Mikhail From python at lucidity.plus.com Sat Feb 18 19:01:48 2017 From: python at lucidity.plus.com (Erik) Date: Sun, 19 Feb 2017 00:01:48 +0000 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: <20170218044326.GN5689@ando.pearwood.info> Message-ID: <961b8106-ac82-f82b-d09c-5a9ff7346fd6@lucidity.plus.com> On 18/02/17 19:35, Mikhail V wrote: > You mean what my proposal would bring > technically better than e.g.: > > for i,e in enumerate(Seq) > > Well, nothing, and I will simply use it, > with only difference it could be: > > for i,e over enumerate(Seq) > > In this case only space holes will be > smoothed out, so pure optical fix. But you also make the language's structure not make sense. For good or bad, English is the language that the keywords are written in so it makes sense for the Python language constructs to follow English constructs. An iterable in Python (something that can be the target of a 'for' loop) is a collection of objects (whether they represent a sequence of integers, a set of unique values, a list of random things, whatever). It is valid English to say "for each object in my collection, I will do the following:". It is not valid English to say "for each object over my collection, I will do the following:". In that respect, "in" is the correct keyword for Python to use. In the physical world, if the "collection" is some coins in your pocket, would you say "for each coin over my pocket, I will take it out and look at it"? Other than that, I also echo Stephen's comments that not all iterables' lengths can be known in advance, and not all iterables can be indexed, so looping using length and indexing is a subset of what the 'for' loop can do today. Why introduce new syntax for a restricted subset of what can already be done? Soon, someone else will propose another syntax for a different subset. This is why people are talking about the "burden" of learning these extra syntaxes. Rather than 10 different syntaxes for 10 different subsets, why not just learn the one syntax for the general case? E. From eric at trueblade.com Sat Feb 18 19:42:54 2017 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 18 Feb 2017 19:42:54 -0500 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) In-Reply-To: <20170218072543.GP5689@ando.pearwood.info> References: <20170218072543.GP5689@ando.pearwood.info> Message-ID: <72a15ac7-bd72-4931-3646-852b809aac09@trueblade.com> On 2/18/2017 2:25 AM, Steven D'Aprano wrote: > On Fri, Feb 17, 2017 at 11:37:04AM -0500, Mark E. Haase wrote: > >> Python has two string formatting mini-languages. > > Four. % string formatting, .format method, f-strings, string.Template > strings. > > But who's counting? :-) Technical correctness is the best kind! But string.Template has no formatting language, and str.format and f-strings are identical as far as what they do with format specifiers (by using the underlying .__format__() machinery). [0] >> Both allow formatting >> flags, for example in "%03d", the "0" (zero) is a flag that means to pad >> with leading zeroes. I propose to add a string format flag to both >> mini-languages that explicitly says, "this argument is callable, and its >> *return value* should be formatted." > > The % string format codes are intentionally limited to more-or-less the > similar codes available in C, and won't be given any newer functionality. I agree it would be difficult to add this to %-formatting, which is unfortunate. The most often cited need for this feature is for logging. Maybe someone smarter than me could devise an interface to logging which supports .format() formatting, and we could add the callable flag to .format(). Maybe return a different type of logger from logging.getLogger that supports .format? > It's really only f-strings and .format method that this change could be > applied to. I'd suggest adding an explicit conversion flag to the .format() string parser. You'll recall the existing flags are !a, !s, and !r. We could add a !c which calls its argument (with zero parameters) before formatting it. For example: '{!c:%Y-%m-%d}'.format(datetime.datetime.now) '2017-02-18' This would have the advantage that the format specifier would apply to the result of the callable. If logging were made smart enough (or a new interface added), then you could write: logger.info('{!c:%Y-%m-%d} error at {}', datetime.datetime.now, lineno) Then .now() would never be called unless the logging module needed the value. (I realize .now() isn't a great example for calling a function in the future. Substitute any expensive function.) > I see three problems: > > (1) It will be a bug magnet. People will accidently write > > logging.debug('%03d %C03d', 1, expensive()) > > > and then not only will their code still be slow, but they'll have to > debug mysterious > > TypeError: 'int' object is not callable > > exceptions, but only *sometimes*. Most insideously, these Heisenbugs > will only occur when they turn the log level all the way up to > debugging, which will crash their program *before* logging the error! I think this is lessened with my proposal to use !c, but it's definitely still an issue. Test your logging! > (2) It requires the expensive calculation to be wrapped in a lambda: > > logging.debug('%03d %C03d', 1, lambda: nth_prime(10**8) + 1) > > which I guess is kind of Python's way of spelling a thunk, but people > don't like using lambda for that. This doesn't bother me so much. > (3) It is useless for delaying evaluation of something that isn't going > to be converted into a string. True. But that's the most common use case I see presented, and it's much easier to reason about and implement than some of the discussions going on in other threads. Just for completeness, I'd add !c to the f-string parser. Like !s, !r, and !a, it would not be needed [1]. Why write: f'{!c:function}' when you could write: f'{function()}' But, it can't hurt to be consistent. Eric. [0]: there is one slight difference in how str.format and f-strings handle expressions, but their format specifiers are identical https://www.python.org/dev/peps/pep-0498/#differences-between-f-string-and-str-format-expressions [1]: https://www.python.org/dev/peps/pep-0498/#id45 From mikhailwas at gmail.com Sat Feb 18 21:58:56 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 19 Feb 2017 03:58:56 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: <961b8106-ac82-f82b-d09c-5a9ff7346fd6@lucidity.plus.com> References: <20170218044326.GN5689@ando.pearwood.info> <961b8106-ac82-f82b-d09c-5a9ff7346fd6@lucidity.plus.com> Message-ID: On 19 February 2017 at 01:01, Erik wrote: > On 18/02/17 19:35, Mikhail V wrote: >> >> You mean what my proposal would bring >> technically better than e.g.: >> >> for i,e in enumerate(Seq) >> >> Well, nothing, and I will simply use it, >> with only difference it could be: >> >> for i,e over enumerate(Seq) >> >> In this case only space holes will be >> smoothed out, so pure optical fix. > > > But you also make the language's structure not make sense. For good or bad, > English is the language that the keywords are written in so it makes sense > for the Python language constructs to follow English constructs. > > An iterable in Python (something that can be the target of a 'for' loop) is > a collection of objects (whether they represent a sequence of integers, a > set of unique values, a list of random things, whatever). > > It is valid English to say "for each object in my collection, I will do the > following:". > > It is not valid English to say "for each object over my collection, I will > do the following:". > > In that respect, "in" is the correct keyword for Python to use. In the > physical world, if the "collection" is some coins in your pocket, would you > say "for each coin over my pocket, I will take it out and look at it"? That is very fair point and very interesting topic. There is even such classification of languages, how 'natural' are they, and Python is definitely a more natural language. At the same time it is sort of unique selling point of the language, for those who look at it or start to read the code for the first time. Right after that however, pure physics comes in play. So memorizing a visual pattern happens in some minutes of active reading, but further reading lasts "till final victory". Obviously proposing something like "amore" as a keyword would cause only laughter. Nevertherless, the language that uses more reading-effective patterns, in physical sense, will be superior to the language that uses "meaningful" words but which are optically bad,e.g. "all" or "through". And with all respect, "in" happens to be one. > Soon someone else will propose another syntax > for a different subset. This is why people are > talking about the "burden" of learning these > extra syntaxes. Rather than 10 different syntaxes > for 10 different subsets, why not just learn > the one syntax for the general case? I suppose you mean this pair: for i,e in enumerate(Seq): for e in Seq: and conditional if e in Seq: Certainly "over" will not suit in the last example, that is true. But probably swap "in" to "amore" :-) I will be totally ok with it, and it is a good visual pattern. Also it is international, so non-native speakers will learn easier, and +1 to the selling point. None of my words cancel the 'burden', and the 'burden' is tremendous I can imagine. So obviously it is not easy. However most of the burden is not in the word learning . Mikhail From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Feb 19 01:13:06 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 19 Feb 2017 15:13:06 +0900 Subject: [Python-ideas] Efficient debug logging In-Reply-To: <20170216143915.26a7fcf2@subdivisions.wooz.org> References: <81C7E2E9-A666-4840-89A9-E6EE48F79279@barrys-emacs.org> <7b7857c0-ab9b-b1fb-3999-7fcc7cd09c95@egenix.com> <20170216143915.26a7fcf2@subdivisions.wooz.org> Message-ID: <22697.14194.76203.959711@turnbull.sk.tsukuba.ac.jp> Barry Warsaw writes: > On Feb 16, 2017, at 03:20 PM, M.-A. Lemburg wrote: > > >I know some people will disagree, but IMO using "assert" is the wrong > >approach in such situations - it's meant for development and testing > >only, not as short-cut to avoid having to write a proper error > >handler :-) > > I use assertions for "things that can never happen", although sometimes they > do due to an incomplete understanding of the code, the problem, or the > environment. We could interpret that to mean you don't consider that code is ever out of "development and testing" phase, although it's good enough to install in production. I think that is a good philosophy, and I don't see it as a contradiction of what MAL wrote. From pavol.lisy at gmail.com Sun Feb 19 04:23:31 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sun, 19 Feb 2017 10:23:31 +0100 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) In-Reply-To: <72a15ac7-bd72-4931-3646-852b809aac09@trueblade.com> References: <20170218072543.GP5689@ando.pearwood.info> <72a15ac7-bd72-4931-3646-852b809aac09@trueblade.com> Message-ID: On 2/19/17, Eric V. Smith wrote: > On 2/18/2017 2:25 AM, Steven D'Aprano wrote: >> I see three problems: >> >> (1) It will be a bug magnet. People will accidently write >> >> logging.debug('%03d %C03d', 1, expensive()) >> >> >> and then not only will their code still be slow, but they'll have to >> debug mysterious >> >> TypeError: 'int' object is not callable >> >> exceptions, but only *sometimes*. Most insideously, these Heisenbugs >> will only occur when they turn the log level all the way up to >> debugging, which will crash their program *before* logging the error! > > I think this is lessened with my proposal to use !c, but it's definitely > still an issue. Test your logging! 1. I think that error message could be understandable at least as this: '{:g}'.format('a') ValueError: Unknown format code 'g' for object of type 'str' with something like isinstance(arg, callable) 2. Static checker could check bracket presence if there is !c format specifier and fire warning. 3. There could be another problem if return type is callable too... From srkunze at mail.de Sun Feb 19 05:59:11 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Sun, 19 Feb 2017 11:59:11 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: <09c3bd0c-38bb-6648-feb6-ca6aa67c233c@mail.de> I would like to add another view of this feature might be very useful for cleaning up existing code bases: Just have a look at https://pypi.org/project/xfork/ and specifically I would like to point you to the following lines https://github.com/srkunze/fork/blob/afecde0/fork.py#L216 till #419. As you can see these whooping 203 lines are for the sake of implementing a delaying proxy class (of a 520 lines project). If this feature gets added, the almost the whole implementation could *shrink to a mere line* of (I hope): delayed: future.result() That would be awesome! Btw. adding parameters to 'delayed' like lambda would also be useful. [I gather that delayed is a mere placeholder for now.] On 18.02.2017 12:10, Nathaniel Smith wrote: > On Fri, Feb 17, 2017 at 9:34 PM, David Mertz wrote: >> On Fri, Feb 17, 2017 at 6:20 PM, Nathaniel Smith wrote: >>> value = delayed: some_dict.get("whatever") >>> if value is None: >>> ... >>> >>> I.e., the question is, how does 'is' work on delayed objects? I guess >>> it has to force the promise and walk the proxy chain in each input and >>> then do an 'is' on the base objects? >> >> You've explained the semantics exactly. That's not confusing at all. > Okay... so what if I want to check if two objects refer to the same > delayed computation? I guess you can say that's just not supported, > but that's *extraordinarily weird* for Python. It's new to Python yes, but it's not weird. You already can implement such proxies today as I've demonstrated. > And at the implementation level... so you just added two type checks > and two branches to every 'is' call; this seems concerning. And now > 'is' can raise an error, which it never could before. You also AFAICT > have to modify every single C extension function to check for and > handle these things, which is probably impossible even if the overhead > of all the checks is acceptable, which isn't obvious. I'm just not > seeing how this could be implemented. I don't share your concerns here. What you describe is the nature of *delayed*. It's not only applicable to 'is' but to all operations which evaluate delayed objects. My point of view from the other side: me and other people need delayed proxies, so they implement them, and all of us make the same mistakes over and over gain. Cheers, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Feb 19 06:29:52 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 19 Feb 2017 22:29:52 +1100 Subject: [Python-ideas] More classical for-loop In-Reply-To: References: <20170218044326.GN5689@ando.pearwood.info> <961b8106-ac82-f82b-d09c-5a9ff7346fd6@lucidity.plus.com> Message-ID: <20170219112952.GR5689@ando.pearwood.info> On Sun, Feb 19, 2017 at 03:58:56AM +0100, Mikhail V wrote: > Right after that however, pure physics comes in play. > So memorizing a visual pattern happens in some minutes > of active reading, but further reading lasts "till final victory". You think that learning to read is "pure physics". That's very ignorant of physiology, neuroscience and pedagogy. [...] > Nevertherless, the language that uses more > reading-effective patterns, in physical sense, > will be superior to the language that uses "meaningful" > words but which are optically bad,e.g. "all" or "through". > And with all respect, "in" happens to be one. You think that "in" is "optically bad" and "over" is "optically good". Please tell us the optics formula used to determine the optical "goodness" and "badness" of a word. I want to see this physics formula that tells us how good or bad a word is "optically", and I want to know the names of at least a dozen widely available text books that teach about this theory of the optical goodness and badness of words. I want to know who discovered this formula, where it has been published for peer-review by other scientists, and what evidence exists that the formula is factually correct. Unless you do all that, don't bother responding. Your personal opinion about the optical properties of words is not a fact, and we aren't interested. Go start a blog and write about it there for people who care about your theories, and stop wasting our time here. The purpose of this mailing list is to improve the Python language, not to make it match your pseudo-scientific fantasies. There is absolutely no chance that the "in" keyword is going to be replaced by "over". Zero. None. ??????????? ????. This is the last I have to say about this proposal. Don't bother responding to argue, I'm not listening, and I doubt anyone else is either. This proposal is dead in the water, its not going anywhere. -- Steve From edk141 at gmail.com Sun Feb 19 08:03:32 2017 From: edk141 at gmail.com (Ed Kellett) Date: Sun, 19 Feb 2017 13:03:32 +0000 Subject: [Python-ideas] Light-weight call-by-name syntax in Python In-Reply-To: References: Message-ID: On Fri, 17 Feb 2017 at 10:23 Stephan Houben wrote: > Proposal: Light-weight call-by-name syntax in Python > > The following syntax > a : b > is to be interpreted as: > a(lambda: b) > > Effectively, this gives a "light-weight macro system" to Python, > since it allows with little syntax to indicate that the argument to > a function is not to be immediately invoked. > This is, in my view, one case where Python's existing lambda syntax is perfectly sufficient. (I'd even argue it might be the *only* case...) If you could logger.debug(lambda: expensive_to_compute_message_here), I don't think that the delayed-expression proposal would ever have existed. Ed -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sun Feb 19 11:21:07 2017 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 19 Feb 2017 11:21:07 -0500 Subject: [Python-ideas] String Format Callable Flag (Was: Efficient Debug Logging) In-Reply-To: References: <20170218072543.GP5689@ando.pearwood.info> <72a15ac7-bd72-4931-3646-852b809aac09@trueblade.com> Message-ID: On 2/19/2017 4:23 AM, Pavol Lisy wrote: > On 2/19/17, Eric V. Smith wrote: >> On 2/18/2017 2:25 AM, Steven D'Aprano wrote: > >>> I see three problems: >>> >>> (1) It will be a bug magnet. People will accidently write >>> >>> logging.debug('%03d %C03d', 1, expensive()) >>> >>> >>> and then not only will their code still be slow, but they'll have to >>> debug mysterious >>> >>> TypeError: 'int' object is not callable >>> >>> exceptions, but only *sometimes*. Most insideously, these Heisenbugs >>> will only occur when they turn the log level all the way up to >>> debugging, which will crash their program *before* logging the error! >> >> I think this is lessened with my proposal to use !c, but it's definitely >> still an issue. Test your logging! > > 1. I think that error message could be understandable at least as this: > '{:g}'.format('a') > ValueError: Unknown format code 'g' for object of type 'str' > > with something like isinstance(arg, callable) Sure, there would be a reasonable error message. The concern (as with logging anything), is that the error case logging is typically poorly tested. > 2. Static checker could check bracket presence if there is !c format > specifier and fire warning. But it's valid to have a function that returns a callable, so these could be false positives. But linters can do whatever they want, I guess. > 3. There could be another problem if return type is callable too... I don't think that's a problem. You'd just print the callable, as you can do today: >>> def fn(): pass ... >>> format(fn, '') '' From desmoulinmichel at gmail.com Sun Feb 19 11:24:46 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 19 Feb 2017 17:24:46 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> A great proposal, although now I would have to explain to my students the subtle difference between: res = (print(i * i) for i in range(x)) print('foo') print(res) And res = delayed [print(i * i) for i in range(x)] print('foo') all(res) They seems doing something similar, but they really don't. Overall, I still love it. When I read about it, I immidiatly though about how Django handles translation in models: - define your string in english - mark it with ugettext_lazy and NOT ugettext - the framework delays the translation until a request comes around with data about the user lang The proposed featured would solve the problem nicely. Although I'm not clear on the result of: def stuff(arg=delayed []): Does this mean we create a NEW list everytime in the body function ? Or just a new one the first time than the reference stays in arg ? Because the first behavior would solve a problem Python had with mutable default arguments since the begining. But that would mean the result of "delayed []" is a concrete thing we store in arg. The "delayed" keyword sounds a lot like something used in async io, so I like "lazy" much more. Not only it is shorter, but it convey the meaning of what we are doing better. Talking about async, we need to be clear on what those do: a = (await|yield) lazy stuff a = lazy (await|yield) stuff (should it even allowed ?) a = (lazy stuff(x) for x in stuff) a = None with open(x) as f: a = lazy stuff() # raise IOError print(a) try: a = lazy stuff() # raise except Exception: pass a = lazy f'{name}' + stuff(age) # is there a closure where we store "name" and 'age'? I can see a reasonable outcome for most of this, but it must be very clear. However, I can see several very important things we need to be taking in consederation debugging wise. First, if there is an exception in the lazy expression, Python must indicate in the stack trace where this expression has been defined and where it's evaluated. Pdb must also be able to allow easily to step in those in a coherent manner. Evnetually we also may need to allow this: a = lazy stuff if a is not lazy: print(a) But then lazy can't be used a var name to help with the transition. One last thing: my vote is not dropping the ":" in front of they keyword. From mertz at gnosis.cx Sun Feb 19 12:31:27 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 19 Feb 2017 09:31:27 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: On Sun, Feb 19, 2017 at 8:24 AM, Michel Desmoulin wrote: > A great proposal, although now I would have to explain to my students > the subtle difference between: > > res = (print(i * i) for i in range(x)) > res = delayed [print(i * i) for i in range(x)] > They seems doing something similar, but they really don't. Well, at the introductory level they are kinda similar. I know the mechanism would have to be different. But at a first brush it's the difference between delaying the whole concrete collection and delaying one item at a time. That wouldn't be terrible for a first Compsci lesson. > def stuff(arg=delayed []): > > Does this mean we create a NEW list every time in the body function ? Or > just a new one the first time than the reference stays in arg ? > I think this cannot make a new list each time. Of course, I'm one of those people who have used the mutable default deliberately, albeit now it's mostly superseded by functools.lru_cache(). But the idea of a "delayed object" is one that transforms into a concrete (or at least *different* value) on first access. In a sense, the transformation from a delayed object to an iterator is still keeping it lazy; and clearly `x = delayed my_gen()` is a possible pattern. The pattern of `def stuff(arg=delayed expensive_computation(): ...` is important to have. But as in my longer example, `arg` might or might not be accessed in the function body depending on condition execution paths. Still, once `expensive_computation()` happens one time, that should be it, we have a result. Obviously `list()` is not an expensive operation, but the syntax cannot make a boundary for "how costly." > The "delayed" keyword sounds a lot like something used in async io, so I > like "lazy" much more. Not only it is shorter, but it convey the meaning > of what we are doing better. > I like `lazy` too. > a = (await|yield) lazy stuff > a = lazy (await|yield) stuff (should it even allowed ?) > a = (lazy stuff(x) for x in stuff) > a = lazy f'{name}' + stuff(age) # is there a closure where we store "name" > and 'age'? I don't quite have a clear intuition about how lazy/delayed and await/yield/async should interact. I think it would be perfectly consistent with other Python patterns if we decided some combinations cannot be used together. Likewise you can't write `x = await yield from foo`, and that's fine, even though `yield from` is an expression. > First, if there is an exception in the lazy expression, Python must > indicate in the stack trace where this expression has been defined and > where it's evaluated. > Yes. I mentioned that there needs to be *some* way, even if it's an ugly construct, to find out that something is delayed without concretizing it. I think the best idea is hinted at in my preliminary thought. I.e. we can have a special member of a delayed object that does not concretize the object on access. So maybe `object._delayed_code` of something similar. Since it's the interpreter itself, we can say that accessing that member of the object is not a concretization, unlike accessing any other member. Every object that is *not* a delayed/lazy one should probably have None for that value. But delayed ones should have, I guess, the closure that would get executed on access (then once accessed, the object becomes whatever the result of the expression is, with `._delayed_code` then set to None on that transformed object). > a = lazy stuff > if a is not lazy: > print(a) > So my spelling would be: a = lazy stuff if a._delayed_code is not None: print(a) > One last thing: my vote is not dropping the ":" in front of they keyword. > I think the colon has parser problems, as I showed in some examples. Plus I don't like how it looks. But I'd much rather have `a = lazy: stuff` than not have the construct at all, nonetheless. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Sun Feb 19 12:36:14 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 19 Feb 2017 18:36:14 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: <87d3a91b-a6a1-6661-15a0-3b4abd719663@gmail.com> > > One last thing: my vote is not dropping the ":" in front of they > keyword. > > > I think the colon has parser problems, as I showed in some examples. > Plus I don't like how it looks. But I'd much rather have `a = lazy: > stuff` than not have the construct at all, nonetheless. > > This was a typo on my part. I prefer to AVOID the ":" in front of the keyword. From josephhackman at gmail.com Sun Feb 19 13:13:02 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Sun, 19 Feb 2017 13:13:02 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: Michel- Thanks for the feedback! On 19 February 2017 at 11:24, Michel Desmoulin wrote: > A great proposal, although now I would have to explain to my students > the subtle difference between: > > res = (print(i * i) for i in range(x)) > print('foo') > print(res) > > And > > res = delayed [print(i * i) for i in range(x)] > print('foo') > all(res) > > They seems doing something similar, but they really don't. > > Overall, I still love it. > > When I read about it, I immidiatly though about how Django handles > translation in models: > > - define your string in english > - mark it with ugettext_lazy and NOT ugettext > - the framework delays the translation until a request comes around with > data about the user lang > > The proposed featured would solve the problem nicely. > > Although I'm not clear on the result of: > > def stuff(arg=delayed []): > > Does this mean we create a NEW list everytime in the body function ? Or > just a new one the first time than the reference stays in arg ? > > Because the first behavior would solve a problem Python had with mutable > default arguments since the begining. But that would mean the result of > "delayed []" is a concrete thing we store in arg. > My honest preference would be that the [] is evaluated fresh each time the function is called. def stuff(arg=delayed f()): would result in f() being called every time stuff() is. This seems more valuable to me than just doing it once when the function is first called. > > The "delayed" keyword sounds a lot like something used in async io, so I > like "lazy" much more. Not only it is shorter, but it convey the meaning > of what we are doing better. > I'm fine with either delayed or lazy. > Talking about async, we need to be clear on what those do: > > a = (await|yield) lazy stuff > suggestion: a = await lazy stuff # same as await stuff, the await forces the lazy to be evaluated. a = yield lazy stuff # yields a lazy expression that will be evaluated when read, a is still set on the push as usual. > a = lazy (await|yield) stuff (should it even allowed ?) > a = lazy await stuff # returns a lazy expression that, when evaluated will await stuff. I know this is dangerous, but I think it fits the pattern and Python is a 'consenting adults' language. If it attempts to evaluate outside a coroutine, I'm fine with it raising an exception.I'm also totally cool with this not being allowed. a = lazy yield stuff # the generator doesn't yield/pause until a is read from. see above. a = (lazy stuff(x) for x in stuff) > a generator that returns lazy expressions that are not executed unless read. > a = None > ? > with open(x) as f: > a = lazy stuff() # raise IOError > print(a) > try: > a = lazy stuff() # raise > except Exception: > pass > I think this is one of the best points. My guess is that the exception should be raised where the expression is evaluated. We're all consenting adults here, and if you want to cause an uncaught exception somewhere, who am I to stop you? a = lazy f'{name}' + stuff(age) # is there a closure where we store > "name" and 'age'? > I suggest yes, where possible/reasonable. I can see a reasonable outcome for most of this, but it must be very clear. > > However, I can see several very important things we need to be taking in > consederation debugging wise. > > First, if there is an exception in the lazy expression, Python must > indicate in the stack trace where this expression has been defined and > where it's evaluated. > > Pdb must also be able to allow easily to step in those in a coherent > manner. > > Evnetually we also may need to allow this: > > a = lazy stuff > if a is not lazy: > print(a) > I do think that this is probably the best greenfield solution for the problem. My only strong feeling is that it should be VERY difficult to 'accidentally' inspect a lazy, rather than evaluating it. > But then lazy can't be used a var name to help with the transition. > Yeah. :( > One last thing: my vote is not dropping the ":" in front of they keyword. I don't understand what your meaning is here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Feb 19 13:33:10 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 19 Feb 2017 10:33:10 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: On Sun, Feb 19, 2017 at 10:13 AM, Joseph Hackman wrote: > > My honest preference would be that the [] is evaluated fresh each time the > function is called. > def stuff(arg=delayed f()): > would result in f() being called every time stuff() is. This seems more > valuable to me than just doing it once when the function is first called. > This doesn't make sense. Function definition time is very different than function execution time. Changing that distinction is a WAY bigger change than I think we should contemplate. Moreover, there is a completely obvious way to spell the behavior you want: def stuff(): arg = f() # ... whatever ... This is exactly the obvious way to spell "f() is called every time stuff() is". -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Sun Feb 19 13:47:34 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Sun, 19 Feb 2017 13:47:34 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: > > This doesn't make sense. Function definition time is very different than > function execution time. Changing that distinction is a WAY bigger change > than I think we should contemplate. > Moreover, there is a completely obvious way to spell the behavior you want: > def stuff(): > arg = f() > # ... whatever ... > This is exactly the obvious way to spell "f() is called every time stuff() > is". I think it would be useful, but yeah, it really doesn't fit in with the rest of lazy/delayed. The present format of defaulting to none and then doing if arg is None: is totally functional. On the flip side, doing a lazy in the function definition would save time evaluating defaults while also fitting in. Your argument has convinced me, and I now take (what i believe to be) your position: def stuff(arg = lazy f()): should result in a function where the default value of arg is not evaluated until first function call, and then the value of the expression is used as the default. Now, back to what Michel was probably actually asking. In the case of: def stuff(arg = lazy []): is the default value of arg a new list with each execution? (i.e. the resulting value of the expression is `make a new list`) I would say that for consistency's sake, not, which I believe would be consistent with the logic behind why default values being [] are kept between calls. a = lazy [] b = a a.append('a') print(b) # expected behavior is ['a'] I maintain that it would be nice for there to be a way to say (the default value of this argument is to run some expression *every time*), but delayed/lazy probably isn't that. On 19 February 2017 at 13:33, David Mertz wrote: > On Sun, Feb 19, 2017 at 10:13 AM, Joseph Hackman > wrote: >> >> My honest preference would be that the [] is evaluated fresh each time >> the function is called. >> def stuff(arg=delayed f()): >> would result in f() being called every time stuff() is. This seems more >> valuable to me than just doing it once when the function is first called. >> > > This doesn't make sense. Function definition time is very different than > function execution time. Changing that distinction is a WAY bigger change > than I think we should contemplate. > > Moreover, there is a completely obvious way to spell the behavior you want: > > def stuff(): > > arg = f() > > # ... whatever ... > > > This is exactly the obvious way to spell "f() is called every time stuff() > is". > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Feb 19 14:27:38 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 19 Feb 2017 11:27:38 -0800 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: On Sun, Feb 19, 2017 at 10:47 AM, Joseph Hackman wrote: > Your argument has convinced me, and I now take (what i believe to be) your >> position: >> > > def stuff(arg = lazy f()): > > should result in a function where the default value of arg is not > evaluated until first function call, and then the value of the expression > is used as the default. > Indeed. And in particular, f() *might not* be excuted even during that first (or any) function call, depending on what conditional path are taken within the function body. That's the crucial part. The function may have perfectly good uses where you don't want to take the computational time, or have the side-effects, but other uses where you need that deferred value or action. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Sun Feb 19 15:25:02 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 19 Feb 2017 21:25:02 +0100 Subject: [Python-ideas] More classical for-loop In-Reply-To: <20170219112952.GR5689@ando.pearwood.info> References: <20170218044326.GN5689@ando.pearwood.info> <961b8106-ac82-f82b-d09c-5a9ff7346fd6@lucidity.plus.com> <20170219112952.GR5689@ando.pearwood.info> Message-ID: On 19 February 2017 at 12:29, Steven D'Aprano wrote: > > Please tell us the optics formula used to determine the optical > "goodness" and "badness" of a word. I want to see this physics formula > that tells us how good or bad a word is "optically", and I want to know > the names of at least a dozen widely available text books that teach > about this theory of the optical goodness and badness of words. I've understood already that ther is no interest in proposal and that is fine. If you want something about theories, the only way is that you PM me and I'll see what I can do for you free of charge. Mikhail From pavol.lisy at gmail.com Sun Feb 19 17:15:14 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sun, 19 Feb 2017 23:15:14 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: On 2/19/17, David Mertz wrote: > On Sun, Feb 19, 2017 at 10:13 AM, Joseph Hackman > wrote: >> >> My honest preference would be that the [] is evaluated fresh each time >> the >> function is called. >> def stuff(arg=delayed f()): >> would result in f() being called every time stuff() is. This seems more >> valuable to me than just doing it once when the function is first called. >> > > This doesn't make sense. Function definition time is very different than > function execution time. Changing that distinction is a WAY bigger change > than I think we should contemplate. > > Moreover, there is a completely obvious way to spell the behavior you want: > > def stuff(): > > arg = f() > > # ... whatever ... > > > This is exactly the obvious way to spell "f() is called every time stuff() > is". A few more complicated will be set non default value to argument. But how to spell same behaviour in this case? -> def stuff(arg=delayed locals()) From pavol.lisy at gmail.com Sun Feb 19 17:54:38 2017 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Sun, 19 Feb 2017 23:54:38 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> References: <4a6ef568-589a-84bb-666d-e6bf6d2534b4@gmail.com> Message-ID: On 2/19/17, Michel Desmoulin wrote: > Evnetually we also may need to allow this: > > a = lazy stuff > if a is not lazy: > print(a) > > But then lazy can't be used a var name to help with the transition. What about this? if not inspect.islazy(a): print(a) Next idea is probably obvious: class Busy_Beaver: ''' we want to be sure that beaver is disturbed only if it is really necessary ''' def __call_me_later__(self, n): return too_expensive(n) From desmoulinmichel at gmail.com Mon Feb 20 03:30:17 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 20 Feb 2017 09:30:17 +0100 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: I wrote a blog post about this, and someone asked me if it meant allowing lazy imports to make optional imports easier. Someting like: lazy import foo lazy from foo import bar So now if I don't use the imports, the module is not loaded, which could also significantly speed up applications starting time with a lot of imports. From tritium-list at sdamon.com Mon Feb 20 06:18:19 2017 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Mon, 20 Feb 2017 06:18:19 -0500 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: Message-ID: <159901d28b6b$0aef9280$20ceb780$@hotmail.com> > -----Original Message----- > From: Python-ideas [mailto:python-ideas-bounces+tritium- > list=sdamon.com at python.org] On Behalf Of Michel Desmoulin > Sent: Monday, February 20, 2017 3:30 AM > To: python-ideas at python.org > Subject: Re: [Python-ideas] Delayed Execution via Keyword > > I wrote a blog post about this, and someone asked me if it meant > allowing lazy imports to make optional imports easier. > > Someting like: > > lazy import foo > lazy from foo import bar > > So now if I don't use the imports, the module is not loaded, which could > also significantly speed up applications starting time with a lot of > imports. Would that not also make a failure to import an error at the time of executing the imported piece of code rather than at the place of import? And how would optional imports work if they are not loaded until use? Right now, optional imports are done by wrapping the import statement in a try/except, would you not need to do that handling everywhere the imported object is used instead? (I haven't been following the entire thread, and I don't know if this is a forest/tress argument) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rymg19 at gmail.com Mon Feb 20 15:54:34 2017 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 20 Feb 2017 14:54:34 -0600 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: Apologies if this has already been covered! Right now, if you want to get multiple elements in a list, you have to do: elements = [mylist[a], mylist[b]] My proposal is two-folded: - Right now, a[b,c] is already valid syntax, since it's just indexing a with the tuple (b, c). The proposal is to make this a specialization in the grammar, and also allow stuff like a[b:c, d:e] (like `a.__getitem__(slice(b, c), slice(d, e))`). - Add support for indexing via tuples in list.__getitem__. list.__getitem__(tuple) would roughly be equivalent to map(list.__getitem__, tuple). The first part is solely so that slices would be allowed in the syntax, but if you guys don't like the idea, the second part still stands. Thoughts? *ducks from flying tomatoes* -- Ryan (????) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rainventions at gmail.com Mon Feb 20 16:04:28 2017 From: rainventions at gmail.com (Ryan Birmingham) Date: Mon, 20 Feb 2017 16:04:28 -0500 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: So, to make sure I have this right: your proposal says array should be indexable by a list of indexes as they're currently done, in a tuple, right? Would this also mean that something like (1:4, 8:10, 13) should be an acceptable constructor for a tuple? -Ryan Birmingham On 20 February 2017 at 15:54, Ryan Gonzalez wrote: > Apologies if this has already been covered! > > Right now, if you want to get multiple elements in a list, you have to do: > > elements = [mylist[a], mylist[b]] > > My proposal is two-folded: > > - Right now, a[b,c] is already valid syntax, since it's just indexing a > with the tuple (b, c). The proposal is to make this a specialization in the > grammar, and also allow stuff like a[b:c, d:e] (like > `a.__getitem__(slice(b, c), slice(d, e))`). > > - Add support for indexing via tuples in list.__getitem__. > list.__getitem__(tuple) would roughly be equivalent to > map(list.__getitem__, tuple). > > The first part is solely so that slices would be allowed in the syntax, > but if you guys don't like the idea, the second part still stands. > > Thoughts? *ducks from flying tomatoes* > > -- > Ryan (????) > Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else > http://refi64.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcgoble3 at gmail.com Mon Feb 20 16:05:57 2017 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 20 Feb 2017 21:05:57 +0000 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: On Mon, Feb 20, 2017 at 3:55 PM Ryan Gonzalez wrote: > - Right now, a[b,c] is already valid syntax, since it's just indexing a > with the tuple (b, c). The proposal is to make this a specialization in the > grammar, and also allow stuff like a[b:c, d:e] (like > `a.__getitem__(slice(b, c), slice(d, e))`). > The syntax/grammar already permits slices to be used in this fashion: Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> class Tester: ... def __getitem__(self, *args): ... print('object was indexed with these args:', args) ... >>> obj = Tester() >>> obj[3:5, 2:6, 7, -15:6:3] object was indexed with these args: ((slice(3, 5, None), slice(2, 6, None), 7, slice(-15, 6, 3)),) -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Mon Feb 20 16:14:33 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Mon, 20 Feb 2017 22:14:33 +0100 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: On 20 February 2017 at 22:05, Jonathan Goble wrote: > On Mon, Feb 20, 2017 at 3:55 PM Ryan Gonzalez wrote: > >> - Right now, a[b,c] is already valid syntax, since it's just indexing a >> with the tuple (b, c). The proposal is to make this a specialization in the >> grammar, and also allow stuff like a[b:c, d:e] (like >> `a.__getitem__(slice(b, c), slice(d, e))`). >> > > The syntax/grammar already permits slices to be used in this fashion: > Moreover, this syntax is already extensively used by numpy and means something different - multidimensional slicing. Having a different semantics for lists would be therefore confusing. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 20 17:12:37 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 20 Feb 2017 22:12:37 +0000 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: On 20 February 2017 at 20:54, Ryan Gonzalez wrote: > Apologies if this has already been covered! > > Right now, if you want to get multiple elements in a list, you have to do: > > elements = [mylist[a], mylist[b]] > > My proposal is two-folded: > > - Right now, a[b,c] is already valid syntax, since it's just indexing a with > the tuple (b, c). The proposal is to make this a specialization in the > grammar, and also allow stuff like a[b:c, d:e] (like `a.__getitem__(slice(b, > c), slice(d, e))`). I'm not sure what you mean by a "specialisation in the grammar". This is currently valid syntax. It's only list.__getitem__ that rejects it: >>> lst[4:5,6] Traceback (most recent call last): File "", line 1, in TypeError: list indices must be integers or slices, not tuple That error comes from lst.__getitem__, not from the grammar. > - Add support for indexing via tuples in list.__getitem__. > list.__getitem__(tuple) would roughly be equivalent to map(list.__getitem__, > tuple). > > The first part is solely so that slices would be allowed in the syntax, but > if you guys don't like the idea, the second part still stands. But they already are... > Thoughts? *ducks from flying tomatoes* Thoughts on the second path, then. I presume you're aware this can be done right now using a helper function. And indeed you point out yourself that it's basically map(lst.__getitem__, seq). So I guess the key question is what *additional* benefit do you see to this proposal that justifies adding it to the builtin list type *now*, when people have managed fine this far without it. Paul From abedillon at gmail.com Mon Feb 20 19:52:49 2017 From: abedillon at gmail.com (Abe Dillon) Date: Mon, 20 Feb 2017 18:52:49 -0600 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: <159901d28b6b$0aef9280$20ceb780$@hotmail.com> References: <159901d28b6b$0aef9280$20ceb780$@hotmail.com> Message-ID: On Fri, Feb 17, 2017, Steven D'Aprano wrote: > JIT compilation delays *compiling* the code to run-time. This is a > proposal for delaying *running* the code until such time as some other > piece of code actually needs the result. My thought was that if a compiler is capable of determining what needs to be compiled just in time, then an interpreter might be able to determine what expressions need to be evaluated just when their results are actually used. So if you had code that looked like: >>> log.debug("data: %s", expensive()) The interpreter could skip evaluating the expensive function if the result is never used. It would only evaluate it "just in time". This would almost certainly require just in time compilation as well, otherwise the byte code that calls the "log.debug" function would be unaware of the byte code that implements the function. This is probably a pipe-dream, though; because the interpreter would have to be aware of side effects. On Mon, Feb 20, 2017 at 5:18 AM, wrote: > > > > -----Original Message----- > > From: Python-ideas [mailto:python-ideas-bounces+tritium- > > list=sdamon.com at python.org] On Behalf Of Michel Desmoulin > > Sent: Monday, February 20, 2017 3:30 AM > > To: python-ideas at python.org > > Subject: Re: [Python-ideas] Delayed Execution via Keyword > > > > I wrote a blog post about this, and someone asked me if it meant > > allowing lazy imports to make optional imports easier. > > > > Someting like: > > > > lazy import foo > > lazy from foo import bar > > > > So now if I don't use the imports, the module is not loaded, which could > > also significantly speed up applications starting time with a lot of > > imports. > > Would that not also make a failure to import an error at the time of > executing the imported piece of code rather than at the place of import? > And how would optional imports work if they are not loaded until use? > Right > now, optional imports are done by wrapping the import statement in a > try/except, would you not need to do that handling everywhere the imported > object is used instead? > > (I haven't been following the entire thread, and I don't know if this is a > forest/tress argument) > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.morton13 at gmail.com Mon Feb 20 20:07:30 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Tue, 21 Feb 2017 01:07:30 +0000 Subject: [Python-ideas] Delayed Execution via Keyword In-Reply-To: References: <159901d28b6b$0aef9280$20ceb780$@hotmail.com> Message-ID: This comes from a bit of a misunderstanding of how an interpreter figures out what needs to be compiled. Most (all?) JIT compilers run code in an interpreted manner, and then compile subsections down to efficient machine code when they notice that the same code path is taken repeatedly, so in pypy something like x = 0 for i in range(100000): x += 1 would, get, after 10-20 runs through the loop, turned into assembly that looked like what you'd write in pure C, instead of the very indirection and pointer heavy code that such a loop would be if you could take it and convert it to cpython actually executes, for example. So the "hot" code is still run. All that said, this is a bit of an off topic discussion and probably shouldn't be on list. What you really do want is functional purity, which is a different concept and one that python as a language can't easily provide no matter what. --Josh On Mon, Feb 20, 2017 at 7:53 PM Abe Dillon wrote: > On Fri, Feb 17, 2017, Steven D'Aprano wrote: > > JIT compilation delays *compiling* the code to run-time. This is a > proposal for delaying *running* the code until such time as some other > piece of code actually needs the result. > > > My thought was that if a compiler is capable of determining what needs to > be compiled just in time, then an interpreter might be able to determine > what expressions need to be evaluated just when their results are actually > used. > > So if you had code that looked like: > > >>> log.debug("data: %s", expensive()) > > The interpreter could skip evaluating the expensive function if the result > is never used. It would only evaluate it "just in time". This would almost > certainly require just in time compilation as well, otherwise the byte code > that calls the "log.debug" function would be unaware of the byte code that > implements the function. > > This is probably a pipe-dream, though; because the interpreter would have > to be aware of side effects. > > > > On Mon, Feb 20, 2017 at 5:18 AM, wrote: > > > > > -----Original Message----- > > From: Python-ideas [mailto:python-ideas-bounces+tritium- > > list=sdamon.com at python.org] On Behalf Of Michel Desmoulin > > Sent: Monday, February 20, 2017 3:30 AM > > To: python-ideas at python.org > > Subject: Re: [Python-ideas] Delayed Execution via Keyword > > > > I wrote a blog post about this, and someone asked me if it meant > > allowing lazy imports to make optional imports easier. > > > > Someting like: > > > > lazy import foo > > lazy from foo import bar > > > > So now if I don't use the imports, the module is not loaded, which could > > also significantly speed up applications starting time with a lot of > > imports. > > Would that not also make a failure to import an error at the time of > executing the imported piece of code rather than at the place of import? > And how would optional imports work if they are not loaded until use? > Right > now, optional imports are done by wrapping the import statement in a > try/except, would you not need to do that handling everywhere the imported > object is used instead? > > (I haven't been following the entire thread, and I don't know if this is a > forest/tress argument) > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Feb 20 23:34:48 2017 From: mertz at gnosis.cx (David Mertz) Date: Mon, 20 Feb 2017 23:34:48 -0500 Subject: [Python-ideas] List indexing multiple elements In-Reply-To: References: Message-ID: On Mon, Feb 20, 2017 at 12:54 PM, Ryan Gonzalez wrote: > elements = [mylist[a], mylist[b]] > - Right now, a[b,c] is already valid syntax, since it's just indexing a > with the tuple (b, c). The proposal is to make this a specialization in the > grammar, and also allow stuff like a[b:c, d:e] (like > `a.__getitem__(slice(b, c), slice(d, e))`). > This part is DOA. As someone else notes, use of tuples as indexers is commonplace in NumPy (and Pandas, XArray, Dask, etc). In those collection types, the comma refers to dimensions of the data. However, as someone else notes, you can create whatever meaning you want for indexing with a tuple. I think it would be confusing to give a meaning very different from that in NumPy; but it's perfectly syntactical. class MultisliceList(list): def __getitem__(self, arg): if isinstance(arg, int) or isinstance(arg, slice): return list.__getitem__(self, arg) elif isinstance(arg, tuple): indices = set() for x in arg: if isinstance(x, int): indices.add(x) elif isinstance(x, slice): for i in range(x.start or 0, x.stop, x.step or 1): indices.add(i) else: raise NotImplementedError("Can only index with ints and slices") else: raise NotImplementedError("Can only index with ints and slices") return MultisliceList([list.__getitem__(self, i) for i in sorted(indices)])) >>> l = MultisliceList(range(1000,0,-5)) >>> l[10:15], type(l[10:15]) ([950, 945, 940, 935, 930], list) >>> l[9], type(l[9]) (955, int) >>> msl = l[9,110:115,10:15,100] >>> msl, type(msl) ([955, 950, 945, 940, 935, 930, 500, 450, 445, 440, 435, 430], __main__.MultisliceList) I decided that the various index positions indicated should be in sorted order (there might be overlap in tuple items). Also I didn't make a single slice stay as the special class. You can write your version however you like. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagenaarhenkjaap at gmail.com Thu Feb 23 08:37:15 2017 From: wagenaarhenkjaap at gmail.com (Henk-Jaap Wagenaar) Date: Thu, 23 Feb 2017 13:37:15 +0000 Subject: [Python-ideas] if in for-loop statement Message-ID: Hi all, Often I have typed something like for x in range(100) if is_prime(x): # do things with x to find that this does not work, instead resorting to: for x in range(100): if is_prime(x): # do things with x or for x in range(100): if not is_prime(x): continue # do things with x Other solutions to another case of this 'problem' are discussed has been discussed on StackOverflow ( http://stackoverflow.com/questions/6981717/pythonic-way-to-combine-for-loop-and-if-statement) where it is suggested one uses a generator expression before the loop. None of these solutions seem very Pythonic to me. I appreciate there is a cost associated with changing the language syntax, and I do not understand all the finer details of the inner workings involved with the Python language development, however in my limited understanding in it would be: - fully backwards compatible, - require one to change "expression_list" to "or_test [comp_iter]" in the syntax of the for statement (if I got it right). - it would mean there is a Pythonic solution to a current 'problem' that does not have one. A few problems I foresee: - One wants for loops to bleed their target_list (that is the point normally), so this is different from generators, - This allows for nesting of generators, like in a generator expression which might be hard to implement? Note that this has been suggested before at least once ( https://mail.python.org/pipermail/python-dev/2007-November/075257.html), and that thread itself suggests it has been suggested before and shutdown by Guido (though no source is given for this). All the best, Henk-Jaap Wagenaar -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 23 08:46:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Feb 2017 13:46:04 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: On 23 February 2017 at 13:37, Henk-Jaap Wagenaar wrote: > Note that this has been suggested before at least once > (https://mail.python.org/pipermail/python-dev/2007-November/075257.html), > and that thread itself suggests it has been suggested before and shutdown by > Guido (though no source is given for this). Thanks for your interest, but as you note this has already been proposed and rejected (from my personal recollection, more than once). So the key question is surely, what are you proposing that wasn't already discussed in previous threads, and why do you think that the new aspect of your proposal is likely to change Guido's mind? Sorry to be blunt, but rehashing an already extensively discussed idea isn't that productive for anyone. Paul From dmoisset at machinalis.com Thu Feb 23 08:54:37 2017 From: dmoisset at machinalis.com (Daniel Moisset) Date: Thu, 23 Feb 2017 13:54:37 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: Just as a reference, I think the most recent reincarnation of this thread was: https://mail.python.org/pipermail/python-ideas/2016-September/042270.html On 23 February 2017 at 13:46, Paul Moore wrote: > On 23 February 2017 at 13:37, Henk-Jaap Wagenaar > wrote: > > Note that this has been suggested before at least once > > (https://mail.python.org/pipermail/python-dev/2007-November/075257.html > ), > > and that thread itself suggests it has been suggested before and > shutdown by > > Guido (though no source is given for this). > > Thanks for your interest, but as you note this has already been > proposed and rejected (from my personal recollection, more than once). > > So the key question is surely, what are you proposing that wasn't > already discussed in previous threads, and why do you think that the > new aspect of your proposal is likely to change Guido's mind? > > Sorry to be blunt, but rehashing an already extensively discussed idea > isn't that productive for anyone. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagenaarhenkjaap at gmail.com Thu Feb 23 09:20:34 2017 From: wagenaarhenkjaap at gmail.com (Henk-Jaap Wagenaar) Date: Thu, 23 Feb 2017 14:20:34 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: Hi Paul, Daniel, others, That is fair enough and the right amount of blunt. Let me go through the points that were made in that thread (I have not found any other threads, if someone links to them, I will go through them as well): ----- From: https://mail.python.org/pipermail/python-dev/2007-November/075258.html, alternative solution (in current language): > for x in (x for x in range if is_prime(x)): > # do something with x This statement seems overly complicated for what it is trying to achieve and is semantically much harder to parse than the proposed alternative, e.g. there is repetition of the string "for x in". ---- A similar point to where I started from in https://mail.python.org/piperm ail/python-ideas/2016-September/042600.html: >"Sorry to re-raise this thread--I'm inclined to agree that the case >doesn't really warrant new syntax. I just wanted to add that I think >the very fact that this syntax is supported by list comprehensions is >an argument *in its favor*. > >I could easily see a Python newbie being confused that they can write >"for x in y if z" inside a list comprehension, but not in a bare >for-statement. Sure they'd learn quickly enough that the filtering >syntax is unique to list comprehensions. But to anyone who doesn't >know the historical progression of the Python language that would seem >highly arbitrary and incongruous I would think. > >Just $0.02 USD from a pedagogical perspective. > >Erik" ---- The thread Daniel mentioned was ended by the following post https://mail.python.org/pipermail/python-ideas/2016-September/042285.html >Generally speaking, we only add new syntax in cases where we're >prepared to say "In all cases where the new syntax applies, it should >be used in preference to existing alternative spellings". > >Special casing a single "if-continue" in a loop body doesn't meet that >(deliberately high) bar, with Paul Moore's email going into some more >detail on the specifics of that. > >Cheers, >Nick." I am not sure I agree with what he is saying (or I am not following it...). I think no-one in that thread particularly demonstrated why this could not supersede all equivalent solutions, even if the if is longer (but not complex), one can simply do: for x in range(10) \ if is_prime(x) and is_overabundant(x) and is_perfect(x) and x != 5: # do something with x where you would line up the if with the range/... on the previous line (this example is silly, because the conditions cannot be satisfied). This in my view would still be clearer and more concise than the alternative solutions. The emails in the threads seem to indicate that almost everyone is a shade of grey here, where some really like it (but can life without), some think it is alright but don't particularly mind, and some think it is not the worth the cost or would allow abuse. No-one seems to fundamentally hate the idea, or in a alternative world where it was already in Python, would suggest taking it out (even if it a backwards-incompatible epoch like 2->3 was approaching) though my reading between the lines of emails can be wrong. I think Erik's email (see above) shows this most clearly. In a straw poll at the company I work at everyone was in favour, though they obviously are not in charge of implementing or changing documentation so that is easy for them to say, they've got no skin in the game. I don't know whether it is common for such an idea to be brought up again and again by newcomers/those who don't strolls the archives enough, but if it keeps being brought up, and the main argument against is it would take time and effort to document and implement for little benefit, if Python sticks around for a long enough time, it will end up taking less time simply implement it! Best, Henk-Jaap On 23 February 2017 at 13:54, Daniel Moisset wrote: > Just as a reference, I think the most recent reincarnation of this thread > was: https://mail.python.org/pipermail/python-ideas/2016-Sep > tember/042270.html > > On 23 February 2017 at 13:46, Paul Moore wrote: > >> On 23 February 2017 at 13:37, Henk-Jaap Wagenaar >> wrote: >> > Note that this has been suggested before at least once >> > (https://mail.python.org/pipermail/python-dev/2007-November/075257.html >> ), >> > and that thread itself suggests it has been suggested before and >> shutdown by >> > Guido (though no source is given for this). >> >> Thanks for your interest, but as you note this has already been >> proposed and rejected (from my personal recollection, more than once). >> >> So the key question is surely, what are you proposing that wasn't >> already discussed in previous threads, and why do you think that the >> new aspect of your proposal is likely to change Guido's mind? >> >> Sorry to be blunt, but rehashing an already extensively discussed idea >> isn't that productive for anyone. >> >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Daniel F. Moisset - UK Country Manager > www.machinalis.com > Skype: @dmoisset > On 23 February 2017 at 13:54, Daniel Moisset wrote: > Just as a reference, I think the most recent reincarnation of this thread > was: https://mail.python.org/pipermail/python-ideas/2016-Sep > tember/042270.html > > On 23 February 2017 at 13:46, Paul Moore wrote: > >> On 23 February 2017 at 13:37, Henk-Jaap Wagenaar >> wrote: >> > Note that this has been suggested before at least once >> > (https://mail.python.org/pipermail/python-dev/2007-November/075257.html >> ), >> > and that thread itself suggests it has been suggested before and >> shutdown by >> > Guido (though no source is given for this). >> >> Thanks for your interest, but as you note this has already been >> proposed and rejected (from my personal recollection, more than once). >> >> So the key question is surely, what are you proposing that wasn't >> already discussed in previous threads, and why do you think that the >> new aspect of your proposal is likely to change Guido's mind? >> >> Sorry to be blunt, but rehashing an already extensively discussed idea >> isn't that productive for anyone. >> >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Daniel F. Moisset - UK Country Manager > www.machinalis.com > Skype: @dmoisset > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jelle.zijlstra at gmail.com Thu Feb 23 09:51:12 2017 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Thu, 23 Feb 2017 06:51:12 -0800 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: 2017-02-23 5:37 GMT-08:00 Henk-Jaap Wagenaar : > > Hi all, > > Often I have typed something like > > for x in range(100) if is_prime(x): > # do things with x > > to find that this does not work, instead resorting to: > > for x in range(100): > if is_prime(x): > # do things with x > > or > > for x in range(100): > if not is_prime(x): > continue > # do things with x > > Other solutions to another case of this 'problem' are discussed has been discussed on StackOverflow (http://stackoverflow.com/questions/6981717/pythonic-way-to-combine-for-loop-and-if-statement) where it is suggested one uses a generator expression before the loop. None of these solutions seem very Pythonic to me. > > I appreciate there is a cost associated with changing the language syntax, and I do not understand all the finer details of the inner workings involved with the Python language development, however in my limited understanding in it would be: > - fully backwards compatible, > - require one to change "expression_list" to "or_test [comp_iter]" in the syntax of the for statement (if I got it right). > - it would mean there is a Pythonic solution to a current 'problem' that does not have one. > > A few problems I foresee: > - One wants for loops to bleed their target_list (that is the point normally), so this is different from generators, > - This allows for nesting of generators, like in a generator expression which might be hard to implement? > I think it might also be difficult to parse. Python currently supports this syntax: In [8]: for x in [1, 2] if True else [1, 2, 3]: ...: print(x) ...: 1 2 I'm not sure the parser could distinguish this structure from the one you propose (which would have a very different meaning) without making it significantly more complicated. Changes that make the parser more complicated than LL(1) have been consistently rejected in the past; see https://www.python.org/dev/peps/pep-3099/. > > Note that this has been suggested before at least once (https://mail.python.org/pipermail/python-dev/2007-November/075257.html), and that thread itself suggests it has been suggested before and shutdown by Guido (though no source is given for this). > > All the best, > > Henk-Jaap Wagenaar > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Thu Feb 23 10:02:18 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Feb 2017 15:02:18 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: On 23 February 2017 at 14:20, Henk-Jaap Wagenaar wrote: > > In a straw poll at the company I work at everyone was in favour, though they > obviously are not in charge of implementing or changing documentation so > that is easy for them to say, they've got no skin in the game. I don't know > whether it is common for such an idea to be brought up again and again by > newcomers/those who don't strolls the archives enough, but if it keeps being > brought up, and the main argument against is it would take time and effort > to document and implement for little benefit, if Python sticks around for a > long enough time, it will end up taking less time simply implement it! We really need a FAQ for this, if there isn't one already. You quoted Nick referring to a post I'd already made on this, and while I'm afraid I don't have time to find a link for you, I'll try to summarise again here. The bar for syntax changes to Python (and also library changes, although to a slightly lesser extent) is deliberately very high. This is not because people don't like proposals, but rather because the proposers consistently and drastically underestimate the cost of adding new features to the language. Things that need to be factored into the cost of a change: 1. Written materials, such as books, training courses, blog posts will be out of date, at best offering obsolete advice, at worst being outright wrong. Either the authors have extra work to do to change them, or the readers need to learn the "new truth", probably by making mistakes that they could have avoided if the material were correct. 2. Core developers need to understand the implementation of the new feature in order to support it. 3. Unintended consequences and/or problems with the design of the new feature (= bugs) need to be fixed. 4. People writing code that needs to support multiple versions of Python probably won't benefit, as they'll have to avoid the feature. But they will pay costs: a) They need to review that decision over time - is the new feature compelling enough that we should drop support for older versions yet? b) If there's an existing way of writing the construct, why ever change to the new version at all? c) PRs written by people used to the new feature will need reviewing and fixing for compatibility. 5. People not interested in the new feature will encounter it in other people's code and will need to understand it. 6. Someone has to code it. It's often not at all clear whether the proposer is offering to spend the time implementing the feature, and we typically don't bluntly say "well, will you write the code?" both because it's a bit aggressive, and also because an answer of "yes" isn't enough by itself - so we don't want to mislead people that the *only* problem is finding someone to code the change. There's probably others that I have missed. In addition, there's a couple of other points, not directly related to cost but still downsides of new features: 1. People need to be able to search for the new feature and learn it. For syntax in particular, that's hard, as you can't search for something unless you know the right terms. Often a library function is better than syntax for precisely this reason. 2. If a new syntax is defined as "equivalent to XXX which is how you write it now", then (a) it immediately violates the "there should be one obvious way to do it" rule, *unless* the new syntax is so compelling as to immediately be seen as the new "obvious" way to do things. And (b) the people having to support older versions of Python (i.e. a *lot* of people) have no incentive to use the new construct because then they'd have to drop support for existing Python versions. Now, none of the above are insurmountable. We do get new syntax features - and they aren't always massive like async or typing. Unpacking generalisations and f-strings are recent new syntax. But you have to have a *really* good story in terms of benefits to pass the bar. And that means a lot more than just "most people on the list weren't actively against it" in the case of a proposal that's already been made and rejected. Regarding your other points, it *is* hard for newcomers to know what's an old idea. We do what we can (even to the extent of sometimes recommending PEPs just so that ideas can be formally rejected for reference) but not everyone reads up on the history, and we try to be forgiving of that. Your original post showed that you had done some research, so thanks for that. Hopefully, we can work on clarifying for people like you why the "it keeps coming up" argument isn't sufficient by itself. As regards "it will take less time in the end to implement it", if only that were true :-) Sadly, most of the people contributing to discussions here (and even though I'm a core dev, I include myself here) don't actually write code for the Python core that often - whether because time to write emails is easier to find than time to code, or for other reasons, isn't that important. So even we were all to spend a week off from Python-ideas, that wouldn't necessarily translate into new patches for Python. I hope that puts the discussion into context. Paul From wagenaarhenkjaap at gmail.com Thu Feb 23 10:12:14 2017 From: wagenaarhenkjaap at gmail.com (Henk-Jaap Wagenaar) Date: Thu, 23 Feb 2017 15:12:14 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: One does not seem to be able to do this in a generator expression: foo = (x for x in [1, 2] if True else [1, 2, 3]) gives a syntax error, however, adding parenthesis 'solves' this: foo = (x for x in [1, 2] if True else [1, 2, 3]) In the for-loop version, either works. Though I guess this would be even more frowned upon, one can even do: for x in [], print("Here be dragons"): pass Whether it can be implemented in an LL(1) parser, my gut says it could, but this does complicate matters if one were to continue supporting it and create a whirl of confusion. If anything, this shows that there could have been scope for more consistency between the for-statement and generator expression by enforcing parenthesis in the for-loop as well but I think the grammar as it is will be here to stay, unless there is going to be a Python 4 like there is Python 3... I think to be honest this is a pretty big nail in the coffin. H-J On 23 February 2017 at 14:51, Jelle Zijlstra wrote: > 2017-02-23 5:37 GMT-08:00 Henk-Jaap Wagenaar : > > > > Hi all, > > > > Often I have typed something like > > > > for x in range(100) if is_prime(x): > > # do things with x > > > > to find that this does not work, instead resorting to: > > > > for x in range(100): > > if is_prime(x): > > # do things with x > > > > or > > > > for x in range(100): > > if not is_prime(x): > > continue > > # do things with x > > > > Other solutions to another case of this 'problem' are discussed has been > discussed on StackOverflow (http://stackoverflow.com/ > questions/6981717/pythonic-way-to-combine-for-loop-and-if-statement) > where it is suggested one uses a generator expression before the loop. None > of these solutions seem very Pythonic to me. > > > > I appreciate there is a cost associated with changing the language > syntax, and I do not understand all the finer details of the inner workings > involved with the Python language development, however in my limited > understanding in it would be: > > - fully backwards compatible, > > - require one to change "expression_list" to "or_test [comp_iter]" in > the syntax of the for statement (if I got it right). > > - it would mean there is a Pythonic solution to a current 'problem' that > does not have one. > > > > A few problems I foresee: > > - One wants for loops to bleed their target_list (that is the point > normally), so this is different from generators, > > - This allows for nesting of generators, like in a generator expression > which might be hard to implement? > > > I think it might also be difficult to parse. Python currently supports > this syntax: > > In [8]: for x in [1, 2] if True else [1, 2, 3]: > ...: print(x) > ...: > 1 > 2 > > I'm not sure the parser could distinguish this structure from the one > you propose (which would have a very different meaning) without making > it significantly more complicated. Changes that make the parser more > complicated than LL(1) have been consistently rejected in the past; > see https://www.python.org/dev/peps/pep-3099/. > > > > > Note that this has been suggested before at least once ( > https://mail.python.org/pipermail/python-dev/2007-November/075257.html), > and that thread itself suggests it has been suggested before and shutdown > by Guido (though no source is given for this). > > > > All the best, > > > > Henk-Jaap Wagenaar > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wagenaarhenkjaap at gmail.com Thu Feb 23 10:18:54 2017 From: wagenaarhenkjaap at gmail.com (Henk-Jaap Wagenaar) Date: Thu, 23 Feb 2017 15:18:54 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: Hi Paul, Thanks for typing that all out and taking the time to respond to my emails. I think as you say, it might be good to put this somewhere obvious. I did find https://docs.python.org/devguide/langchanges.html and http://www.curiousefficiency.org/posts/2011/02/justifying-python-language-changes.html but the things you wrote would be useful to add the former (or be linked from there). It would also be good if there was a counter-point to that blog post of changes (like this one) that are not sufficiently thought through or significant enough to be accepted: it is always good to have examples on both sides. In terms of your "it keeps coming up", maybe there should be a general PEP (or something else) that simply outlines a bunch of ideas (and will be kept adding to) that keep cropping up but have been rejected/received unsympathetically many times? Including maybe links to these threads? Maybe that is something that could save you (and others subscribed) time and effort? H-J On 23 February 2017 at 15:02, Paul Moore wrote: > On 23 February 2017 at 14:20, Henk-Jaap Wagenaar > wrote: > > > > In a straw poll at the company I work at everyone was in favour, though > they > > obviously are not in charge of implementing or changing documentation so > > that is easy for them to say, they've got no skin in the game. I don't > know > > whether it is common for such an idea to be brought up again and again by > > newcomers/those who don't strolls the archives enough, but if it keeps > being > > brought up, and the main argument against is it would take time and > effort > > to document and implement for little benefit, if Python sticks around > for a > > long enough time, it will end up taking less time simply implement it! > > We really need a FAQ for this, if there isn't one already. You quoted > Nick referring to a post I'd already made on this, and while I'm > afraid I don't have time to find a link for you, I'll try to summarise > again here. > > The bar for syntax changes to Python (and also library changes, > although to a slightly lesser extent) is deliberately very high. This > is not because people don't like proposals, but rather because the > proposers consistently and drastically underestimate the cost of > adding new features to the language. > > Things that need to be factored into the cost of a change: > > 1. Written materials, such as books, training courses, blog posts will > be out of date, at best offering obsolete advice, at worst being > outright wrong. Either the authors have extra work to do to change > them, or the readers need to learn the "new truth", probably by making > mistakes that they could have avoided if the material were correct. > 2. Core developers need to understand the implementation of the new > feature in order to support it. > 3. Unintended consequences and/or problems with the design of the new > feature (= bugs) need to be fixed. > 4. People writing code that needs to support multiple versions of > Python probably won't benefit, as they'll have to avoid the feature. > But they will pay costs: > a) They need to review that decision over time - is the new feature > compelling enough that we should drop support for older versions yet? > b) If there's an existing way of writing the construct, why ever > change to the new version at all? > c) PRs written by people used to the new feature will need reviewing > and fixing for compatibility. > 5. People not interested in the new feature will encounter it in other > people's code and will need to understand it. > 6. Someone has to code it. It's often not at all clear whether the > proposer is offering to spend the time implementing the feature, and > we typically don't bluntly say "well, will you write the code?" both > because it's a bit aggressive, and also because an answer of "yes" > isn't enough by itself - so we don't want to mislead people that the > *only* problem is finding someone to code the change. > > There's probably others that I have missed. > > In addition, there's a couple of other points, not directly related to > cost but still downsides of new features: > > 1. People need to be able to search for the new feature and learn it. > For syntax in particular, that's hard, as you can't search for > something unless you know the right terms. Often a library function is > better than syntax for precisely this reason. > 2. If a new syntax is defined as "equivalent to XXX which is how you > write it now", then (a) it immediately violates the "there should be > one obvious way to do it" rule, *unless* the new syntax is so > compelling as to immediately be seen as the new "obvious" way to do > things. And (b) the people having to support older versions of Python > (i.e. a *lot* of people) have no incentive to use the new construct > because then they'd have to drop support for existing Python versions. > > Now, none of the above are insurmountable. We do get new syntax > features - and they aren't always massive like async or typing. > Unpacking generalisations and f-strings are recent new syntax. But you > have to have a *really* good story in terms of benefits to pass the > bar. And that means a lot more than just "most people on the list > weren't actively against it" in the case of a proposal that's already > been made and rejected. > > Regarding your other points, it *is* hard for newcomers to know what's > an old idea. We do what we can (even to the extent of sometimes > recommending PEPs just so that ideas can be formally rejected for > reference) but not everyone reads up on the history, and we try to be > forgiving of that. Your original post showed that you had done some > research, so thanks for that. Hopefully, we can work on clarifying for > people like you why the "it keeps coming up" argument isn't sufficient > by itself. > > As regards "it will take less time in the end to implement it", if > only that were true :-) Sadly, most of the people contributing to > discussions here (and even though I'm a core dev, I include myself > here) don't actually write code for the Python core that often - > whether because time to write emails is easier to find than time to > code, or for other reasons, isn't that important. So even we were all > to spend a week off from Python-ideas, that wouldn't necessarily > translate into new patches for Python. > > I hope that puts the discussion into context. > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 23 10:33:32 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Feb 2017 15:33:32 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: On 23 February 2017 at 15:18, Henk-Jaap Wagenaar wrote: > Thanks for typing that all out and taking the time to respond to my emails. > I think as you say, it might be good to put this somewhere obvious. > > I did find https://docs.python.org/devguide/langchanges.html and > http://www.curiousefficiency.org/posts/2011/02/justifying-python-language-changes.html > but the things you wrote would be useful to add the former (or be linked > from there). It would also be good if there was a counter-point to that blog > post of changes (like this one) that are not sufficiently thought through or > significant enough to be accepted: it is always good to have examples on > both sides. > > In terms of your "it keeps coming up", maybe there should be a general PEP > (or something else) that simply outlines a bunch of ideas (and will be kept > adding to) that keep cropping up but have been rejected/received > unsympathetically many times? Including maybe links to these threads? > > Maybe that is something that could save you (and others subscribed) time and > effort? Ha. I never even *thought* of putting something like this in the devguide. Thanks for the pointer. I'll try to make some time to polish up my comments and put a PR together for that. I've thought we should have some sort of "list info" page describing things like this and giving some context for posters before now. Possibly linked from the automatic footer the list adds. I don't know who would be able to do something like that, though. Thanks for your suggestions. Paul From srkunze at mail.de Thu Feb 23 11:00:59 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 23 Feb 2017 17:00:59 +0100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: Hi Henk-Jaap, thanks for your "if in for" proposal. Paul's comments are all "motherhood statements" against a generic proposal. It's nothing that would prevent your specific proposal from being accepted or not. And there's no rejected PEP for this feature as far as I can see. Skimming through the other thread about this topic, I remember me not being very offensive against it. I would use it often. IIRC, one result of former discussions were that it would be technically possible. But the other proposer did not convince most thread participants back then. I for one, though, didn't buy the counter-arguments and still don't. They still sound to me like abstract fears. On 23.02.2017 16:18, Henk-Jaap Wagenaar wrote: > I did find https://docs.python.org/devguide/langchanges.html and > http://www.curiousefficiency.org/posts/2011/02/justifying-python-language-changes.html > but the things you wrote would be useful to add the former (or be > linked from there). It would also be good if there was a counter-point > to that blog post of changes (like this one) that are not sufficiently > thought through or significant enough to be accepted: it is always > good to have examples on both sides. > > In terms of your "it keeps coming up", maybe there should be a general > PEP (or something else) that simply outlines a bunch of ideas (and > will be kept adding to) that keep cropping up but have been > rejected/received unsympathetically many times? Including maybe links > to these threads? Recently, we had a similar discussion about writing an "official" document which might prevent proposal such as yours. I wasn't the only one being offended by this kind of thinking. There are, as you noted, two sides of the table and just because "it keeps coming up" and "was received unsympathetically many times" doesn't mean we shouldn't consider it. Point is, it actually seems sympathetic to many people and this is why it keeps coming up. Nevertheless, a proposal needs consensus. Regards, Sven From steve at pearwood.info Thu Feb 23 13:01:15 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Feb 2017 05:01:15 +1100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: <20170223180115.GA5689@ando.pearwood.info> On Thu, Feb 23, 2017 at 03:02:18PM +0000, Paul Moore wrote: > We really need a FAQ for this, if there isn't one already. Indeed. Earlier this year, I started a thread on Things That Won't Change In Python. If I recall correctly, Nick disagreed that it should be a PEP, I started to write up a response to everyone's feedback, and then my PC crashed and I lost the response. I never came back to it. I think its time to do so. I'm fine with the idea of it being a FAQ. https://mail.python.org/pipermail/python-ideas/2017-January/044201.html -- Steve From steve at pearwood.info Thu Feb 23 13:07:20 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Feb 2017 05:07:20 +1100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: <20170223180720.GB5689@ando.pearwood.info> On Thu, Feb 23, 2017 at 03:33:32PM +0000, Paul Moore wrote: > Ha. I never even *thought* of putting something like this in the > devguide. Thanks for the pointer. I'll try to make some time to polish > up my comments and put a PR together for that. Paul, at the moment I have neither the time nor the mental energy to deal with learning the brave New World of Github's way of doing things, but if you want to adapt my "Things That Won't Change" proto-PEP from last month, please feel free to do so! I have an update to the original version, over the next couple of days, time permitting, I'll clean it up and send it to the list. > I've thought we should have some sort of "list info" page describing > things like this and giving some context for posters before now. > Possibly linked from the automatic footer the list adds. I don't know > who would be able to do something like that, though. The obvious places to link to this page are: - the Python-Ideas signup page; - the devguide; - the FAQs. -- Steve From steve at pearwood.info Thu Feb 23 13:25:46 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Feb 2017 05:25:46 +1100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: Message-ID: <20170223182546.GC5689@ando.pearwood.info> On Thu, Feb 23, 2017 at 01:37:15PM +0000, Henk-Jaap Wagenaar wrote: [...] > Other solutions to another case of this 'problem' are discussed has been > discussed on StackOverflow ( > http://stackoverflow.com/questions/6981717/pythonic-way-to-combine-for-loop-and-if-statement) > where it is suggested one uses a generator expression before the loop. None > of these solutions seem very Pythonic to me. Indeed not. The Pythonic solution is exactly the one you used: DON'T combine the for-loop and if-statement. for x in range(100): if is_prime(x): ... If the use of two lines and two indents is truly a problem, to the point that you really need to refactor to a single line, that's a code smell: your function is probably too big, too complex and does too much, and should be refactored. > - it would mean there is a Pythonic solution to a current 'problem' that > does not have one. It does have a solution: "Don't do it". One common argument is that people can write a for...if in a single expression as part of comprehensions: [... for x in range(100) if is_prime(x)] so they should be able to write the same outside. We can write: [... for x in seqA for y in seqB if y for z in seqC] Does that mean we should be able to write this as a combined statement? for x in seqA for y in seqB if y for z in seqC: ... Certainly not! The fact that comprehensions allow such ugly code is not a feature to be emulated, but something to be discouraged. Comprehensions are limited to being a single expression, and so we have to compromise on good design in order to get the full functionality required. No such compromise is needed with the statement forms of for/if. We can draw the line right at the start, and insist that each statement falls on a line on its own. Deeply nested for...for...for...if...if...if...for... blocks look ugly because they are ugly. Allowing them all on one line makes the Python language worse, not better. It is unfortunate that comprehensions, by their nature, must allow that sort of thing, but that doesn't mean we have to allow it elsewhere. -- Steve From rmartin at astek.fr Thu Feb 23 13:11:36 2017 From: rmartin at astek.fr (=?UTF-8?Q?R=c3=a9gis_Martin?=) Date: Thu, 23 Feb 2017 19:11:36 +0100 Subject: [Python-ideas] if in for-loop statement Message-ID: <3bb55e86-611c-cbb1-81c3-f9e8de56ad2e@astek.fr> Hello, why not using the filter function? for x in filter(is_prime, range(100)): # do something with x This is equivalent as was mentionned at first: for x in range(100) if is_prime(x): # do things with x R?gis -- R?gis Martin Directeur de Projets Astek Industrie Tel: 06-83-53-15-05 http://www.groupeastek.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: rmartin.vcf Type: text/x-vcard Size: 182 bytes Desc: not available URL: From wagenaarhenkjaap at gmail.com Thu Feb 23 16:11:19 2017 From: wagenaarhenkjaap at gmail.com (Henk-Jaap Wagenaar) Date: Thu, 23 Feb 2017 21:11:19 +0000 Subject: [Python-ideas] if in for-loop statement In-Reply-To: <3bb55e86-611c-cbb1-81c3-f9e8de56ad2e@astek.fr> References: <3bb55e86-611c-cbb1-81c3-f9e8de56ad2e@astek.fr> Message-ID: I think, in that particular example, that is an ok solution, however if you are combining multiple booleans to create a lambda or separate function that is quite a bit of syntactical sugar to add (unless it is re-used). H-J On 23 February 2017 at 18:11, R?gis Martin wrote: > Hello, > why not using the filter function? > > for x in filter(is_prime, range(100)): > # do something with x > > This is equivalent as was mentionned at first: > > for x in range(100) if is_prime(x): > # do things with x > > R?gis > > -- > R?gis Martin > Directeur de Projets Astek Industrie > Tel: 06-83-53-15-05 > http://www.groupeastek.com/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu Feb 23 17:07:49 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 23 Feb 2017 23:07:49 +0100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: <20170223182546.GC5689@ando.pearwood.info> References: <20170223182546.GC5689@ando.pearwood.info> Message-ID: Hey Steven, On 23.02.2017 19:25, Steven D'Aprano wrote: > Indeed not. The Pythonic solution is exactly the one you used: DON'T > combine the for-loop and if-statement. > > for x in range(100): > if is_prime(x): > ... > > If the use of two lines and two indents is truly a problem, to the point > that you really need to refactor to a single line, that's a code smell: > your function is probably too big, too complex and does too much, and > should be refactored. May I disagree with you here? I write many functions with a single for loop + a single if+continue on a daily basis (usually generators). They don't seem to look like code-smell to me. Not saying they justify this feature but it's easier to read a positive (including) if than a negative one (skipping). > Deeply nested for...for...for...if...if...if...for... blocks look ugly > because they are ugly. Allowing them all on one line makes the Python > language worse, not better. It is unfortunate that comprehensions, by > their nature, must allow that sort of thing, but that doesn't mean we > have to allow it elsewhere. Another disagreement here. Just because it's possible to do ugly stuff with a hammer shouldn't mean we cannot allow hammers. It's an argument neither in favor of nor against the proposal. Sven From chris.barker at noaa.gov Thu Feb 23 23:01:50 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 23 Feb 2017 20:01:50 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: On Mon, Feb 6, 2017 at 5:42 AM, Juraj Sukop wrote: > Do you mean something like: > > isclose(f(x), 0.0, rel_tol, abs_tol) > > If so, what should `rel_tol` and `abs_tol` be? > isclose is mostly about "relative" closeness, so rel_tol is more-or-less the number of decimal digits you want the same: In [13]: isclose(1.11111111119, 1.11111111118, rel_tol=1e-11) Out[13]: True 11 digits are the same In [14]: isclose(1.11111111119, 1.11111111118, rel_tol=1e-12) Out[14]: False 12 aren't ( I say more or less, because FP values are in binary and we tend to think in decimal) You might think you'd want to know if the two values are as close as possible given the FP representation -- which is what nextafter() would give you -- but it is very, very, rare that a computation accurately preserves ALL the digits -- and if it did, you could use the regular old ==, <, > checks. But NOTHING is relatively close to zero, so if you want to compare to zero you need an absolute tolerance -- what is close to zero in your application: In [19]: isclose(1e-100, 0.0, abs_tol = 1e-101) Out[19]: False In [20]: isclose(1e-100, 0.0, abs_tol = 1e-100) Out[20]: True Again, you probably don't want the smallest possible representable FP value -- cause if your computation was that perfect, why not just check for zero? Figuring out what value makes sense in your use case requires either careful numerical analysis (that few of us are capable of!), or experimentation -- how small a value can you use and get the algorithm to converge? -Chris See PEP 485 for more detail: https://www.python.org/dev/peps/pep-0485/ > On Mon, Feb 6, 2017 at 2:16 PM, M.-A. Lemburg wrote: > >> On 06.02.2017 13:22, Juraj Sukop wrote: >> > On Mon, Feb 6, 2017 at 11:29 AM, M.-A. Lemburg wrote: >> > >> >> >> >> Juraj: Could you provide some use cases, where such a function >> >> would help in Python applications ? (I can see use cases >> >> written in C, but due to the low level, find it hard to >> >> believe that people would use this at the Python level) >> >> >> > >> > In my case, `nextafter` would be used to check if a number is close to >> > polynomial zero, e.g.: >> > >> > def f(x): >> > return 2.0*x**3 - 3.0*x**2 + 5.0*x - 7.0 >> > >> > # x = 1.4455284586795218 >> > x = 1.445528458679522 >> > # x = 1.4455284586795223 >> > # x = 1.4455284586795225 >> > >> > left = nextafter(x, -float('inf')) >> > right = nextafter(x, float('inf')) >> > >> > print((f(left) < 0.0) != (f(x) < 0.0) or (f(x) < 0.0) != (f(right) < >> > 0.0)) >> >> Isn't this something you can do with math.isclose() ? >> >> This would even give you a predefined error range, >> not a dynamic one. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts (#1, Feb 06 2017) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> Python Database Interfaces ... http://products.egenix.com/ >> >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> http://www.malemburg.com/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Feb 23 23:23:40 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 23 Feb 2017 20:23:40 -0800 Subject: [Python-ideas] Fwd: Define a method or function attribute outside of a class with the dot operator In-Reply-To: <589E0E45.4070504@stoneleaf.us> References: <20170210101327.GA4796@ando.pearwood.info> <20170210152506.GE4796@ando.pearwood.info> <589E0E45.4070504@stoneleaf.us> Message-ID: Has this REALLY not been discussed and rejected long ago????? > But I'm +1 on writing a PEP -- collect all these pros and cons in one > place to save on future discussion. (And (good) PEP writing is a way to > earn valuable Python Points!) > Exactly -- this is obvious enough that it WILL come up again, and I'm sure it has (but my memory gets fuzzy more than a few months back....) It would be great to document it even if it is headed for rejection. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Feb 23 23:28:10 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Feb 2017 15:28:10 +1100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: <20170223182546.GC5689@ando.pearwood.info> Message-ID: <20170224042810.GD5689@ando.pearwood.info> On Thu, Feb 23, 2017 at 11:07:49PM +0100, Sven R. Kunze wrote: > >If the use of two lines and two indents is truly a problem, to the point > >that you really need to refactor to a single line, that's a code smell: > >your function is probably too big, too complex and does too much, and > >should be refactored. > > May I disagree with you here? I write many functions with a single for > loop + a single if+continue on a daily basis (usually generators). > They don't seem to look like code-smell to me. I'm sorry, I didn't explain myself well enough. There is nothing wrong with a nested for...if pair of statements. But that does take two lines, and two indents, rather than one: block for ... if ... block versus hypothetical: block for ... if ... block The proposed syntax saves one line, and one indent. That is a saving, but it is not a big saving. If one line and one indent *really* makes a difference to a piece of code, it is probably because you are already deeply nested: class ... def ... def ... try ... try ... while ... try ... if ... with ... if ... while ... for ... if ... # not enough # room here That is already so deeply nested that saving one indent might be important. But the problem isn't the for...if section, it is the ten or a dozen *previous* statements. "I have run out of horizontal space" is the code-smell, not "I have a for loop and an if branch". Most of the time, "save one line of code" or "save one indent" is not a big win. It it insignificant: most of the time, extra lines are cheap, and there is plenty of room for all the indent levels needed. There is one obvious exception to the rule: if ... else: if ... else: if ... else: if ... That can get pretty costly in terms of indent levels very quickly. And that is why Python has elif: if ... elif ... elif ... elif ... So don't misunderstand me. I do appreciate that sometimes saving indents and lines is important. But what I am saying is that in *this* case, it is rarely important enough to matter, and when it does matter, your code probably has bigger problems than just the for...if blocks. > Not saying they justify this feature but it's easier to read a positive > (including) if than a negative one (skipping). > > >Deeply nested for...for...for...if...if...if...for... blocks look ugly > >because they are ugly. Allowing them all on one line makes the Python > >language worse, not better. It is unfortunate that comprehensions, by > >their nature, must allow that sort of thing, but that doesn't mean we > >have to allow it elsewhere. > > Another disagreement here. Just because it's possible to do ugly stuff > with a hammer shouldn't mean we cannot allow hammers. It's an argument > neither in favor of nor against the proposal. Since code is read more often than it is written, good programming languages are pretty programming languages. That's why most of us choose Python: its not the most efficient language, or fastest running language, or most powerful language, but it is fast enough, efficient enough, powerful enough, and most importantly, it is pretty. By which I mean it is relatively easy to read (at least for English-readers), which makes it easy to write, debug and maintain. If we want to keep Python pretty, we should discourage things which are ugly. Discourage, or even outright ban them. Some features are on the borderline. There is a grey area where it comes to the subjective aesthetic judgement of the language Dictator. I think that this is one of those areas, and I hope that we are correctly channelling the BDFL. Because for...if...for...if... is on the boundary, we can allow it in one case (comprehensions) and forbid it in another (statements): - in a series of nested statements, the benefit of allowing the for...if all on one line is not enough to make up for the risk of people misusing it; - in a comprehension, the benefit is greater, which is enough to make up for the risk of people misusing it. Because a comprehension is an expression, adding an extra line and an extra indent is not usually helpful. It may mess up the expression it is embedded in and lead to messy code: result = some_function(arg, [comprehension that goes over multiple lines and indents], spam, eggs, cheese) + 1 print(result) That's probably not helpful, and possibly even confusing. So for comprehensions, we can relax the rule about one indent level per for or if statement. The reason we have different syntax for comprehensions and statements is that they are different kinds of code, used in different ways. -- Steve From rosuav at gmail.com Thu Feb 23 23:45:17 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 24 Feb 2017 15:45:17 +1100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: <20170224042810.GD5689@ando.pearwood.info> References: <20170223182546.GC5689@ando.pearwood.info> <20170224042810.GD5689@ando.pearwood.info> Message-ID: On Fri, Feb 24, 2017 at 3:28 PM, Steven D'Aprano wrote: > There is nothing wrong with a nested for...if pair of statements. But > that does take two lines, and two indents, rather than one: > > block > for ... > if ... > block > > versus hypothetical: > > block > for ... if ... > block > > > The proposed syntax saves one line, and one indent. That is a saving, > but it is not a big saving. If one line and one indent *really* makes a > difference to a piece of code, it is probably because you are already > deeply nested: > > class ... > def ... > def ... > try ... > try ... > while ... > try ... > if ... > with ... > if ... > while ... > for ... > if ... > # not enough > # room here Don't forget that you can rewrite a "for-if" using two additional lines and no indents, rather than one line and one indent: for ...: if not (...): continue ... ... So you can take your pick which version you want. Granted, I can still see value in the for-if statement, but not enough to justify new syntax. ChrisA From ncoghlan at gmail.com Fri Feb 24 04:13:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Feb 2017 19:13:56 +1000 Subject: [Python-ideas] math.nextafter In-Reply-To: <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> Message-ID: On 6 February 2017 at 20:29, M.-A. Lemburg wrote: > On 04.02.2017 12:59, Stephan Houben wrote: > > Hi all, > > > > Visual C++ 2015 supports this one: > > > > https://msdn.microsoft.com/en-us/library/h0dff77w.aspx > > > > In any case, this is easy to implement an efficient fallback in C, unlike > > the fma() function we discussed some time ago. > > > > To put this in a bit wider perspective: would it be useful to investigate > > how much of the C99 math library could > > be supported in Python in general? > > +1 from me for those features which can be emulated for > platforms which don't have the math lib function > available and are not too esoteric (though many of those > have already been added), e.g. cbt() may be useful. > > Now, with respect to the one mentioned in the subject, > I'm not sure how useful this would be in the stdlib, > since it's very much tied to whatever float type Python > happens to use on a platform. > Just from an API point of view, a more idiomatically-Pythonic concept seems to me to be a "floatrange()" construct, where the default step was defined by the internal representation (i.e. it would increment by the smallest possible value). I'm not sure of any use cases outside exploring the behaviour of numerical algorithm implementations in the presence of mathematical discontinuities, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From juraj.sukop at gmail.com Fri Feb 24 04:40:13 2017 From: juraj.sukop at gmail.com (Juraj Sukop) Date: Fri, 24 Feb 2017 10:40:13 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: On Fri, Feb 24, 2017 at 5:01 AM, Chris Barker wrote: > cause if your computation was that perfect, why not just check for zero? > > A polynomial root may simply be not representable in double precision floating-point format. Per the example I posted above, the best one can hope for in such situation is to find (x, nextafter(x, float('inf'))) interval which has opposite function value signs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Fri Feb 24 04:58:15 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 24 Feb 2017 10:58:15 +0100 Subject: [Python-ideas] if in for-loop statement In-Reply-To: References: <20170223182546.GC5689@ando.pearwood.info> <20170224042810.GD5689@ando.pearwood.info> Message-ID: <226a1eab-5368-d351-2bc2-9b59e8eb393c@mail.de> On 24.02.2017 05:45, Chris Angelico wrote: > Don't forget that you can rewrite a "for-if" using two additional > lines and no indents, rather than one line and one indent: > > for ...: > if not (...): > continue > ... > ... That's exactly what I meant by "for+if+continue". At work we even set a guideline for this to prevent excessive indentation for longer loop bodies. > So you can take your pick which version you want. Granted, I can still > see value in the for-if statement, but not enough to justify new > syntax. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Feb 24 05:29:02 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 24 Feb 2017 11:29:02 +0100 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> Message-ID: <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> On 24.02.2017 10:13, Nick Coghlan wrote: > On 6 February 2017 at 20:29, M.-A. Lemburg wrote: > >> On 04.02.2017 12:59, Stephan Houben wrote: >>> Hi all, >>> >>> Visual C++ 2015 supports this one: >>> >>> https://msdn.microsoft.com/en-us/library/h0dff77w.aspx >>> >>> In any case, this is easy to implement an efficient fallback in C, unlike >>> the fma() function we discussed some time ago. >>> >>> To put this in a bit wider perspective: would it be useful to investigate >>> how much of the C99 math library could >>> be supported in Python in general? >> >> +1 from me for those features which can be emulated for >> platforms which don't have the math lib function >> available and are not too esoteric (though many of those >> have already been added), e.g. cbt() may be useful. >> >> Now, with respect to the one mentioned in the subject, >> I'm not sure how useful this would be in the stdlib, >> since it's very much tied to whatever float type Python >> happens to use on a platform. >> > > Just from an API point of view, a more idiomatically-Pythonic concept seems > to me to be a "floatrange()" construct, where the default step was defined > by the internal representation (i.e. it would increment by the smallest > possible value). > > I'm not sure of any use cases outside exploring the behaviour of numerical > algorithm implementations in the presence of mathematical discontinuities, > though. Perhaps closeto() could be extended to address the use case: "Match anything within N number of smallest float representable intervals around float value x" https://www.python.org/dev/peps/pep-0485/ This could then be used to detect cases where it doesn't make sense to run additional rounds of refinement to find roots or local minima, since IEEE floats simply don't provide enough accuracy to dig deeper. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 24 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From chris.barker at noaa.gov Fri Feb 24 13:12:53 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 24 Feb 2017 10:12:53 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: On Fri, Feb 24, 2017 at 1:40 AM, Juraj Sukop wrote: > > > On Fri, Feb 24, 2017 at 5:01 AM, Chris Barker > wrote: > >> cause if your computation was that perfect, why not just check for zero? >> >> > A polynomial root may simply be not representable in double precision > floating-point format. > Indeed. > Per the example I posted above, the best one can hope for in such > situation is to find (x, nextafter(x, float('inf'))) interval which has > opposite function value signs. > I'm not much of a numerical analyst, but I think the number of applications in which the computation is accurate to the limit of float precision is vanishingly small -- so you need something larger than the smallest representable non-zero value anyway. Also many time when one is looking for a "zero", you are really looking for the difference between two values to be zero -- in which case isclose() is the right tool for the job. Thus this kind of thing is useful primarily for "exploring the behaviour of numerical algorithm implementations" as Nick said. Which doesn't mean it wouldn't' be useful to have in Python. Someone said: " it's very much tied to whatever float type Python happens to use on a platform." Which is the whole point -- if one could assume that Python is using IEEE 754 64 bit floats, then you could write these functions yourself but it's built-in then the implementation can make sure it's correct for the platform/etc that you are currently running on. By the way, it looks like math doesn't have machine epsilon either: https://en.wikipedia.org/wiki/Machine_epsilon which would be handy as well. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mahmoud at hatnote.com Fri Feb 24 13:27:44 2017 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Fri, 24 Feb 2017 10:27:44 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: By the way, it looks like math doesn't have machine epsilon either: > > > https://en.wikipedia.org/wiki/Machine_epsilon > > which would be handy as well. > > -CHB > > Pretty sure machine epsilon is in the sys module's float_info object. Or are you saying it would be handy to alias sys.float_info.epsilon over to the math module too? Mahmoud -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Feb 24 20:11:41 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 24 Feb 2017 17:11:41 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <01a2a360-c81d-6cbe-be86-53a370f2ed21@egenix.com> Message-ID: <7132427928815104353@unknownmsgid> On Feb 24, 2017, at 10:28 AM, Mahmoud Hashemi wrote: By the way, it looks like math doesn't have machine epsilon either: > > > https://en.wikipedia.org/wiki/Machine_epsilon > > Pretty sure machine epsilon is in the sys module's float_info object. Ahh, thanks! I though I remembered it was somewhere. Or are you saying it would be handy to alias sys.float_info.epsilon over to the math module too? That might be a good idea, yes. Particularly if other related functions are added as being discussed. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Fri Feb 24 20:29:15 2017 From: mertz at gnosis.cx (David Mertz) Date: Fri, 24 Feb 2017 20:29:15 -0500 Subject: [Python-ideas] math.nextafter In-Reply-To: <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> Message-ID: Marc-Andr? slightly misspelled the recent-ish addition of math.isclose(), but I agree that it is absolutely where a "nextafter" belongs. The function signature is already relatively complex to cover several different but related use cases. I.e.: is_close(a, b, *, rel_tol=1e-09, abs_tol=0.0) -> bool I think an additional keyword-only argument for `nextafter_tol` (maybe a different spelling) would fit very naturally there. This would allow someone to specify 1 for that degree of closeness, but it would also allow them to specify some integer larger than 1 without needed to write a loop calling `nextafter()` repeatedly. Yours, David... On Fri, Feb 24, 2017 at 5:29 AM, M.-A. Lemburg wrote: > Perhaps closeto() could be extended to address the use case: > > "Match anything within N number of smallest float representable > intervals around float value x" > > https://www.python.org/dev/peps/pep-0485/ > > This could then be used to detect cases where it doesn't > make sense to run additional rounds of refinement to > find roots or local minima, since IEEE floats simply don't > provide enough accuracy to dig deeper. > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Feb 24 21:43:01 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 Feb 2017 18:43:01 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> Message-ID: On Feb 24, 2017 5:29 PM, "David Mertz" wrote: Marc-Andr? slightly misspelled the recent-ish addition of math.isclose(), but I agree that it is absolutely where a "nextafter" belongs. The function signature is already relatively complex to cover several different but related use cases. I.e.: is_close(a, b, *, rel_tol=1e-09, abs_tol=0.0) -> bool I think an additional keyword-only argument for `nextafter_tol` (maybe a different spelling) would fit very naturally there. This would allow someone to specify 1 for that degree of closeness, but it would also allow them to specify some integer larger than 1 without needed to write a loop calling `nextafter()` repeatedly. My 2c: I disagree -- numerical tolerance based closeness is fundamentally different than floating point representation based closeness (often discussed with the term "ulp" = "unit in the last place". For example, the number that's closest to 1.0 from above is twice as far away in numerical terms as the number that's closest to it from below. Mixing these two concepts in one function is just confusing, and we're not going to run out of names. It's also a little weird to jump from nextafter to isclose, since AFAIK they have pretty much non-overlapping use cases... FWIW, numpy provides all of the following as separate functions: * an isclose equivalent * nextafter * a function for counting the number of ulps between two floats * a function for checking that two floats differ by at most N ulps I'm not enough of an expert on numerical analysis to have an opinion on how useful these would be for Python itself. They certainly are part of a complete IEEE754 implementation, and useful for exploring the details of how floats work, if nothing else. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 25 01:53:58 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Feb 2017 16:53:58 +1000 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> Message-ID: On 25 February 2017 at 12:43, Nathaniel Smith wrote: > FWIW, numpy provides all of the following as separate functions: > > * an isclose equivalent > * nextafter > * a function for counting the number of ulps between two floats > * a function for checking that two floats differ by at most N ulps > "Does NumPy offer it?" is a useful data point when asking "Is this feature useful for numerical analysis?", so it'd helpful to know that it does have them. > I'm not enough of an expert on numerical analysis to have an opinion on > how useful these would be for Python itself. They certainly are part of a > complete IEEE754 implementation, and useful for exploring the details of > how floats work, if nothing else. > It seems like a reasonable thing to expose to me, rather than asking folks to derive their own equivalents based on the information in sys.float_info. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sat Feb 25 12:27:58 2017 From: mertz at gnosis.cx (David Mertz) Date: Sat, 25 Feb 2017 09:27:58 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> Message-ID: I do agree that a math.count_ulps() would be very nice to have. Or whatever spelling NumPy users for that concept. I see nextafter as more similar. Yes, it's not a uniform increment, which is the whole point. If you want a convergence that is as good as it can be in float, that's the relevant measure. Likewise for "almost as good" being N ulps. On Feb 24, 2017 9:43 PM, "Nathaniel Smith" wrote: On Feb 24, 2017 5:29 PM, "David Mertz" wrote: Marc-Andr? slightly misspelled the recent-ish addition of math.isclose(), but I agree that it is absolutely where a "nextafter" belongs. The function signature is already relatively complex to cover several different but related use cases. I.e.: is_close(a, b, *, rel_tol=1e-09, abs_tol=0.0) -> bool I think an additional keyword-only argument for `nextafter_tol` (maybe a different spelling) would fit very naturally there. This would allow someone to specify 1 for that degree of closeness, but it would also allow them to specify some integer larger than 1 without needed to write a loop calling `nextafter()` repeatedly. My 2c: I disagree -- numerical tolerance based closeness is fundamentally different than floating point representation based closeness (often discussed with the term "ulp" = "unit in the last place". For example, the number that's closest to 1.0 from above is twice as far away in numerical terms as the number that's closest to it from below. Mixing these two concepts in one function is just confusing, and we're not going to run out of names. It's also a little weird to jump from nextafter to isclose, since AFAIK they have pretty much non-overlapping use cases... FWIW, numpy provides all of the following as separate functions: * an isclose equivalent * nextafter * a function for counting the number of ulps between two floats * a function for checking that two floats differ by at most N ulps I'm not enough of an expert on numerical analysis to have an opinion on how useful these would be for Python itself. They certainly are part of a complete IEEE754 implementation, and useful for exploring the details of how floats work, if nothing else. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From vamsi_ism at outlook.com Sun Feb 26 08:39:22 2017 From: vamsi_ism at outlook.com (Vamsi Krishna Avula) Date: Sun, 26 Feb 2017 13:39:22 +0000 Subject: [Python-ideas] Make Path objects iterable Message-ID: This is somewhere between a question and a proposal. I'm just trying to understand why Path objects use an explicit iterdir method. Why not make them iterable? -- Vamsi From storchaka at gmail.com Sun Feb 26 10:13:00 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 26 Feb 2017 17:13:00 +0200 Subject: [Python-ideas] Make Path objects iterable In-Reply-To: References: Message-ID: On 26.02.17 15:39, Vamsi Krishna Avula wrote: > This is somewhere between a question and a proposal. I'm just trying to understand why Path objects use an explicit iterdir method. > Why not make them iterable? Because this is ambiguous. Iterating can mean many different things: 1. Iterate characters of string representation of the path (str(path)). 2. Iterate path components (path.parts). 3. Open a file and iterate its lines (path.open()). 4. Iterate files in the directory (path.iterdir()). From qhlonline at 163.com Sun Feb 26 22:07:33 2017 From: qhlonline at 163.com (qhlonline) Date: Mon, 27 Feb 2017 11:07:33 +0800 (CST) Subject: [Python-ideas] suggestion about the sort() function of the list instance Message-ID: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> Hi, all I have a suggestion that, the sort() member method of the list instance, should return the 'self' as the result of list.sort() call. Now list.sort() returns nothing, so that I can NOT write code like this: res = {item: func(item) for item in item_list.sort()} It feels bad. Regards! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rainventions at gmail.com Sun Feb 26 22:26:47 2017 From: rainventions at gmail.com (Ryan Birmingham) Date: Sun, 26 Feb 2017 22:26:47 -0500 Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> Message-ID: It sorts the list in place. Can you use sorted(item_list)? -Ryan Birmingham On 26 February 2017 at 22:07, qhlonline wrote: > Hi, all > I have a suggestion that, the sort() member method of the list > instance, should return the 'self' as the result of list.sort() call. > Now list.sort() returns nothing, so that I can NOT write code like > this: > > res = {item: func(item) for item in item_list.sort()} > > It feels bad. > > Regards! > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcgoble3 at gmail.com Sun Feb 26 22:37:24 2017 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Mon, 27 Feb 2017 03:37:24 +0000 Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> Message-ID: On Sun, Feb 26, 2017 at 10:23 PM qhlonline wrot > Hi, all > I have a suggestion that, the sort() member method of the list > instance, should return the 'self' as the result of list.sort() call. > Now list.sort() returns nothing, so that I can NOT write code like > this: > > res = {item: func(item) for item in item_list.sort()} > > It feels bad. > > Regards! > list.sort() does not return anything on purpose, to remind you that it operates by side effect (i.e. that calling list.sort() mutates the list in-place). If you want it to return the list, use sorted() to get a copy of the list. Also, for your specific example, I wouldn't use a comprehension in the first place. I'd probably do something like this: item_list.sort() res = dict(zip(item_list, map(func, item_list))) But sorting the item_list is pointless in this case, since dictionaries are unordered [1], so the sorting has no real effect in this case, other than the fact that the list is now sorted for the benefit of subsequent code, if that should be relevant. [1] Dictionaries are ordered in CPython 3.6, but this behavior cannot be relied upon, as doing so means your code won't work on older versions of CPython, nor is it guaranteed to work in other implementations of Python 3.6. -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Mon Feb 27 03:32:02 2017 From: flying-sheep at web.de (Philipp A.) Date: Mon, 27 Feb 2017 08:32:02 +0000 Subject: [Python-ideas] Make Path objects iterable In-Reply-To: References: Message-ID: the same is true for files: Deciding to iterate them line-wise is relatively arbitrary, byte/char-wise would be equally intitive. So something could be chosen. But I think being explicit in the case of paths is really not that inconvenient. Serhiy Storchaka schrieb am So., 26. Feb. 2017 um 16:14 Uhr: On 26.02.17 15:39, Vamsi Krishna Avula wrote: > This is somewhere between a question and a proposal. I'm just trying to understand why Path objects use an explicit iterdir method. > Why not make them iterable? Because this is ambiguous. Iterating can mean many different things: 1. Iterate characters of string representation of the path (str(path)). 2. Iterate path components (path.parts). 3. Open a file and iterate its lines (path.open()). 4. Iterate files in the directory (path.iterdir()). -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Feb 28 06:54:26 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 12:54:26 +0100 Subject: [Python-ideas] get() method for list and tuples Message-ID: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> dict.get() is a very useful method, but for lists and tuples, we need to rely on try/except instead. Can we get list.get and tuple.get as well? Also, for list, a list.setdefault like the dict.setdefault would be logical. From desmoulinmichel at gmail.com Tue Feb 28 07:04:51 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 13:04:51 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword Message-ID: The debate on the 'lazy' keyword seems to have settled, but I don't know if somebody is trying to write a PEP about it. Anyway, I was doing something like this the other day: conf.get('setting_name', load_from_db('setting_name')) And then realized I could save a query not doing the load_from_db() call most of the time. But dict.get didn't accept callable so I couldn't do: conf.get('setting_name', lambda key: load_from_db('setting_name')) Which is standard practice in a lot of popular Python libs on Pypi. Instead I did: val = conf.get('setting_name') if val is None: val = load_from_db('setting_name') Which is way more verbose. It also has a bug if None is a valid configuration value or if I expect my code to be thread safe. It was not a problem for me, but in that case one would even have to do: try: val = conf['setting_name'] except KeyError: val = load_from_db('setting_name') Which is even more verbose. I was going to suggest to python-ideas to update the dict.get() signature to accept a callable, but it would break compatibility with many code actually getting a callable. We could do it for Python 4, but it's far way. Instead, I think it's a good example of were 'lazy' could help. You can't get simpler than: conf.get('setting_name', lazy load_from_db('setting_name')) From desmoulinmichel at gmail.com Tue Feb 28 07:17:49 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 13:17:49 +0100 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins Message-ID: We have the immutable frozenset for sets and and tuples for lists. But we also have something to manipulate dict as immutable datastructures: >>> from types import MappingProxyType as idict >>> d = idict({'a':1, 'b':2, 'c':3}) >>> d['a'] = 4 Traceback (most recent call last): File "", line 1, in d['a'] = 4 TypeError: 'mappingproxy' object does not support item assignment We could expose this as a built type to allow the last of the most important data structure in Python to be easily immutable. From rosuav at gmail.com Tue Feb 28 07:19:54 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Feb 2017 23:19:54 +1100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: On Tue, Feb 28, 2017 at 11:04 PM, Michel Desmoulin wrote: > Instead, I think it's a good example of were 'lazy' could help. You > can't get simpler than: > > conf.get('setting_name', lazy load_from_db('setting_name')) > Alternatively, you could define 'conf' as a subclass of dict with a __missing__ method: class Config(dict): def __missing__(self, key): self[key] = load_from_db(key) return self[key] conf = Config() Then it becomes even simpler AND less redundant: conf['setting_name'] ChrisA From desmoulinmichel at gmail.com Tue Feb 28 07:24:36 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 13:24:36 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: Le 28/02/2017 ? 13:19, Chris Angelico a ?crit : > On Tue, Feb 28, 2017 at 11:04 PM, Michel Desmoulin > wrote: >> Instead, I think it's a good example of were 'lazy' could help. You >> can't get simpler than: >> >> conf.get('setting_name', lazy load_from_db('setting_name')) >> > > Alternatively, you could define 'conf' as a subclass of dict with a > __missing__ method: > > class Config(dict): > def __missing__(self, key): > self[key] = load_from_db(key) > return self[key] > conf = Config() > > Then it becomes even simpler AND less redundant: > > conf['setting_name'] Yes but this assumes: - I have access to the code instantiating conf; - all code using conf are using load_from_db as a default value; - load_from_db exists for all code using the conf object There is always a solution to all problems, as Python is turing complete. You don't need list comprehension, you can use a for loop. You don't need upacking you can uses indexing. And you don't need lazy, it's just convenient. From markusmeskanen at gmail.com Tue Feb 28 07:27:32 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Tue, 28 Feb 2017 14:27:32 +0200 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: Well you could just use a dict subclass here with get() that takes callable... On Feb 28, 2017 14:25, "Michel Desmoulin" wrote: > > > Le 28/02/2017 ? 13:19, Chris Angelico a ?crit : > > On Tue, Feb 28, 2017 at 11:04 PM, Michel Desmoulin > > wrote: > >> Instead, I think it's a good example of were 'lazy' could help. You > >> can't get simpler than: > >> > >> conf.get('setting_name', lazy load_from_db('setting_name')) > >> > > > > Alternatively, you could define 'conf' as a subclass of dict with a > > __missing__ method: > > > > class Config(dict): > > def __missing__(self, key): > > self[key] = load_from_db(key) > > return self[key] > > conf = Config() > > > > Then it becomes even simpler AND less redundant: > > > > conf['setting_name'] > > Yes but this assumes: > > - I have access to the code instantiating conf; > - all code using conf are using load_from_db as a default value; > - load_from_db exists for all code using the conf object > > There is always a solution to all problems, as Python is turing complete. > > You don't need list comprehension, you can use a for loop. > > You don't need upacking you can uses indexing. > > And you don't need lazy, it's just convenient. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Feb 28 07:36:07 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 13:36:07 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> Le 28/02/2017 ? 13:27, Markus Meskanen a ?crit : > Well you could just use a dict subclass here with get() that takes > callable... > As I said to Angelico: Yes but this assumes: - I have access to the code instantiating conf; - all code using conf are using load_from_db as a default value; - load_from_db exists for all code using the conf object There is always a solution to all problems, as Python is turing complete. You don't need list comprehension, you can use a for loop. You don't need upacking you can use indexing. And you don't need lazy, it's just convenient. From markusmeskanen at gmail.com Tue Feb 28 07:47:09 2017 From: markusmeskanen at gmail.com (Markus Meskanen) Date: Tue, 28 Feb 2017 14:47:09 +0200 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> Message-ID: > > You don't need list comprehension, you can use a for loop. > > You don't need upacking you can use indexing. > > And you don't need lazy, it's just convenient. > With this sole argument you can add almost anything to the language as long as it has at least one use case where it's better than existing tools. I'm not against the lazy proposal in general, but attempting to justify a new keyword to replace this is just silly to me: class Config(dict): def lazy_get(self, key, callable=None): if key in self: return self[key] return callable(key) -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Feb 28 09:46:27 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 15:46:27 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> Message-ID: <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Le 28/02/2017 ? 13:47, Markus Meskanen a ?crit : > You don't need list comprehension, you can use a for loop. > > You don't need upacking you can use indexing. > > And you don't need lazy, it's just convenient. > > > With this sole argument you can add almost anything to the language as > long as it has at least one use case where it's better than existing > tools. I'm not against the lazy proposal in general, but attempting to > justify a new keyword to replace this is just silly to me: > > class Config(dict): > def lazy_get(self, key, callable=None): > if key in self: > return self[key] > return callable(key) With this sole argument you can reject await / async because we could do it with yield from and decorator. With this solve argument you can reject comprehension because we have map and filter. With this solve argument you can reject defauldict because we can code one ourself. I mean come on, a decorator is just: @decorator def decorated(): Instead of decorated = decorator(decorated) It's just convenience and elegance. The sweet spot for lazy is a use case where you can't easily modify the code to get your getter and setter. You example again makes the assumption you can easily add such a wrapper. But if you can't, then you have to import it and wrap the config object in every module using it. Test all that. Ensure you don't pass the wrapper by accident to something that would break on it, etc. And how many time do you have to do that ? You will write your wrapper for lists ? And tuples ? And will you test those ? document those ? Explain to others that to get a callable, you need to pass a callable returning a callable ? Then again for your next project or you do a library with all the stuff it implies ? It's neither convenient nor elegant. lazy is not only practical, but it's also beautiful. It reads well. It solves a problem we all have on a regular basis. Yes we can live without it. I mean, Python is already incredibly convenient, of course whatever we suggest now is going to be a cherry on top of the language cake. And yes I got a big mouth because it's a lot of work to implement it and it's work I'm not capable to do since I can't code in C to save my life. Does it removes the merits from lazy ? Wouldn't you like it to just be able to drop "lazy" in your next 3.7 code ? From steve at pearwood.info Tue Feb 28 09:45:19 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 01:45:19 +1100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> Message-ID: <20170228144519.GK5689@ando.pearwood.info> On Tue, Feb 28, 2017 at 12:54:26PM +0100, Michel Desmoulin wrote: > dict.get() is a very useful method, but for lists and tuples, we need to > rely on try/except instead. No you don't. You can use slicing. alist = [1, 2, 3] print(alist[99:100]) # get the item at position 99 In my experience, dict.get is very useful, but list.get only *seems* useful. I've written my own version: def get(alist, pos, default=None): try: return alist[pos] except IndexError: return default but then struggled to find a good use for it. It seems like it ought to be useful, but in practice I found that it was only covering up bugs in my code. If I was indexing a list outside of the range of existing items, that's a bug, and using get() just made it hard to fix. > Can we get list.get and tuple.get as well? > > Also, for list, a list.setdefault like the dict.setdefault would be logical. What would it do? For example, given: alist = [] y = alist.setdefault(10, 'a') what will alist equal? -- Steve From mlet_it_bew at 126.com Tue Feb 28 08:59:51 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Tue, 28 Feb 2017 21:59:51 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects Message-ID: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> Hi! I write a '<'-based immutable set class. But it is quit different from the standard set class. I wish collections.abc.Set be more friendly to immutable tree sets or Python add new syntax to unify such difference. good example: a = [] a += a # "a" is the original list a = () a += a # "a" is a new tuple bad example: a = set() a.add(1) # return None; "a" changed e = a.pop(); a = frozen_tree_set() a = a.add(1) # return another set; e, a = a.ipop() # return two objects instead of one! solution I used: a <<= 1 # <==> a = a.add(1) but "a.ipop()" .... my current solution is to write a wrapper class to turn immutable set into mutable set, but it is so noisy to box and unbox. solution that I wish: a :=.add(1) # "=." mimic "+="; return the result object e ~ a :=.pop() d[key] :=.add(1) # in dict if only python add following features: 1) modify the original object 1-0) 1) define: def .method(self):... # "." means "intend to modify self" # return any thing # to avoid immutable method # which was intended to return new object # leave self unchanged 2) invoke: r = a..method(); 1-1) ignore result # no matter what returned, discard it a.-.method(); # <==> [a..method(), None][-1] 1-2) return self # no matter what returned, return self a.>.method().>.method();# <==> [a..method(), a..method(), a][-1] 2) create new object 2-0) 1) define # intend to return (result, new object) def ^method():... 2)invoke: r, a' = a.^method(); 2-1) return other, discard result a.-^method().-^method();# <==> a.^method()[1].^method()[1]; 2-2) assign other to original variable a=^method(); # <==> a = a.^method()[1]; 3) unify both: a :=.method(); # if defined ".method" then "a..method();" # elif defined "^method" then "a = a.^method()[1];" # elif defined "method" then "a.method();" # else error r ~ a :=.method(); # if defined ".method" then "r = a..method();" # elif defined "^method" then "r, a = a.^method();" # elif defined "method" then "r = a.method();" # else error -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Tue Feb 28 10:08:46 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Tue, 28 Feb 2017 10:08:46 -0500 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: Yes I do think this is precisely where a lazy or delayed feature would work well. I'm working on writing up a complete PEP now, and have made some progress on the things left unsolved in the other thread. I will post again when the PEP is closer to complete. Fundamentally, anything can be done, but I don't think that altering the code of others for one case is Pythonic, and though it happens, I think there is value in making it less necessary. Cheers! Joseph > On Feb 28, 2017, at 9:46 AM, Michel Desmoulin wrote: > > >> Le 28/02/2017 ? 13:47, Markus Meskanen a ?crit : >> You don't need list comprehension, you can use a for loop. >> >> You don't need upacking you can use indexing. >> >> And you don't need lazy, it's just convenient. >> >> >> With this sole argument you can add almost anything to the language as >> long as it has at least one use case where it's better than existing >> tools. I'm not against the lazy proposal in general, but attempting to >> justify a new keyword to replace this is just silly to me: >> >> class Config(dict): >> def lazy_get(self, key, callable=None): >> if key in self: >> return self[key] >> return callable(key) > > With this sole argument you can reject await / async because we could do > it with yield from and decorator. > > With this solve argument you can reject comprehension because we have > map and filter. > > With this solve argument you can reject defauldict because we can code > one ourself. > > I mean come on, a decorator is just: > > @decorator > def decorated(): > > Instead of > > decorated = decorator(decorated) > > It's just convenience and elegance. > > The sweet spot for lazy is a use case where you can't easily modify the > code to get your getter and setter. You example again makes the > assumption you can easily add such a wrapper. But if you can't, then you > have to import it and wrap the config object in every module using it. > Test all that. Ensure you don't pass the wrapper by accident to > something that would break on it, etc. > > And how many time do you have to do that ? > > You will write your wrapper for lists ? And tuples ? > > And will you test those ? document those ? Explain to others that to get > a callable, you need to pass a callable returning a callable ? > > Then again for your next project or you do a library with all the stuff > it implies ? > > It's neither convenient nor elegant. > > lazy is not only practical, but it's also beautiful. It reads well. It > solves a problem we all have on a regular basis. > > Yes we can live without it. I mean, Python is already incredibly > convenient, of course whatever we suggest now is going to be a cherry on > top of the language cake. > > And yes I got a big mouth because it's a lot of work to implement it and > it's work I'm not capable to do since I can't code in C to save my life. > > Does it removes the merits from lazy ? > > Wouldn't you like it to just be able to drop "lazy" in your next 3.7 code ? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From desmoulinmichel at gmail.com Tue Feb 28 10:16:28 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 16:16:28 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <20170228144519.GK5689@ando.pearwood.info> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> Message-ID: Le 28/02/2017 ? 15:45, Steven D'Aprano a ?crit : > On Tue, Feb 28, 2017 at 12:54:26PM +0100, Michel Desmoulin wrote: >> dict.get() is a very useful method, but for lists and tuples, we need to >> rely on try/except instead. > > No you don't. You can use slicing. > > alist = [1, 2, 3] > print(alist[99:100]) # get the item at position 99 No this gives you a list of one item or an empty list. dict.get('key', default_value) let you get a SCALAR value, OR a default value if it doesn't exist. It's a very different use case. > > In my experience, dict.get is very useful, but list.get only *seems* > useful. I've written my own version: > > def get(alist, pos, default=None): > try: > return alist[pos] > except IndexError: > return default > > Based on your rational, we would just reject dict.get as well the first time it's implemented. > but then struggled to find a good use for it. It seems like it ought to > be useful, but in practice I found that it was only covering up bugs in > my code. How so ? "get the element x or a default value if it doesn't exist" seem at the contrary, a very robust approach. Plus it's consistent. It's only fair to expect it to exists after you learn about dict.get. First places where I missed it at the top of my head was *args, sys.argv personnaly. If I was indexing a list outside of the range of existing > items, that's a bug, and using get() just made it hard to fix. > > > >> Can we get list.get and tuple.get as well? >> >> Also, for list, a list.setdefault like the dict.setdefault would be logical. > > What would it do? > > For example, given: > > alist = [] > y = alist.setdefault(10, 'a') > > what will alist equal? Fair enough. From rainventions at gmail.com Tue Feb 28 10:19:03 2017 From: rainventions at gmail.com (Ryan Birmingham) Date: Tue, 28 Feb 2017 10:19:03 -0500 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> Message-ID: I'm sorry for the confusion, but what is frozen_tree_set() here, and what is ipop? frozensets don't have pop or 'ipop', so my apologies that I'm a bit lost here. -Ryan Birmingham On 28 February 2017 at 08:59, ????? wrote: > > Hi! > I write a '<'-based immutable set class. > But it is quit different from the standard set class. > I wish collections.abc.Set be more friendly to immutable tree sets or > Python add new syntax to unify such difference. > > > good example: > a = [] > a += a # "a" is the original list > > a = () > a += a # "a" is a new tuple > > bad example: > a = set() > a.add(1) # return None; "a" changed > e = a.pop(); > > a = frozen_tree_set() > a = a.add(1) # return another set; > e, a = a.ipop() # return two objects instead of one! > > solution I used: > a <<= 1 # <==> a = a.add(1) > but "a.ipop()" .... > my current solution is to write a wrapper class > to turn immutable set into mutable set, > but it is so noisy to box and unbox. > > > solution that I wish: > a :=.add(1) # "=." mimic "+="; return the result object > e ~ a :=.pop() > > d[key] :=.add(1) # in dict > > > > if only python add following features: > 1) modify the original object > 1-0) > 1) define: > def .method(self):... > # "." means "intend to modify self" > # return any thing > # to avoid immutable method > # which was intended to return new object > # leave self unchanged > > 2) invoke: > r = a..method(); > 1-1) ignore result > # no matter what returned, discard it > a.-.method(); # <==> [a..method(), None][-1] > 1-2) return self > # no matter what returned, return self > a.>.method().>.method();# <==> [a..method(), a..method(), a][-1] > > > 2) create new object > 2-0) > 1) define > # intend to return (result, new object) > def ^method():... > > 2)invoke: > r, a' = a.^method(); > 2-1) return other, discard result > a.-^method().-^method();# <==> a.^method()[1].^method()[1]; > 2-2) assign other to original variable > a=^method(); # <==> a = a.^method()[1]; > > 3) unify both: > a :=.method(); > # if defined ".method" then "a..method();" > # elif defined "^method" then "a = a.^method()[1];" > # elif defined "method" then "a.method();" > # else error > > r ~ a :=.method(); > # if defined ".method" then "r = a..method();" > # elif defined "^method" then "r, a = a.^method();" > # elif defined "method" then "r = a.method();" > # else error > > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Feb 28 10:19:26 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 16:19:26 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: <870625e8-d57d-a2aa-5b1f-06f2b0566b94@gmail.com> Good luck Le 28/02/2017 ? 16:08, Joseph Hackman a ?crit : > Yes I do think this is precisely where a lazy or delayed feature would work well. I'm working on writing up a complete PEP now, and have made some progress on the things left unsolved in the other thread. I will post again when the PEP is closer to complete. > > Fundamentally, anything can be done, but I don't think that altering the code of others for one case is Pythonic, and though it happens, I think there is value in making it less necessary. > > Cheers! > Joseph > From mehaase at gmail.com Tue Feb 28 10:21:23 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 28 Feb 2017 10:21:23 -0500 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: This could be solved with a null-coalescing operator, e.g. PEP-505. >>> val = conf.get('setting_name') ?? load_from_db('setting_name') The right side isn't evaluated unless the left side is None. It's similar to this: >>> val = conf.get('setting_name') or load_from_db('setting_name') Except that using "or" would result in any false-y value (0, "", [], etc.) being overridden by the result of `load_from_db(...)`. I'm not strongly opposed to "lazy", but I think null-coalescing is easier to reason about. On Tue, Feb 28, 2017 at 7:04 AM, Michel Desmoulin wrote: > The debate on the 'lazy' keyword seems to have settled, but I don't know > if somebody is trying to write a PEP about it. > > Anyway, I was doing something like this the other day: > > conf.get('setting_name', load_from_db('setting_name')) > > And then realized I could save a query not doing the load_from_db() call > most of the time. > > But dict.get didn't accept callable so I couldn't do: > > conf.get('setting_name', lambda key: load_from_db('setting_name')) > > Which is standard practice in a lot of popular Python libs on Pypi. > > Instead I did: > > val = conf.get('setting_name') > if val is None: > val = load_from_db('setting_name') > > Which is way more verbose. It also has a bug if None is a valid > configuration value or if I expect my code to be thread safe. > > It was not a problem for me, but in that case one would even have to do: > > try: > val = conf['setting_name'] > except KeyError: > val = load_from_db('setting_name') > > Which is even more verbose. > > I was going to suggest to python-ideas to update the dict.get() > signature to accept a callable, but it would break compatibility with > many code actually getting a callable. > > We could do it for Python 4, but it's far way. > > Instead, I think it's a good example of were 'lazy' could help. You > can't get simpler than: > > conf.get('setting_name', lazy load_from_db('setting_name')) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Tue Feb 28 10:24:29 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Tue, 28 Feb 2017 10:24:29 -0500 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: References: Message-ID: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> +1 I think this makes a lot of sense. What would you name the built in? -Joseph > On Feb 28, 2017, at 7:17 AM, Michel Desmoulin wrote: > > We have the immutable frozenset for sets and and tuples for lists. > > But we also have something to manipulate dict as immutable datastructures: > >>>> from types import MappingProxyType as idict >>>> d = idict({'a':1, 'b':2, 'c':3}) >>>> d['a'] = 4 > Traceback (most recent call last): > File "", line 1, in > d['a'] = 4 > TypeError: 'mappingproxy' object does not support item assignment > > We could expose this as a built type to allow the last of the most > important data structure in Python to be easily immutable. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From desmoulinmichel at gmail.com Tue Feb 28 10:28:34 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 28 Feb 2017 16:28:34 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: <0dc13b0e-7ed2-2ce0-a3a9-74d6375f839d@gmail.com> Only if "None" is not a proper value from .get(). And this doest solve the problems with more general callbacks that are not trying to get a value (cf Django lazy_gettext). Besides, "??" does not exist yet for Python, so let's counter the argument for lazy with the potential benefits of something we don't have ATM. A proposal is already hard enough to defenD on Python-ideas :) Le 28/02/2017 ? 16:21, Mark E. Haase a ?crit : > This could be solved with a null-coalescing operator, e.g. PEP-505. > > >>> val = conf.get('setting_name') ?? load_from_db('setting_name') > > The right side isn't evaluated unless the left side is None. It's > similar to this: > > >>> val = conf.get('setting_name') or load_from_db('setting_name') > > Except that using "or" would result in any false-y value (0, "", [], > etc.) being overridden by the result of `load_from_db(...)`. > > I'm not strongly opposed to "lazy", but I think null-coalescing is > easier to reason about. > From levkivskyi at gmail.com Tue Feb 28 10:40:40 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 28 Feb 2017 16:40:40 +0100 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> References: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> Message-ID: Searching MappingProxyType on GitHub gives over 10,000 results. I think it probably makes sense to make mappingproxy more "visible", maybe move it to collections module? (where OrderedDict lives) I am not sure if it makes sense to move it to builtins. (for comparison frozenset gives around 1000,000 results) -- Ivan On 28 February 2017 at 16:24, Joseph Hackman wrote: > +1 > > I think this makes a lot of sense. What would you name the built in? > > -Joseph > > > On Feb 28, 2017, at 7:17 AM, Michel Desmoulin > wrote: > > > > We have the immutable frozenset for sets and and tuples for lists. > > > > But we also have something to manipulate dict as immutable > datastructures: > > > >>>> from types import MappingProxyType as idict > >>>> d = idict({'a':1, 'b':2, 'c':3}) > >>>> d['a'] = 4 > > Traceback (most recent call last): > > File "", line 1, in > > d['a'] = 4 > > TypeError: 'mappingproxy' object does not support item assignment > > > > We could expose this as a built type to allow the last of the most > > important data structure in Python to be easily immutable. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mehaase at gmail.com Tue Feb 28 10:41:53 2017 From: mehaase at gmail.com (Mark E. Haase) Date: Tue, 28 Feb 2017 10:41:53 -0500 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <4DE2E169-8419-4689-AF3E-6C1982D67AC9@gmail.com> References: <4DE2E169-8419-4689-AF3E-6C1982D67AC9@gmail.com> Message-ID: There are plenty of PEP-505 threads in the archives, plus the PEP itself. I was only pointing out that the use case in thread might also be solved with an existing proposal*, so I won't derail this thread any further. *There are actually several proposals for short-circuit evaluation, including PEP-531 and PEP-532. On Tue, Feb 28, 2017 at 10:29 AM, Joseph Hackman wrote: > I like null coalesce too. :) > > I know that BDFL has said no to ? Before, but was that for the if-then > shorthand only? > > Perhaps submit a new thread to this list so people can discuss/find? > > -Joseph > > On Feb 28, 2017, at 10:21 AM, Mark E. Haase wrote: > > This could be solved with a null-coalescing operator, e.g. PEP-505. > > >>> val = conf.get('setting_name') ?? load_from_db('setting_name') > > The right side isn't evaluated unless the left side is None. It's similar > to this: > > >>> val = conf.get('setting_name') or load_from_db('setting_name') > > Except that using "or" would result in any false-y value (0, "", [], etc.) > being overridden by the result of `load_from_db(...)`. > > I'm not strongly opposed to "lazy", but I think null-coalescing is easier > to reason about. > > > On Tue, Feb 28, 2017 at 7:04 AM, Michel Desmoulin < > desmoulinmichel at gmail.com> wrote: > >> The debate on the 'lazy' keyword seems to have settled, but I don't know >> if somebody is trying to write a PEP about it. >> >> Anyway, I was doing something like this the other day: >> >> conf.get('setting_name', load_from_db('setting_name')) >> >> And then realized I could save a query not doing the load_from_db() call >> most of the time. >> >> But dict.get didn't accept callable so I couldn't do: >> >> conf.get('setting_name', lambda key: load_from_db('setting_name')) >> >> Which is standard practice in a lot of popular Python libs on Pypi. >> >> Instead I did: >> >> val = conf.get('setting_name') >> if val is None: >> val = load_from_db('setting_name') >> >> Which is way more verbose. It also has a bug if None is a valid >> configuration value or if I expect my code to be thread safe. >> >> It was not a problem for me, but in that case one would even have to do: >> >> try: >> val = conf['setting_name'] >> except KeyError: >> val = load_from_db('setting_name') >> >> Which is even more verbose. >> >> I was going to suggest to python-ideas to update the dict.get() >> signature to accept a callable, but it would break compatibility with >> many code actually getting a callable. >> >> We could do it for Python 4, but it's far way. >> >> Instead, I think it's a good example of were 'lazy' could help. You >> can't get simpler than: >> >> conf.get('setting_name', lazy load_from_db('setting_name')) >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 10:44:38 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 10:44:38 -0500 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: If you'd like help with the PEP, I'd be happy to assist (subject to time, as always the problem). I think this idea got pretty good support in the discussion (of course, I'm biased because it's one of the few proposed here that I actually really like). Few core developers really chimed in on the matter, so it's hard to say what they might opine about the implementation or supportability. On Tue, Feb 28, 2017 at 10:08 AM, Joseph Hackman wrote: > Yes I do think this is precisely where a lazy or delayed feature would > work well. I'm working on writing up a complete PEP now, and have made some > progress on the things left unsolved in the other thread. I will post again > when the PEP is closer to complete. > > Fundamentally, anything can be done, but I don't think that altering the > code of others for one case is Pythonic, and though it happens, I think > there is value in making it less necessary. > > Cheers! > Joseph > > > On Feb 28, 2017, at 9:46 AM, Michel Desmoulin > wrote: > > > > > >> Le 28/02/2017 ? 13:47, Markus Meskanen a ?crit : > >> You don't need list comprehension, you can use a for loop. > >> > >> You don't need upacking you can use indexing. > >> > >> And you don't need lazy, it's just convenient. > >> > >> > >> With this sole argument you can add almost anything to the language as > >> long as it has at least one use case where it's better than existing > >> tools. I'm not against the lazy proposal in general, but attempting to > >> justify a new keyword to replace this is just silly to me: > >> > >> class Config(dict): > >> def lazy_get(self, key, callable=None): > >> if key in self: > >> return self[key] > >> return callable(key) > > > > With this sole argument you can reject await / async because we could do > > it with yield from and decorator. > > > > With this solve argument you can reject comprehension because we have > > map and filter. > > > > With this solve argument you can reject defauldict because we can code > > one ourself. > > > > I mean come on, a decorator is just: > > > > @decorator > > def decorated(): > > > > Instead of > > > > decorated = decorator(decorated) > > > > It's just convenience and elegance. > > > > The sweet spot for lazy is a use case where you can't easily modify the > > code to get your getter and setter. You example again makes the > > assumption you can easily add such a wrapper. But if you can't, then you > > have to import it and wrap the config object in every module using it. > > Test all that. Ensure you don't pass the wrapper by accident to > > something that would break on it, etc. > > > > And how many time do you have to do that ? > > > > You will write your wrapper for lists ? And tuples ? > > > > And will you test those ? document those ? Explain to others that to get > > a callable, you need to pass a callable returning a callable ? > > > > Then again for your next project or you do a library with all the stuff > > it implies ? > > > > It's neither convenient nor elegant. > > > > lazy is not only practical, but it's also beautiful. It reads well. It > > solves a problem we all have on a regular basis. > > > > Yes we can live without it. I mean, Python is already incredibly > > convenient, of course whatever we suggest now is going to be a cherry on > > top of the language cake. > > > > And yes I got a big mouth because it's a lot of work to implement it and > > it's work I'm not capable to do since I can't code in C to save my life. > > > > Does it removes the merits from lazy ? > > > > Wouldn't you like it to just be able to drop "lazy" in your next 3.7 > code ? > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From josephhackman at gmail.com Tue Feb 28 10:29:37 2017 From: josephhackman at gmail.com (Joseph Hackman) Date: Tue, 28 Feb 2017 10:29:37 -0500 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: <4DE2E169-8419-4689-AF3E-6C1982D67AC9@gmail.com> I like null coalesce too. :) I know that BDFL has said no to ? Before, but was that for the if-then shorthand only? Perhaps submit a new thread to this list so people can discuss/find? -Joseph > On Feb 28, 2017, at 10:21 AM, Mark E. Haase wrote: > > This could be solved with a null-coalescing operator, e.g. PEP-505. > > >>> val = conf.get('setting_name') ?? load_from_db('setting_name') > > The right side isn't evaluated unless the left side is None. It's similar to this: > > >>> val = conf.get('setting_name') or load_from_db('setting_name') > > Except that using "or" would result in any false-y value (0, "", [], etc.) being overridden by the result of `load_from_db(...)`. > > I'm not strongly opposed to "lazy", but I think null-coalescing is easier to reason about. > > >> On Tue, Feb 28, 2017 at 7:04 AM, Michel Desmoulin wrote: >> The debate on the 'lazy' keyword seems to have settled, but I don't know >> if somebody is trying to write a PEP about it. >> >> Anyway, I was doing something like this the other day: >> >> conf.get('setting_name', load_from_db('setting_name')) >> >> And then realized I could save a query not doing the load_from_db() call >> most of the time. >> >> But dict.get didn't accept callable so I couldn't do: >> >> conf.get('setting_name', lambda key: load_from_db('setting_name')) >> >> Which is standard practice in a lot of popular Python libs on Pypi. >> >> Instead I did: >> >> val = conf.get('setting_name') >> if val is None: >> val = load_from_db('setting_name') >> >> Which is way more verbose. It also has a bug if None is a valid >> configuration value or if I expect my code to be thread safe. >> >> It was not a problem for me, but in that case one would even have to do: >> >> try: >> val = conf['setting_name'] >> except KeyError: >> val = load_from_db('setting_name') >> >> Which is even more verbose. >> >> I was going to suggest to python-ideas to update the dict.get() >> signature to accept a callable, but it would break compatibility with >> many code actually getting a callable. >> >> We could do it for Python 4, but it's far way. >> >> Instead, I think it's a good example of were 'lazy' could help. You >> can't get simpler than: >> >> conf.get('setting_name', lazy load_from_db('setting_name')) >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 28 10:57:25 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Feb 2017 16:57:25 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: On 28.02.2017 15:46, Michel Desmoulin wrote: > lazy is not only practical, but it's also beautiful. It reads well. It > solves a problem we all have on a regular basis. The only practical use case I ever ran into for lazy evaluation are lazy imports, simply because imports cause a lot of startup overhead. It's a case which the new keyword wouldn't help with, though. For the discussion, it would help if you'd write up a definition of where the lazy evaluation should finally happen, which use cases would be allowed or not and how the compiler could detect these. IMO, there are much better ways to write code which only evaluates expensive code when really needed. I don't see how "lazy" could automate this in a well defined, helpful and obvious way, simply because the side effects of moving evaluation from the place of definition to an arbitrary other place in the code are not easy to manage. Even if you do manage to clearly define when to evaluate a lazy expression, the context in which it gets evaluated may have already changed, giving wrong results, causing exceptions due to missing information. Delaying exceptions can also break the exception handling in your application. PS: You run into the same issues with lazy imports, which is why special care has to be taken when using these. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 28 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mertz at gnosis.cx Tue Feb 28 11:35:49 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 08:35:49 -0800 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: Clearly there is SOME semantics that is consistent and coherent since many languages have either a lazy or eager declaration syntax, with different defaults between languages but both being built in. There are a lot of ways that Python isn't Haskell, obviously. But both Scheme and OCaml are eager by default with a lazy declaration (and Haskell or Miranda have an eager declaration correspondingly). It might be worth looking at their semantics in the PEP. On Tue, Feb 28, 2017 at 7:57 AM, M.-A. Lemburg wrote: > On 28.02.2017 15:46, Michel Desmoulin wrote: > > lazy is not only practical, but it's also beautiful. It reads well. It > > solves a problem we all have on a regular basis. > > The only practical use case I ever ran into for lazy evaluation > are lazy imports, simply because imports cause a lot of > startup overhead. It's a case which the new keyword wouldn't > help with, though. > > For the discussion, it would help if you'd write up a definition > of where the lazy evaluation should finally happen, which use > cases would be allowed or not and how the compiler could detect > these. > > IMO, there are much better ways to write code which only evaluates > expensive code when really needed. > > I don't see how "lazy" could automate this in a well defined, > helpful and obvious way, simply because the side effects of moving > evaluation from the place of definition to an arbitrary other place > in the code are not easy to manage. > > Even if you do manage to clearly define when to evaluate a lazy > expression, the context in which it gets evaluated may > have already changed, giving wrong results, causing exceptions > due to missing information. Delaying exceptions can also > break the exception handling in your application. > > PS: You run into the same issues with lazy imports, which is > why special care has to be taken when using these. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Feb 28 2017) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 28 11:54:18 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Feb 2017 17:54:18 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> Message-ID: <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> On 28.02.2017 17:35, David Mertz wrote: > Clearly there is SOME semantics that is consistent and coherent since many > languages have either a lazy or eager declaration syntax, with different > defaults between languages but both being built in. There are a lot of > ways that Python isn't Haskell, obviously. But both Scheme and OCaml are > eager by default with a lazy declaration (and Haskell or Miranda have an > eager declaration correspondingly). > > It might be worth looking at their semantics in the PEP. Scheme, for example, uses an explicit approach to turning a promise into a value: http://www.shido.info/lisp/scheme_lazy_e.html This makes a lot of sense, but you can already have the same in Python using generators. Haskell is lazy by default, but this only works because you can pass parameters by name rather than value. https://wiki.haskell.org/Lazy_evaluation In Python we only have pass by value (parameters to functions get pushed onto the VM stack). Pass by name can also be had, but requires explicit indirection, e.g. by using a generator as wrapper. > On Tue, Feb 28, 2017 at 7:57 AM, M.-A. Lemburg wrote: > >> On 28.02.2017 15:46, Michel Desmoulin wrote: >>> lazy is not only practical, but it's also beautiful. It reads well. It >>> solves a problem we all have on a regular basis. >> >> The only practical use case I ever ran into for lazy evaluation >> are lazy imports, simply because imports cause a lot of >> startup overhead. It's a case which the new keyword wouldn't >> help with, though. >> >> For the discussion, it would help if you'd write up a definition >> of where the lazy evaluation should finally happen, which use >> cases would be allowed or not and how the compiler could detect >> these. >> >> IMO, there are much better ways to write code which only evaluates >> expensive code when really needed. >> >> I don't see how "lazy" could automate this in a well defined, >> helpful and obvious way, simply because the side effects of moving >> evaluation from the place of definition to an arbitrary other place >> in the code are not easy to manage. >> >> Even if you do manage to clearly define when to evaluate a lazy >> expression, the context in which it gets evaluated may >> have already changed, giving wrong results, causing exceptions >> due to missing information. Delaying exceptions can also >> break the exception handling in your application. >> >> PS: You run into the same issues with lazy imports, which is >> why special care has to be taken when using these. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts (#1, Feb 28 2017) >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> Python Database Interfaces ... http://products.egenix.com/ >>>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> http://www.malemburg.com/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 28 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mertz at gnosis.cx Tue Feb 28 12:10:00 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 09:10:00 -0800 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> Message-ID: On Tue, Feb 28, 2017 at 8:54 AM, M.-A. Lemburg wrote: > On 28.02.2017 17:35, David Mertz wrote: > > Clearly there is SOME semantics that is consistent and coherent since > many > > languages have either a lazy or eager declaration syntax, with different > > defaults between languages but both being built in. There are a lot of > > ways that Python isn't Haskell, obviously. But both Scheme and OCaml are > > eager by default with a lazy declaration (and Haskell or Miranda have an > > eager declaration correspondingly). > > > > It might be worth looking at their semantics in the PEP. > > Scheme, for example, uses an explicit approach to turning > a promise into a value: > > http://www.shido.info/lisp/scheme_lazy_e.html > > This makes a lot of sense, but you can already have the > same in Python using generators. > I think the closer equivalent is a lambda. And it's quite true that Scheme's `force` is pretty much the same thing as making a call to the lambda later. E.g. def force(x): if callable(x): return x() else: return x In Python we only have pass by value (parameters to functions > get pushed onto the VM stack). Pass by name can also be had, > but requires explicit indirection, e.g. by using a generator > as wrapper. I wouldn't want to change pass-by-value semantics. What I imagine instead is a new type `lazy` or `delayed` or `deferred` (whatever spelling). Delayed objects would have that type, and access to objects would need to do an implementation-level call to a "_force" that looks something like: def _force(x): if isinstance(x, delayed): return _concretize(x) else: return x I *do* see the obvious likely problem here. Object access is perhaps the most common thing that happens in Python, and adding a new conditional check to every one of those could very well slow down all the non-lazy behavior far too much. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 12:12:41 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 09:12:41 -0800 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: References: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> Message-ID: The difference in hits might be because MappingProxyType has a funny name and is in a hidden-ish location. I.e. not necessarily because it *would be* less used or useful if it were more exposed. In either case, the name that makes sense to me would be `frozendict`. That could very well live in `collections` of course. On Tue, Feb 28, 2017 at 7:40 AM, Ivan Levkivskyi wrote: > Searching MappingProxyType on GitHub gives over 10,000 results. > I think it probably makes sense to make mappingproxy more "visible", > maybe move it to collections module? (where OrderedDict lives) > > I am not sure if it makes sense to move it to builtins. (for comparison > frozenset gives around 1000,000 results) > > -- > Ivan > > > > On 28 February 2017 at 16:24, Joseph Hackman > wrote: > >> +1 >> >> I think this makes a lot of sense. What would you name the built in? >> >> -Joseph >> >> > On Feb 28, 2017, at 7:17 AM, Michel Desmoulin < >> desmoulinmichel at gmail.com> wrote: >> > >> > We have the immutable frozenset for sets and and tuples for lists. >> > >> > But we also have something to manipulate dict as immutable >> datastructures: >> > >> >>>> from types import MappingProxyType as idict >> >>>> d = idict({'a':1, 'b':2, 'c':3}) >> >>>> d['a'] = 4 >> > Traceback (most recent call last): >> > File "", line 1, in >> > d['a'] = 4 >> > TypeError: 'mappingproxy' object does not support item assignment >> > >> > We could expose this as a built type to allow the last of the most >> > important data structure in Python to be easily immutable. >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 12:18:53 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 09:18:53 -0800 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> Message-ID: On Tue, Feb 28, 2017 at 7:16 AM, Michel Desmoulin wrote: > Le 28/02/2017 ? 15:45, Steven D'Aprano a ?crit : > > No you don't. You can use slicing. > > alist = [1, 2, 3] > > print(alist[99:100]) # get the item at position 99 > > No this gives you a list of one item or an empty list. > > dict.get('key', default_value) let you get a SCALAR value, OR a default > value if it doesn't exist. > x = (alist[pos:pos+1] or [default_val])[0] > How so ? "get the element x or a default value if it doesn't exist" seem > at the contrary, a very robust approach. > Yes, and easily written as above. What significant advantage would it have to spell the above as: x = alist.get(pos, default_val) It's a couple characters shorter in the proposed version. I guess I'll concede that needing the odd indexing at the end to get the scalar is slightly ugly. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 12:23:54 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 09:23:54 -0800 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> Message-ID: On Tue, Feb 28, 2017 at 5:59 AM, ????? wrote: > bad example: > a = set() > a.add(1) # return None; "a" changed > e = a.pop(); > That is simply misspelled for your intent. a = set() a |= {1} e = a.pop() -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Feb 28 12:30:22 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Feb 2017 18:30:22 +0100 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> Message-ID: <5404e149-7260-ea19-8ea9-feb7df363702@egenix.com> On 28.02.2017 17:54, M.-A. Lemburg wrote: > On 28.02.2017 17:35, David Mertz wrote: >> Clearly there is SOME semantics that is consistent and coherent since many >> languages have either a lazy or eager declaration syntax, with different >> defaults between languages but both being built in. There are a lot of >> ways that Python isn't Haskell, obviously. But both Scheme and OCaml are >> eager by default with a lazy declaration (and Haskell or Miranda have an >> eager declaration correspondingly). >> >> It might be worth looking at their semantics in the PEP. > > Scheme, for example, uses an explicit approach to turning > a promise into a value: > > http://www.shido.info/lisp/scheme_lazy_e.html > > This makes a lot of sense, but you can already have the > same in Python using generators. Here's an example similar to OCaml's lazy evaluation, which uses a simple lazy proxy object. import sys import time ### OCaml like lazy evaluation class Lazy: def __init__(self, code, frame): self.code = code self.globals = frame.f_globals self.locals = frame.f_locals def force(self): return eval(self.code, self.globals, self.locals) def lazy(code): return Lazy(code, sys._getframe(1)) ### def log(level, b, c): if level > 100: return if isinstance(c, Lazy): c = c.force() print ('%04i: %s' % (level, b % c)) def expensive(x): time.sleep(2.0) return x value = 1 log(1000, 'Hello %i', lazy("expensive(value)")) log(10, 'Error %i', lazy("expensive(value)")) ### Everything is nice and explicitly defined in the example. You can see where the deferred evaluation is requested and where it's eventually run. There are no surprises. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 28 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From mertz at gnosis.cx Tue Feb 28 12:52:27 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 09:52:27 -0800 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: <5404e149-7260-ea19-8ea9-feb7df363702@egenix.com> References: <92c8ce8d-83e9-635e-fbfc-45f53e2ba708@gmail.com> <73557614-5b17-d693-2f18-20a3f9c112f2@gmail.com> <4bc3cce6-0cfb-b344-ce09-8e03b46cbef0@egenix.com> <5404e149-7260-ea19-8ea9-feb7df363702@egenix.com> Message-ID: On Tue, Feb 28, 2017 at 9:30 AM, M.-A. Lemburg wrote: > Here's an example similar to OCaml's lazy evaluation, which > uses a simple lazy proxy object. > ### OCaml like lazy evaluation > class Lazy: > def __init__(self, code, frame): > self.code = code > self.globals = frame.f_globals > self.locals = frame.f_locals > def force(self): > return eval(self.code, self.globals, self.locals) > This is a nice implementation. def log(level, b, c): > if level > 100: > return > if isinstance(c, Lazy): > c = c.force() > print ('%04i: %s' % (level, b % c)) > > value = 1 > log(1000, 'Hello %i', lazy("expensive(value)")) > log(10, 'Error %i', lazy("expensive(value)")) > I agree this is very explicit. It's also more code which has a minor smell to it. In the hypothetical new construct I think it would look like this: def log(level, b, c): if level <= 100: print ('%04i: %s' % (level, b % c)) log(1000, 'Hello %i', delayed expensive(value)) log(10, 'Error %i', delayed expensive(value)) To me the second reads much better. But indeed there *might be* surprises. We might access the delayed object somewhere other than in the log() function (not in this example since it's not named, but in other cases). Understanding where that happens adds to debugging work. Where I believe this is more important is for Dask-like code structures. So perhaps I write: x = delayed expensive(1,2) y = delayed expensive(x,7) z = delayed expensive(x,y) w = delayed expensive(y,z) v = delayed expensive(w,x) print(z) If I want to concretize `z` I don't want to do all the work of also concretizing `w` and `v` (but I need to do `x` and `y`). In your explicit proxy approach, a lot of extra code is needed here. But maybe Dask's spelling is perfectly fine: from dask import delayed x = delayed(expensive)(1,2) y = delayed(expensive)(x,7) z = delayed(expensive)(x,y) w = delayed(expensive)(y,z) v = delayed(expensive)(w,x) print(z.compute()) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Feb 28 12:55:06 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 28 Feb 2017 09:55:06 -0800 Subject: [Python-ideas] math.nextafter In-Reply-To: References: <013401d27edd$234b8630$69e29290$@hotmail.com> <7ed2141a-9b77-ef9e-c4cb-2ee640689da0@egenix.com> <00594e7e-245c-846f-bf3d-636ac9e0a81f@egenix.com> Message-ID: On Fri, Feb 24, 2017 at 6:43 PM, Nathaniel Smith wrote: > On Feb 24, 2017 5:29 PM, "David Mertz" wrote: > > Marc-Andr? slightly misspelled the recent-ish addition of math.isclose(), > but I agree that it is absolutely where a "nextafter" belongs. > > > My 2c: I disagree -- numerical tolerance based closeness is fundamentally > different than floating point representation based closeness > agreed -- isclose() is a convenience function to provide folks a way to "do a reasonable thing" without delving into floating point representation issues, etc. INdeed, essentially the same approach could be used for Decimal and Fraction, though they aren't supported by math.isclose() due to the math module being written in C, and the rest of math being all about floats. It's also a little weird to jump from nextafter to isclose, since AFAIK > they have pretty much non-overlapping use cases... > Exactly -- and you can tell by this this thread that confusion is easy with this stuff -- putting them together will only sow more confusion. "Floating Point is Hard" Note how many really smart people on this list say things like " I'm no expert in numerical analysis.. " * nextafter > * a function for counting the number of ulps between two floats > * a function for checking that two floats differ by at most N ulps > > I'm not enough of an expert on numerical analysis to have an opinion on > how useful these would be for Python itself. They certainly are part of a > complete IEEE754 implementation, and useful for exploring the details of > how floats work, if nothing else. > I think there is little question that these are useful for numerical analysis. I think the question is if enough people want to use Python for that kind of analysis to add it to the stdlib. My tendency is to say yes -- if someone wants to write the code, it shouldn't be that hard to maintain -- it's mostly going to be calls to the underlying C lib, yes? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 13:13:38 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 02:13:38 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> Message-ID: <415196cb.c2.15a85ed7ce7.Coremail.mlet_it_bew@126.com> > but what is frozen_tree_set() here, frozen_tree_set is my set class. > frozensets don't have pop or 'ipop' "frozen" means we don't modify the object, but we can use it as prototype to create new object. e.g. in Haskell: val {attr = 1} or in Python: namedtuple._replace frozensets can have pop(), but the hash implement will be O(n). > what is ipop? when we pop element from frozensets, we get a element and a new set. At 2017-02-28 23:19:03, "Ryan Birmingham" wrote: I'm sorry for the confusion, but what is frozen_tree_set() here, and what is ipop? frozensets don't have pop or 'ipop', so my apologies that I'm a bit lost here. -Ryan Birmingham On 28 February 2017 at 08:59, ????? wrote: Hi! I write a '<'-based immutable set class. But it is quit different from the standard set class. I wish collections.abc.Set be more friendly to immutable tree sets or Python add new syntax to unify such difference. good example: a = [] a += a # "a" is the original list a = () a += a # "a" is a new tuple bad example: a = set() a.add(1) # return None; "a" changed e = a.pop(); a = frozen_tree_set() a = a.add(1) # return another set; e, a = a.ipop() # return two objects instead of one! solution I used: a <<= 1 # <==> a = a.add(1) but "a.ipop()" .... my current solution is to write a wrapper class to turn immutable set into mutable set, but it is so noisy to box and unbox. solution that I wish: a :=.add(1) # "=." mimic "+="; return the result object e ~ a :=.pop() d[key] :=.add(1) # in dict if only python add following features: 1) modify the original object 1-0) 1) define: def .method(self):... # "." means "intend to modify self" # return any thing # to avoid immutable method # which was intended to return new object # leave self unchanged 2) invoke: r = a..method(); 1-1) ignore result # no matter what returned, discard it a.-.method(); # <==> [a..method(), None][-1] 1-2) return self # no matter what returned, return self a.>.method().>.method();# <==> [a..method(), a..method(), a][-1] 2) create new object 2-0) 1) define # intend to return (result, new object) def ^method():... 2)invoke: r, a' = a.^method(); 2-1) return other, discard result a.-^method().-^method();# <==> a.^method()[1].^method()[1]; 2-2) assign other to original variable a=^method(); # <==> a = a.^method()[1]; 3) unify both: a :=.method(); # if defined ".method" then "a..method();" # elif defined "^method" then "a = a.^method()[1];" # elif defined "method" then "a.method();" # else error r ~ a :=.method(); # if defined ".method" then "r = a..method();" # elif defined "^method" then "r, a = a.^method();" # elif defined "method" then "r = a.method();" # else error _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 13:13:59 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 02:13:59 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> Message-ID: <79f37868.c3.15a85edd106.Coremail.mlet_it_bew@126.com> > That is simply misspelled for your intent. Take a look at the following function in Haskell (GHC) Data.Set minView :: Set a -> Maybe (a, Set a) We always want the new set after a immutable set pop(). "e = a.pop()" is fine for mutable set, but not for immutable case. we need "e, a = a.pop()" to update "a". ? 2017-03-01 01:23:54?"David Mertz" ??? On Tue, Feb 28, 2017 at 5:59 AM, ????? wrote: bad example: a = set() a.add(1) # return None; "a" changed e = a.pop(); That is simply misspelled for your intent. a = set() a |= {1} e = a.pop() -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Feb 28 13:10:15 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 28 Feb 2017 19:10:15 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> Message-ID: <1efd6852-d481-20ad-a485-2e552200215c@mail.de> On 28.02.2017 18:18, David Mertz wrote: > > Yes, and easily written as above. What significant advantage would it > have to spell the above as: > > x = alist.get(pos, default_val) > > It's a couple characters shorter in the proposed version. I guess > I'll concede that needing the odd indexing at the end to get the > scalar is slightly ugly. 1. advantage: it looks like dict access -> allows duck typing (oh how often I'd missed that) 2. advantage: no try except 3. advantage: no weird workaround with slices and additional item access Thanks for bringing this up, Michel. First point would be most important to me. Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 13:48:18 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 10:48:18 -0800 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: <415196cb.c2.15a85ed7ce7.Coremail.mlet_it_bew@126.com> References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> <415196cb.c2.15a85ed7ce7.Coremail.mlet_it_bew@126.com> Message-ID: So it sounds like what you want is this: In [30]: class FrozenSet(frozenset): ...: def pop(self): ...: item = next(iter(self)) ...: return item, self-{item} ...: In [31]: a = FrozenSet({1,2,3,4,5}) In [32]: a.pop() Out[32]: (1, frozenset({2, 3, 4, 5})) I think I'm +1 on `frozenset` itself growing that method. On Tue, Feb 28, 2017 at 10:13 AM, ????? wrote: > > > but what is frozen_tree_set() here, > frozen_tree_set is my set class. > > > frozensets don't have pop or 'ipop' > "frozen" means we don't modify the object, but we can use it as prototype > to create new object. > e.g. in Haskell: val {attr = 1} or in Python: namedtuple._replace > frozensets can have pop(), but the hash implement will be O(n). > > > what is ipop? > when we pop element from frozensets, we get a element and a new set. > > > > At 2017-02-28 23:19:03, "Ryan Birmingham" wrote: > > I'm sorry for the confusion, but what is frozen_tree_set() here, and what > is ipop? frozensets don't have pop or 'ipop', so my apologies that I'm a > bit lost here. > > -Ryan Birmingham > > On 28 February 2017 at 08:59, ????? wrote: > >> >> Hi! >> I write a '<'-based immutable set class. >> But it is quit different from the standard set class. >> I wish collections.abc.Set be more friendly to immutable tree sets or >> Python add new syntax to unify such difference. >> >> >> good example: >> a = [] >> a += a # "a" is the original list >> >> a = () >> a += a # "a" is a new tuple >> >> bad example: >> a = set() >> a.add(1) # return None; "a" changed >> e = a.pop(); >> >> a = frozen_tree_set() >> a = a.add(1) # return another set; >> e, a = a.ipop() # return two objects instead of one! >> >> solution I used: >> a <<= 1 # <==> a = a.add(1) >> but "a.ipop()" .... >> my current solution is to write a wrapper class >> to turn immutable set into mutable set, >> but it is so noisy to box and unbox. >> >> >> solution that I wish: >> a :=.add(1) # "=." mimic "+="; return the result object >> e ~ a :=.pop() >> >> d[key] :=.add(1) # in dict >> >> >> >> if only python add following features: >> 1) modify the original object >> 1-0) >> 1) define: >> def .method(self):... >> # "." means "intend to modify self" >> # return any thing >> # to avoid immutable method >> # which was intended to return new object >> # leave self unchanged >> >> 2) invoke: >> r = a..method(); >> 1-1) ignore result >> # no matter what returned, discard it >> a.-.method(); # <==> [a..method(), None][-1] >> 1-2) return self >> # no matter what returned, return self >> a.>.method().>.method();# <==> [a..method(), a..method(), a][-1] >> >> >> 2) create new object >> 2-0) >> 1) define >> # intend to return (result, new object) >> def ^method():... >> >> 2)invoke: >> r, a' = a.^method(); >> 2-1) return other, discard result >> a.-^method().-^method();# <==> a.^method()[1].^method()[1]; >> 2-2) assign other to original variable >> a=^method(); # <==> a = a.^method()[1]; >> >> 3) unify both: >> a :=.method(); >> # if defined ".method" then "a..method();" >> # elif defined "^method" then "a = a.^method()[1];" >> # elif defined "method" then "a.method();" >> # else error >> >> r ~ a :=.method(); >> # if defined ".method" then "r = a..method();" >> # elif defined "^method" then "r, a = a.^method();" >> # elif defined "method" then "r = a.method();" >> # else error >> >> >> >> >> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 13:52:47 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 10:52:47 -0800 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <1efd6852-d481-20ad-a485-2e552200215c@mail.de> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <1efd6852-d481-20ad-a485-2e552200215c@mail.de> Message-ID: On Tue, Feb 28, 2017 at 10:10 AM, Sven R. Kunze wrote: > Yes, and easily written as above. What significant advantage would it > have to spell the above as: > > x = alist.get(pos, default_val) > > It's a couple characters shorter in the proposed version. I guess I'll > concede that needing the odd indexing at the end to get the scalar is > slightly ugly. > > 1. advantage: it looks like dict access -> allows duck typing (oh how > often I'd missed that) > 2. advantage: no try except > 3. advantage: no weird workaround with slices and additional item access > How often would you duck-type "access either an integer position or a named key in a collection?" I'm all for duck typing, but it feels like those are a pretty different pattern. For example, I know that if a list has something at index 10 it also has something at index 9. I absolutely *do not* know that if a dict has something at key 'g' it also has something at key 'f'. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Feb 28 14:53:36 2017 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 28 Feb 2017 19:53:36 +0000 Subject: [Python-ideas] Another use case for the 'lazy' (aka 'delayed') keyword In-Reply-To: References: Message-ID: <26578983-4e59-022f-03bb-fdcdf2997203@mrabarnett.plus.com> On 2017-02-28 12:04, Michel Desmoulin wrote: > The debate on the 'lazy' keyword seems to have settled, but I don't know > if somebody is trying to write a PEP about it. > > Anyway, I was doing something like this the other day: > > conf.get('setting_name', load_from_db('setting_name')) > > And then realized I could save a query not doing the load_from_db() call > most of the time. > > But dict.get didn't accept callable so I couldn't do: > > conf.get('setting_name', lambda key: load_from_db('setting_name')) > > Which is standard practice in a lot of popular Python libs on Pypi. > > Instead I did: > > val = conf.get('setting_name') > if val is None: > val = load_from_db('setting_name') > > Which is way more verbose. It also has a bug if None is a valid > configuration value or if I expect my code to be thread safe. > [snip] If None is a valid value, then use a sentinel: MISSING = object() val = conf.get('setting_name', MISSING) if val is MISSING: val = load_from_db('setting_name') From matt at getpattern.com Tue Feb 28 15:05:37 2017 From: matt at getpattern.com (Matt Gilson) Date: Tue, 28 Feb 2017 12:05:37 -0800 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: References: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> Message-ID: I've implemented `frozendict` a few times for various projects and never knew about MappingProxyType. I always used `collections.abc.Mapping` ... On Tue, Feb 28, 2017 at 9:12 AM, David Mertz wrote: > The difference in hits might be because MappingProxyType has a funny name > and is in a hidden-ish location. I.e. not necessarily because it *would > be* less used or useful if it were more exposed. > > It also might be because you can use `frozenset` in python2.x code -- And there's lots of that lying around... > In either case, the name that makes sense to me would be `frozendict`. > That could very well live in `collections` of course. > Yes, I agree. Though it'd also probably need to be hashable if we were to give it that name. I'm not 100% sure that `MappingProxyType` works there as it's just a view into another mapping. If the first mapping changes, so does the hash. This is the same problem we have hashing tuple in some sense -- But tuple can say "Nope, sorry. I can't hash this because it's got an unhashable member". I don't think we can really do the same thing with a MappingProxy since most of the time, it'll be constructed from something else. I suppose the workaround is pretty simple though: class frozendict(MappingProxyType): def __init__(self, proxy): super().__init__(proxy.copy()) # Copy the proxy! -- Maybe need `copy.copy()` instead? def __hash__(self): return hash(frozenset(self.items())) This could likely be done better, but hopefully it gets the idea across... > > On Tue, Feb 28, 2017 at 7:40 AM, Ivan Levkivskyi > wrote: > >> Searching MappingProxyType on GitHub gives over 10,000 results. >> I think it probably makes sense to make mappingproxy more "visible", >> maybe move it to collections module? (where OrderedDict lives) >> >> I am not sure if it makes sense to move it to builtins. (for comparison >> frozenset gives around 1000,000 results) >> >> -- >> Ivan >> >> >> >> On 28 February 2017 at 16:24, Joseph Hackman >> wrote: >> >>> +1 >>> >>> I think this makes a lot of sense. What would you name the built in? >>> >>> -Joseph >>> >>> > On Feb 28, 2017, at 7:17 AM, Michel Desmoulin < >>> desmoulinmichel at gmail.com> wrote: >>> > >>> > We have the immutable frozenset for sets and and tuples for lists. >>> > >>> > But we also have something to manipulate dict as immutable >>> datastructures: >>> > >>> >>>> from types import MappingProxyType as idict >>> >>>> d = idict({'a':1, 'b':2, 'c':3}) >>> >>>> d['a'] = 4 >>> > Traceback (most recent call last): >>> > File "", line 1, in >>> > d['a'] = 4 >>> > TypeError: 'mappingproxy' object does not support item assignment >>> > >>> > We could expose this as a built type to allow the last of the most >>> > important data structure in Python to be easily immutable. >>> > >>> > _______________________________________________ >>> > Python-ideas mailing list >>> > Python-ideas at python.org >>> > https://mail.python.org/mailman/listinfo/python-ideas >>> > Code of Conduct: http://python.org/psf/codeofconduct/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- [image: pattern-sig.png] Matt Gilson // SOFTWARE ENGINEER E: matt at getpattern.com // P: 603.892.7736 We?re looking for beta testers. Go here to sign up! -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Feb 28 16:02:51 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 28 Feb 2017 15:02:51 -0600 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <1efd6852-d481-20ad-a485-2e552200215c@mail.de> Message-ID: I wouldn't see myself using it for an arbitrary value in the middle of the list, but perhaps for the 0th or -1st (first/last) element of a list that might be empty. Maybe also second-to-first/last if I'm initializing some comparison between sequential items? Some of the examples don't work with negative indexes, boo. On Tue, Feb 28, 2017 at 12:52 PM, David Mertz wrote: > On Tue, Feb 28, 2017 at 10:10 AM, Sven R. Kunze wrote: > >> Yes, and easily written as above. What significant advantage would it >> have to spell the above as: >> >> x = alist.get(pos, default_val) >> >> It's a couple characters shorter in the proposed version. I guess I'll >> concede that needing the odd indexing at the end to get the scalar is >> slightly ugly. >> >> 1. advantage: it looks like dict access -> allows duck typing (oh how >> often I'd missed that) >> 2. advantage: no try except >> 3. advantage: no weird workaround with slices and additional item access >> > > How often would you duck-type "access either an integer position or a > named key in a collection?" > > I'm all for duck typing, but it feels like those are a pretty different > pattern. For example, I know that if a list has something at index 10 it > also has something at index 9. I absolutely *do not* know that if a dict > has something at key 'g' it also has something at key 'f'. > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 28 16:17:31 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 22:17:31 +0100 Subject: [Python-ideas] Positional-only parameters Message-ID: Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor From lists at whitehouse.kiwi.nz Tue Feb 28 16:27:08 2017 From: lists at whitehouse.kiwi.nz (Aaron) Date: Tue, 28 Feb 2017 21:27:08 +0000 Subject: [Python-ideas] Optional argument to fnmatch functions treat * and ** differently Message-ID: <8c514f76-515b-073c-3651-04c1dec2ca80@whitehouse.kiwi.nz> Hello, I would like fnmatch.fnmatch to have a mode where: ** matches any files and zero or more directories and subdirectories (i.e. the current behaviour of *) * matches in one path level/only to the next path separator This is essentially achievable in glob.glob from version 3.5 (with recursive=True), but glob is not suitable for my use case, as I need to work with strings and not files. I would like to gauge whether adding such behaviour behind an optional argument similar to glob's "recursive" (e.g. "glob_asterisks=False") is likely to be accepted into the standard library, before I start working on a patch. I have seen people raise the question of how to achieve this in a number of places. Somebody has also forked the Python 2 standard library fnmatch module to implement this as a standalone library: https://github.com/kianxineki/python-wildcard (which shows that the actual changes are likely to be pretty small). This request is also discussed in the following bug: http://bugs.python.org/issue28718 Not directly on point, but some may find the discussion interesting: http://stackoverflow.com/questions/18341848/fnmatch-and-recursive-path-match-with http://stackoverflow.com/questions/27726545/python-glob-but-against-a-list-of-strings-rather-than-the-filesystem I would appreciate any thoughts. Kind regards, Aaron From tjreedy at udel.edu Tue Feb 28 16:49:38 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Feb 2017 16:49:38 -0500 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> <415196cb.c2.15a85ed7ce7.Coremail.mlet_it_bew@126.com> Message-ID: On 2/28/2017 1:48 PM, David Mertz wrote: > So it sounds like what you want is this: > > In [30]: class FrozenSet(frozenset): > ...: def pop(self): > ...: item = next(iter(self)) > ...: return item, self-{item} > ...: > > In [31]: a = FrozenSet({1,2,3,4,5}) > > In [32]: a.pop() > Out[32]: (1, frozenset({2, 3, 4, 5})) > > > I think I'm +1 on `frozenset` itself growing that method. To me, 'pop' implies mutation. Tuples do not have a pop method, and it is not obvious to me that either tuples or frozensets should. What are the use cases that are not better served by converting to list or set? -- Terry Jan Reedy From yselivanov.ml at gmail.com Tue Feb 28 17:03:01 2017 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 28 Feb 2017 17:03:01 -0500 Subject: [Python-ideas] Positional-only parameters In-Reply-To: References: Message-ID: <4dbe3271-e865-12fa-ee7e-a2c3c142756b@gmail.com> I'm +0.5 to add positional-only parameters. Pros: * A lot of people don't know what '/' currently means in functions signatures rendered by `help` and docs. Because it's not a real syntax, it's really hard to find what it means. * Some APIs do look better with positional-only parameters, especially functions with one argument. * We already have this feature half-implemented: some builtin methods have positional-only arguments, inspect.signature API supports it already. Cons: * Function declarations will become a bit more complex, making a bump in Python learning curve. * Performance? I'm not sure if adding another set of checks will make a huge impact, but we'll need to benchmark this. Yury On 2017-02-28 4:17 PM, Victor Stinner wrote: > Hi, > > For technical reasons, many functions of the Python standard libraries > implemented in C have positional-only parameters. Example: > ------- > $ ./python > Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>>> help(str.replace) > replace(self, old, new, count=-1, /) # <== notice "/" at the end > ... >>>> "a".replace("x", "y") # ok > 'a' > >>>> "a".replace(old="x", new="y") # ERR! > TypeError: replace() takes at least 2 arguments (0 given) > ------- > > When converting the methods of the builtin str type to the internal > "Argument Clinic" tool (tool to generate the function signature, > function docstring and the code to parse arguments in C), I asked if > we should add support for keyword arguments in str.replace(). The > answer was quick: no! It's a deliberate design choice. > > Quote of Yury Selivanov's message: > """ > I think Guido explicitly stated that he doesn't like the idea to > always allow keyword arguments for all methods. I.e. `str.find('aaa')` > just reads better than `str.find(needle='aaa')`. Essentially, the idea > is that for most of the builtins that accept one or two arguments, > positional-only parameters are better. > """ > http://bugs.python.org/issue29286#msg285578 > > I just noticed a module on PyPI to implement this behaviour on Python functions: > > https://pypi.python.org/pypi/positional > > My question is: would it make sense to implement this feature in > Python directly? If yes, what should be the syntax? Use "/" marker? > Use the @positional() decorator? > > Do you see concrete cases where it's a deliberate choice to deny > passing arguments as keywords? > > Don't you like writing int(x="123") instead of int("123")? :-) (I know > that Serhiy Storshake hates the name of the "x" parameter of the int > constructor ;-)) > > By the way, I read that "/" marker is unknown by almost all Python > developers, and [...] syntax should be preferred, but > inspect.signature() doesn't support this syntax. Maybe we should fix > signature() and use [...] format instead? > > Replace "replace(self, old, new, count=-1, /)" with "replace(self, > old, new[, count=-1])" (or maybe even not document the default > value?). > > Python 3.5 help (docstring) uses "S.replace(old, new[, count])". > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From frazer at frazermclean.co.uk Tue Feb 28 17:07:27 2017 From: frazer at frazermclean.co.uk (Frazer McLean) Date: Tue, 28 Feb 2017 23:07:27 +0100 Subject: [Python-ideas] Positional-only parameters In-Reply-To: References: Message-ID: <8B59BB5C-18FE-4C09-9CBE-ED6C79DD9A86@frazermclean.co.uk> On 28 Feb 2017, at 22:17, Victor Stinner wrote: > I just noticed a module on PyPI to implement this behaviour on Python > functions: > > https://pypi.python.org/pypi/positional Tangential to the main topic, but this module doesn?t enforce positional-only arguments. It allows you enforce keyword-only arguments like on Python 3: >>> from positional import positional >>> @positional(1) ... def replace(old, new): ... ... ... >>> replace(old='a', new='b') >>> replace('a', 'b') Traceback (most recent call last): File "", line 1, in File ??/site-packages/positional/__init__.py", line 97, in inner raise TypeError(message) TypeError: replace takes at most 1 positional argument (2 given) Frazer From bussonniermatthias at gmail.com Tue Feb 28 17:08:53 2017 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Tue, 28 Feb 2017 14:08:53 -0800 Subject: [Python-ideas] Positional-only parameters In-Reply-To: <4dbe3271-e865-12fa-ee7e-a2c3c142756b@gmail.com> References: <4dbe3271-e865-12fa-ee7e-a2c3c142756b@gmail.com> Message-ID: I think that was started as pep 457: https://www.python.org/dev/peps/pep-0457/ (Syntax For Positional-Only Parameters) Still informal. +1, it's likely possible to backport it to previous version using a decorator and faking __signature__. -- M On Tue, Feb 28, 2017 at 2:03 PM, Yury Selivanov wrote: > I'm +0.5 to add positional-only parameters. > > Pros: > > * A lot of people don't know what '/' currently means in > functions signatures rendered by `help` and docs. Because > it's not a real syntax, it's really hard to find what it > means. > > * Some APIs do look better with positional-only parameters, > especially functions with one argument. > > * We already have this feature half-implemented: some > builtin methods have positional-only arguments, > inspect.signature API supports it already. > > Cons: > > * Function declarations will become a bit more complex, > making a bump in Python learning curve. > > * Performance? I'm not sure if adding another set of checks > will make a huge impact, but we'll need to benchmark this. > > Yury > > > > On 2017-02-28 4:17 PM, Victor Stinner wrote: >> >> Hi, >> >> For technical reasons, many functions of the Python standard libraries >> implemented in C have positional-only parameters. Example: >> ------- >> $ ./python >> Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>>>> >>>>> help(str.replace) >> >> replace(self, old, new, count=-1, /) # <== notice "/" at the end >> ... >>>>> >>>>> "a".replace("x", "y") # ok >> >> 'a' >> >>>>> "a".replace(old="x", new="y") # ERR! >> >> TypeError: replace() takes at least 2 arguments (0 given) >> ------- >> >> When converting the methods of the builtin str type to the internal >> "Argument Clinic" tool (tool to generate the function signature, >> function docstring and the code to parse arguments in C), I asked if >> we should add support for keyword arguments in str.replace(). The >> answer was quick: no! It's a deliberate design choice. >> >> Quote of Yury Selivanov's message: >> """ >> I think Guido explicitly stated that he doesn't like the idea to >> always allow keyword arguments for all methods. I.e. `str.find('aaa')` >> just reads better than `str.find(needle='aaa')`. Essentially, the idea >> is that for most of the builtins that accept one or two arguments, >> positional-only parameters are better. >> """ >> http://bugs.python.org/issue29286#msg285578 >> >> I just noticed a module on PyPI to implement this behaviour on Python >> functions: >> >> https://pypi.python.org/pypi/positional >> >> My question is: would it make sense to implement this feature in >> Python directly? If yes, what should be the syntax? Use "/" marker? >> Use the @positional() decorator? >> >> Do you see concrete cases where it's a deliberate choice to deny >> passing arguments as keywords? >> >> Don't you like writing int(x="123") instead of int("123")? :-) (I know >> that Serhiy Storshake hates the name of the "x" parameter of the int >> constructor ;-)) >> >> By the way, I read that "/" marker is unknown by almost all Python >> developers, and [...] syntax should be preferred, but >> inspect.signature() doesn't support this syntax. Maybe we should fix >> signature() and use [...] format instead? >> >> Replace "replace(self, old, new, count=-1, /)" with "replace(self, >> old, new[, count=-1])" (or maybe even not document the default >> value?). >> >> Python 3.5 help (docstring) uses "S.replace(old, new[, count])". >> >> Victor >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Tue Feb 28 17:11:47 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Feb 2017 14:11:47 -0800 Subject: [Python-ideas] Positional-only parameters In-Reply-To: References: Message-ID: <58B5F5A3.3080801@stoneleaf.us> https://www.python.org/dev/peps/pep-0457/ https://mail.python.org/pipermail/python-dev/2014-January/131865.html -- ~Ethan~ From mal at egenix.com Tue Feb 28 17:17:50 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Feb 2017 23:17:50 +0100 Subject: [Python-ideas] Positional-only parameters In-Reply-To: References: Message-ID: <91626164-0290-d575-c049-43ff846314d4@egenix.com> On 28.02.2017 22:17, Victor Stinner wrote: > Hi, > > For technical reasons, many functions of the Python standard libraries > implemented in C have positional-only parameters. Keyword argument handling is comparatively slow and rarely needed for non-optional positional parameters of built-ins. You might make a case for optional ones, but then: how often are these used in practice to warrant the drop in performance ? Note: All this is different for Python methods/functions. The overhead of keyword parsing is small compared to what Python has to do to setup new frames for execution. > Example: > ------- > $ ./python > Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>>> help(str.replace) > replace(self, old, new, count=-1, /) # <== notice "/" at the end > ... >>>> "a".replace("x", "y") # ok > 'a' > >>>> "a".replace(old="x", new="y") # ERR! > TypeError: replace() takes at least 2 arguments (0 given) > ------- > > When converting the methods of the builtin str type to the internal > "Argument Clinic" tool (tool to generate the function signature, > function docstring and the code to parse arguments in C), I asked if > we should add support for keyword arguments in str.replace(). The > answer was quick: no! It's a deliberate design choice. > > Quote of Yury Selivanov's message: > """ > I think Guido explicitly stated that he doesn't like the idea to > always allow keyword arguments for all methods. I.e. `str.find('aaa')` > just reads better than `str.find(needle='aaa')`. Essentially, the idea > is that for most of the builtins that accept one or two arguments, > positional-only parameters are better. > """ > http://bugs.python.org/issue29286#msg285578 > > I just noticed a module on PyPI to implement this behaviour on Python functions: > > https://pypi.python.org/pypi/positional > > My question is: would it make sense to implement this feature in > Python directly? If yes, what should be the syntax? Use "/" marker? > Use the @positional() decorator? > > Do you see concrete cases where it's a deliberate choice to deny > passing arguments as keywords? > > Don't you like writing int(x="123") instead of int("123")? :-) (I know > that Serhiy Storshake hates the name of the "x" parameter of the int > constructor ;-)) ... and that's another reason why to avoid them: the naming of positional parameters was never really taken into account when writing the C functions and it does even change sometimes without warning. > By the way, I read that "/" marker is unknown by almost all Python > developers, and [...] syntax should be preferred, but > inspect.signature() doesn't support this syntax. Maybe we should fix > signature() and use [...] format instead? +1 > Replace "replace(self, old, new, count=-1, /)" with "replace(self, > old, new[, count=-1])" (or maybe even not document the default > value?). > > Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Using "count=-1" looks confusing when the method doesn't allow keyword arguments, so it's probably better to use the 3.5 style and document the defaults in the doc-string. AFAIR, some of these don't even have a (documented) default value. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 28 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From victor.stinner at gmail.com Tue Feb 28 17:19:39 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 23:19:39 +0100 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: References: Message-ID: 2017-02-28 13:17 GMT+01:00 Michel Desmoulin : > We have the immutable frozenset for sets and and tuples for lists. > > But we also have something to manipulate dict as immutable datastructures: > >>>> from types import MappingProxyType as idict >>>> d = idict({'a':1, 'b':2, 'c':3}) >>>> d['a'] = 4 > Traceback (most recent call last): > File "", line 1, in > d['a'] = 4 > TypeError: 'mappingproxy' object does not support item assignment > > We could expose this as a built type to allow the last of the most > important data structure in Python to be easily immutable. Hello, Such builtin type exists! It's called types.MappingProxyType! ... Sorry, I don't understand your proposition. Do you want to not have to write "from types import MappingProxyType" and put a new type in __builtins__? If yes, I don't think that it's worth it, it's not that hard to add an import to each file using such type. By the way, try to not abuse this type please ;-) (Don't try to put it everywhere?) Victor From bussonniermatthias at gmail.com Tue Feb 28 17:31:30 2017 From: bussonniermatthias at gmail.com (Matthias Bussonnier) Date: Tue, 28 Feb 2017 14:31:30 -0800 Subject: [Python-ideas] Positional-only parameters In-Reply-To: <91626164-0290-d575-c049-43ff846314d4@egenix.com> References: <91626164-0290-d575-c049-43ff846314d4@egenix.com> Message-ID: > I just noticed a module on PyPI to implement this behaviour on Python functions: > https://pypi.python.org/pypi/positional This library seem to do the opposite of what you ask and force kwarg only. In [11]: @positional.positional(1) ...: def foo(a=1,b=2): ...: print(a,b) ...: In [12]: foo(a=1,b=2) # should have raised. 1 2 In [13]: foo(1,b=2) 1 2 This seem to be the right thing: https://pypi.python.org/pypi/positionalonly In [1]: @positionalonly(1) ...: def foo(a=1,b=2): ...: print(a,b) In [2]: foo(1, 2) 1 2 In [3]: foo(a=1,b=2) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () ----> 1 foo(a=1,b=2) /Users/bussonniermatthias/anaconda/lib/python3.5/site-packages/positionalonly/__init__.py in fun(*args, **kwargs) 208 ]) 209 raise TypeError("The following parameters of `{}` are positional only.\n" --> 210 "They were used as keywords arguments:\n{}".format(function.__qualname__, lst)) 211 if insertn and len(args) >= insertn: 212 args = args[:insertn] + (None,) + args[insertn:] TypeError: The following parameters of `foo` are positional only. They were used as keywords arguments: - 'a' (1) should be in 0th position -- M On Tue, Feb 28, 2017 at 2:17 PM, M.-A. Lemburg wrote: > On 28.02.2017 22:17, Victor Stinner wrote: >> Hi, >> >> For technical reasons, many functions of the Python standard libraries >> implemented in C have positional-only parameters. > > Keyword argument handling is comparatively slow and rarely > needed for non-optional positional parameters of built-ins. > > You might make a case for optional ones, but then: how often are > these used in practice to warrant the drop in performance ? > > Note: All this is different for Python methods/functions. The > overhead of keyword parsing is small compared to what Python > has to do to setup new frames for execution. > >> Example: >> ------- >> $ ./python >> Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>>>> help(str.replace) >> replace(self, old, new, count=-1, /) # <== notice "/" at the end >> ... >>>>> "a".replace("x", "y") # ok >> 'a' >> >>>>> "a".replace(old="x", new="y") # ERR! >> TypeError: replace() takes at least 2 arguments (0 given) >> ------- >> >> When converting the methods of the builtin str type to the internal >> "Argument Clinic" tool (tool to generate the function signature, >> function docstring and the code to parse arguments in C), I asked if >> we should add support for keyword arguments in str.replace(). The >> answer was quick: no! It's a deliberate design choice. >> >> Quote of Yury Selivanov's message: >> """ >> I think Guido explicitly stated that he doesn't like the idea to >> always allow keyword arguments for all methods. I.e. `str.find('aaa')` >> just reads better than `str.find(needle='aaa')`. Essentially, the idea >> is that for most of the builtins that accept one or two arguments, >> positional-only parameters are better. >> """ >> http://bugs.python.org/issue29286#msg285578 >> >> I just noticed a module on PyPI to implement this behaviour on Python functions: >> >> https://pypi.python.org/pypi/positional >> >> My question is: would it make sense to implement this feature in >> Python directly? If yes, what should be the syntax? Use "/" marker? >> Use the @positional() decorator? >> >> Do you see concrete cases where it's a deliberate choice to deny >> passing arguments as keywords? >> >> Don't you like writing int(x="123") instead of int("123")? :-) (I know >> that Serhiy Storshake hates the name of the "x" parameter of the int >> constructor ;-)) > > ... and that's another reason why to avoid them: the naming > of positional parameters was never really taken into account > when writing the C functions and it does even change > sometimes without warning. > >> By the way, I read that "/" marker is unknown by almost all Python >> developers, and [...] syntax should be preferred, but >> inspect.signature() doesn't support this syntax. Maybe we should fix >> signature() and use [...] format instead? > > +1 > >> Replace "replace(self, old, new, count=-1, /)" with "replace(self, >> old, new[, count=-1])" (or maybe even not document the default >> value?). >> >> Python 3.5 help (docstring) uses "S.replace(old, new[, count])". > > Using "count=-1" looks confusing when the method doesn't > allow keyword arguments, so it's probably better to use the 3.5 > style and document the defaults in the doc-string. > > AFAIR, some of these don't even have a (documented) default value. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Feb 28 2017) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> Python Database Interfaces ... http://products.egenix.com/ >>>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mlet_it_bew at 126.com Tue Feb 28 17:57:15 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 06:57:15 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects Message-ID: <38e2b548.256.15a86f126b6.Coremail.mlet_it_bew@126.com> > To me, 'pop' implies mutation. Tuples do not have a pop method, and it > is not obvious to me that either tuples or frozensets should. What are > the use cases that are not better served by converting to list or set? > -- > Terry Jan Reedy 1) coverting to set or list is O(n) in time 2) if I have to keep the old copy, standard set solution will be O(n) both in time and space! working examples: 1) priority queue: insert and pop occur 2) share immutable data to difference subroutines: each one can modify local copy safely and concurrency. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 17:59:52 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 06:59:52 +0800 (CST) Subject: [Python-ideas] __repr__: to support pprint Message-ID: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> Hi, everyone! Oftenly, __repr__ return "{type}({args}, {kws})" problem: 1) to call .format() myself is tedious I have seen someone do it wrong: "T(1,)"! I write a helper function myself, but to import such tiny function everywhere isnot good. 2) pprint cannot dive into string that from repr() To use pprint, I sometimes have to recursively turn objects into builtin containers: (type_name, args...) solution: allow __repr__ to return str OR tuple: (args, kws) Better, isn't it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 18:02:23 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 07:02:23 +0800 (CST) Subject: [Python-ideas] add __contains__ into the "type" object Message-ID: <1a6bb123.269.15a86f5d855.Coremail.mlet_it_bew@126.com> where we use types? almost: isinstance(obj, T); # issubclass(S, T); Note that TYPE is SET; if we add __contains__ and __le__ into "type", then things become: obj in T; # S <= T; # if only not misleading to a total ordering example: def __getitems__(self, i): if i in Integral: ... elif i in slice: ... # Save "(,)". Really, I prefer to type "lambda:;" than "()". # I fail to modify the "type" object, since it is a C-object. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 18:14:33 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 15:14:33 -0800 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: <38e2b548.256.15a86f126b6.Coremail.mlet_it_bew@126.com> References: <38e2b548.256.15a86f126b6.Coremail.mlet_it_bew@126.com> Message-ID: On Tue, Feb 28, 2017 at 2:57 PM, ????? wrote: > 1) coverting to set or list is O(n) in time > A hypothetical frozenset.pop() is also necessarily O(N). It needs to copy N-1 elements into the new (smaller) frozenset object. So this isn't an argument. > 2) if I have to keep the old copy, > standard set solution will be O(n) both in time and space! > Again, nothing gained here. Same applies to frozenset.pop() (assuming it returns both the item popped and a reduced set). If you just want "something from frozenset" without creating a new almost-copy frozenset, that is spelled `next(iter(fs))`. > working examples: > 1) priority queue: > insert and pop occur > 2) share immutable data to difference subroutines: > each one can modify local copy safely and concurrency. > Sounds like `collections.deque` might be what you want (for concurrency, not for immutability). But a local copy will require, by definition, a *copy* operation either way. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 28 18:12:50 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 10:12:50 +1100 Subject: [Python-ideas] add __contains__ into the "type" object In-Reply-To: <1a6bb123.269.15a86f5d855.Coremail.mlet_it_bew@126.com> References: <1a6bb123.269.15a86f5d855.Coremail.mlet_it_bew@126.com> Message-ID: <20170228231250.GN5689@ando.pearwood.info> On Wed, Mar 01, 2017 at 07:02:23AM +0800, ????? wrote: > > where we use types? > almost: > isinstance(obj, T); > # issubclass(S, T); > > Note that TYPE is SET; What does that mean? I don't understand. > if we add __contains__ and __le__ into "type", > then things become: > obj in T; But obj is **not** in T, since T is a type, not a container. "is-a" tests are not the same as "in" tests. They are completely unrelated comparisons. http://www.w3resource.com/java-tutorial/inheritance-composition-relationship.php The Wikipedia page on is-a is terribly complicated, but folks may get something from it: https://en.wikipedia.org/wiki/Is-a -- Steve From contact at nicolas-cellier.net Tue Feb 28 18:31:19 2017 From: contact at nicolas-cellier.net (Nicolas Cellier) Date: Wed, 1 Mar 2017 00:31:19 +0100 Subject: [Python-ideas] lazy use for optional import Message-ID: I have seen some interest into lazy functionality implementation. I wondered if it can be linked with optional import. PEP 8 authoritatively states: Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants. So, if we want to stick to PEP8 with non mandatory import, we have to catch the import errors, or jail the class or function using extra functionnality. Why not using the potential lazy keyword to have a nice way to deal with it? For example: lazy import pylab as pl # do nothing for now > > # do stuff > > def plot(*args): > pl.figure() # Will raise an ImportError at this point > pl.plot(...) > That way, our library will raise an ImportError only on plot func usage with an explicit traceback : if matplotlib is not installed, we will have the line where it is used for the first time and we will have the name of the faulty library. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jelle.zijlstra at gmail.com Tue Feb 28 18:35:31 2017 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Tue, 28 Feb 2017 15:35:31 -0800 Subject: [Python-ideas] add __contains__ into the "type" object In-Reply-To: <20170228231250.GN5689@ando.pearwood.info> References: <1a6bb123.269.15a86f5d855.Coremail.mlet_it_bew@126.com> <20170228231250.GN5689@ando.pearwood.info> Message-ID: 2017-02-28 15:12 GMT-08:00 Steven D'Aprano : > On Wed, Mar 01, 2017 at 07:02:23AM +0800, ????? wrote: >> >> where we use types? >> almost: >> isinstance(obj, T); >> # issubclass(S, T); >> >> Note that TYPE is SET; > > What does that mean? I don't understand. > > >> if we add __contains__ and __le__ into "type", >> then things become: >> obj in T; > > But obj is **not** in T, since T is a type, not a container. > But in type theory, types are sets in some sense. For example, the bool type is the set {True, False}, and the int type is the infinite set {..., -1, 0, 1, ...}. Similarly, typing.py has a Union type: Union[A, B] is the union of the types A and B. Subclasses are subsets of their parent classes, because their set of possible values is a subset of the possible values of their parent class. The OP seems to be proposing that we reflect this identity between types and sets in Python by spelling "isinstance(obj, T)" as "obj in T" and "issubclass(S, T)" as "S <= T". This proposal has some solid theory behind it and I don't think it would be hard to implement, but it doesn't seem like a particularly useful change to me. It wouldn't really enable anything we can't do now, and it may be confusing to people reading code that "obj in list" does something completely different from "obj in list()". > "is-a" tests are not the same as "in" tests. They are completely > unrelated comparisons. > > http://www.w3resource.com/java-tutorial/inheritance-composition-relationship.php > > The Wikipedia page on is-a is terribly complicated, but folks may get > something from it: > > https://en.wikipedia.org/wiki/Is-a > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mlet_it_bew at 126.com Tue Feb 28 18:47:35 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 07:47:35 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: References: <38e2b548.256.15a86f126b6.Coremail.mlet_it_bew@126.com> Message-ID: <262c49f5.3a0.15a871f3b36.Coremail.mlet_it_bew@126.com> > A hypothetical frozenset.pop() is also necessarily O(N). It needs to copy N-1 elements into the new (smaller) frozenset object. So this isn't an argument. Pop tuple/frozenset(standard one) gain no benefit. # O(n) It is a different story for balanced tree. # O(log n) > Sounds like `collections.deque` might be what you want (for concurrency, not for immutability). But a local copy will require, by definition, a *copy* operation either way. My intent is to unify "SET" interface, not for to using deque. I want something that is SET can use anywhere regardless mutable or not. And the idiom SHOULD work for other type. WHY set.add / list.sort return None? if return self, someone may think it don't modify the orignal object. so, mutable object will have different methods. Such differences are good UNLESS we want to ignore it explictly. We need a uniform way to make a interface suitable for both cases. At 2017-03-01 07:14:33, "David Mertz" wrote: On Tue, Feb 28, 2017 at 2:57 PM, ????? wrote: 1) coverting to set or list is O(n) in time A hypothetical frozenset.pop() is also necessarily O(N). It needs to copy N-1 elements into the new (smaller) frozenset object. So this isn't an argument. 2) if I have to keep the old copy, standard set solution will be O(n) both in time and space! Again, nothing gained here. Same applies to frozenset.pop() (assuming it returns both the item popped and a reduced set). If you just want "something from frozenset" without creating a new almost-copy frozenset, that is spelled `next(iter(fs))`. working examples: 1) priority queue: insert and pop occur 2) share immutable data to difference subroutines: each one can modify local copy safely and concurrency. Sounds like `collections.deque` might be what you want (for concurrency, not for immutability). But a local copy will require, by definition, a *copy* operation either way. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 28 18:56:16 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 10:56:16 +1100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> Message-ID: <20170228235616.GP5689@ando.pearwood.info> On Tue, Feb 28, 2017 at 04:16:28PM +0100, Michel Desmoulin wrote: > > > Le 28/02/2017 ? 15:45, Steven D'Aprano a ?crit : > > On Tue, Feb 28, 2017 at 12:54:26PM +0100, Michel Desmoulin wrote: > >> dict.get() is a very useful method, but for lists and tuples, we need to > >> rely on try/except instead. > > > > No you don't. You can use slicing. > > > > alist = [1, 2, 3] > > print(alist[99:100]) # get the item at position 99 > > No this gives you a list of one item or an empty list. I am aware of that. I'm just saying you don't have to use try...except, you can use slicing instead. Obviously you have to adapt the code since you are getting a list not a single item: # may fail, if alist is empty if alist[0] == "spam": ... # cannot fail if alist[0:1] == ["spam"]: ... This is *especially* useful for strings since a slice of a string is a string, and an item from a string is still a string. > dict.get('key', default_value) let you get a SCALAR value, OR a default > value if it doesn't exist. > > It's a very different use case. For a default, use "or": first_item = (alist[0:1] or ["ham"])[0] But honestly, in my experience the number of times I actually needed something like that is tiny. > > In my experience, dict.get is very useful, but list.get only *seems* > > useful. I've written my own version: > > > > def get(alist, pos, default=None): > > try: > > return alist[pos] > > except IndexError: > > return default > > > > > > Based on your rational, we would just reject dict.get as well the first > time it's implemented. I just said that dict.get is proven to be useful. There's no doubt that it is useful. > > but then struggled to find a good use for it. It seems like it ought to > > be useful, but in practice I found that it was only covering up bugs in > > my code. > > How so ? "get the element x or a default value if it doesn't exist" seem > at the contrary, a very robust approach. A very robust approach **for what** ? What are you trying to do? Why are you indexing into arbitrary positions of a list without knowing whether or not there is something there? List are not dicts and they are used differently. It is very common, and useful, to want to look up a key in a dict without knowing if it exists, and do something with a default if it doesn't exist. This is so useful that Python gives us *at least* four different ways to avoid using a try...except: - dict.get - dict.setdefault - dict subclasses with __missing__ defined - collections.defaultdict all of which solve slightly different use-cases. But *in my experience* it is rare to need to look up some arbitrary item in a list that may not even exist, and if I find myself doing so, it probably means my code is badly designed and I'm going to have trouble later on. For the tiny number of exceptions, I can use the existing solutions: either try...except, or slicing as above. If your experience is different from mine, please explain your use-case. > Plus it's consistent. It's only fair to expect it to exists after you > learn about dict.get. As quoted in PEP 8, "A foolish consistency is the bugbear of little minds." Lists and dicts aren't the same thing and don't offer the same interface. There's no dict.sort or list.update, dicts don't support slicing, concatenation or repetition, there's no list.popitem. Dicts have keys and values, lists have items or elements. list.get has to prove its usefulness on its own, not just because dict.get is useful. > First places where I missed it at the top of my head was *args, sys.argv > personnaly. Missed it for what? What are you trying to do? Don't assume that it is so obvious that everyone will instantly guess your use-case. -- Steve From steve at pearwood.info Tue Feb 28 19:02:54 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 11:02:54 +1100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <1efd6852-d481-20ad-a485-2e552200215c@mail.de> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <1efd6852-d481-20ad-a485-2e552200215c@mail.de> Message-ID: <20170301000254.GQ5689@ando.pearwood.info> On Tue, Feb 28, 2017 at 07:10:15PM +0100, Sven R. Kunze wrote: > 1. advantage: it looks like dict access -> allows duck typing (oh how > often I'd missed that) Dicts and lists don't duck-type because they are very different things. We know what ["item"]*10 does. What would {"key": "value"}*10 do? What would list.values() iterate over? -- Steve From mlet_it_bew at 126.com Tue Feb 28 19:04:40 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 08:04:40 +0800 (CST) Subject: [Python-ideas] add __contains__ into the "type" object Message-ID: <3f0a1ab9.491.15a872edeb8.Coremail.mlet_it_bew@126.com> TYPE is not a collection! But conceptually it is a SET, set of all its possible instances. quot: "Types and Programming Languages (2002)(Benjamin C. Pierce)" [page 92] Chapter 8 Typed Arithmetic Expressions "a term t has type T" (or "t belongs to T," or "t is an element of T") An object obj has a type T or obj is a type T <==> obj belongs to T So, "obj in T" is fine. e.g. "1 in int" means 1 is in the whole number set. > > Note that TYPE is SET; > What does that mean? I don't understand. > > obj in T; > But obj is **not** in T, since T is a type, not a container. > "is-a" tests are not the same as "in" tests. They are completely > unrelated comparisons. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 19:12:28 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 08:12:28 +0800 (CST) Subject: [Python-ideas] add variable "__this_func__" inside all functions' locals Message-ID: <483eaf6.553.15a873604e1.Coremail.mlet_it_bew@126.com> How a function refer itself? def f(): f() # fine... really??? consider these situations: 1) decorator @xxx def f():... # how can I access the wrapped version? 2) replace by other def f(): f() # call second!! a = T(f) # to call first def f():... 3) refactor: rename function # first version def f():f() ------------------- # second version, wrap it def f(): log(); try: # to avoid indent _f_impl() except... def _f_impl(): _f_impl() # hadnot we rename it! solution: # add __this_func__ class C: @xxxx def f(self): self.f() __class__.f(self) __this_func__(self) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 28 19:13:39 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 11:13:39 +1100 Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> Message-ID: <20170301001338.GR5689@ando.pearwood.info> On Mon, Feb 27, 2017 at 11:07:33AM +0800, qhlonline wrote: > Hi, all > I have a suggestion that, the sort() member method of the list > instance, should return the 'self' as the result of list.sort() > call. Having list.sort() and list.reverse() return self is a perfectly good design. The advantage is you can write things like this: list.sort().reverse() but the disadvantage is that it may fool people into thinking it returns a *copy* of the list. Python avoids that trap by returning None, so that you cannot write: sorted_items = items.sort() but instead people write: items = items.sort() so it seems that whatever we do, it will confuse some people. > Now list.sort() returns nothing, so that I can NOT write > code like this: > > res = {item: func(item) for item in item_list.sort()} What is the purpose of the sort? Because dicts are unordered, the results will be no different if you just write: d = {item: func(item) for item in item_list} -- Steve From fakedme+py at gmail.com Tue Feb 28 19:30:37 2017 From: fakedme+py at gmail.com (Soni L.) Date: Tue, 28 Feb 2017 21:30:37 -0300 Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: <20170301001338.GR5689@ando.pearwood.info> References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> <20170301001338.GR5689@ando.pearwood.info> Message-ID: On 28/02/17 09:13 PM, Steven D'Aprano wrote: > On Mon, Feb 27, 2017 at 11:07:33AM +0800, qhlonline wrote: >> Hi, all >> I have a suggestion that, the sort() member method of the list >> instance, should return the 'self' as the result of list.sort() >> call. > Having list.sort() and list.reverse() return self is a perfectly good > design. The advantage is you can write things like this: > > list.sort().reverse() > > but the disadvantage is that it may fool people into thinking it returns > a *copy* of the list. Python avoids that trap by returning None, so that > you cannot write: > > sorted_items = items.sort() > > but instead people write: > > items = items.sort() > > so it seems that whatever we do, it will confuse some people. > > >> Now list.sort() returns nothing, so that I can NOT write >> code like this: >> >> res = {item: func(item) for item in item_list.sort()} > What is the purpose of the sort? Because dicts are unordered, the > results will be no different if you just write: > > d = {item: func(item) for item in item_list} > > Stateful functions? From eric at trueblade.com Tue Feb 28 18:47:06 2017 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 28 Feb 2017 18:47:06 -0500 Subject: [Python-ideas] add __contains__ into the "type" object In-Reply-To: References: <1a6bb123.269.15a86f5d855.Coremail.mlet_it_bew@126.com> <20170228231250.GN5689@ando.pearwood.info> Message-ID: <5876dd6a-5688-d7c3-95d9-cb43c0b03852@trueblade.com> On 2/28/2017 6:35 PM, Jelle Zijlstra wrote: > 2017-02-28 15:12 GMT-08:00 Steven D'Aprano : >> On Wed, Mar 01, 2017 at 07:02:23AM +0800, ????? wrote: >>> >>> where we use types? >>> almost: >>> isinstance(obj, T); >>> # issubclass(S, T); >>> >>> Note that TYPE is SET; >> >> What does that mean? I don't understand. >> >> >>> if we add __contains__ and __le__ into "type", >>> then things become: >>> obj in T; >> >> But obj is **not** in T, since T is a type, not a container. >> > > But in type theory, types are sets in some sense. For example, the > bool type is the set {True, False}, and the int type is the infinite > set {..., -1, 0, 1, ...}. Similarly, typing.py has a Union type: > Union[A, B] is the union of the types A and B. Subclasses are subsets > of their parent classes, because their set of possible values is a > subset of the possible values of their parent class. > > The OP seems to be proposing that we reflect this identity between > types and sets in Python by spelling "isinstance(obj, T)" as "obj in > T" and "issubclass(S, T)" as "S <= T". This proposal has some solid > theory behind it and I don't think it would be hard to implement, but > it doesn't seem like a particularly useful change to me. It wouldn't > really enable anything we can't do now, and it may be confusing to > people reading code that "obj in list" does something completely > different from "obj in list()". Also, you go from the easy to google "python issubclass" to ... ? Syntax is hard to search for. Eric. From lucas.bourneuf at laposte.net Tue Feb 28 19:16:21 2017 From: lucas.bourneuf at laposte.net (lucas) Date: Wed, 1 Mar 2017 01:16:21 +0100 Subject: [Python-ideas] Expose a child factory using MappingProxyType in builtins In-Reply-To: References: <58015C0D-ED48-42A4-BD94-5BDEB8E98C91@gmail.com> Message-ID: I would be very happy to see a frozendict in collections :) Just for curiosity ; apart caching, is there any optimization that can be done on a frozendict implementation (over dict) ? I wonder if frozendict would be implemented as dict without modification methods, or as a particular object that by design does not easily allow modifications. On 28/02/2017 21:05, Matt Gilson wrote: > I've implemented `frozendict` a few times for various projects and never > knew about MappingProxyType. I always used `collections.abc.Mapping` ... > > > On Tue, Feb 28, 2017 at 9:12 AM, David Mertz wrote: > >> The difference in hits might be because MappingProxyType has a funny name >> and is in a hidden-ish location. I.e. not necessarily because it *would >> be* less used or useful if it were more exposed. >> >> > It also might be because you can use `frozenset` in python2.x code -- And > there's lots of that lying around... > > >> In either case, the name that makes sense to me would be `frozendict`. >> That could very well live in `collections` of course. >> > > Yes, I agree. Though it'd also probably need to be hashable if we were to > give it that name. I'm not 100% sure that `MappingProxyType` works there > as it's just a view into another mapping. If the first mapping changes, so > does the hash. This is the same problem we have hashing tuple in some > sense -- But tuple can say "Nope, sorry. I can't hash this because it's > got an unhashable member". I don't think we can really do the same thing > with a MappingProxy since most of the time, it'll be constructed from > something else. I suppose the workaround is pretty simple though: > > class frozendict(MappingProxyType): > def __init__(self, proxy): > super().__init__(proxy.copy()) # Copy the proxy! -- Maybe need > `copy.copy()` instead? > def __hash__(self): > return hash(frozenset(self.items())) > > This could likely be done better, but hopefully it gets the idea across... > > >> >> On Tue, Feb 28, 2017 at 7:40 AM, Ivan Levkivskyi >> wrote: >> >>> Searching MappingProxyType on GitHub gives over 10,000 results. >>> I think it probably makes sense to make mappingproxy more "visible", >>> maybe move it to collections module? (where OrderedDict lives) >>> >>> I am not sure if it makes sense to move it to builtins. (for comparison >>> frozenset gives around 1000,000 results) >>> >>> -- >>> Ivan >>> >>> >>> >>> On 28 February 2017 at 16:24, Joseph Hackman >>> wrote: >>> >>>> +1 >>>> >>>> I think this makes a lot of sense. What would you name the built in? >>>> >>>> -Joseph >>>> >>>>> On Feb 28, 2017, at 7:17 AM, Michel Desmoulin < >>>> desmoulinmichel at gmail.com> wrote: >>>>> >>>>> We have the immutable frozenset for sets and and tuples for lists. >>>>> >>>>> But we also have something to manipulate dict as immutable >>>> datastructures: >>>>> >>>>>>>> from types import MappingProxyType as idict >>>>>>>> d = idict({'a':1, 'b':2, 'c':3}) >>>>>>>> d['a'] = 4 >>>>> Traceback (most recent call last): >>>>> File "", line 1, in >>>>> d['a'] = 4 >>>>> TypeError: 'mappingproxy' object does not support item assignment >>>>> >>>>> We could expose this as a built type to allow the last of the most >>>>> important data structure in Python to be easily immutable. >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. Intellectual property is >> to the 21st century what the slave trade was to the 16th. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Tue Feb 28 19:50:35 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 11:50:35 +1100 Subject: [Python-ideas] __repr__: to support pprint In-Reply-To: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> References: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> Message-ID: <20170301005035.GS5689@ando.pearwood.info> On Wed, Mar 01, 2017 at 06:59:52AM +0800, ????? wrote: > solution: > allow __repr__ to return > str > OR tuple: (args, kws) > > Better, isn't it? No. -- Steve From rob.cliffe at btinternet.com Tue Feb 28 19:22:38 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 1 Mar 2017 00:22:38 +0000 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> Message-ID: On 28/02/2017 11:54, Michel Desmoulin wrote: > dict.get() is a very useful method, but for lists and tuples, we need to > rely on try/except instead. > > Can we get list.get and tuple.get as well? If PEP 463 "Exception-catching expressions" were accepted and implemented, we wouldn't need any of them: val = myDict[k] except KeyError: default val = myList[n] except IndexError: default Rob Cliffe > > Also, for list, a list.setdefault like the dict.setdefault would be logical. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Tue Feb 28 20:18:47 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 1 Mar 2017 02:18:47 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> Message-ID: I love this proposal but Guido rejected it. Fighting for it right now would probably be detrimental to the current proposed feature which could potentially be more easily accepted. At least let's make it a separate thread. I would love to restart the debate about this one. This is one of my most wanted feature ever in Python. Le 01/03/2017 ? 01:22, Rob Cliffe a ?crit : > > > On 28/02/2017 11:54, Michel Desmoulin wrote: >> dict.get() is a very useful method, but for lists and tuples, we need to >> rely on try/except instead. >> >> Can we get list.get and tuple.get as well? > If PEP 463 "Exception-catching expressions" were accepted and > implemented, we wouldn't need any of them: > val = myDict[k] except KeyError: default > val = myList[n] except IndexError: default > Rob Cliffe >> >> Also, for list, a list.setdefault like the dict.setdefault would be >> logical. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Tue Feb 28 20:23:41 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Feb 2017 17:23:41 -0800 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> Message-ID: <58B6229D.10203@stoneleaf.us> On 02/28/2017 05:18 PM, Michel Desmoulin wrote: > I love this proposal but Guido rejected it. Fighting for it right now > would probably be detrimental to the current proposed feature which > could potentially be more easily accepted. PEP 463 has a better chance of being accepted than this one does, for reasons that D'Aprano succinctly summarized. -- ~Ethan~ From desmoulinmichel at gmail.com Tue Feb 28 20:26:18 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 1 Mar 2017 02:26:18 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <20170301000254.GQ5689@ando.pearwood.info> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <1efd6852-d481-20ad-a485-2e552200215c@mail.de> <20170301000254.GQ5689@ando.pearwood.info> Message-ID: <6b254e05-b4b3-ed3d-0279-6276e77337e1@gmail.com> Le 01/03/2017 ? 01:02, Steven D'Aprano a ?crit : > On Tue, Feb 28, 2017 at 07:10:15PM +0100, Sven R. Kunze wrote: > >> 1. advantage: it looks like dict access -> allows duck typing (oh how >> often I'd missed that) > > Dicts and lists don't duck-type because they are very different things. > > We know what ["item"]*10 does. What would {"key": "value"}*10 do? > > What would list.values() iterate over? > > The fact the API is not exactly the same doesn't prevent duck typing. Duck typing is precesily about incomplete but good enough similar API. For the dict and list: - you can iterate on both - you can index both - you can size both Hence I can see very well functions working with both. E.G: helper to extract x elements or a default value: def extract(data, *args, default="None"): for x in args: try: yield data[x] except (KeyError, ValueError): yield default Usage: a, b, c = extract(scores, "foo", "bar", "doh") x, y, z = extract(items, 2, 5, 8, default=0) I actually have this helper function. With list.get and tuple.get, this would become: def extract(data, *args, default="None"): return (data.get(x, default) for x in args) From boekewurm at gmail.com Tue Feb 28 20:30:56 2017 From: boekewurm at gmail.com (Matthias welp) Date: Wed, 1 Mar 2017 02:30:56 +0100 Subject: [Python-ideas] add variable "__this_func__" inside all functions' locals In-Reply-To: <483eaf6.553.15a873604e1.Coremail.mlet_it_bew@126.com> References: <483eaf6.553.15a873604e1.Coremail.mlet_it_bew@126.com> Message-ID: Hi, On 1 March 2017 at 01:12, ????? wrote: > > How a function refer itself? > def f(): > f() # fine... really??? I understand your question as the following: "Should functions be allowed to point to themselves/the as of construction time unbound variable in the function body, as they are not yet bound to a variable at construction time" I think they should be allowed, as only when the function is executed the variable lookups (and therefore function lookups) should happen. Object.function() would have, well, interesting behaviour if you were to pre-insert all function calls in the code: d = {'hello': 'world'} def f(): d.keys = lambda: {} print(d.keys()) would result in "['keys']" being printed, while you explicitly said that the keys method on variable d has to return an empty dict. I hope this helps you with your question. -Matthias From mertz at gnosis.cx Tue Feb 28 20:32:06 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 17:32:06 -0800 Subject: [Python-ideas] unify usage of mutable and immutable objects In-Reply-To: References: <57e3a098.b1a0.15a85052519.Coremail.mlet_it_bew@126.com> <415196cb.c2.15a85ed7ce7.Coremail.mlet_it_bew@126.com> Message-ID: On Tue, Feb 28, 2017 at 1:49 PM, Terry Reedy wrote: > On 2/28/2017 1:48 PM, David Mertz wrote: > >> In [30]: class FrozenSet(frozenset): >> ...: def pop(self): >> ...: item = next(iter(self)) >> ...: return item, self-{item} >> >> In [31]: a = FrozenSet({1,2,3,4,5}) >> >> In [32]: a.pop() >> Out[32]: (1, frozenset({2, 3, 4, 5})) >> > > To me, 'pop' implies mutation. Tuples do not have a pop method, and it is > not obvious to me that either tuples or frozensets should. What are the > use cases that are not better served by converting to list or set? Yeah, I guess you are right. I was thinking that having a compatible API for sets and frozensets would be good. But this really isn't it anyway. `.pop()` is one of those few standard library methods that is used for simultaneous mutation and return value. My FrozenSet class does something quite different with the method; possibly something useful enough to have under a different name, but it doesn't help with duck typing set-like objects. The OP suggests: Pop tuple/frozenset(standard one) gain no benefit. # O(n) > It is a different story for balanced tree. # O(log n) This is true. You can make a slightly changed copy of an immutable tree in O(log N) time since most pointers are to existing subtrees. But then, pretty much everything else one does is also O(log N) which loses to the O(1) of most set operations. But that's moot for Python. A custom class implementing a balanced tree is free to define its own semantics for a `.pop()` method. This is a very different data structure than Python's set/frozenset which are based on hashes. We are not going to change the implementation of sets so fundamentally, mostly because having O(1) behaviors is so nice (add, remove, membership check, etc). -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qhlonline at 163.com Tue Feb 28 20:31:43 2017 From: qhlonline at 163.com (qhlonline) Date: Wed, 1 Mar 2017 09:31:43 +0800 (CST) Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: <20170301001338.GR5689@ando.pearwood.info> References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> <20170301001338.GR5689@ando.pearwood.info> Message-ID: <1b70bbd4.2499.15a877e9111.Coremail.qhlonline@163.com> My code example is not proper, Yes, may be this is better: list.sort().revers(). Other languages do this differently. JavaScript may return the sorted, while C++ STL returns nothing. I think that it maybe more important to let user have good knowledge about this function then to have fluent code on some occasions. I prefer to draw back this suggestion. Regards? At 2017-03-01 08:13:39, "Steven D'Aprano" wrote: >On Mon, Feb 27, 2017 at 11:07:33AM +0800, qhlonline wrote: >> Hi, all >> I have a suggestion that, the sort() member method of the list >> instance, should return the 'self' as the result of list.sort() >> call. > >Having list.sort() and list.reverse() return self is a perfectly good >design. The advantage is you can write things like this: > >list.sort().reverse() > >but the disadvantage is that it may fool people into thinking it returns >a *copy* of the list. Python avoids that trap by returning None, so that >you cannot write: > >sorted_items = items.sort() > >but instead people write: > >items = items.sort() > >so it seems that whatever we do, it will confuse some people. > > >> Now list.sort() returns nothing, so that I can NOT write >> code like this: >> >> res = {item: func(item) for item in item_list.sort()} > >What is the purpose of the sort? Because dicts are unordered, the >results will be no different if you just write: > > d = {item: func(item) for item in item_list} > > >-- >Steve >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From qhlonline at 163.com Tue Feb 28 20:18:29 2017 From: qhlonline at 163.com (qhlonline) Date: Wed, 1 Mar 2017 09:18:29 +0800 (CST) Subject: [Python-ideas] __repr__: to support pprint In-Reply-To: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> References: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> Message-ID: <4e5598e6.1ea2.15a87727516.Coremail.qhlonline@163.com> Yes, We are both python users with languages OTHER then English, It is a headache when looking at the print output of dicts with other language encoding. They are just byte array. I am using python2.7 Regards! At 2017-03-01 06:59:52, "?????" wrote: Hi, everyone! Oftenly, __repr__ return "{type}({args}, {kws})" problem: 1) to call .format() myself is tedious I have seen someone do it wrong: "T(1,)"! I write a helper function myself, but to import such tiny function everywhere isnot good. 2) pprint cannot dive into string that from repr() To use pprint, I sometimes have to recursively turn objects into builtin containers: (type_name, args...) solution: allow __repr__ to return str OR tuple: (args, kws) Better, isn't it? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 28 20:32:07 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 12:32:07 +1100 Subject: [Python-ideas] suggestion about the sort() function of the list instance In-Reply-To: References: <10473a54.3a6d.15a7d899712.Coremail.qhlonline@163.com> <20170301001338.GR5689@ando.pearwood.info> Message-ID: <20170301013207.GT5689@ando.pearwood.info> On Tue, Feb 28, 2017 at 09:30:37PM -0300, Soni L. wrote: > Stateful functions? What? -- Steve From desmoulinmichel at gmail.com Tue Feb 28 20:56:44 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 1 Mar 2017 02:56:44 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <20170228235616.GP5689@ando.pearwood.info> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <20170228235616.GP5689@ando.pearwood.info> Message-ID: <58e46182-1210-438f-b85a-02aa1bd9dc9e@gmail.com> > > I am aware of that. I'm just saying you don't have to use try...except, > you can use slicing instead. Obviously you have to adapt the code since > you are getting a list not a single item: > > # may fail, if alist is empty > if alist[0] == "spam": ... > > # cannot fail > if alist[0:1] == ["spam"]: ... > ... > > first_item = (alist[0:1] or ["ham"])[0] Come on, I've been doing Python for more than a decade and never saw anybody doing that. Even reading it in a code would make me scratch my head for a moment with a "what is it doing that for?". You are trying to hard to provide a counter argument here. > > But honestly, in my experience the number of times I actually needed > something like that is tiny. ... > > A very robust approach **for what** ? > > What are you trying to do? Why are you indexing into arbitrary positions > of a list without knowing whether or not there is something there? > > List are not dicts and they are used differently. It is very common, and > useful, to want to look up a key in a dict without knowing if it exists, > and do something with a default if it doesn't exist. This is so useful Maybe your missions involve working with people doing properly their job. Me, I have to deal SOAP government systems, mongodb based API built by teenagers, geographer data set exports and FTP + CSV in marina systems (which I happen to work on right now). 3rd party CSV, XML and JSON processing are just a hundred of lines of try/except on indexing because they have many listings, data positions is important and a lot of system got it wrong, giving you inconsistent output with missing data and terrible labeling. And because life is unfair, the data you can extract is often a mix of heterogeneous mappings and lists / tuples. And your tool must manage the various versions of the data format they send to you, some with additional fields, or missing ones. Some named, other found by position. This summer, I had to convert a data set provided by polls in africa through an android form, generated from an XML schema, send as json using Ajax, then stored in mongodb... to an excel spread sheet (and also an HTML table and some JS graphs for good measure). Needingless to say I dealt with a looooot of IndexError. Grepping the project gives me: grep -R IndexError | wc -l 33 In contrast I have 32 KeyError (most of them to allow lazy default value), and 3 decorators. If special syntax exist for decorators, then surely we can spare a .get() for lists and tuples. Apparently IndexError is an important error because if I grep the virtualenv of the project: grep -R IndexError | wc -l 733 Ok, it's a pretty large project with 154 dependancies, but it's still almost 7 IndexError by package on average. So it's not a rare use case. I also see it regularly in my classes. Students try it because they learned it works with dict. It makes sense. Don't dismiss a use case because you don't have it. Python is so versatile it's used in many diverse areas. From boekewurm at gmail.com Tue Feb 28 20:58:35 2017 From: boekewurm at gmail.com (Matthias welp) Date: Wed, 1 Mar 2017 02:58:35 +0100 Subject: [Python-ideas] __repr__: to support pprint In-Reply-To: <4e5598e6.1ea2.15a87727516.Coremail.qhlonline@163.com> References: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> <4e5598e6.1ea2.15a87727516.Coremail.qhlonline@163.com> Message-ID: Hi, On 1 March 2017 at 02:18, qhlonline wrote: > Yes, We are both python users with languages OTHER then English, It is a > headache when looking at the print output of dicts with other language > encoding. They are just byte array. I am using python2.7 Please note that unless this is still a problem in python 3, this is unlikely to be changed. __repr__ is (as per the docs) 'a printable string representation of the object'. If this were to get changed, many APIs might be broken due to tuples not having the same methods and/or features as strings, e.g. you cannot use them as a path. Not saying that they will, but it's potentially breaking a lot of code which interacts with other programs, and maybe even some internals. You are free to experiment with overriding/extending __repr__ for your internal usage, but please note that it might break external libraries depending on obj.__repr__ or repr(obj), and print() might break when using built-in types as containers for your objects. I am a -1 on changing __repr__'s signature. -Matthias From desmoulinmichel at gmail.com Tue Feb 28 21:02:57 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 1 Mar 2017 03:02:57 +0100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <58B6229D.10203@stoneleaf.us> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <58B6229D.10203@stoneleaf.us> Message-ID: <764b7a4e-04f1-bfda-8e65-f750a3281af6@gmail.com> Le 01/03/2017 ? 02:23, Ethan Furman a ?crit : > On 02/28/2017 05:18 PM, Michel Desmoulin wrote: > >> I love this proposal but Guido rejected it. Fighting for it right now >> would probably be detrimental to the current proposed feature which >> could potentially be more easily accepted. > > PEP 463 has a better chance of being accepted than this one does, for > reasons that D'Aprano succinctly summarized. > > -- > ~Ethan~ The debate is not even over and you are already declaring a winner. That's not really fair. Give the idea a chance and read until the end. D'Aprano's argument is mostly "I don't encounter IndexError really often and when I do I have this twisted one liner to get away it". Well, that's not really a good reason to reject things for Python because it's a language with a very diverse user base. Some bankers, some web dev, some geographers, some mathematicians, some students, some 3D graphists, etc. And the language value obvious, readable, predictable code for all. Most people on this list have a specialty, because their speciality don't see a use for the feature doesn't mean there is not one. So I provided on my last answer an explanation of what I would use it for. From rosuav at gmail.com Tue Feb 28 21:04:00 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Mar 2017 13:04:00 +1100 Subject: [Python-ideas] __repr__: to support pprint In-Reply-To: References: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> <4e5598e6.1ea2.15a87727516.Coremail.qhlonline@163.com> Message-ID: On Wed, Mar 1, 2017 at 12:58 PM, Matthias welp wrote: > You are free to experiment with overriding/extending __repr__ for your > internal usage, but please note that it might break external libraries > depending on obj.__repr__ or repr(obj), and print() might break when > using built-in types as containers for your objects. Given that this started out with a post about pprint, maybe a more viable approach would be a dedicated __pprint__ hook? That might be easier to push through. (I'm not pushing for it though.) ChrisA From rosuav at gmail.com Tue Feb 28 21:07:59 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Mar 2017 13:07:59 +1100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <764b7a4e-04f1-bfda-8e65-f750a3281af6@gmail.com> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <58B6229D.10203@stoneleaf.us> <764b7a4e-04f1-bfda-8e65-f750a3281af6@gmail.com> Message-ID: On Wed, Mar 1, 2017 at 1:02 PM, Michel Desmoulin wrote: > Le 01/03/2017 ? 02:23, Ethan Furman a ?crit : >> On 02/28/2017 05:18 PM, Michel Desmoulin wrote: >> >>> I love this proposal but Guido rejected it. Fighting for it right now >>> would probably be detrimental to the current proposed feature which >>> could potentially be more easily accepted. >> >> PEP 463 has a better chance of being accepted than this one does, for >> reasons that D'Aprano succinctly summarized. >> >> -- >> ~Ethan~ > > The debate is not even over and you are already declaring a winner. > That's not really fair. Give the idea a chance and read until the end. This channel doesn't exist to be fair. It exists to discuss improvements to a programming language. On that basis, Ethan's post is perfectly appropriate (and, in my opinion, accurate). ChrisA From steve at pearwood.info Tue Feb 28 21:10:34 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Mar 2017 13:10:34 +1100 Subject: [Python-ideas] get() method for list and tuples In-Reply-To: <6b254e05-b4b3-ed3d-0279-6276e77337e1@gmail.com> References: <2b7ac126-881e-f79b-6157-c45100904707@gmail.com> <20170228144519.GK5689@ando.pearwood.info> <1efd6852-d481-20ad-a485-2e552200215c@mail.de> <20170301000254.GQ5689@ando.pearwood.info> <6b254e05-b4b3-ed3d-0279-6276e77337e1@gmail.com> Message-ID: <20170301021034.GU5689@ando.pearwood.info> On Wed, Mar 01, 2017 at 02:26:18AM +0100, Michel Desmoulin wrote: > The fact the API is not exactly the same doesn't prevent duck typing. > Duck typing is precesily about incomplete but good enough similar API. Indeed. But the relationship goes this way: # Duck-typing done the right way. Types P and Q both quack like ducks, so if all we need is something that quacks, either P or Q will do. not this way: Types P and Q both quack like ducks. I want something that swims like a duck, like P. Q doesn't swim at all, so we need to add Q.swim() so we can duck-type P or Q. If we are going to propose Q.swim(), it must be because it makes sense for Q instances to swim, regardless of what P does. > For the dict and list: > > - you can iterate on both > - you can index both > - you can size both Right -- because all these operations make sense for both dicts and lists. Does get() make sense for both? It certainly makes sense for dicts. It makes *some* sense for lists, but (in my opinion) not enough to justify making it a built-in method of the type. Making it a built-in method isn't just a convenience, it is also blessing this as "the right thing to do". As I've said, in my experience trying to index into arbitrary positions of a sequence (list or tuple) without knowing whether that index exists or not is rarely the right thing to do. (That makes it very different from key lookup in a mapping or dict.) I believe that the way to argue for list.get() is not because it will make it easy to duck-type lists and dicts. It is (in my experience) very rare to need to duck-type lists and dicts. I believe you should identify code that handles lists that would benefit from this change. Under what circumstances do you ask for the 17th item of a list which may only contain 9 items? (For arbitrary values of 17 and 9.) -- Steve From mlet_it_bew at 126.com Tue Feb 28 21:14:27 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 10:14:27 +0800 (CST) Subject: [Python-ideas] add variable "__this_func__" inside all functions' locals In-Reply-To: References: <483eaf6.553.15a873604e1.Coremail.mlet_it_bew@126.com> Message-ID: <4493c171.2648.15a87a5b373.Coremail.mlet_it_bew@126.com> We need not care other functions, just the "current" one. Other functions are definitely out of our control. My last example distinguish 3 cases: self.f() # object/overloaded version __class__.f(self) # decorated version __this_func__(self) # prime version At 2017-03-01 09:30:56, "Matthias welp" wrote: >Hi, > >On 1 March 2017 at 01:12, ????? wrote: >> >> How a function refer itself? >> def f(): >> f() # fine... really??? > >I understand your question as the following: "Should functions be >allowed to point to themselves/the as of construction time unbound >variable in the function body, as they are not yet bound to a variable >at construction time" > >I think they should be allowed, as only when the function is executed >the variable lookups (and therefore function lookups) should happen. >Object.function() would have, well, interesting behaviour if you were >to pre-insert all function calls in the code: > > d = {'hello': 'world'} > def f(): > d.keys = lambda: {} > print(d.keys()) > >would result in "['keys']" being printed, while you explicitly said >that the keys method on variable d has to return an empty dict. > >I hope this helps you with your question. > >-Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 21:20:23 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 10:20:23 +0800 (CST) Subject: [Python-ideas] unify usage of mutable and immutable objects Message-ID: <6c4bf1f7.2872.15a87ab1f50.Coremail.mlet_it_bew@126.com> > We are not going to change the implementation of sets so fundamentally, > mostly because having O(1) behaviors is so nice (add, remove, membership > check, etc). What I want are such interfaces (no matter with standard set): class IDynSet: # "Dyn" means it have methods like ipop_both def ipop_both(self): if hasattr(self, 'pop_mutable'): elem = self.pop_mutable() elif hasattr(self, 'ipop_immutable'): elem, self = self.ipop_immutable() else: raise return elem, self class IMutableDynSet(IDynSet): def pop_mutable(self): return self.pop() class IImutableDynSet(IDynSet): def ipop_immutable(self): elem, new = self.ipop() # ipop not pop return elem, new -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 21:27:21 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 10:27:21 +0800 (CST) Subject: [Python-ideas] __repr__: to support pprint In-Reply-To: References: <320dd54e.260.15a86f38c41.Coremail.mlet_it_bew@126.com> <4e5598e6.1ea2.15a87727516.Coremail.qhlonline@163.com> Message-ID: <4fd10d87.2ad7.15a87b18138.Coremail.mlet_it_bew@126.com> It has not to break current code: def repr(obj, allowed_tuple = False): r = type(obj).__repr__() if allowed_tuple: return r if isinstance(r, tuple): r = make_str(type(obj), r) return r At 2017-03-01 09:58:35, "Matthias welp" wrote: >Hi, > >On 1 March 2017 at 02:18, qhlonline wrote: >> Yes, We are both python users with languages OTHER then English, It is a >> headache when looking at the print output of dicts with other language >> encoding. They are just byte array. I am using python2.7 > >Please note that unless this is still a problem in python 3, this is >unlikely to be changed. > >__repr__ is (as per the docs) 'a printable string representation of >the object'. If this were to get changed, many APIs might be broken >due to tuples not having the same methods and/or features as strings, >e.g. you cannot use them as a path. Not saying that they will, but >it's potentially breaking a lot of code which interacts with other >programs, and maybe even some internals. > >You are free to experiment with overriding/extending __repr__ for your >internal usage, but please note that it might break external libraries >depending on obj.__repr__ or repr(obj), and print() might break when >using built-in types as containers for your objects. > >I am a -1 on changing __repr__'s signature. > >-Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From mlet_it_bew at 126.com Tue Feb 28 21:56:56 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 10:56:56 +0800 (CST) Subject: [Python-ideas] a bad feature in Python syntax Message-ID: <75139e7e.34e0.15a87cc97f4.Coremail.mlet_it_bew@126.com> I'm bited once: >>> '' in {} == False False >>> ('' in {}) == False True # '' in {} == False ==>> ('' in {}) and ({} == False) ==>> False! I think only compare operations should be chained. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Tue Feb 28 21:54:14 2017 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Tue, 28 Feb 2017 20:54:14 -0600 Subject: [Python-ideas] add variable "__this_func__" inside all functions' locals In-Reply-To: <4493c171.2648.15a87a5b373.Coremail.mlet_it_bew@126.com> References: <483eaf6.553.15a873604e1.Coremail.mlet_it_bew@126.com> <4493c171.2648.15a87a5b373.Coremail.mlet_it_bew@126.com> Message-ID: > On Feb 28, 2017, at 8:14 PM, ????? wrote: > > We need not care other functions, just the "current" one. > Other functions are definitely out of our control. > My last example distinguish 3 cases: > self.f() # object/overloaded version > __class__.f(self) # decorated version > __this_func__(self) # prime version If your decorator uses `functools.wraps` or `functools.update_wrapper` (it should, for lots of reasons, and many, perhaps most, third party wrappers do), then you can access the wrapped function as `decorated.__wrapped__`. See https://docs.python.org/3/library/functools.html#functools.update_wrapper -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Feb 28 22:04:59 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Mar 2017 14:04:59 +1100 Subject: [Python-ideas] a bad feature in Python syntax In-Reply-To: <75139e7e.34e0.15a87cc97f4.Coremail.mlet_it_bew@126.com> References: <75139e7e.34e0.15a87cc97f4.Coremail.mlet_it_bew@126.com> Message-ID: On Wed, Mar 1, 2017 at 1:56 PM, ????? wrote: > I'm bited once: > >>> '' in {} == False > False > >>> ('' in {}) == False > True > > # '' in {} == False ==>> ('' in {}) and ({} == False) ==>> False! > > I think only compare operations should be chained. I do feel your pain, but generally, you shouldn't be using "== False" to negate a condition. That's what the "not" operator is for - or inverted conditions. >>> '' not in {} True Much better. :) ChrisA From mlet_it_bew at 126.com Tue Feb 28 21:44:22 2017 From: mlet_it_bew at 126.com (=?GBK?B?0+/R1MbGy+m0pg==?=) Date: Wed, 1 Mar 2017 10:44:22 +0800 (CST) Subject: [Python-ideas] add a always turn on "assert" Message-ID: <3c613d9b.30cc.15a87c11743.Coremail.mlet_it_bew@126.com> "assert" is good, but it is used as a guard frequently. We can make such usage legal by adding a new syntax: assert bool_expr, ExceptionType, True suggest reasons: 1) even "__debug__" turn off, assert is working assertion as guard. 2) to avoid boilerplate code I write code like this: if pred(....) or pred(....): raise ValueError('pred(....) or pred(....)') Simplifed: assert pred(...), ValueError, True # the above line will be printed when error. # I need not to copy the condition! 3) future: "assert bool_expr, ET, False" To aid static tool, like Proof System. Better document. For complicate algorithm, I actually add a invariant outline comment above every statement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Feb 28 23:50:21 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Feb 2017 20:50:21 -0800 Subject: [Python-ideas] a bad feature in Python syntax In-Reply-To: <75139e7e.34e0.15a87cc97f4.Coremail.mlet_it_bew@126.com> References: <75139e7e.34e0.15a87cc97f4.Coremail.mlet_it_bew@126.com> Message-ID: OK, I'm impressed! I've written about and taught Python for almost 20 years. I never realized `in` was a chained comparison. I'm pretty sure I've never seen it used that way "in the wild." I also never tried using `is` in a chained way until just now. That said, there are at least three things perverse about the examples below, and they should *definitely* never be used in real code. This is vaguely plausible (for variable defined in some more interesting way where the substring relation is not so obvious; say content read from files): >>> a = "a" >>> b = "abc" >>> c = "abcde" >>> a in b in c True On Tue, Feb 28, 2017 at 6:56 PM, ????? wrote: > I'm bited once: > >>> '' in {} == False > False > >>> ('' in {}) == False > True > > # '' in {} == False ==>> ('' in {}) and ({} == False) ==>> False! > > I think only compare operations should be chained. > > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: