[Python-ideas] Better comand line version of python -c

[Russell Stewart]

Andrew,

You've brought up many design points that I might have done differently if
I were to start over. Things got locked in after the first few hundred
users, but bringing things under the primary python command would free up a
lot more room without backwards compatibility.

Heres some thoughts on a few things you said, assuming a brand new version
without backwards compatibility would be the way forward:

Json handling:
I think I would remove this if I did it again. I originally found it quite
useful, but the fact that it doesn't support multi-line json inputs makes
it weird. Also, all the functionality can be accessed through the json
module with python (e.g. py -l 'json.loads(l)').

Split input/output:
This one is actually super useful for parsing lines of text that you get
from other unix commands (e.g. py --si 'special: ' -x x[1][:4] will grab
the first 4 characters after the word special in any input file). It's also
much easier to provide complete access to what people want to do with the
single delimeter flag, so its much better than the --ji --jo flags in that
respect.

-csv option:
I considered this at length, and originally the --si was motivated by
commas. But ultimately --si is good for lots of things other than commas. A
separate csv command seems nice, until you realize that there are too many
options to specify in parameters, just like in the json case. It's better
to let people parse csv docs with py -l 'csv.Reader(l) ... '.

Windows py.exe issue:
Yea, I'm aware of this. I have a pythonwpy for windows that is run with
wpy, but no one seems to care about comand line tools on windows.

py vs python -m pythonpy ...:
Yes, using this with python -m is a very real option. People could simply
alias py='python -m pythonpy' to avoid typing issues. This is probably the
most straightforward distribution option. It would be great to keep the
ipython style tab completion, but I'm not sure that its necessary.

py -fx flag:
Originally, I didn't want to name this -f because I didn't want the active
variable x to be magically introduced. When you type -fx, its clear where
the variable x comes from. This was probably a mistake, because everyone
hates 2 character single-dash flags. Should be changed in the case of an
overhaul. This command is also only marginally useful, due to the fact that
grep already handles this use case so well, and because you can get the
functionality with py -x 'x if foo(x) else None' just as well.

pythonpy should be Iterable based, not list-based:
I agree somewhat. The current dictionary functionality is weird, only
printing the list of keys. Again, I don't want to change this for backwards
compat reasons. This could be done a lot of different ways, and I think the
current strategy is pretty good, but leaves a little bit to be desired in
terms of simplicity. The -x and -l flags capture a lot of the list-printing
that you would want to do, so its not clear you need to do list-printing
with no arguments. Using a separate flag for list printing would be a good
design consideration here. Keep in mind that the current motivation for all
the list strangeness is so that pythonpy will play well with unix pipes,
which it does. That is really important for some core usecases.

Daemon mode:
I hadn't thought of this one before. Maybe it could be done well, but
daemon modes tend incur a high cost of mental overhead on the user. I've
worked hard to keep the startup time within a factor of 2 from the python
interpreter, so I do care a lot about that.

--i flag:
Again, the naming here is up for change. It's useful in a few cases, where
python throws exceptions easily (e.g. py -x re.match(r"b", x).group(0)).
But I haven't used it all that much, and it really could be removed.

So in general I like the python -m idea. Support for tab completion under
this would be really nice. I've long thought that python -c and python -m
should have ipython style tab completion anyways, so that one could type
python -m Sim<tab> and get python -m SimpleHTTPServer. Another idea would
be to try and get a shorter binary name distributed with python. It sounds
superficial, but is ultimately quite important if python wants to compete
with ruby, perl, and sed.

Best,
Russell

On Thu, Jan 8, 2015 at 2:35 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Thursday, January 8, 2015 1:11 PM, Russell Stewart <
> Russell.S.Stewart at gmail.com> wrote:
>
>
> >I am the original dev for pypi.python.org/pypi/pythonpy, which offers
> much of the same functionality as python -c, but in a more convenient form.
> I'd be really interested in figuring out whether the core python tools
> could support many of the use cases that pythonpy covers, so that users
> wouldn't have to install it as a separate tool. I don't have a concrete
> idea of what it would look like, but many of my users have said they would
> much prefer an officially supported tool. Perhaps someone on this mailing
> list can provide some insight into the available options.
>
> This looks pretty cool to me—so I have lots of questions and comments. :)
>
> The obvious problem is what to call it. `py` (well, `py.exe`) is already
> the name of the PEP 397 launcher tool (the thing that lets you use shbang
> lines on Windows), so this would need a new name.
>
> Maybe it would be better as just a different flag or set of flags on
> `python` itself, like `--pprint`/`-p`? Of course then there's the problem
> of where to stick the flags for `-p` (for GNU-style long names this is
> pretty easy: `--print=filter,lines`, but obviously for something meant to
> be used for quick&dirty command-line usage you need short names as well…).
> And the "backport" to 3.4 (and 2.x) would act differently from the standard
> 3.5+ version. But it might be worth looking at anyway.
>
>
> Or, of course, just make this a stdlib module, so it's just `python
> [PYTHON OPTIONS] -m YOUR_MODULENAME [YOUR MODULE OPTIONS]`, and you can
> alias that to whatever you want.
>
> Is `-fx` a multi-character single-hyphen flag (in which case it's very
> weird to mix those with GNU-style long arguments in the same program, or is
> it a combination of `-f` and `-x`? And is the filter expression an argument
> to `-fx`, or does `-f` just change the interpretation of the argument?
>
>
> Speaking of GNU-style long arguments, why `--i` instead of just `-i` (as a
> short name for `--ignore_exceptions` or something)
>
> The documentation is pretty Python 2-specific: the majority of the
> examples use the `range` function and depend on it returning a list.
>
>
> You seem to have built a non-trivial custom pretty-printer for this tool
> (e.g., to print lists row by row) as well as an auto-importer (e.g., to use
> `collections.Counter` without an `import collections`); maybe some or all
> of that should be separated out and made available to Python code somewhere
> in the stdlib, and then the script (or flag) could just use that function?
>
> This seems to be pretty strongly list-based for no good reason. Why not
> print _any_ iterable line by line, make -l set `l = sys.stdin` instead of
> `l = list(sys.stdin)` (I realize that doesn't work for the specific example
> of `py -l 'l[::-1]'`, but `py -l reversed(l)` would work just as well in
> that case, and it's hard to think of other examples where there would even
> be a problem), etc.?
>
> It might be nice to have an option to see the `repr` instead of the `str`
> of everything, to match what you'd see from the interactive interpreter.
>
> While --si is cool, people already misuse `str.split` and `re.split` to
> try to parse CSV and similar input that has quotes or escapes; it might be
> nice to have a `--csv` mode to parse the input with `csv.reader` (and that
> could still take an optional delimiter, of course; passing other dialect
> flags to `reader` might be out of scope).
>
> A columnar reading mode might also be nice, since that's one of those
> things that's novices have a hard time writing without statements.
>
> An option for pretty JSON output instead of compact JSON output might be
> nice.
>
> And maybe an option to read multi-line JSON input—not just a single JSON
> value (which is trivial with `json.loads(l)`, but, since JSON is
> self-delimiting, still a stream of them, without the requirement of
> one/line (e.g., by looping over `json.JSONDecoder.raw_decode`).
>
> I assume you're evaluating the expression with `eval`. Are you passing it
> a custom locals and globals (so any internals of your script itself aren't
> available), or not? I could see disadvantages of both (the former may
> prevent some useful quick&dirty hacks; the latter could raise safety
> concerns), but either way, it's probably worth documenting.
>
> From the summary help, it looks like you can apply both `-x` and `-fx`. If
> you do that, do they run in a specific order, or in the order specified?
> For that matter, can you apply multiple mappings and/or filterings? (If
> not, it seems like that could be handy.)
>
> It might be useful to have some kind of "daemon mode", where you start up
> an interpreter in the background and then pipe input to it. Then you can
> pipe multiple things to the same interpreter session, both for persistence,
> and to avoid the process startup cost on platforms where it matters (like
> Windows). Maybe with an optional timeout, where the daemon stops a few
> minutes after last use, so you don't have to remember to `py -d
> sys.exit(0)` or similar. And maybe the magic `_` should work in daemon
> mode, as it does in the interactive interpreter. (I could see writing some
> one-liner that takes 90 seconds to run, then realizing I wanted it in JSON
> format, so `py --daemon -jo _` would be handy to avoid re-running
> everything…)
>
> It would be really cool if, when run under PowerShell, this could handle
> scriptlet input and output instead of plain text (so, e.g., you pass it an
> array of strings and it processes each string).
>
> Being able to somehow pass command-line arguments through to `python`
> itself (assuming this isn't merged into the main interpreter, of course)
> might be handy. In particular, I could see `-u` being useful, but there
> might be others (including platform/implementation-specific `-X` options).
> Of course that would come for free with the `-m` interface, but since I'm
> not sure that's the best option...
>
> Instead of just being able to ignore exceptions, it might be nice to
> enable one-line exception output (e.g., print just the type and message, no
> traceback, and to stdout instead of stderr).
>
> With `-x`, an option to prefix each line of output with the corresponding
> line of input could be handy, similar to the `-v` option to commands like
> `cp`, `tar`, etc.
>
> A way to feed input files in could be handy to allow the kind of
> one-liners people often fall back to perl for, although I'm not sure
> exactly what you'd want this to look like.
>
> Some of these are probably way out of scope even for a third-party tool,
> much less for a built-in stdlib version, but I'm not sure exactly which, so
> I've just dumped everything on you to let you sort them out. :)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150108/ef14fc78/attachment-0001.html>