Network/multi-user program

[Chris Angelico]

On Tue, Jul 22, 2014 at 5:54 PM, Monte Milanuk <memilanuk at invalid.com> wrote:
> Well... thats part of where my lack of experience with js or complex
> projects using anything other than just python is going to show:
> initially I thought javascript was just for buttons/effects in the
> client browser as thats all the trivial examples I looked at years ago
> did.  The bits n pieces I'm seeing of these 'modern' javascript MVC
> frameworks like sencha, angular, etc. is making me think that a lot of
> the 'work' is moving from the server to the client via the javascript...
> which just blurs the heck out of things and confuses me as to what
> should be in the browser and what should be on the 'server'?  And where
> the testing goes?  If more of the 'heavy lifting' is being done on the
> client, is there a need for a full-service python framework like django
> or would something lightweight like flask be more appropriate?  Again, I
> know almost nothing about pyramid and where it falls into the mix.  I
> can read the propaganda on their respective web sites, but that is
> necessarily skewed. :/

Lots of things getting messed in together here, so I'm going to start
right back at some basics. You probably know some of what I'm going to
say already, but I'm not sure exactly how much, so I'll just say it
anyway. Hopefully this won't be TOO long. :)

Communication between the server and the client is over HTTP.
(Usually. I'm ignoring SPDY and other protocols, and I'm also ignoring
the distinction with HTTPS, which just adds an encryption layer and
doesn't affect any of this.) HTTP is a stateless protocol, where the
client sends a request to the server and the server sends back a
response. Simple, and works really beautifully for the simple case
where you just want a page. If you go to
http://rosuav.com/hymns/LetHerGo2.png in your browser, it sends a
request "GET /hymns/LetHerGo2.png" to my server, and my server
responds with "\x89PNG" and about 100KB of PNG-encoded data, which
your browser then displays as an image. So far, so good.

For something more interactive, though, HTTP isn't really ideal. The
server can send back a frames-based page (either with <FRAME> or with
<IFRAME>), and then with links <A HREF="..." TARGET="..."> modify
specific parts of that page, but it's still pretty clunky, and it's
not going to look perfectly smooth. (I've built systems that do
exactly this; if your use-case happens to fit this, it can work. But
it's unusual to fit into that.) No, what you really want is two
things: First, a way to change what's on the page without throwing it
all away and fetching an entire new page; and second, a way for the
browser to talk to the server without throwing away its current page
and replacing it.

The first one fundamentally requires client-side scripting. You have
to have the server provide something executable which the client will
run. And that, in itself, adds another bunch of problems (security,
sandboxing, and stuff), which is why ECMAScript/JavaScript is the only
properly-supported language; there've been various attempts to get
another language supported across all browsers, but never successfully
enough to supplant JS. And due to backward-compatibility requirements
(a modern browser needs to be able to run JS code written in the 90s),
the language's worst design flaws (like UTF-16 strings) simply cannot
be changed. But the power is there for code to do whatever it likes to
the displayed page: remove stuff, add stuff, change styles, fiddle
with text, anything at all.

Fortunately, the second requirement is fairly easily covered. JS code
can issue additional HTTP requests to the server, and do whatever it
likes with the responses. The server still does whatever it does in
response to those requests, maybe some database changes or lookups, or
other work that can only be done on the server. And then it can send
back either preformatted HTML for the JS to insert into the page, or
enough information for the JS to build the content itself. (In some
cases, the server might send back nothing more than a flag saying
"Success" or "Failure", and the JS does all the rest, possibly
including a retry loop.)

But there's still a fundamental disconnection between the web browser
and the database, which can be handled ONLY by the web server.
(Technically you could let the browser talk directly to the DBMS. If
you really need that, you know enough to know that I've been
simplifying quite a bit of this discussion.) To keep your server-side
code clean, especially as it gets bigger and more complex, you often
want to make use of a web framework like Django, Flask, etc, etc, etc.
You don't have to send back HTML from the server; if you have a
dedicated "script request" endpoint, you can send back a simpler form
of the information (maybe JSON, or plain text, or something), and
possibly you can have uglier error checking (eg if you get the
parameters wrong, it just spits back an HTTP 500 instead of
courteously telling you which one you omitted - because anyone who
gets the parameters wrong is fiddling with things). Think of each
serverside action the client wants to do as an endpoint, same as a
page is, and you'll have an idea of how much complexity your server
has. The only "heavy lifting" that you're taking off the server is the
prettiness of some of it; for the most part, a big and complex site
will still be a big and complex site.

Here are two Python-based web sites that I've built:

https://github.com/Rosuav/Yosemite
https://github.com/Rosuav/Flask1

The first one is a self-contained HTTP server, built on
http.server.BaseHTTPRequestHandler
(BaseHTTPServer.BaseHTTPRequestHandler in Python 2). It's tiny, it's
for a trusted LAN, and it's not so much a "web site" as an application
that happens to use a web browser as its UI. It hardly uses JS at all;
I could completely avoid JS by having form buttons instead of script
buttons for the top-right control block. (No point doing so, but
that's how little client-side scripting there is.) The code's a bit
messy, for several reasons; it's old code from when I didn't know as
much (and, I think, from when Python 2.5 or *maybe* 2.6 was current!),
and it tries to support Py2 and Py3, and Windows and several Linuxes,
and several different ways of accomplishing its goals. But it works,
and we use it every day here. (They've just finished watching Wreck It
Ralph in the other room, controlled by Yosemite, and I just used it to
fire up my playlist of Frozen musical clips - part of my standard
coding collection.)

The second is much more recent. I had a pair of PHP scripts to
populate and view a MySQL database of quotes from a MUD. Later I
changed the back end to PostgreSQL, but left the actual code in PHP...
until I had reason to learn Flask, and decided to rewrite my 1 as an
exercise. It's pretty simple, and it doesn't use anything like all of
Flask's features, plus it's probably doing things in less-than-optimal
ways, but again, it works. (This is live at http://rosuav.com/1/ -
although it probably won't be of great interest to you unless you're
into RPGs.)

These two examples are roughly the same complexity. Dig into the code
and see if you can find what's doing what. Does Flask make the code
more readable? I would say it does, and that doesn't change if most of
your site is done with AJAX. Would Django (or some other framework) be
better or worse than Flask? Well, that's something I can't answer. But
I'm pretty happy with Flask, and other people are pretty happy with
other frameworks, so just look at the features and make a choice. I
don't think the shift to "heavy lifting on the client" makes any
significant difference to the choice of framework.

As to testing... Unit tests would be ideal for testing the server, as
it's pretty hard to unit-test a JavaScript UI; but your primary
user-based testing will be of the entire system as a whole - sit
someone down with the web browser and have him/her actually use the
system the way it's meant to be used. (And how it's meant not to be
abused, that's part of testing too. But you know what I mean.)

Hope that's of at least some value! This did turn out a bit long, sorry... :|

ChrisA