[Edu-sig] DocTest Quiz

[Michael Tobis]

On 4/23/06, Ian Bicking <ianb at colorstudy.com> wrote:
> Have you guys contacted the people doing sandboxed Python on the web?
> I've seen two of them, but I can't for the life of me remember where and
> my searching has been fruitless so far.  I think both people hang out on
> comp.lang.python.

Is this the thread you are remembering?

http://tinyurl.com/f94u8

see especially

http://www.mired.org/home/mwm/try_python/

I asked some questions on that thread about sandboxing prompted by
Jeff's idea, and got some interesting responses.

That said:

Ultimately to get to the big one-room python schoolhouse we all seem
to want, it really has to be a peer relationship with peers picking
which puzzles are interesting.

However, to test kids' passing microtests in a school curriculum, you
don;t need that.
All you need is a way for the kid to pass the doctest and ask a server
for a credential.

Specifically:

It is possible to do this entirely through HTTP requests on the
client. The student would be doing conventional doctests during
development, but would be running a validator that hits a server with
HTTP requests ("validate my code for exercise 42") gets HTTP responses
('send me the output from "foo(42)" with this ticket T ') and
resubmits ('here's ticket T, foo(42) returns "don't talk to me about
life") and the server responds ("OK, you are validated in the
database, and here is ticket U which will allow you to request
exercise 43).

You DON'T want the client code to be a real XML-RPC server, because it
has to be an HTTP client, since these kids will be running in an
arbitrarily locked-down requirement, except for outbound HTTP
requests. But, except for being the client at the lowest level,  it
looks a lot like an XML-RPC server, interpreting known requests

Security advantages abound. The validation server address will be
hardwired into the code that the client runs, and will be a trusted
machine on the part of the school system that delivers those codes to
the users.. That code can, like a vanilla XML-RPC server, only run
registered procedures, if you are worried about executing arbitrary
code from a hacked server. ALL TESTS RUN ON THE CLIENT. You need not
fear for any horror that the kid the kid's evil older cousin comes up
with, because they will never get to run any scripts server side.

Probably you will want the server code to be directly munging doctests
to make its requests. That way, the teacher can specify conventional
doctests, and it will be the server's job to parse them into requests.

Again, this is to say nothing against solving the sandbox problem, as
even bigger things become possible with true sandboxes. My intent is
to knock this thing out in a few days, though, so we can spend the
summer producing curriculum for Jeff to try out. The first version
would probably just wrap XML-RPC on the client and doctest on the
server. Unfortunately, those few days haven't materialized yet, and
won't for a couple of weeks.

This was the inspiration that knowcked me over the head at PyCon. I
hope there's still room to go with it.

> If there were student-contributed doctests this
> seems like a potential concern.

Yes, this is the problem with my approach. I don't handle that, and
that is why a sandbox solution is still a good idea.

But my approach solves Jeff's immediate functional requirements very
simply without any security problems at all.

mt