building an online judge to evaluate Python programs

[Modulok]

On Fri, Sep 20, 2013 at 11:28 AM, Jabba Laci <jabba.laci at gmail.com> wrote:

> Hi,
>
> In our school I have an introductory Python course. I have collected a
> large list of exercises for the students and I would like them to be
> able to test their solutions with an online judge (
> http://en.wikipedia.org/wiki/Online_judge ). At the moment I have a
> very simple web application that is similar to Project Euler: you
> provide the ID of the exercise and the output of the program, and it
> tells you if it's correct or not. However, it can only be used with
> programs that produce an output (usually a short string or a number).
>
> In the next step I would like to do the following. The user can upload
> his/her script, and the system tests it with various inputs and tells
> you if it's OK or not (like checkio.org for instance). How to get
> started with this?
>
> There are several questions:
> * What is someone sends an infinite loop? There should be a time limit.
> * What is someone sends a malicious code? The script should be run in a
> sandbox.
>
> All tips are appreciated.
>
> Thanks,
>
> Laszlo
> --
> https://mail.python.org/mailman/listinfo/python-list



How much time is it worth?

Here's some ideas:

Build a server just for grading student work that contains no valuable data,
probably a virtual server on another machine. Something like virtualBox,
etc.
The code they submit is executed as a less privileged operating system user.
For added security, you could look into using something like FreeBSD process
jails but there's a learning curve there.

I would not be overly concerned with security - put out a bounty as extra
credit:
He who hacks the server through the judge program, and provides details on
how they did it, wins something great, even if it's just class notoriety.

What is someone sends an infinite loop? There should be a time limit:

    Run their code as a separate process. If the process or its children run
    too long, kill them. On similar grounds you can use operating system
    features to define limits on maximum memory, disk usage, maximum number
of
    processes, etc. On FreeBSD this can done via the `limits` command, other
    operating systems have their own versions of the same thing. Limits are
    useful, they prevent things like exhausting physical memory.


What is someone sends a malicious code?

    If the server contains no valuable data, there's not much damage they
can
    do other than using it for email spam or perhaps shutting it down. You
    could prevent email spam by a few firewall rules. To protect the judge
    program itself from being hacked set its permission bits to read only.

    It is *extremely* difficult to create a sandbox without using operating
    system (kernel enforced) features and access controls i.e. permission
bits,
    process jails, kernel enforced hardware resource limits, etc. Don't be
    tempted to try it without these or you will likely fail.

    So long as the student program is run in a process owned by another,
less
    privileged system user account, it shouldn't be able to modify the judge
    process which started it. For extra security you could even put the
judge
    program on its own file system mounted as read-only. The resulting
grades
    themselves would have to be stored elsewhere obviously, perhaps another
    server that has strict check constraints on what constitutes valid
input no
    different than any web service.

    You can have yet-another process or even a process on another server
    watchdog your judge system. Send it a few test programs every x minutes.
    If it fails to respond with the expected results in the expected time
frame
    - kill it and restart it. Test again. If it still fails consider it
hacked
    (or broken).

Do any graduate students or teacher's assistants owe you favors?
-Modulok-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20130921/ab7734ca/attachment.html>