[PyAR2] PyAR2 Digest, Vol 14, Issue 1

[W W]

On Thu, Dec 11, 2008 at 8:33 AM, <paul.hermeneutic at gmail.com> wrote:

> <snip>I assume that you are going to run the submitted code to see that
> correct results are produced.  As the challenges get more complex, it
> will become more difficult to ensure that no one has sent any
> malicious code.


Indeed - even with the ease of which you can /usually/ tell, that's not
always the case. One doesn't just have to use the os module to do some wonky
things. Although using a linux account with specific permissions (or a
virtual env) is an easy way to get most things secure, it's not foolproof.
And as most programmers know, if you make something foolproof, the universe
makes a better fool ;)


> Is there already, or could we create, an automated system where users
> could upload their source code?  I have seen such web sites for C/C++
> code.  The system must first ensure that the code is not malicious.
> This alone might be significant effort.  It would then then execute
> the code and assess the results.  Several metrics would be generated
> that would give an overview of the entry.
>
> Correctness of results


Depending on the challenge, unless the results are explicitly defined, you
can have problems even with this. After all, in my code I get the factors in
two different ways: by pair, and unique. I mentioned even the "tree" model.
Which one of these is correct or is each one? In the "assignment" it wasn't
explicitly declared.

Another question is what type should the results be returned as? A list of
tuples? A list of integers? A string? A dictionary? Unicode? The hex
representation? The more complex the problem, the more opportunities for
interpretation, when the results aren't explicitly declared. And as always,
sample results are rather helpful. If you say: output should be an ordered
list of unique integers, i.e.

print factors => [1, 3, 5, 15]

Then it leaves little question if print factors=> [{'1':'15', '15':'1',
'3':'5', '5':'3'}] is a correct result. While both are lists of unique
factors, only one is /only/ a list of unique factors.


> Speed performance

Number of source code lines (LOC)
> Ratio of comments to LOC


All rather easy to generate (although it also leaves unanswered whether a
docstring counts as a comment, what the count is for end of line codes, i.e.
print x    # This should output a list of unique factors), etc.


> Are there other metrics by which source code can be evaluated?


One requires the "human touch" - but ease of understanding (for the most
perl-ish award). If your code has zero comments, that could be a good thing
- if you programmed it well.

Consider the following two statements:

print x

print factors

One of them you probably have no idea what it means, the other one you can
guess that it probably prints the factors. You may not know what type
factors /is/, but you can be fairly sure that it's some type of aggregation
of factors.

Anyhow... probably about 3¢ worth there ;)
-Wayne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/mailman/private/pyar2/attachments/20081211/3eb42b4f/attachment.htm>