From metatracker at psf.upfronthosting.co.za  Mon Feb  1 21:22:28 2016
From: metatracker at psf.upfronthosting.co.za (Andrew Barnert)
Date: Tue, 02 Feb 2016 02:22:28 +0000
Subject: [Tracker-discuss] [issue439] Email notifictations not received for
 rietveld review
In-Reply-To: <1331683700.81.0.418164698072.issue439@psf.upfronthosting.co.za>
Message-ID: <1454379748.08.0.56538107531.issue439@psf.upfronthosting.co.za>


Andrew Barnert added the comment:

I suspect this is caused by either #554, #553 and its various dups/associates, or (if it's not caused by one of those two) #562.

My money is on #554. Rietveld messages use the developer's address for From. That makes it look like a forged message. That means it's guaranteed to be bounced if the developer is on a DMARC=reject domain, and quarantined if the developer is on a DMARC=quarantine domain, and that it will probably be marked as spam if the developer is on any of the major (and therefore heavily-forged) public hosting domains. Everything else from Python's servers, including bug tracker nosy messages, uses "Developer Name <reports at bugs.python.org>" instead of "Developer Name <developer at developer.address.example.com>", so it doesn't have this problem.

----------
nosy: +abarnert

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue439>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Fri Feb 12 17:04:30 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Fri, 12 Feb 2016 22:04:30 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
Message-ID: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>


New submission from Maciej Szulik:

Add github field to user profile.

----------
assignedto: ezio.melotti
files: gh.diff
messages: 3001
nosy: ezio.melotti, maciej.szulik
priority: feature
status: in-progress
title: Add github field in user profile

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________
-------------- next part --------------
diff --git a/html/user.item.html b/html/user.item.html
--- a/html/user.item.html
+++ b/html/user.item.html
@@ -105,7 +105,12 @@
 
  <tr tal:define="name string:homepage; label string:Homepage; value context/homepage">
   <th metal:use-macro="th_label">Home page</th>
-  <td><input name="organisation" metal:use-macro="normal_input"></td>
+  <td><input name="homepage" metal:use-macro="normal_input"></td>
+ </tr>
+
+ <tr tal:define="name string:github; label string:GitHub Name; value context/github">
+  <th metal:use-macro="th_label">GitHub Name</th>
+  <td><input name="github" metal:use-macro="normal_input"></td>
  </tr>
 
  <tr>
diff --git a/schema.py b/schema.py
--- a/schema.py
+++ b/schema.py
@@ -96,7 +96,8 @@
              contrib_form_date=Date(),
              openids=String(), # space separated list
              iscommitter=Boolean(),
-             homepage=String()
+             homepage=String(),
+             github=String()
              )
 user.setkey("username")
 db.security.addPermission(name='Register', klass='user',

From metatracker at psf.upfronthosting.co.za  Fri Feb 12 23:50:11 2016
From: metatracker at psf.upfronthosting.co.za (Ezio Melotti)
Date: Sat, 13 Feb 2016 04:50:11 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1455339011.37.0.902212177099.issue579@psf.upfronthosting.co.za>


Ezio Melotti added the comment:

Thanks for the patch!
Have you checked that users can edit their own github username and that they can not edit the username of other users?
I'm not sure what the permissions for viewing the username should be, maybe it should be hidden to anonymous and regular users and be viewable to developers and coordinators.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Mon Feb 15 05:46:07 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Mon, 15 Feb 2016 10:46:07 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1455533167.52.0.188345468501.issue579@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

No, I have not :/ Lemme you give it a try and check those details.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Mon Feb 15 17:09:35 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Mon, 15 Feb 2016 22:09:35 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1455574175.36.0.121782544427.issue579@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

Updated so that users can see each others gh name, but edit only his own. By the way I've fixed homepage settings, which weren't correct either.

It would be also worth extending search with gh name on user list, but if we need that I'm gonna need your help

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________
-------------- next part --------------
diff --git a/html/user.item.html b/html/user.item.html
--- a/html/user.item.html
+++ b/html/user.item.html
@@ -105,7 +105,12 @@
 
  <tr tal:define="name string:homepage; label string:Homepage; value context/homepage">
   <th metal:use-macro="th_label">Home page</th>
-  <td><input name="organisation" metal:use-macro="normal_input"></td>
+  <td><input name="homepage" metal:use-macro="normal_input"></td>
+ </tr>
+
+ <tr tal:define="name string:github; label string:GitHub Name; value context/github">
+  <th metal:use-macro="th_label">GitHub Name</th>
+  <td><input name="github" metal:use-macro="normal_input"></td>
  </tr>
 
  <tr>
diff --git a/schema.py b/schema.py
--- a/schema.py
+++ b/schema.py
@@ -96,7 +96,8 @@
              contrib_form_date=Date(),
              openids=String(), # space separated list
              iscommitter=Boolean(),
-             homepage=String()
+             homepage=String(),
+             github=String()
              )
 user.setkey("username")
 db.security.addPermission(name='Register', klass='user',
@@ -349,7 +350,7 @@
 p = db.security.addPermission(name='View', klass='user',
         properties=('id', 'username', 'address', 'realname', 'phone',
                     'organisation', 'alternate_addresses', 'timezone',
-                    'roles', 'contrib_form', 'iscommitter', 'homepage'))
+                    'roles', 'contrib_form', 'iscommitter', 'homepage', 'github'))
 db.security.addPermissionToRole('User', p)
 db.security.addPermissionToRole('Developer', p)
 # Coordinator may view all user properties.
@@ -374,8 +375,8 @@
                 'address', 'realname',
                 'phone', 'organisation',
                 'alternate_addresses',
-                'queries',
-                'timezone')) # Note: 'roles' excluded - users should not be able to edit their own roles.
+                'queries', 'timezone',
+                'homepage', 'github')) # Note: 'roles' excluded - users should not be able to edit their own roles.
                              # Also excluded: contrib_form, contrib_form_date, iscommitter
 for r in 'User', 'Developer':
     db.security.addPermissionToRole(r, p)

From metatracker at psf.upfronthosting.co.za  Fri Feb 19 17:45:52 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Fri, 19 Feb 2016 22:45:52 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1455921952.0.0.946622593634.issue579@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

I did initial check whether patch from https://bitbucket.org/kinggreedy1991/roundup-bpo/branc
hes/compare/kinggreedy1991/roundup-bpo:tip%0Dezio_melotti/roundup-bpo:bugs.python.org#diff works correctly. From initial glimpses it's working ok, though I reserve some more time to have a deeper analysis, still.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Sun Feb 21 15:52:59 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Sun, 21 Feb 2016 20:52:59 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1456087979.59.0.56625364967.issue579@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

I've added the search for github user name according to the discussion on IRC. I'm including the patch, but for the record, it's also being available at https://bitbucket.org/soltysh/python-dev/branch/github

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________
-------------- next part --------------
diff --git a/extensions/pydevutils.py b/extensions/pydevutils.py
--- a/extensions/pydevutils.py
+++ b/extensions/pydevutils.py
@@ -20,6 +20,11 @@
     db = request.client.db
     return 'Coordinator' in db.user.get(user, 'roles')

+def is_developer(request):
+    user = request.client.userid
+    db = request.client.db
+    return 'Developer' in db.user.get(user, 'roles')
+
 def clean_ok_message(ok_message):
     """Remove nosy_count and message_count from the ok_message."""
     pattern = '\s*(?:nosy|message)_count,|,\s*(?:nosy|message)_count(?= edited)'
@@ -60,6 +65,7 @@
 def init(instance):
     instance.registerUtil('is_history_ok', is_history_ok)
     instance.registerUtil('is_coordinator', is_coordinator)
+    instance.registerUtil('is_developer', is_developer)
     instance.registerUtil('clean_ok_message', clean_ok_message)
     instance.registerUtil('issueid_and_action_from_class',
                           issueid_and_action_from_class)
diff --git a/html/style.css b/html/style.css
--- a/html/style.css
+++ b/html/style.css
@@ -177,6 +177,10 @@
   white-space: nowrap;
 }

+table.form th:last-child {
+  border-right: #ddd solid 1px;
+}
+
 table.form th.header {
   font-weight: bold;
   text-align: left;
diff --git a/html/user.index.html b/html/user.index.html
--- a/html/user.index.html
+++ b/html/user.index.html
@@ -19,7 +19,7 @@
 <table class="form" tal:define="
        search_input templates/page/macros/search_input;">

-   <tr><th class="header" colspan="5">Search for users</th></tr>
+   <tr><th class="header" colspan="7">Search for users</th></tr>
    <tr>
        <th class="header">Username</th>
        <td tal:define="name string:username">
@@ -33,6 +33,14 @@
                          name name;
                          id name"/>
        </td>
+       <tal:if condition="python:utils.is_developer(request)">
+       <th class="header">Github</th>
+       <td tal:define="name string:github">
+           <input tal:attributes="value python:request.form.getvalue(name) or nothing;
+                         name name;
+                         id name"/>
+       </td>
+       </tal:if>

        <td><input class="form-small" type="submit" value="Search" i18n:attributes="value"/></td>
    </tr>
diff --git a/html/user.item.html b/html/user.item.html
--- a/html/user.item.html
+++ b/html/user.item.html
@@ -105,7 +105,12 @@

  <tr tal:define="name string:homepage; label string:Homepage; value context/homepage">
   <th metal:use-macro="th_label">Home page</th>
-  <td><input name="organisation" metal:use-macro="normal_input"></td>
+  <td><input name="homepage" metal:use-macro="normal_input"></td>
+ </tr>
+
+ <tr tal:define="name string:github; label string:GitHub Name; value context/github">
+  <th metal:use-macro="th_label">GitHub Name</th>
+  <td><input name="github" metal:use-macro="normal_input"></td>
  </tr>

  <tr>
diff --git a/schema.py b/schema.py
--- a/schema.py
+++ b/schema.py
@@ -96,7 +96,8 @@
              contrib_form_date=Date(),
              openids=String(), # space separated list
              iscommitter=Boolean(),
-             homepage=String()
+             homepage=String(),
+             github=String()
              )
 user.setkey("username")
 db.security.addPermission(name='Register', klass='user',
@@ -349,7 +350,7 @@
 p = db.security.addPermission(name='View', klass='user',
         properties=('id', 'username', 'address', 'realname', 'phone',
                     'organisation', 'alternate_addresses', 'timezone',
-                    'roles', 'contrib_form', 'iscommitter', 'homepage'))
+                    'roles', 'contrib_form', 'iscommitter', 'homepage', 'github'))
 db.security.addPermissionToRole('User', p)
 db.security.addPermissionToRole('Developer', p)
 # Coordinator may view all user properties.
@@ -374,8 +375,8 @@
                 'address', 'realname',
                 'phone', 'organisation',
                 'alternate_addresses',
-                'queries',
-                'timezone')) # Note: 'roles' excluded - users should not be able to edit their own roles.
+                'queries', 'timezone',
+                'homepage', 'github')) # Note: 'roles' excluded - users should not be able to edit their own roles.
                              # Also excluded: contrib_form, contrib_form_date, iscommitter
 for r in 'User', 'Developer':
     db.security.addPermissionToRole(r, p)

From metatracker at psf.upfronthosting.co.za  Sun Feb 21 17:00:38 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Sun, 21 Feb 2016 22:00:38 +0000
Subject: [Tracker-discuss] [issue579] Add github field in user profile
In-Reply-To: <1455314670.59.0.81694080109.issue579@psf.upfronthosting.co.za>
Message-ID: <1456092038.58.0.480290618129.issue579@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

After further investigating into the rest patch that I temporarily applied here https://bitbucket.org/soltysh/roundup/branch/rest my current findings are:
- we return too much information on GET, any authenticated user gets ALL of the details from any user, including password (hashed but still it's there). 
- I'm not clear how the POST works, it does not accept JSON as input but rather tries to parse incoming arguments, which fails.

I'll try cutting this patch into something smaller. In the first place I'd suggest removing all modifications actions and start with just read ones. Ezio I'll ping you tomorrow on IRC to discuss this more.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue579>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Tue Feb 23 05:19:35 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Tue, 23 Feb 2016 10:19:35 +0000
Subject: [Tracker-discuss] [issue580] CSV Injection Vulnerability
Message-ID: <1456222775.65.0.0851572486272.issue580@psf.upfronthosting.co.za>


New submission from Maciej Szulik:

Copied from http://bugs.python.org/issue26399:

The "Download as CSV " feature of bugs.python.org does not properly "escape" fields. This allows an adversary to turn a field into active content so when we download the csv and opens it, the active content gets executed. Here is more information about this issue:
http://www.contextis.com/resources/blog/comma-separated-vulnerabilities/

Steps to Reproduce.
1. Enter the title with the payload : -2+3+cmd|' /C calc'!A0
2. Download the bugs as CSV
3. Open it with excel and Calc will get prompted.

Depending upon the system user privileges, an attacker can perform various tasks using the same.
If the user is with high privilege, it is easy to change the system password as mentioned below
-2+3+cmd|' /C net user administrator lol at 123'!A0

Mitigations:
Ensure all fields are properly "escaped" before returning the CSV file to the user.

Regards,
Acid

Impact of this one is high, as download as CSV is present for guest user as well. Means anyone can download the bugs using "Download as CSV " function and as the file is downloaded from the trusted resource so the possibility is high the code will get executed.

----------
messages: 3008
nosy: maciej.szulik
priority: urgent
status: unread
title: CSV Injection Vulnerability

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue580>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Tue Feb 23 08:20:39 2016
From: metatracker at psf.upfronthosting.co.za (R David Murray)
Date: Tue, 23 Feb 2016 13:20:39 +0000
Subject: [Tracker-discuss] [issue580] CSV Injection Vulnerability
In-Reply-To: <1456222775.65.0.0851572486272.issue580@psf.upfronthosting.co.za>
Message-ID: <1456233639.58.0.0102772476484.issue580@psf.upfronthosting.co.za>


R David Murray added the comment:

This should be reported to Roundup upstream.  The fix should be simple (just changing the csv dialect), so it doesn't really matter who develops the patch as long as both upstream and we apply it :)

----------
nosy: +r.david.murray
status: unread -> chatting

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue580>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Sat Feb 27 17:44:49 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Sat, 27 Feb 2016 22:44:49 +0000
Subject: [Tracker-discuss] [issue581] Add rest endpoint to roundup
Message-ID: <1456613089.91.0.963732827198.issue581@psf.upfronthosting.co.za>


New submission from Maciej Szulik:

This issue is meant to address new feature which is adding REST endpoint to roundup.

----------
assignedto: maciej.szulik
messages: 3010
nosy: ezio.melotti, maciej.szulik
priority: feature
status: in-progress
title: Add rest endpoint to roundup

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue581>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Sat Feb 27 18:10:34 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Sat, 27 Feb 2016 23:10:34 +0000
Subject: [Tracker-discuss] [issue581] Add rest endpoint to roundup
In-Reply-To: <1456613089.91.0.963732827198.issue581@psf.upfronthosting.co.za>
Message-ID: <1456614634.4.0.953116530371.issue581@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

This partially derived from #579. 

Previous problem I've described was that any user was able to see others full details, including password. I've finally nailed what was the problem when calling hasPermissions item_id was not passed. I've fixed that and added some more tests. 

As discussed previously I've removed all PUT/POST/PATCH actions leaving just GETs.

My work is under https://bitbucket.org/soltysh/roundup/branch/rest 
Yet I'm attaching current patch here.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue581>
_______________________________________________________
-------------- next part --------------
diff --git a/roundup/cgi/client.py b/roundup/cgi/client.py
--- a/roundup/cgi/client.py
+++ b/roundup/cgi/client.py
@@ -22,6 +22,7 @@
 from roundup.mailer import Mailer, MessageSendError, encode_quopri
 from roundup.cgi import accept_language
 from roundup import xmlrpc
+from roundup import rest
 
 from roundup.anypy.cookie_ import CookieError, BaseCookie, SimpleCookie, \
     get_cookie_date
@@ -378,6 +379,8 @@
         try:
             if self.path == 'xmlrpc':
                 self.handle_xmlrpc()
+            elif self.path == 'rest' or self.path[:5] == 'rest/':
+                self.handle_rest()
             else:
                 self.inner_main()
         finally:
@@ -419,6 +422,24 @@
         self.setHeader("Content-Length", str(len(output)))
         self.write(output)
 
+    def handle_rest(self):
+        # Set the charset and language
+        self.determine_charset()
+        self.determine_language()
+        # Open the database as the correct user.
+        # TODO: add everything to RestfulDispatcher
+        self.determine_user()
+        self.check_anonymous_access()
+
+        # Call rest library to handle the request
+        handler = rest.RestfulInstance(self, self.db)
+        output = handler.dispatch(self.env['REQUEST_METHOD'], self.path,
+                                  self.form)
+
+        # self.setHeader("Content-Type", "text/xml")
+        self.setHeader("Content-Length", str(len(output)))
+        self.write(output)
+
     def add_ok_message(self, msg, escape=True):
         add_message(self._ok_message, msg, escape)
 
diff --git a/roundup/rest.py b/roundup/rest.py
new file mode 100644
--- /dev/null
+++ b/roundup/rest.py
@@ -0,0 +1,551 @@
+"""
+Restful API for Roundup
+
+This module is free software, you may redistribute it
+and/or modify under the same terms as Python.
+"""
+
+import urlparse
+import os
+import json
+import pprint
+import sys
+import time
+import traceback
+import re
+
+from roundup import hyperdb
+from roundup import date
+from roundup import actions
+from roundup.exceptions import *
+from roundup.cgi.exceptions import *
+
+
+def _data_decorator(func):
+    """Wrap the returned data into an object."""
+    def format_object(self, *args, **kwargs):
+        # get the data / error from function
+        try:
+            code, data = func(self, *args, **kwargs)
+        except NotFound, msg:
+            code = 404
+            data = msg
+        except IndexError, msg:
+            code = 404
+            data = msg
+        except Unauthorised, msg:
+            code = 403
+            data = msg
+        except UsageError, msg:
+            code = 400
+            data = msg
+        except (AttributeError, Reject), msg:
+            code = 405
+            data = msg
+        except ValueError, msg:
+            code = 409
+            data = msg
+        except NotImplementedError:
+            code = 402  # nothing to pay, just a mark for debugging purpose
+            data = 'Method under development'
+        except:
+            exc, val, tb = sys.exc_info()
+            code = 400
+            ts = time.ctime()
+            data = '%s: An error occurred. Please check the server log' \
+                   ' for more information.' % ts
+            # out to the logfile
+            print 'EXCEPTION AT', ts
+            traceback.print_exc()
+
+        # decorate it
+        self.client.response_code = code
+        if code >= 400:  # any error require error format
+            result = {
+                'error': {
+                    'status': code,
+                    'msg': data
+                }
+            }
+        else:
+            result = {
+                'data': data
+            }
+        return result
+    return format_object
+
+
+def parse_accept_header(accept):
+    """
+    Parse the Accept header *accept*, returning a list with 3-tuples of
+    [(str(media_type), dict(params), float(q_value)),] ordered by q values.
+
+    If the accept header includes vendor-specific types like::
+        application/vnd.yourcompany.yourproduct-v1.1+json
+
+    It will actually convert the vendor and version into parameters and
+    convert the content type into `application/json` so appropriate content
+    negotiation decisions can be made.
+
+    Default `q` for values that are not specified is 1.0
+
+    # Based on https://gist.github.com/samuraisam/2714195
+    # Also, based on a snipped found in this project:
+    #   https://github.com/martinblech/mimerender
+    """
+    result = []
+    for media_range in accept.split(","):
+        parts = media_range.split(";")
+        media_type = parts.pop(0).strip()
+        media_params = []
+        # convert vendor-specific content types into something useful (see
+        # docstring)
+        typ, subtyp = media_type.split('/')
+        # check for a + in the sub-type
+        if '+' in subtyp:
+            # if it exists, determine if the subtype is a vendor-specific type
+            vnd, sep, extra = subtyp.partition('+')
+            if vnd.startswith('vnd'):
+                # and then... if it ends in something like "-v1.1" parse the
+                # version out
+                if '-v' in vnd:
+                    vnd, sep, rest = vnd.rpartition('-v')
+                    if len(rest):
+                        # add the version as a media param
+                        try:
+                            version = media_params.append(('version',
+                                                           float(rest)))
+                        except ValueError:
+                            version = 1.0  # could not be parsed
+                # add the vendor code as a media param
+                media_params.append(('vendor', vnd))
+                # and re-write media_type to something like application/json so
+                # it can be used usefully when looking up emitters
+                media_type = '{}/{}'.format(typ, extra)
+        q = 1.0
+        for part in parts:
+            (key, value) = part.lstrip().split("=", 1)
+            key = key.strip()
+            value = value.strip()
+            if key == "q":
+                q = float(value)
+            else:
+                media_params.append((key, value))
+        result.append((media_type, dict(media_params), q))
+    result.sort(lambda x, y: -cmp(x[2], y[2]))
+    return result
+
+
+class Routing(object):
+    __route_map = {}
+    __var_to_regex = re.compile(r"<:(\w+)>")
+    url_to_regex = r"([\w.\-~!$&'()*+,;=:@\%%]+)"
+
+    @classmethod
+    def route(cls, rule, methods='GET'):
+        """A decorator that is used to register a view function for a
+        given URL rule:
+            @self.route('/')
+            def index():
+                return 'Hello World'
+
+        rest/ will be added to the beginning of the url string
+
+        Args:
+            rule (string): the URL rule
+            methods (string or tuple or list): the http method
+        """
+        # strip the '/' character from rule string
+        rule = rule.strip('/')
+
+        # add 'rest/' to the rule string
+        if not rule.startswith('rest/'):
+            rule = '^rest/' + rule + '$'
+
+        if isinstance(methods, basestring):  # convert string to tuple
+            methods = (methods,)
+        methods = set(item.upper() for item in methods)
+
+        # convert a rule to a compiled regex object
+        # so /data/<:class>/<:id> will become
+        #    /data/([charset]+)/([charset]+)
+        # and extract the variable names to a list [(class), (id)]
+        func_vars = cls.__var_to_regex.findall(rule)
+        rule = re.compile(cls.__var_to_regex.sub(cls.url_to_regex, rule))
+
+        # then we decorate it:
+        # route_map[regex][method] = func
+        def decorator(func):
+            rule_route = cls.__route_map.get(rule, {})
+            func_obj = {
+                'func': func,
+                'vars': func_vars
+            }
+            for method in methods:
+                rule_route[method] = func_obj
+            cls.__route_map[rule] = rule_route
+            return func
+        return decorator
+
+    @classmethod
+    def execute(cls, instance, path, method, input):
+        # format the input
+        path = path.strip('/').lower()
+        method = method.upper()
+
+        # find the rule match the path
+        # then get handler match the method
+        for path_regex in cls.__route_map:
+            match_obj = path_regex.match(path)
+            if match_obj:
+                try:
+                    func_obj = cls.__route_map[path_regex][method]
+                except KeyError:
+                    raise Reject('Method %s not allowed' % method)
+
+                # retrieve the vars list and the function caller
+                list_vars = func_obj['vars']
+                func = func_obj['func']
+
+                # zip the varlist into a dictionary, and pass it to the caller
+                args = dict(zip(list_vars, match_obj.groups()))
+                args['input'] = input
+                return func(instance, **args)
+        raise NotFound('Nothing matches the given URI')
+
+
+class RestfulInstance(object):
+    """The RestfulInstance performs REST request from the client"""
+
+    __default_patch_op = "replace"  # default operator for PATCH method
+    __accepted_content_type = {
+        "application/json": "json",
+        "*/*": "json"
+        # "application/xml": "xml"
+    }
+    __default_accept_type = "json"
+
+    def __init__(self, client, db):
+        self.client = client
+        self.db = db
+        self.translator = client.translator
+        self.actions = client.instance.actions.copy()
+        self.actions.update({'retire': actions.Retire})
+
+        protocol = 'http'
+        host = self.client.env['HTTP_HOST']
+        tracker = self.client.env['TRACKER_NAME']
+        self.base_path = '%s://%s/%s/rest' % (protocol, host, tracker)
+        self.data_path = self.base_path + '/data'
+
+    def error_obj(self, status, msg, source=None):
+        """Return an error object"""
+        self.client.response_code = status
+        result = {
+            'error': {
+                'status': status,
+                'msg': msg
+            }
+        }
+        if source is not None:
+            result['error']['source'] = source
+
+        return result
+
+    @Routing.route("/data/<:class_name>", 'GET')
+    @_data_decorator
+    def get_collection(self, class_name, input):
+        """GET resource from class URI.
+
+        This function returns only items have View permission
+        class_name should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            list: list of reference item in the class
+                id: id of the object
+                link: path to the object
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % self.db.classes)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name
+        ):
+            raise Unauthorised('Permission to view %s denied' % class_name)
+
+        class_obj = self.db.getclass(class_name)
+        class_path = '%s/%s/' % (self.data_path, class_name)
+
+        # Handle filtering and pagination
+        filter_props = {}
+        page = {
+            'size': None,
+            'index': None
+        }
+        for form_field in input.value:
+            key = form_field.name
+            value = form_field.value
+            if key.startswith("where_"):  # serve the filter purpose
+                key = key[6:]
+                filter_props[key] = [
+                    getattr(self.db, key).lookup(p)
+                    for p in value.split(",")
+                ]
+            elif key.startswith("page_"):  # serve the paging purpose
+                key = key[5:]
+                value = int(value)
+                page[key] = value
+
+        if not filter_props:
+            obj_list = class_obj.list()
+        else:
+            obj_list = class_obj.filter(None, filter_props)
+
+        # extract result from data
+        result = [
+            {'id': item_id, 'link': class_path + item_id}
+            for item_id in obj_list
+            if self.db.security.hasPermission(
+                'View', self.db.getuid(), class_name, itemid=item_id
+            )
+        ]
+
+        # pagination
+        if page['size'] is not None and page['index'] is not None:
+            page_start = max(page['index'] * page['size'], 0)
+            page_end = min(page_start + page['size'], len(result))
+            result = result[page_start:page_end]
+
+        self.client.setHeader("X-Count-Total", str(len(result)))
+        return 200, result
+
+    @Routing.route("/data/<:class_name>/<:item_id>", 'GET')
+    @_data_decorator
+    def get_element(self, class_name, item_id, input):
+        """GET resource from object URI.
+
+        This function returns only properties have View permission
+        class_name and item_id should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            item_id (string): id of the resource (Ex: 12, 15)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            dict: a dictionary represents the object
+                id: id of the object
+                type: class name of the object
+                link: link to the object
+                attributes: a dictionary represent the attributes of the object
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name, itemid=item_id
+        ):
+            raise Unauthorised(
+                'Permission to view %s/%s denied' % (class_name, item_id)
+            )
+
+        class_obj = self.db.getclass(class_name)
+        props = None
+        for form_field in input.value:
+            key = form_field.name
+            value = form_field.value
+            if key == "fields":
+                props = value.split(",")
+
+        if props is None:
+            props = class_obj.properties.keys()
+
+        props.sort()  # sort properties
+
+        try:
+            result = [
+                (prop_name, class_obj.get(item_id, prop_name))
+                for prop_name in props
+                if self.db.security.hasPermission(
+                    'View', self.db.getuid(), class_name, prop_name, item_id
+                )
+            ]
+        except KeyError, msg:
+            raise UsageError("%s field not valid" % msg)
+        result = {
+            'id': item_id,
+            'type': class_name,
+            'link': '%s/%s/%s' % (self.data_path, class_name, item_id),
+            'attributes': dict(result)
+        }
+
+        return 200, result
+
+    @Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'GET')
+    @_data_decorator
+    def get_attribute(self, class_name, item_id, attr_name, input):
+        """GET resource from attribute URI.
+
+        This function returns only attribute has View permission
+        class_name should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            item_id (string): id of the resource (Ex: 12, 15)
+            attr_name (string): attribute of the resource (Ex: title, nosy)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            list: a dictionary represents the attribute
+                id: id of the object
+                type: class name of the attribute
+                link: link to the attribute
+                data: data of the requested attribute
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name, attr_name, item_id
+        ):
+            raise Unauthorised(
+                'Permission to view %s/%s %s denied' %
+                (class_name, item_id, attr_name)
+            )
+
+        class_obj = self.db.getclass(class_name)
+        data = class_obj.get(item_id, attr_name)
+        result = {
+            'id': item_id,
+            'type': type(data),
+            'link': "%s/%s/%s/%s" %
+                    (self.data_path, class_name, item_id, attr_name),
+            'data': data
+        }
+
+        return 200, result
+
+    @Routing.route("/data/<:class_name>", 'OPTIONS')
+    @_data_decorator
+    def options_collection(self, class_name, input):
+        """OPTION return the HTTP Header for the class uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        return 204, ""
+
+    @Routing.route("/data/<:class_name>/<:item_id>", 'OPTIONS')
+    @_data_decorator
+    def options_element(self, class_name, item_id, input):
+        """OPTION return the HTTP Header for the object uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        self.client.setHeader(
+            "Accept-Patch",
+            "application/x-www-form-urlencoded, multipart/form-data"
+        )
+        return 204, ""
+
+    @Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'OPTIONS')
+    @_data_decorator
+    def option_attribute(self, class_name, item_id, attr_name, input):
+        """OPTION return the HTTP Header for the attribute uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        self.client.setHeader(
+            "Accept-Patch",
+            "application/x-www-form-urlencoded, multipart/form-data"
+        )
+        return 204, ""
+
+    def dispatch(self, method, uri, input):
+        """format and process the request"""
+        # if X-HTTP-Method-Override is set, follow the override method
+        headers = self.client.request.headers
+        method = headers.getheader('X-HTTP-Method-Override') or method
+
+        # parse Accept header and get the content type
+        accept_header = parse_accept_header(headers.getheader('Accept'))
+        accept_type = "invalid"
+        for part in accept_header:
+            if part[0] in self.__accepted_content_type:
+                accept_type = self.__accepted_content_type[part[0]]
+
+        # get the request format for response
+        # priority : extension from uri (/rest/issue.json),
+        #            header (Accept: application/json, application/xml)
+        #            default (application/json)
+        ext_type = os.path.splitext(urlparse.urlparse(uri).path)[1][1:]
+        data_type = ext_type or accept_type or self.__default_accept_type
+
+        # check for pretty print
+        try:
+            pretty_output = input['pretty'].value.lower() == "true"
+        except KeyError:
+            pretty_output = False
+
+        # add access-control-allow-* to support CORS
+        self.client.setHeader("Access-Control-Allow-Origin", "*")
+        self.client.setHeader(
+            "Access-Control-Allow-Headers",
+            "Content-Type, Authorization, X-HTTP-Method-Override"
+        )
+        self.client.setHeader(
+            "Allow",
+            "HEAD, OPTIONS, GET"
+        )
+        self.client.setHeader(
+            "Access-Control-Allow-Methods",
+            "HEAD, OPTIONS, GET"
+        )
+
+        # Call the appropriate method
+        try:
+            output = Routing.execute(self, uri, method, input)
+        except NotFound, msg:
+            output = self.error_obj(404, msg)
+        except Reject, msg:
+            output = self.error_obj(405, msg)
+
+        # Format the content type
+        if data_type.lower() == "json":
+            self.client.setHeader("Content-Type", "application/json")
+            if pretty_output:
+                indent = 4
+            else:
+                indent = None
+            output = RoundupJSONEncoder(indent=indent).encode(output)
+        else:
+            self.client.response_code = 406
+            output = "Content type is not accepted by client"
+
+        return output
+
+
+class RoundupJSONEncoder(json.JSONEncoder):
+    """RoundupJSONEncoder overrides the default JSONEncoder to handle all
+    types of the object without returning any error"""
+    def default(self, obj):
+        try:
+            result = json.JSONEncoder.default(self, obj)
+        except TypeError:
+            result = str(obj)
+        return result
diff --git a/roundup/scripts/roundup_server.py b/roundup/scripts/roundup_server.py
--- a/roundup/scripts/roundup_server.py
+++ b/roundup/scripts/roundup_server.py
@@ -251,7 +251,7 @@
         else:
             return self.run_cgi()
 
-    do_GET = do_POST = do_HEAD = run_cgi_outer
+    do_GET = do_POST = do_HEAD = do_PUT = do_DELETE = do_PATCH = do_OPTIONS = run_cgi_outer
 
     def index(self):
         ''' Print up an index of the available trackers
diff --git a/test/test_rest.py b/test/test_rest.py
new file mode 100644
--- /dev/null
+++ b/test/test_rest.py
@@ -0,0 +1,301 @@
+import unittest
+import os
+import shutil
+import errno
+
+from roundup.cgi.exceptions import *
+from roundup import password, hyperdb
+from roundup.rest import RestfulInstance
+from roundup.backends import list_backends
+from roundup.cgi import client
+import random
+
+import db_test_base
+
+NEEDS_INSTANCE = 1
+
+
+class TestCase(unittest.TestCase):
+
+    backend = None
+
+    def setUp(self):
+        self.dirname = '_test_rest'
+        # set up and open a tracker
+        self.instance = db_test_base.setupTracker(self.dirname, self.backend)
+
+        # open the database
+        self.db = self.instance.open('admin')
+
+        # Get user id (user4 maybe). Used later to get data from db.
+        self.joeid = self.db.user.create(
+            username='joe',
+            password=password.Password('random'),
+            address='random at home.org',
+            realname='Joe Random',
+            roles='User'
+        )
+
+        self.db.commit()
+        self.db.close()
+
+        env = {
+            'PATH_INFO': 'http://localhost/rounduptest/rest/',
+            'HTTP_HOST': 'localhost',
+            'TRACKER_NAME': 'rounduptest'
+        }
+        self.dummy_client = client.Client(self.instance, None, env, [], None)
+        self.empty_form = cgi.FieldStorage()
+
+    def tearDown(self):
+        self.db.close()
+        try:
+            shutil.rmtree(self.dirname)
+        except OSError, error:
+            if error.errno not in (errno.ENOENT, errno.ESRCH):
+                raise
+
+    def open(self, user='joe'):
+        """
+        Opens database as given user.
+        """
+        self.db = self.instance.open(user)
+
+        self.db.tx_Source = 'web'
+
+        self.db.issue.addprop(tx_Source=hyperdb.String())
+        self.db.msg.addprop(tx_Source=hyperdb.String())
+
+        self.db.post_init()
+
+        thisdir = os.path.dirname(__file__)
+        vars = {}
+        execfile(os.path.join(thisdir, "tx_Source_detector.py"), vars)
+        vars['init'](self.db)
+        self.server = RestfulInstance(self.dummy_client, self.db)
+
+    def testGetSelf(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # Retrieve all three users.
+        results = self.server.get_collection('user', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), 3)
+
+        # Obtain data for 'joe'.
+        results = self.server.get_element('user', self.joeid, self.empty_form)
+        results = results['data']
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(results['attributes']['username'], 'joe')
+        self.assertEqual(results['attributes']['realname'], 'Joe Random')
+
+        # Obtain data for 'joe'.
+        results = self.server.get_attribute(
+            'user', self.joeid, 'username', self.empty_form
+        )
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(results['data']['data'], 'joe')
+
+    def testGetAdmin(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        results = self.server.get_element('user', '1', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertFalse(results['data']['attributes'].has_key('openids'))
+        self.assertFalse(results['data']['attributes'].has_key('password'))
+
+    def testGetSelfAttribute(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        results = self.server.get_attribute('user', self.joeid, 'password', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIsNotNone(results['data']['data'])
+
+    def testGetAdminAttribute(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        # Retrieve all three users.
+        results = self.server.get_attribute('user', '1', 'password', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 403)
+
+    def testGetAnonymous(self):
+        """
+        Anonymous should not get users.
+        """
+        self.open('anonymous')
+        results = self.server.get_collection('user', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 403)
+
+    def testFilter(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # create sample data
+        try:
+            self.db.status.create(name='open')
+        except ValueError:
+            pass
+        try:
+            self.db.status.create(name='closed')
+        except ValueError:
+            pass
+        try:
+            self.db.priority.create(name='normal')
+        except ValueError:
+            pass
+        try:
+            self.db.priority.create(name='critical')
+        except ValueError:
+            pass
+        self.db.issue.create(
+            title='foo4',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('critical')
+        )
+        self.db.issue.create(
+            title='foo1',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_open_norm = self.db.issue.create(
+            title='foo2',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_closed_norm = self.db.issue.create(
+            title='foo3',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_closed_crit = self.db.issue.create(
+            title='foo4',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('critical')
+        )
+        issue_open_crit = self.db.issue.create(
+            title='foo5',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('critical')
+        )
+        base_path = self.dummy_client.env['PATH_INFO'] + 'data/issue/'
+
+        # Retrieve all issue status=open
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'open')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_open_norm), results['data'])
+        self.assertIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(
+            get_obj(base_path, issue_closed_norm), results['data']
+        )
+
+        # Retrieve all issue status=closed and priority=critical
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'closed'),
+            cgi.MiniFieldStorage('where_priority', 'critical')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_closed_crit), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(
+            get_obj(base_path, issue_closed_norm), results['data']
+        )
+
+        # Retrieve all issue status=closed and priority=normal,critical
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'closed'),
+            cgi.MiniFieldStorage('where_priority', 'normal,critical')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_closed_crit), results['data'])
+        self.assertIn(get_obj(base_path, issue_closed_norm), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_norm), results['data'])
+
+    def testPagination(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # create sample data
+        for i in range(0, random.randint(5, 10)):
+            self.db.issue.create(title='foo' + str(i))
+
+        # Retrieving all the issues
+        results = self.server.get_collection('issue', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        total_length = len(results['data'])
+
+        # Pagination will be 70% of the total result
+        page_size = total_length * 70 // 100
+        page_zero_expected = page_size
+        page_one_expected = total_length - page_zero_expected
+
+        # Retrieve page 0
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 0)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), page_zero_expected)
+
+        # Retrieve page 1
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 1)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), page_one_expected)
+
+        # Retrieve page 2
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 2)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), 0)
+
+
+def get_obj(path, id):
+    return {
+        'id': id,
+        'link': path + id
+    }
+
+
+def test_suite():
+    suite = unittest.TestSuite()
+    for l in list_backends():
+        dct = dict(backend=l)
+        subcls = type(TestCase)('TestCase_%s' % l, (TestCase,), dct)
+        suite.addTest(unittest.makeSuite(subcls))
+    return suite
+
+if __name__ == '__main__':
+    runner = unittest.TextTestRunner()
+    unittest.main(testRunner=runner)

From metatracker at psf.upfronthosting.co.za  Sat Feb 27 18:13:08 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Sat, 27 Feb 2016 23:13:08 +0000
Subject: [Tracker-discuss] [issue581] Add rest endpoint to roundup
In-Reply-To: <1456613089.91.0.963732827198.issue581@psf.upfronthosting.co.za>
Message-ID: <1456614788.8.0.0310398724873.issue581@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

Still needs fixing:
- auth error

Additionally, if we want full-REST support we need:
- decide if we want to have delete at all
- post using json, not just form-data, since we return GET data with json
- fix validation when post-ing

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue581>
_______________________________________________________

From brett at python.org  Sat Feb 27 11:58:02 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Feb 2016 16:58:02 +0000
Subject: [Tracker-discuss] This mailing list is now under the PSF CoC
Message-ID: <CAP1=2W4nmf61Q2QW0Ywu1-0SkXDaG6bx=RPZGV1oDhJdMW7iwA@mail.gmail.com>

I have updated the list's info page as well as the footer for this mailing
list to denote this fact.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tracker-discuss/attachments/20160227/110d25c5/attachment.html>

From metatracker at psf.upfronthosting.co.za  Mon Feb 29 16:43:53 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Mon, 29 Feb 2016 21:43:53 +0000
Subject: [Tracker-discuss] [issue581] Add rest endpoint to roundup
In-Reply-To: <1456613089.91.0.963732827198.issue581@psf.upfronthosting.co.za>
Message-ID: <1456782233.76.0.833927746742.issue581@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

The auth error seems to be minor with the fact that passing non-existent auth data gives me full access to any data. Working on it now...

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue581>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Mon Feb 29 17:14:53 2016
From: metatracker at psf.upfronthosting.co.za (Maciej Szulik)
Date: Mon, 29 Feb 2016 22:14:53 +0000
Subject: [Tracker-discuss] [issue581] Add rest endpoint to roundup
In-Reply-To: <1456613089.91.0.963732827198.issue581@psf.upfronthosting.co.za>
Message-ID: <1456784093.84.0.74186934969.issue581@psf.upfronthosting.co.za>


Maciej Szulik added the comment:

I was too fast with my previous judgment, thankfully. It's working ok. I've additionally fixed the message returned upon auth error. I think we can consider this done atm. Ezio ptal.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue581>
_______________________________________________________
-------------- next part --------------
diff --git a/roundup/cgi/client.py b/roundup/cgi/client.py
--- a/roundup/cgi/client.py
+++ b/roundup/cgi/client.py
@@ -22,6 +22,7 @@
 from roundup.mailer import Mailer, MessageSendError, encode_quopri
 from roundup.cgi import accept_language
 from roundup import xmlrpc
+from roundup import rest
 
 from roundup.anypy.cookie_ import CookieError, BaseCookie, SimpleCookie, \
     get_cookie_date
@@ -378,6 +379,8 @@
         try:
             if self.path == 'xmlrpc':
                 self.handle_xmlrpc()
+            elif self.path == 'rest' or self.path[:5] == 'rest/':
+                self.handle_rest()
             else:
                 self.inner_main()
         finally:
@@ -419,6 +422,33 @@
         self.setHeader("Content-Length", str(len(output)))
         self.write(output)
 
+    def handle_rest(self):
+        # Set the charset and language
+        self.determine_charset()
+        self.determine_language()
+        # Open the database as the correct user.
+        # TODO: add everything to RestfulDispatcher
+        try:
+            self.determine_user()
+        except LoginError, err:
+            self.response_code = http_.client.UNAUTHORIZED
+            self.write({'error': {
+                    'status': http_.client.UNAUTHORIZED,
+                    'msg': 'Unauthorized'
+                }})
+            return
+
+        self.check_anonymous_access()
+
+        # Call rest library to handle the request
+        handler = rest.RestfulInstance(self, self.db)
+        output = handler.dispatch(self.env['REQUEST_METHOD'], self.path,
+                                  self.form)
+
+        # self.setHeader("Content-Type", "text/xml")
+        self.setHeader("Content-Length", str(len(output)))
+        self.write(output)
+
     def add_ok_message(self, msg, escape=True):
         add_message(self._ok_message, msg, escape)
 
@@ -1165,7 +1195,7 @@
         if name is None:
             name = 'home'
 
-        tplname = name     
+        tplname = name
         if view:
             tplname = '%s.%s' % (name, view)
 
diff --git a/roundup/rest.py b/roundup/rest.py
new file mode 100644
--- /dev/null
+++ b/roundup/rest.py
@@ -0,0 +1,551 @@
+"""
+Restful API for Roundup
+
+This module is free software, you may redistribute it
+and/or modify under the same terms as Python.
+"""
+
+import urlparse
+import os
+import json
+import pprint
+import sys
+import time
+import traceback
+import re
+
+from roundup import hyperdb
+from roundup import date
+from roundup import actions
+from roundup.exceptions import *
+from roundup.cgi.exceptions import *
+
+
+def _data_decorator(func):
+    """Wrap the returned data into an object."""
+    def format_object(self, *args, **kwargs):
+        # get the data / error from function
+        try:
+            code, data = func(self, *args, **kwargs)
+        except NotFound, msg:
+            code = 404
+            data = msg
+        except IndexError, msg:
+            code = 404
+            data = msg
+        except Unauthorised, msg:
+            code = 403
+            data = msg
+        except UsageError, msg:
+            code = 400
+            data = msg
+        except (AttributeError, Reject), msg:
+            code = 405
+            data = msg
+        except ValueError, msg:
+            code = 409
+            data = msg
+        except NotImplementedError:
+            code = 402  # nothing to pay, just a mark for debugging purpose
+            data = 'Method under development'
+        except:
+            exc, val, tb = sys.exc_info()
+            code = 400
+            ts = time.ctime()
+            data = '%s: An error occurred. Please check the server log' \
+                   ' for more information.' % ts
+            # out to the logfile
+            print 'EXCEPTION AT', ts
+            traceback.print_exc()
+
+        # decorate it
+        self.client.response_code = code
+        if code >= 400:  # any error require error format
+            result = {
+                'error': {
+                    'status': code,
+                    'msg': data
+                }
+            }
+        else:
+            result = {
+                'data': data
+            }
+        return result
+    return format_object
+
+
+def parse_accept_header(accept):
+    """
+    Parse the Accept header *accept*, returning a list with 3-tuples of
+    [(str(media_type), dict(params), float(q_value)),] ordered by q values.
+
+    If the accept header includes vendor-specific types like::
+        application/vnd.yourcompany.yourproduct-v1.1+json
+
+    It will actually convert the vendor and version into parameters and
+    convert the content type into `application/json` so appropriate content
+    negotiation decisions can be made.
+
+    Default `q` for values that are not specified is 1.0
+
+    # Based on https://gist.github.com/samuraisam/2714195
+    # Also, based on a snipped found in this project:
+    #   https://github.com/martinblech/mimerender
+    """
+    result = []
+    for media_range in accept.split(","):
+        parts = media_range.split(";")
+        media_type = parts.pop(0).strip()
+        media_params = []
+        # convert vendor-specific content types into something useful (see
+        # docstring)
+        typ, subtyp = media_type.split('/')
+        # check for a + in the sub-type
+        if '+' in subtyp:
+            # if it exists, determine if the subtype is a vendor-specific type
+            vnd, sep, extra = subtyp.partition('+')
+            if vnd.startswith('vnd'):
+                # and then... if it ends in something like "-v1.1" parse the
+                # version out
+                if '-v' in vnd:
+                    vnd, sep, rest = vnd.rpartition('-v')
+                    if len(rest):
+                        # add the version as a media param
+                        try:
+                            version = media_params.append(('version',
+                                                           float(rest)))
+                        except ValueError:
+                            version = 1.0  # could not be parsed
+                # add the vendor code as a media param
+                media_params.append(('vendor', vnd))
+                # and re-write media_type to something like application/json so
+                # it can be used usefully when looking up emitters
+                media_type = '{}/{}'.format(typ, extra)
+        q = 1.0
+        for part in parts:
+            (key, value) = part.lstrip().split("=", 1)
+            key = key.strip()
+            value = value.strip()
+            if key == "q":
+                q = float(value)
+            else:
+                media_params.append((key, value))
+        result.append((media_type, dict(media_params), q))
+    result.sort(lambda x, y: -cmp(x[2], y[2]))
+    return result
+
+
+class Routing(object):
+    __route_map = {}
+    __var_to_regex = re.compile(r"<:(\w+)>")
+    url_to_regex = r"([\w.\-~!$&'()*+,;=:@\%%]+)"
+
+    @classmethod
+    def route(cls, rule, methods='GET'):
+        """A decorator that is used to register a view function for a
+        given URL rule:
+            @self.route('/')
+            def index():
+                return 'Hello World'
+
+        rest/ will be added to the beginning of the url string
+
+        Args:
+            rule (string): the URL rule
+            methods (string or tuple or list): the http method
+        """
+        # strip the '/' character from rule string
+        rule = rule.strip('/')
+
+        # add 'rest/' to the rule string
+        if not rule.startswith('rest/'):
+            rule = '^rest/' + rule + '$'
+
+        if isinstance(methods, basestring):  # convert string to tuple
+            methods = (methods,)
+        methods = set(item.upper() for item in methods)
+
+        # convert a rule to a compiled regex object
+        # so /data/<:class>/<:id> will become
+        #    /data/([charset]+)/([charset]+)
+        # and extract the variable names to a list [(class), (id)]
+        func_vars = cls.__var_to_regex.findall(rule)
+        rule = re.compile(cls.__var_to_regex.sub(cls.url_to_regex, rule))
+
+        # then we decorate it:
+        # route_map[regex][method] = func
+        def decorator(func):
+            rule_route = cls.__route_map.get(rule, {})
+            func_obj = {
+                'func': func,
+                'vars': func_vars
+            }
+            for method in methods:
+                rule_route[method] = func_obj
+            cls.__route_map[rule] = rule_route
+            return func
+        return decorator
+
+    @classmethod
+    def execute(cls, instance, path, method, input):
+        # format the input
+        path = path.strip('/').lower()
+        method = method.upper()
+
+        # find the rule match the path
+        # then get handler match the method
+        for path_regex in cls.__route_map:
+            match_obj = path_regex.match(path)
+            if match_obj:
+                try:
+                    func_obj = cls.__route_map[path_regex][method]
+                except KeyError:
+                    raise Reject('Method %s not allowed' % method)
+
+                # retrieve the vars list and the function caller
+                list_vars = func_obj['vars']
+                func = func_obj['func']
+
+                # zip the varlist into a dictionary, and pass it to the caller
+                args = dict(zip(list_vars, match_obj.groups()))
+                args['input'] = input
+                return func(instance, **args)
+        raise NotFound('Nothing matches the given URI')
+
+
+class RestfulInstance(object):
+    """The RestfulInstance performs REST request from the client"""
+
+    __default_patch_op = "replace"  # default operator for PATCH method
+    __accepted_content_type = {
+        "application/json": "json",
+        "*/*": "json"
+        # "application/xml": "xml"
+    }
+    __default_accept_type = "json"
+
+    def __init__(self, client, db):
+        self.client = client
+        self.db = db
+        self.translator = client.translator
+        self.actions = client.instance.actions.copy()
+        self.actions.update({'retire': actions.Retire})
+
+        protocol = 'http'
+        host = self.client.env['HTTP_HOST']
+        tracker = self.client.env['TRACKER_NAME']
+        self.base_path = '%s://%s/%s/rest' % (protocol, host, tracker)
+        self.data_path = self.base_path + '/data'
+
+    def error_obj(self, status, msg, source=None):
+        """Return an error object"""
+        self.client.response_code = status
+        result = {
+            'error': {
+                'status': status,
+                'msg': msg
+            }
+        }
+        if source is not None:
+            result['error']['source'] = source
+
+        return result
+
+    @Routing.route("/data/<:class_name>", 'GET')
+    @_data_decorator
+    def get_collection(self, class_name, input):
+        """GET resource from class URI.
+
+        This function returns only items have View permission
+        class_name should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            list: list of reference item in the class
+                id: id of the object
+                link: path to the object
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % self.db.classes)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name
+        ):
+            raise Unauthorised('Permission to view %s denied' % class_name)
+
+        class_obj = self.db.getclass(class_name)
+        class_path = '%s/%s/' % (self.data_path, class_name)
+
+        # Handle filtering and pagination
+        filter_props = {}
+        page = {
+            'size': None,
+            'index': None
+        }
+        for form_field in input.value:
+            key = form_field.name
+            value = form_field.value
+            if key.startswith("where_"):  # serve the filter purpose
+                key = key[6:]
+                filter_props[key] = [
+                    getattr(self.db, key).lookup(p)
+                    for p in value.split(",")
+                ]
+            elif key.startswith("page_"):  # serve the paging purpose
+                key = key[5:]
+                value = int(value)
+                page[key] = value
+
+        if not filter_props:
+            obj_list = class_obj.list()
+        else:
+            obj_list = class_obj.filter(None, filter_props)
+
+        # extract result from data
+        result = [
+            {'id': item_id, 'link': class_path + item_id}
+            for item_id in obj_list
+            if self.db.security.hasPermission(
+                'View', self.db.getuid(), class_name, itemid=item_id
+            )
+        ]
+
+        # pagination
+        if page['size'] is not None and page['index'] is not None:
+            page_start = max(page['index'] * page['size'], 0)
+            page_end = min(page_start + page['size'], len(result))
+            result = result[page_start:page_end]
+
+        self.client.setHeader("X-Count-Total", str(len(result)))
+        return 200, result
+
+    @Routing.route("/data/<:class_name>/<:item_id>", 'GET')
+    @_data_decorator
+    def get_element(self, class_name, item_id, input):
+        """GET resource from object URI.
+
+        This function returns only properties have View permission
+        class_name and item_id should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            item_id (string): id of the resource (Ex: 12, 15)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            dict: a dictionary represents the object
+                id: id of the object
+                type: class name of the object
+                link: link to the object
+                attributes: a dictionary represent the attributes of the object
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name, itemid=item_id
+        ):
+            raise Unauthorised(
+                'Permission to view %s/%s denied' % (class_name, item_id)
+            )
+
+        class_obj = self.db.getclass(class_name)
+        props = None
+        for form_field in input.value:
+            key = form_field.name
+            value = form_field.value
+            if key == "fields":
+                props = value.split(",")
+
+        if props is None:
+            props = class_obj.properties.keys()
+
+        props.sort()  # sort properties
+
+        try:
+            result = [
+                (prop_name, class_obj.get(item_id, prop_name))
+                for prop_name in props
+                if self.db.security.hasPermission(
+                    'View', self.db.getuid(), class_name, prop_name, item_id
+                )
+            ]
+        except KeyError, msg:
+            raise UsageError("%s field not valid" % msg)
+        result = {
+            'id': item_id,
+            'type': class_name,
+            'link': '%s/%s/%s' % (self.data_path, class_name, item_id),
+            'attributes': dict(result)
+        }
+
+        return 200, result
+
+    @Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'GET')
+    @_data_decorator
+    def get_attribute(self, class_name, item_id, attr_name, input):
+        """GET resource from attribute URI.
+
+        This function returns only attribute has View permission
+        class_name should be valid already
+
+        Args:
+            class_name (string): class name of the resource (Ex: issue, msg)
+            item_id (string): id of the resource (Ex: 12, 15)
+            attr_name (string): attribute of the resource (Ex: title, nosy)
+            input (list): the submitted form of the user
+
+        Returns:
+            int: http status code 200 (OK)
+            list: a dictionary represents the attribute
+                id: id of the object
+                type: class name of the attribute
+                link: link to the attribute
+                data: data of the requested attribute
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        if not self.db.security.hasPermission(
+            'View', self.db.getuid(), class_name, attr_name, item_id
+        ):
+            raise Unauthorised(
+                'Permission to view %s/%s %s denied' %
+                (class_name, item_id, attr_name)
+            )
+
+        class_obj = self.db.getclass(class_name)
+        data = class_obj.get(item_id, attr_name)
+        result = {
+            'id': item_id,
+            'type': type(data),
+            'link': "%s/%s/%s/%s" %
+                    (self.data_path, class_name, item_id, attr_name),
+            'data': data
+        }
+
+        return 200, result
+
+    @Routing.route("/data/<:class_name>", 'OPTIONS')
+    @_data_decorator
+    def options_collection(self, class_name, input):
+        """OPTION return the HTTP Header for the class uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        return 204, ""
+
+    @Routing.route("/data/<:class_name>/<:item_id>", 'OPTIONS')
+    @_data_decorator
+    def options_element(self, class_name, item_id, input):
+        """OPTION return the HTTP Header for the object uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        self.client.setHeader(
+            "Accept-Patch",
+            "application/x-www-form-urlencoded, multipart/form-data"
+        )
+        return 204, ""
+
+    @Routing.route("/data/<:class_name>/<:item_id>/<:attr_name>", 'OPTIONS')
+    @_data_decorator
+    def option_attribute(self, class_name, item_id, attr_name, input):
+        """OPTION return the HTTP Header for the attribute uri
+
+        Returns:
+            int: http status code 204 (No content)
+            body (string): an empty string
+        """
+        if class_name not in self.db.classes:
+            raise NotFound('Class %s not found' % class_name)
+        self.client.setHeader(
+            "Accept-Patch",
+            "application/x-www-form-urlencoded, multipart/form-data"
+        )
+        return 204, ""
+
+    def dispatch(self, method, uri, input):
+        """format and process the request"""
+        # if X-HTTP-Method-Override is set, follow the override method
+        headers = self.client.request.headers
+        method = headers.getheader('X-HTTP-Method-Override') or method
+
+        # parse Accept header and get the content type
+        accept_header = parse_accept_header(headers.getheader('Accept'))
+        accept_type = "invalid"
+        for part in accept_header:
+            if part[0] in self.__accepted_content_type:
+                accept_type = self.__accepted_content_type[part[0]]
+
+        # get the request format for response
+        # priority : extension from uri (/rest/issue.json),
+        #            header (Accept: application/json, application/xml)
+        #            default (application/json)
+        ext_type = os.path.splitext(urlparse.urlparse(uri).path)[1][1:]
+        data_type = ext_type or accept_type or self.__default_accept_type
+
+        # check for pretty print
+        try:
+            pretty_output = input['pretty'].value.lower() == "true"
+        except KeyError:
+            pretty_output = False
+
+        # add access-control-allow-* to support CORS
+        self.client.setHeader("Access-Control-Allow-Origin", "*")
+        self.client.setHeader(
+            "Access-Control-Allow-Headers",
+            "Content-Type, Authorization, X-HTTP-Method-Override"
+        )
+        self.client.setHeader(
+            "Allow",
+            "HEAD, OPTIONS, GET"
+        )
+        self.client.setHeader(
+            "Access-Control-Allow-Methods",
+            "HEAD, OPTIONS, GET"
+        )
+
+        # Call the appropriate method
+        try:
+            output = Routing.execute(self, uri, method, input)
+        except NotFound, msg:
+            output = self.error_obj(404, msg)
+        except Reject, msg:
+            output = self.error_obj(405, msg)
+
+        # Format the content type
+        if data_type.lower() == "json":
+            self.client.setHeader("Content-Type", "application/json")
+            if pretty_output:
+                indent = 4
+            else:
+                indent = None
+            output = RoundupJSONEncoder(indent=indent).encode(output)
+        else:
+            self.client.response_code = 406
+            output = "Content type is not accepted by client"
+
+        return output
+
+
+class RoundupJSONEncoder(json.JSONEncoder):
+    """RoundupJSONEncoder overrides the default JSONEncoder to handle all
+    types of the object without returning any error"""
+    def default(self, obj):
+        try:
+            result = json.JSONEncoder.default(self, obj)
+        except TypeError:
+            result = str(obj)
+        return result
diff --git a/roundup/scripts/roundup_server.py b/roundup/scripts/roundup_server.py
--- a/roundup/scripts/roundup_server.py
+++ b/roundup/scripts/roundup_server.py
@@ -251,7 +251,7 @@
         else:
             return self.run_cgi()
 
-    do_GET = do_POST = do_HEAD = run_cgi_outer
+    do_GET = do_POST = do_HEAD = do_PUT = do_DELETE = do_PATCH = do_OPTIONS = run_cgi_outer
 
     def index(self):
         ''' Print up an index of the available trackers
diff --git a/test/test_rest.py b/test/test_rest.py
new file mode 100644
--- /dev/null
+++ b/test/test_rest.py
@@ -0,0 +1,301 @@
+import unittest
+import os
+import shutil
+import errno
+
+from roundup.cgi.exceptions import *
+from roundup import password, hyperdb
+from roundup.rest import RestfulInstance
+from roundup.backends import list_backends
+from roundup.cgi import client
+import random
+
+import db_test_base
+
+NEEDS_INSTANCE = 1
+
+
+class TestCase(unittest.TestCase):
+
+    backend = None
+
+    def setUp(self):
+        self.dirname = '_test_rest'
+        # set up and open a tracker
+        self.instance = db_test_base.setupTracker(self.dirname, self.backend)
+
+        # open the database
+        self.db = self.instance.open('admin')
+
+        # Get user id (user4 maybe). Used later to get data from db.
+        self.joeid = self.db.user.create(
+            username='joe',
+            password=password.Password('random'),
+            address='random at home.org',
+            realname='Joe Random',
+            roles='User'
+        )
+
+        self.db.commit()
+        self.db.close()
+
+        env = {
+            'PATH_INFO': 'http://localhost/rounduptest/rest/',
+            'HTTP_HOST': 'localhost',
+            'TRACKER_NAME': 'rounduptest'
+        }
+        self.dummy_client = client.Client(self.instance, None, env, [], None)
+        self.empty_form = cgi.FieldStorage()
+
+    def tearDown(self):
+        self.db.close()
+        try:
+            shutil.rmtree(self.dirname)
+        except OSError, error:
+            if error.errno not in (errno.ENOENT, errno.ESRCH):
+                raise
+
+    def open(self, user='joe'):
+        """
+        Opens database as given user.
+        """
+        self.db = self.instance.open(user)
+
+        self.db.tx_Source = 'web'
+
+        self.db.issue.addprop(tx_Source=hyperdb.String())
+        self.db.msg.addprop(tx_Source=hyperdb.String())
+
+        self.db.post_init()
+
+        thisdir = os.path.dirname(__file__)
+        vars = {}
+        execfile(os.path.join(thisdir, "tx_Source_detector.py"), vars)
+        vars['init'](self.db)
+        self.server = RestfulInstance(self.dummy_client, self.db)
+
+    def testGetSelf(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # Retrieve all three users.
+        results = self.server.get_collection('user', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), 3)
+
+        # Obtain data for 'joe'.
+        results = self.server.get_element('user', self.joeid, self.empty_form)
+        results = results['data']
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(results['attributes']['username'], 'joe')
+        self.assertEqual(results['attributes']['realname'], 'Joe Random')
+
+        # Obtain data for 'joe'.
+        results = self.server.get_attribute(
+            'user', self.joeid, 'username', self.empty_form
+        )
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(results['data']['data'], 'joe')
+
+    def testGetAdmin(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        results = self.server.get_element('user', '1', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertFalse(results['data']['attributes'].has_key('openids'))
+        self.assertFalse(results['data']['attributes'].has_key('password'))
+
+    def testGetSelfAttribute(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        results = self.server.get_attribute('user', self.joeid, 'password', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIsNotNone(results['data']['data'])
+
+    def testGetAdminAttribute(self):
+        """
+        Read admin data.
+        """
+        self.open()
+        # Retrieve all three users.
+        results = self.server.get_attribute('user', '1', 'password', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 403)
+
+    def testGetAnonymous(self):
+        """
+        Anonymous should not get users.
+        """
+        self.open('anonymous')
+        results = self.server.get_collection('user', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 403)
+
+    def testFilter(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # create sample data
+        try:
+            self.db.status.create(name='open')
+        except ValueError:
+            pass
+        try:
+            self.db.status.create(name='closed')
+        except ValueError:
+            pass
+        try:
+            self.db.priority.create(name='normal')
+        except ValueError:
+            pass
+        try:
+            self.db.priority.create(name='critical')
+        except ValueError:
+            pass
+        self.db.issue.create(
+            title='foo4',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('critical')
+        )
+        self.db.issue.create(
+            title='foo1',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_open_norm = self.db.issue.create(
+            title='foo2',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_closed_norm = self.db.issue.create(
+            title='foo3',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('normal')
+        )
+        issue_closed_crit = self.db.issue.create(
+            title='foo4',
+            status=self.db.status.lookup('closed'),
+            priority=self.db.priority.lookup('critical')
+        )
+        issue_open_crit = self.db.issue.create(
+            title='foo5',
+            status=self.db.status.lookup('open'),
+            priority=self.db.priority.lookup('critical')
+        )
+        base_path = self.dummy_client.env['PATH_INFO'] + 'data/issue/'
+
+        # Retrieve all issue status=open
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'open')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_open_norm), results['data'])
+        self.assertIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(
+            get_obj(base_path, issue_closed_norm), results['data']
+        )
+
+        # Retrieve all issue status=closed and priority=critical
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'closed'),
+            cgi.MiniFieldStorage('where_priority', 'critical')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_closed_crit), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(
+            get_obj(base_path, issue_closed_norm), results['data']
+        )
+
+        # Retrieve all issue status=closed and priority=normal,critical
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('where_status', 'closed'),
+            cgi.MiniFieldStorage('where_priority', 'normal,critical')
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertIn(get_obj(base_path, issue_closed_crit), results['data'])
+        self.assertIn(get_obj(base_path, issue_closed_norm), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_crit), results['data'])
+        self.assertNotIn(get_obj(base_path, issue_open_norm), results['data'])
+
+    def testPagination(self):
+        """
+        Retrieve all three users
+        obtain data for 'joe'
+        """
+        self.open()
+        # create sample data
+        for i in range(0, random.randint(5, 10)):
+            self.db.issue.create(title='foo' + str(i))
+
+        # Retrieving all the issues
+        results = self.server.get_collection('issue', self.empty_form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        total_length = len(results['data'])
+
+        # Pagination will be 70% of the total result
+        page_size = total_length * 70 // 100
+        page_zero_expected = page_size
+        page_one_expected = total_length - page_zero_expected
+
+        # Retrieve page 0
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 0)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), page_zero_expected)
+
+        # Retrieve page 1
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 1)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), page_one_expected)
+
+        # Retrieve page 2
+        form = cgi.FieldStorage()
+        form.list = [
+            cgi.MiniFieldStorage('page_size', page_size),
+            cgi.MiniFieldStorage('page_index', 2)
+        ]
+        results = self.server.get_collection('issue', form)
+        self.assertEqual(self.dummy_client.response_code, 200)
+        self.assertEqual(len(results['data']), 0)
+
+
+def get_obj(path, id):
+    return {
+        'id': id,
+        'link': path + id
+    }
+
+
+def test_suite():
+    suite = unittest.TestSuite()
+    for l in list_backends():
+        dct = dict(backend=l)
+        subcls = type(TestCase)('TestCase_%s' % l, (TestCase,), dct)
+        suite.addTest(unittest.makeSuite(subcls))
+    return suite
+
+if __name__ == '__main__':
+    runner = unittest.TextTestRunner()
+    unittest.main(testRunner=runner)