From metatracker at psf.upfronthosting.co.za  Thu Apr  2 03:06:13 2015
From: metatracker at psf.upfronthosting.co.za (shiyao.ma)
Date: Thu, 02 Apr 2015 01:06:13 +0000
Subject: [Tracker-discuss] [issue565] Support for virtualenv (and absolute
	import path in bin)
Message-ID: <1427936773.38.0.936164868729.issue565@psf.upfronthosting.co.za>


New submission from shiyao.ma:

The current roundup instance installs scripts with #! /path/to/your/python/executable

This lacks much flexibility, so I changed that to #! /usr/bin/env python.


This patch also fixes a minor bug in the scripts.
Namely, the path in "sys.path.insert(1, path)" is relative, which is subjective to the os.getcwd().

I changed that to absolute path.

This may more sound like a issue of roundup itself.
But it's never too late to be merged upstream.

----------
assignedto: introom
files: venv.patch
messages: 2955
nosy: ezio.melotti, introom
priority: bug
status: unread
title: Support for virtualenv (and absolute import path in bin)

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue565>
_______________________________________________________
-------------- next part --------------
diff --git a/roundup/dist/command/build_scripts.py b/roundup/dist/command/build_scripts.py
--- a/roundup/dist/command/build_scripts.py
+++ b/roundup/dist/command/build_scripts.py
@@ -68,7 +68,11 @@
             # TODO? allow command-line option
             pass
         if target == sys.platform:
-            self.python_executable = os.path.normpath(sys.executable)
+            # promote virtualenv
+            if os.name == 'posix':
+                self.python_executable = "/usr/bin/env python"
+            else:
+                self.python_executable = os.path.normpath(sys.executable)
         else:
             self.python_executable = "python"
 
@@ -82,7 +86,8 @@
 
         # tweak python path for installations outside main python library
         if cmdopt.get("install", {}).has_key("prefix"):
-            prefix = os.path.expanduser(cmdopt['install']['prefix'][1])
+            prefix = os.path.abspath(
+                os.path.expanduser(cmdopt['install']['prefix'][1]))
             version = '%d.%d'%sys.version_info[:2]
             self.script_preamble = """
 import sys
diff --git a/scripts/schema_diagram.py b/scripts/schema_diagram.py
--- a/scripts/schema_diagram.py
+++ b/scripts/schema_diagram.py
@@ -1,4 +1,4 @@
-#! /usr/bin/python
+#! /usr/bin/env python
 #
 # Schema diagram generator contributed by Stefan Seefeld of the fresco
 # project http://www.fresco.org/.

From metatracker at psf.upfronthosting.co.za  Thu Apr  2 17:14:58 2015
From: metatracker at psf.upfronthosting.co.za (R David Murray)
Date: Thu, 02 Apr 2015 15:14:58 +0000
Subject: [Tracker-discuss] [issue565] Support for virtualenv (and absolute
	import path in bin)
In-Reply-To: <1427936773.38.0.936164868729.issue565@psf.upfronthosting.co.za>
Message-ID: <1427987698.33.0.878261600736.issue565@psf.upfronthosting.co.za>


R David Murray added the comment:

Thanks for the suggestion, but this is an incorrect (insecure) change.  The correct change would be to use the "usual mechanisms" (and frankly I forget what those are) to resolve the absolute path to the executable at install time and use that.

And yes, this belongs upstream, not here.

----------
nosy: +r.david.murray
status: unread -> resolved

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue565>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Thu Apr  2 17:17:28 2015
From: metatracker at psf.upfronthosting.co.za (R David Murray)
Date: Thu, 02 Apr 2015 15:17:28 +0000
Subject: [Tracker-discuss] [issue565] Support for virtualenv (and absolute
	import path in bin)
In-Reply-To: <1427936773.38.0.936164868729.issue565@psf.upfronthosting.co.za>
Message-ID: <1427987848.71.0.515326973605.issue565@psf.upfronthosting.co.za>


R David Murray added the comment:

Actually, rereading your patch it looks like it does the right thing before the patch.  So if there's a bug in venv support somehow, that would be the thing to report upstream.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue565>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Thu Apr  2 22:10:51 2015
From: metatracker at psf.upfronthosting.co.za (shiyao.ma)
Date: Thu, 02 Apr 2015 20:10:51 +0000
Subject: [Tracker-discuss] [issue515] Full text search doesn't return results
In-Reply-To: <1365429816.14.0.5567377404.issue515@psf.upfronthosting.co.za>
Message-ID: <1428005451.74.0.173448547438.issue515@psf.upfronthosting.co.za>


shiyao.ma added the comment:

Hi.

JUST FYI.

When a user submits an issue, the tracker will split the both the title and message into words. The splitting regex is the same as provided by RdM.

Those splitted words are stored in the table _words. Each _words entry is in the form of (a_splitted_word, a_text_id).

A _text_id is the PM key of the table _textids. The _textids entry will tell us what's the associated issue title id or message id.

Thus, an inverted index is formed.

When a user searches, for example, through the web interface. The roundup.cgi.templating.py will re-split the words, search that inverted index, and finally find the result.


What I am thinking is, regex is not powerful enough. We may add fix for this by excluding "." from the word boundary.

But this may be (better) achieved by other means, like specifying the "version".

----------
nosy: +introom

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue515>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Thu Apr  2 22:24:47 2015
From: metatracker at psf.upfronthosting.co.za (shiyao.ma)
Date: Thu, 02 Apr 2015 20:24:47 +0000
Subject: [Tracker-discuss] [issue565] Support for virtualenv (and absolute
	import path in bin)
In-Reply-To: <1427936773.38.0.936164868729.issue565@psf.upfronthosting.co.za>
Message-ID: <1428006286.99.0.0379601217617.issue565@psf.upfronthosting.co.za>


shiyao.ma added the comment:

Let me explain a bit. ;)

Suppose I just hg cloned the roundup repo from hg.p.o

When I install the roundup with: python setup install --prefix ../working/roundup

FOR THE INSTALLED FILES:

If this patch is NOT applied,

Files under working/roundup/bin will all have a "#! /hardcoded/path". The path is the absolute path pointing to the python executable which is used for the installation.

Besides, the files will all contain a line in the form of, sys.path.insert("..working/roundup").


If this patch is applied,

Files under working/roundup/bin will all have a "#! /usr/bin/env python"

The sys.path.insert("../working/bar") will also become sys.path.insert("/abspath/to/working/bar")



The reason I fired this issue is, I am new to psf-meta dev. So I found myself constantly "modify the source; install it; preview the results;re-modify the source;re-install".
Due to the relative "sys.path.insert" and the /hardcoded/path", I must use the virutalenv python to install it, and I must have to stay in the right directory to run rd-admin, which I often foget.

But now I found I can modify the source and directly rd-admin/rd-start based on the source. No more installation!
So this issue doesn't pertain much to me now.

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue565>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Fri Apr  3 00:20:14 2015
From: metatracker at psf.upfronthosting.co.za (R David Murray)
Date: Thu, 02 Apr 2015 22:20:14 +0000
Subject: [Tracker-discuss] [issue565] Support for virtualenv (and absolute
	import path in bin)
In-Reply-To: <1427936773.38.0.936164868729.issue565@psf.upfronthosting.co.za>
Message-ID: <1428013214.97.0.179105608198.issue565@psf.upfronthosting.co.za>


R David Murray added the comment:

I'm glad you found a solution.  Yes, it is true that if you use a virtual environment then you must always use that virtual environment (one way or another).

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue565>
_______________________________________________________

From metatracker at psf.upfronthosting.co.za  Fri Apr  3 23:57:41 2015
From: metatracker at psf.upfronthosting.co.za (shiyao.ma)
Date: Fri, 03 Apr 2015 21:57:41 +0000
Subject: [Tracker-discuss] [issue515] Full text search doesn't return results
In-Reply-To: <1365429816.14.0.5567377404.issue515@psf.upfronthosting.co.za>
Message-ID: <1428098261.54.0.787577368.issue515@psf.upfronthosting.co.za>


shiyao.ma added the comment:

This patch splits text in the following form,

suppose TEXT = "aa bb ee.ff.gg"

the splitted words will be: aa, bb, ee, ff, gg, ee.ff, ff.gg, ee.ff.gg

IOW, new words are connected by the dot.

When searching through web interface, for example if the text is "kk hh.pp", then the splitted words are "kk" and "hh.pp".
IOW, when searching, we take "dot" separated words as a whole.


the handling for csv interface, and the xapian based indexer is not modified. If the above form is okay, I will do the remaining stuff.

----------
nosy: +ezio.melotti

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue515>
_______________________________________________________
-------------- next part --------------
diff --git a/roundup/backends/indexer_common.py b/roundup/backends/indexer_common.py
--- a/roundup/backends/indexer_common.py
+++ b/roundup/backends/indexer_common.py
@@ -23,6 +23,28 @@
         # gibberish (encoded text or somesuch) or shorter than 2 characters
         self.minlength = 2
         self.maxlength = 25
+        self.dot_maxlength = 10
+        self.dot_maxrepeat = 2
+        self.pattern_word = re.compile(
+            r'\b\w{%d,%d}\b'
+                % (self.minlength, self.maxlength),
+            re.UNICODE)
+        self.pattern_dot = re.compile(
+            r'\b(\w{1,%d}\.){1,%d}\w{1,%d}\b' %
+                (self.dot_maxlength, self.dot_maxrepeat, self.dot_maxlength),
+            re.UNICODE)
+
+    def segment_text(self, text):
+        wordlist = [w for w in re.findall(self.pattern_word, text)]
+        match = re.search(self.pattern_dot, text)
+        while match:
+            words = match.group().split('.')
+            for length in range(2, len(words)+1):
+                for idx in range(len(words)-length+1):
+                    wordlist.append('.'.join(words[idx:idx+length]))
+            text = text[match.end():]
+            match = re.search(self.pattern_dot, text)
+        return set(w.encode('utf8') for w in wordlist)
 
     def is_stopword(self, word):
         return word in self.stopwords
diff --git a/roundup/backends/indexer_rdbms.py b/roundup/backends/indexer_rdbms.py
--- a/roundup/backends/indexer_rdbms.py
+++ b/roundup/backends/indexer_rdbms.py
@@ -64,9 +64,7 @@
         if not isinstance(text, unicode):
             text = unicode(text, "utf-8", "replace")
         text = text.upper()
-        wordlist = [w.encode("utf-8")
-                    for w in re.findall(r'(?u)\b\w{%d,%d}\b'
-                                        % (self.minlength, self.maxlength), text)]
+        wordlist = self.segment_text(text)
         words = set()
         for word in wordlist:
             if self.is_stopword(word): continue
diff --git a/roundup/cgi/templating.py b/roundup/cgi/templating.py
--- a/roundup/cgi/templating.py
+++ b/roundup/cgi/templating.py
@@ -2800,7 +2800,7 @@
         if self.search_text:
             matches = self.client.db.indexer.search(
                 [w.upper().encode("utf-8", "replace") for w in re.findall(
-                    r'(?u)\b\w{2,25}\b',
+                    r'(?u)\b(?:\w|\.){2,40}\b',
                     unicode(self.search_text, "utf-8", "replace")
                 )], klass, ignore)
         else:

From metatracker at psf.upfronthosting.co.za  Fri Apr 24 17:35:21 2015
From: metatracker at psf.upfronthosting.co.za (Tim Pierce)
Date: Fri, 24 Apr 2015 15:35:21 +0000
Subject: [Tracker-discuss] [issue566] Google login not working on
	bugs.python.org
Message-ID: <1429889721.65.0.614400211918.issue566@psf.upfronthosting.co.za>


New submission from Tim Pierce:

When I attempt to log in to bugs.python.org using the Google login button, it returns a 500 error:

"An error has occurred

A problem was encountered processing your request. The tracker maintainers have been notified of the problem."

I have reproduced the problem several times, in both Chrome 42.0.2311.90 and Safari 8.0.5 (both on OS X).

----------
messages: 2962
nosy: twpierce
priority: urgent
status: unread
title: Google login not working on bugs.python.org

_______________________________________________________
PSF Meta Tracker <metatracker at psf.upfronthosting.co.za>
<http://psf.upfronthosting.co.za/roundup/meta/issue566>
_______________________________________________________