[Mailman-users] RE: htdig integration problem

[Richard Barrett]

In your mail to mailman-users list of Dec 3, 2003 you said:

 > solaris 9
 > mailman 2.1.3
 > htdig  3.1.6
 > patches: indexing-2.1.3-0.1 and htdig-2.1.3-0.4
 >
 > htdig is working.  from the command line i can do:
 >
 > #/opt/www/htdig/bin/htsearch -c
 > /usr/local/mailman/archives/htdig/mailman.conf
 >
 > as users root , mailman and nobody and get the html search results
 > correctly returned.
 >

I'm trying to see how this fits with the statement below where you say 
rundig is failing with a 134 return code when it is run by 
nightly_htdig for the mailman list. If rundig is failing when executed 
by nightly_htdig then how were the search indices for the list 
concerned constructed, because without these then I would not expect 
htsearch for the list using the per-list htdig.conf to return any 
useful search results? See below for further comment on rundig.

 > When I try and search from the web archive form i get:
 >
 >
 >   htdig Archives Access Failure
 >
 > search failed -12-
 >

This is the CGI script $prefix/Mailman/Cgi/mmsearch.py saying that it 
got no response back when it ran htdig's htsearch; the response you are 
seeing is initiated by code at or near line 224 of that Python file. 
There should be information about the failure logged in 
$prefix/Mailman/logs/error. The logged information may include the exit 
status of htsearch if that was non-zero. Is so, what was that exit 
status?

 > If you want to make another attempt to access a list archive then go 
via
 > the list users information page
 > <http://mailman.sghms.ac.uk/mailman//listinfo/mailman>.
 >
 > If this problem persists then please e-mail the following information 
to
 > the mailman at mailman.sghms.ac.uk <mailto:mailman at 
mailman.sghms.ac.uk>:
 >
 >     http://mailman.sghms.ac.uk/pipermail/mailman/
 >     /mailman/mmsearch/mailman
 >
 > if i run the nightly_htdig script from the command line it hangs with 
'rundig failed for
 > list mailman, exit code 134'

By "hangs" do you mean the script sits staring at the wall until it is 
forcibly terminated by the kill command or Ctl-C or do you mean that it 
ends normally with that error message. I'll assume the latter in which 
case the problem is why will rundig not execute correctly.

Observations:

0. I assume you are running htdig on the same machine as Mailman and 
when running nightly_htdig from the command line you did so as the 
mailman uid or at least as a user who is a member of the mailman gid. 
If not then this may be causing a permissions problem.

1. This may be a permissions problem. There are some comments about 
this topic in the INSTALL.htdig-mm file in the build Mailman directory 
which are worth checking. You also need to consider which user's 
crontab is being used to run the nightly_htdig cron script.

2. Have you run $prefix/bin/check_perms in case that helps.

3. Is the $prefix/archives/htdig/<listname>.conf a symlink to 
$prefix/archives/private/<listname>/htdig/<listname>.conf

4. Does $prefix/archives/private/<listname>/htdig/ contain a 
rundig_last_run file and some db.* files with modification dates that 
makes sense? For instance:

-rw-rw-r--    1 mailman  mailman  53511168 Dec  6 02:42 db.docdb
-rw-rw-r--    1 mailman  mailman    568320 Dec  6 02:42 db.docs.index
-rw-rw-r--    1 mailman  mailman  71370695 Dec  6 02:41 db.wordlist
-rw-rw-r--    1 mailman  mailman  54174720 Dec  6 02:41 db.words.db
-rw-r--r--    1 mailman  mailman      3892 Nov 13 21:58 <listname>.conf
-rw-rw-r--    1 mailman  mailman         0 Dec  6 02:43 rundig_last_run

If all else fails you can try running the rundig command for the list 
from the command line as the mailman user. The command is constructed 
by the mmsearch.py as:

<path-to-rundig> -c <path-to-list-htdig.conf>

rundig is a shell script from htdig which runs various of htdig's 
programs in sequence to build/rebuild list search indices. You can add 
some diagnostic echo commands and such or hack around a copy of the 
script to try and determine why the rundig is returning a non-zero 
exist status to nightly_htdig

 >
 > has anyone got any ideas where i can look to put things right...
 >

If everything looks OK you can try "resetting" things by, as the 
mailman user:

1. (carefully) running $prefix/bin/blow_away_htdig for the list. This 
will remove all per-list htdig related files.

2. sending a test message to the list, which should cause 
reconstruction of the per-list htdig stuff

3. running $prefix/bin/arch --wipe for the list.

4. running nightly_htdig from the command line for the list.

If all is properly installed this sequence should leave you with a 
working htdig searchable archive for the list. If it doesn't, then get 
back to me.

 > thanks
 >
 > paul
-----------------------------------------------------------------------
Richard Barrett                               http://www.openinfo.co.uk