[pypy-svn] r30565 - in pypy/dist/pypy/tool/build: . bin builds test
guido at codespeak.net
guido at codespeak.net
Wed Jul 26 13:18:51 CEST 2006
Author: guido
Date: Wed Jul 26 13:18:25 2006
New Revision: 30565
Added:
pypy/dist/pypy/tool/build/ (props changed)
pypy/dist/pypy/tool/build/README.txt (contents, props changed)
pypy/dist/pypy/tool/build/__init__.py (contents, props changed)
pypy/dist/pypy/tool/build/bin/ (props changed)
pypy/dist/pypy/tool/build/bin/client (contents, props changed)
pypy/dist/pypy/tool/build/bin/path.py (contents, props changed)
pypy/dist/pypy/tool/build/bin/server (contents, props changed)
pypy/dist/pypy/tool/build/bin/startcompile (contents, props changed)
pypy/dist/pypy/tool/build/builds/
pypy/dist/pypy/tool/build/client.py (contents, props changed)
pypy/dist/pypy/tool/build/config.py (contents, props changed)
pypy/dist/pypy/tool/build/conftest.py (contents, props changed)
pypy/dist/pypy/tool/build/execnetconference.py (contents, props changed)
pypy/dist/pypy/tool/build/server.py (contents, props changed)
pypy/dist/pypy/tool/build/test/ (props changed)
pypy/dist/pypy/tool/build/test/fake.py (contents, props changed)
pypy/dist/pypy/tool/build/test/path.py (contents, props changed)
pypy/dist/pypy/tool/build/test/test.zip (contents, props changed)
pypy/dist/pypy/tool/build/test/test_client.py (contents, props changed)
pypy/dist/pypy/tool/build/test/test_pypybuilder.py (contents, props changed)
pypy/dist/pypy/tool/build/test/test_request_storage.py (contents, props changed)
pypy/dist/pypy/tool/build/test/test_server.py (contents, props changed)
Log:
Added 'pypybuilder', a tool to build a 'build farm' from participating clients,
the clients register to a server with information about what they can build,
then the server waits for build requests and dispatches to clients. Clients
send back a build when they're done, on which the server sends out emails to
whoever is waiting for the build. When a build is already available, the
requestor is provided with a path (will be URL in the future) to the build,
if no client is available for a certain request the request is queued until
there is one.
Worked on this from https://merlinux.de/svn/user/guido/pypybuilder before
checking in here.
Added: pypy/dist/pypy/tool/build/README.txt
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/README.txt Wed Jul 26 13:18:25 2006
@@ -0,0 +1,59 @@
+============
+PyPyBuilder
+============
+
+What is this?
+=============
+
+PyPyBuilder is an application that allows people to build PyPy instances on
+demand. If you have a nice idle machine connected to the Internet, and don't
+mind us 'borrowing' it every once in a while, you can start up the client
+script (in bin/client) and have the server send compile jobs to your machine.
+If someone requests a build of PyPy that is not already available on the PyPy
+website, and your machine is capable of making such a build, the server may ask
+your machine to create it. If enough people participate, with diverse enough
+machines, an ad-hoc 'build farm' is created this way.
+
+Components
+==========
+
+The application consists of 3 main components: a server component, a client and
+a small component to start compile jobs, which we'll call 'startcompile' for
+now.
+
+The server waits for clients to register, and for compile job requests. When
+clients register, they pass the server information about what compilations they
+can handle (system info). Then when the 'startcompile' component requests a
+compilation job, the server first checks whether a binary is already available,
+and if so returns that.
+
+If there isn't one, the server walks through a list of connected clients to see
+if there is one that can handle the job, and if so tells it to perform it. If
+there's no client to handle the job, it gets queued until there is.
+
+Once a build is available, the server will send an email to all email addresses
+(it could be more than one person asked for the same build at the same time!)
+passed to it by 'startcompile'.
+
+Installation
+============
+
+Client
+------
+
+Installing the system should not be required: just run './bin/client' to start
+the client. Note that it depends on the `py lib`_.
+
+Server
+------
+
+Also for the server there's no real setup required, and again there's a
+dependency on the `py lib`_.
+
+.. _`py lib`: http://codespeak.net/py
+
+More info
+=========
+
+For more information, bug reports, patches, etc., please send an email to
+guido at merlinux.de.
Added: pypy/dist/pypy/tool/build/__init__.py
==============================================================================
Added: pypy/dist/pypy/tool/build/bin/client
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/bin/client Wed Jul 26 13:18:25 2006
@@ -0,0 +1,58 @@
+#!/usr/bin/python
+
+BUFSIZE = 1024
+
+import path
+import sys
+import random
+from pypy.tool.build import config
+
+# XXX using random values for testing
+modules = ['_stackless', '_socket']
+
+"""
+random.shuffle(modules)
+sysinfo = {
+ 'maxint': random.choice((sys.maxint, (2 ** 63 - 1))),
+ 'use_modules': modules[:random.randrange(len(modules) + 1)],
+ 'byteorder': random.choice(('little', 'big')),
+}
+"""
+
+sysinfo = {
+ 'maxint': sys.maxint,
+ 'use_modules': ['_stackless', '_socket'],
+ 'byteorder': sys.byteorder,
+}
+
+if __name__ == '__main__':
+ from py.execnet import SshGateway
+ from pypy.tool.build.client import init
+ gw = SshGateway(config.server)
+ channel = init(gw, sysinfo, path=config.path, port=config.port)
+ print channel.receive() # welcome message
+ try:
+ while 1:
+ data = channel.receive()
+ if not isinstance(data, dict): # needs more checks here
+ raise ValueError(
+ 'received wrong unexpected data of type %s' % (type(data),)
+ )
+ info = data
+ # XXX we should compile here, using data dict for info
+ print 'compilation requested for info %r, now faking that' % (info,)
+ import time; time.sleep(10)
+
+ # write the zip to the server in chunks to server
+ # XXX we're still faking this
+ zipfp = (path.packagedir / 'test/test.zip').open()
+ while True:
+ chunk = zipfp.read(BUFSIZE)
+ if not chunk:
+ break
+ channel.send(chunk)
+ channel.send(None) # tell the server we're done
+ print 'done with compilation, waiting for next'
+ finally:
+ channel.close()
+ gw.exit()
Added: pypy/dist/pypy/tool/build/bin/path.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/bin/path.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,5 @@
+import py
+
+packagedir = py.magic.autopath().dirpath().dirpath()
+rootpath = packagedir.dirpath().dirpath().dirpath()
+py.std.sys.path.append(str(rootpath))
Added: pypy/dist/pypy/tool/build/bin/server
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/bin/server Wed Jul 26 13:18:25 2006
@@ -0,0 +1,27 @@
+#!/usr/bin/python
+
+import path
+from pypy.tool.build import config
+
+from py.execnet import SshGateway
+
+if __name__ == '__main__':
+ from py.execnet import SshGateway
+ from pypy.tool.build.server import init
+
+ gw = SshGateway(config.server)
+ channel = init(gw, port=config.port, path=config.path,
+ projectname=config.projectname,
+ buildpath=str(config.buildpath),
+ mailhost=config.mailhost,
+ mailport=config.mailport,
+ mailfrom=config.mailfrom)
+
+ try:
+ while 1:
+ data = channel.receive()
+ assert isinstance(data, str)
+ print data
+ finally:
+ channel.close()
+ gw.exit()
Added: pypy/dist/pypy/tool/build/bin/startcompile
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/bin/startcompile Wed Jul 26 13:18:25 2006
@@ -0,0 +1,57 @@
+#!/usr/bin/python
+
+import path
+import sys
+import random
+from pypy.tool.build import config
+
+initcode = """
+ import sys
+ sys.path += %r
+
+ from pypy.tool.build import ppbserver
+ channel.send(ppbserver.compile(%r, %r))
+ channel.close()
+"""
+def init(gw, sysinfo, email, port=12321):
+ from pypy.tool.build import execnetconference
+
+ conference = execnetconference.conference(gw, port, False)
+ channel = conference.remote_exec(initcode % (config.path, email, sysinfo))
+ return channel
+
+if __name__ == '__main__':
+ from py.execnet import SshGateway
+
+ from optparse import OptionParser
+ optparser = OptionParser('%prog [options] email')
+ for args, kwargs in config.options:
+ optparser.add_option(*args, **kwargs)
+ optparser.add_option('-r', '--revision', dest='revision', default='trunk',
+ help='SVN revision (defaults to "trunk")')
+
+ (options, args) = optparser.parse_args()
+
+ if not args or len(args) != 1:
+ optparser.error('please provide an email address')
+
+ sysinfo = dict([(attr, getattr(options, attr)) for attr in dir(options) if
+ not attr.startswith('_') and
+ not callable(getattr(options, attr))])
+
+ print 'going to start compile job with info:'
+ for k, v in sysinfo.items():
+ print '%s: %r' % (k, v)
+ print
+
+ gw = SshGateway(config.server)
+ channel = init(gw, sysinfo, args[0], port=config.port)
+ ispath, data = channel.receive()
+ if ispath:
+ print ('a suitable result is already available, you can find it '
+ 'at "%s"' % (data,))
+ else:
+ print data
+ print 'you will be mailed once it\'s ready'
+ channel.close()
+ gw.exit()
Added: pypy/dist/pypy/tool/build/client.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/client.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,71 @@
+import time
+import thread
+
+class PPBClient(object):
+ def __init__(self, channel, sysinfo, testing=False):
+ self.channel = channel
+ self.sysinfo = sysinfo
+ self.busy_on = None
+ self.testing = testing
+
+ from pypy.tool.build import ppbserver
+ self.server = ppbserver
+ self.server.register(self)
+
+ def sit_and_wait(self):
+ """connect to the host and wait for commands"""
+ self.channel.waitclose()
+ self.channel.close()
+
+ def compile(self, info):
+ """send a compile job to the client side
+
+ this waits until the client is done, and assumes the client sends
+ back the whole binary as a single string (XXX this should change ;)
+ """
+ self.busy_on = info
+ self.channel.send(info)
+ thread.start_new_thread(self.wait_until_done, (info,))
+
+ def wait_until_done(self, info):
+ buildpath = self.server.get_new_buildpath(info)
+
+ fp = buildpath.zipfile.open('w')
+ if not self.testing:
+ try:
+ while True:
+ try:
+ chunk = self.channel.receive()
+ except EOFError:
+ # stop compilation, client has disconnected
+ return
+ if chunk is None:
+ break
+ fp.write(chunk)
+ finally:
+ fp.close()
+
+ self.server.compilation_done(info, buildpath)
+ self.busy_on = None
+
+initcode = """
+ import sys
+ sys.path += %r
+
+ from pypy.tool.build.client import PPBClient
+
+ try:
+ client = PPBClient(channel, %r, %r)
+ client.sit_and_wait()
+ finally:
+ channel.close()
+"""
+def init(gw, sysinfo, path=None, port=12321, testing=False):
+ from pypy.tool.build import execnetconference
+
+ if path is None:
+ path = []
+
+ conference = execnetconference.conference(gw, port, False)
+ channel = conference.remote_exec(initcode % (path, sysinfo, testing))
+ return channel
Added: pypy/dist/pypy/tool/build/config.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/config.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,45 @@
+import py
+
+# general settings, used by both server and client
+server = 'johnnydebris.net'
+port = 12321
+path = ['/home/johnny/temp/pypy-dist']
+
+# option definitions for the startcompile script
+# for now we have them here, we should probably use pypy's config instead
+# though...
+import sys
+def _use_modules_callback(option, opt_str, value, parser):
+ parser.values.use_modules = [m.strip() for m in value.split(',')
+ if m.strip()]
+
+def _maxint_callback(option, opt_str, value, parser):
+ parser.values.maxint = 2 ** (int(value) - 1) - 1
+
+options = [
+ (('-m', '--use-modules'), {'action': 'callback', 'type': 'string',
+ 'callback': _use_modules_callback,
+ 'dest': 'use_modules', 'default': [],
+ 'help': 'select the modules you want to use'}),
+ (('-i', '--maxint'), {'action': 'callback', 'callback': _maxint_callback,
+ 'default': sys.maxint, 'dest': 'maxint',
+ 'type': 'string',
+ 'help': ('size of an int in bits (32/64, '
+ 'defaults to sys.maxint)')}),
+ (('-b', '--byteorder'), {'action': 'store',
+ 'dest': 'byteorder', 'default': sys.byteorder,
+ 'nargs': 1,
+ 'help': ('byte order (little/big, defaults '
+ 'to sys.byteorder)')}),
+]
+
+# settings for the server
+projectname = 'pypy'
+buildpath = '/home/johnny/temp/pypy-dist/pypy/tool/build/builds'
+mailhost = '127.0.0.1'
+mailport = 25
+mailfrom = 'johnny at johnnydebris.net'
+
+# settings for the tests
+testpath = [str(py.magic.autopath().dirpath().dirpath())]
+
Added: pypy/dist/pypy/tool/build/conftest.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/conftest.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,20 @@
+import py
+from py.__.documentation.conftest import Directory as Dir, DoctestText, \
+ ReSTChecker
+mypath = py.magic.autopath().dirpath()
+
+Option = py.test.Config.Option
+option = py.test.Config.addoptions("pypybuilder test options",
+ Option('', '--functional',
+ action="store_true", dest="functional", default=False,
+ help="run pypybuilder functional tests"
+ ),
+)
+
+py.test.pypybuilder_option = option
+
+class Directory(Dir):
+ def run(self):
+ if self.fspath == mypath:
+ return ['README.txt', 'test']
+ return super(Directory, self).run()
Added: pypy/dist/pypy/tool/build/execnetconference.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/execnetconference.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,126 @@
+"""
+An extension to py.execnet to allow multiple programs to exchange information
+via a common server. The idea is that all programs first open a gateway to
+the same server (e.g. an SshGateway), and then call the conference() function
+with a local TCP port number. The first program must pass is_server=True and
+the next ones is_server=False: the first program's remote gateway is used as
+shared server for the next ones.
+
+For all programs, the conference() call returns a new gateway that is
+connected to the Python process of this shared server. Information can
+be exchanged by passing data around within this Python process.
+"""
+import py
+from py.__.execnet.register import InstallableGateway
+
+
+def conference(gateway, port, is_server='auto'):
+ if is_server: # True or 'auto'
+ channel = gateway.remote_exec(r"""
+ import thread
+ from socket import *
+ s = socket(AF_INET, SOCK_STREAM)
+ port = channel.receive()
+ try:
+ s.bind(('', port))
+ s.listen(5)
+ except error:
+ channel.send(0)
+ else:
+ channel.send(1)
+
+ def readall(s, n):
+ result = ''
+ while len(result) < n:
+ t = s.read(n-len(result))
+ if not t:
+ raise EOFError
+ result += t
+ return result
+
+ def handle_connexion(clientsock, address):
+ clientfile = clientsock.makefile('r+b',0)
+ source = clientfile.readline().rstrip()
+ clientfile.close()
+ g = {'clientsock' : clientsock, 'address' : address}
+ source = eval(source)
+ if source:
+ g = {'clientsock' : clientsock, 'address' : address}
+ co = compile(source+'\n', source, 'exec')
+ exec co in g
+
+ while True:
+ conn, addr = s.accept()
+ if addr[0] == '127.0.0.1': # else connexion refused
+ thread.start_new_thread(handle_connexion, (conn, addr))
+ del conn
+ """)
+ channel.send(port)
+ ok = channel.receive()
+ if ok:
+ return gateway
+ if is_server == 'auto':
+ pass # fall-through and try as a client
+ else:
+ raise IOError("cannot listen on port %d (already in use?)" % port)
+
+ if 1: # client
+ channel = gateway.remote_exec(r"""
+ import thread
+ from socket import *
+ s = socket(AF_INET, SOCK_STREAM)
+ port = channel.receive()
+ s.connect(('', port))
+ channel.send(1)
+ def receiver(s, channel):
+ while True:
+ data = s.recv(4096)
+ #print >> open('LOG','a'), 'backward', repr(data)
+ channel.send(data)
+ if not data: break
+ thread.start_new_thread(receiver, (s, channel))
+ try:
+ for data in channel:
+ #print >> open('LOG','a'), 'forward', repr(data)
+ s.sendall(data)
+ finally:
+ s.shutdown(1)
+ """)
+ channel.send(port)
+ ok = channel.receive()
+ assert ok
+ return InstallableGateway(ConferenceChannelIO(channel))
+
+
+class ConferenceChannelIO:
+ server_stmt = """
+io = SocketIO(clientsock)
+"""
+
+ error = (EOFError,)
+
+ def __init__(self, channel):
+ self.channel = channel
+ self.buffer = ''
+
+ def read(self, numbytes):
+ #print >> open('LOG', 'a'), 'read %d bytes' % numbytes
+ while len(self.buffer) < numbytes:
+ t = self.channel.receive()
+ if not t:
+ #print >> open('LOG', 'a'), 'EOFError'
+ raise EOFError
+ self.buffer += t
+ buf, self.buffer = self.buffer[:numbytes], self.buffer[numbytes:]
+ #print >> open('LOG', 'a'), '--->', repr(buf)
+ return buf
+
+ def write(self, data):
+ #print >> open('LOG', 'a'), 'write(%r)' % (data,)
+ self.channel.send(data)
+
+ def close_read(self):
+ pass
+
+ def close_write(self):
+ self.channel.close()
Added: pypy/dist/pypy/tool/build/server.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/server.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,332 @@
+import random
+import time
+import thread
+import smtplib
+import py
+
+def issubdict(d1, d2):
+ """sees whether a dict is a 'subset' of another dict
+
+ dictvalues can be immutable data types and list and dicts of
+ immutable data types and lists and ... (recursive)
+ """
+ for k, v in d1.iteritems():
+ if not k in d2:
+ return False
+ d2v = d2[k]
+ if isinstance(v, dict):
+ if not issubdict(v, d2v):
+ return False
+ elif isinstance(v, list):
+ if not set(v).issubset(set(d2v)):
+ return False
+ elif v != d2v:
+ return False
+ return True
+
+# XXX note that all this should be made thread-safe at some point (meaning it
+# currently isn't)!!
+
+class RequestStorage(object):
+ """simple registry that manages information"""
+ def __init__(self, info_to_path=[]):
+ self._id_to_info = {} # id -> info dict
+ self._id_to_emails = {} # id -> requestor email address
+ self._id_to_path = {} # id -> filepath
+
+ self._last_id = 0
+ self._id_lock = thread.allocate_lock()
+
+ self._build_initial(info_to_path)
+
+ def request(self, email, info):
+ """place a request
+
+ this either returns a path to the binary (if it's available
+ already) or an id for the info
+ """
+ self._normalize(info)
+ infoid = self.get_info_id(info)
+ path = self._id_to_path.get(infoid)
+ if path is not None:
+ return path
+ self._id_to_emails.setdefault(infoid, []).append(email)
+
+ def get_info_id(self, info):
+ """retrieve or create an id for an info dict"""
+ self._id_lock.acquire()
+ try:
+ self._normalize(info)
+ for k, v in self._id_to_info.iteritems():
+ if v == info:
+ return k
+ self._last_id += 1
+ id = self._last_id
+ self._id_to_info[id] = info
+ return id
+ finally:
+ self._id_lock.release()
+
+ def add_build(self, info, path):
+ """store the data for a build and make it available
+
+ returns a list of email addresses for the people that should be
+ warned
+ """
+ self._normalize(info)
+ infoid = self.get_info_id(info)
+ emails = self._id_to_emails.pop(infoid)
+ self._id_to_path[infoid] = path
+ return emails
+
+ def _build_initial(self, info_to_path):
+ """fill the dicts with info about files that are already built"""
+ for info, path in info_to_path:
+ id = self.get_info_id(info)
+ self._id_to_path[id] = path
+
+ def _normalize(self, info):
+ for k, v in info.iteritems():
+ if isinstance(v, list):
+ v.sort()
+
+from py.__.path.local.local import LocalPath
+class BuildPath(LocalPath):
+ def _info(self):
+ info = getattr(self, '_info_value', {})
+ if info:
+ return info
+ infopath = self / 'info.txt'
+ if not infopath.check():
+ return {}
+ for line in infopath.readlines():
+ line = line.strip()
+ if not line:
+ continue
+ chunks = line.split(':')
+ key = chunks.pop(0)
+ value = ':'.join(chunks)
+ info[key] = eval(value)
+ self._info_value = info
+ return info
+
+ def _set_info(self, info):
+ self._info_value = info
+ infopath = self / 'info.txt'
+ infopath.ensure()
+ fp = infopath.open('w')
+ try:
+ for key, value in info.iteritems():
+ fp.write('%s: %r\n' % (key, value))
+ finally:
+ fp.close()
+
+ info = property(_info, _set_info)
+
+ def _zipfile(self):
+ return py.path.local(self / 'data.zip')
+
+ def _set_zipfile(self, iterable):
+ # XXX not in use right now...
+ fp = self._zipfile().open('w')
+ try:
+ for chunk in iterable:
+ fp.write(chunk)
+ finally:
+ fp.close()
+
+ zipfile = property(_zipfile, _set_zipfile)
+
+class PPBServer(object):
+ retry_interval = 10
+
+ def __init__(self, projname, channel, builddir, mailhost=None,
+ mailport=None, mailfrom=None):
+ self._projname = projname
+ self._channel = channel
+ self._builddir = builddir
+ self._mailhost = mailhost
+ self._mailport = mailport
+ self._mailfrom = mailfrom
+
+ self._buildpath = py.path.local(builddir)
+ self._clients = []
+ info_to_path = [(p.info, str(p)) for p in
+ self._get_buildpaths(builddir)]
+ self._requeststorage = RequestStorage(info_to_path)
+ self._queued = []
+
+ self._queuelock = thread.allocate_lock()
+ self._namelock = thread.allocate_lock()
+
+ def register(self, client):
+ self._clients.append(client)
+ self._channel.send('registered %s with info %r' % (
+ client, client.sysinfo))
+ client.channel.send('welcome')
+
+ def compile(self, requester_email, info):
+ """start a compilation
+
+ returns a tuple (ispath, data)
+
+ if there's already a build available for info, this will return
+ a tuple (True, path), if not, this will return (False, message),
+ where message describes what is happening with the request (is
+ a build made rightaway, or is there no client available?)
+
+ in any case, if the first item of the tuple returned is False,
+ an email will be sent once the build is available
+ """
+ path = self._requeststorage.request(requester_email, info)
+ if path is not None:
+ self._channel.send('already a build for this info available')
+ return (True, path)
+ for client in self._clients:
+ if client.busy_on == info:
+ self._channel.send('build for %r currently in progress' %
+ (info,))
+ return (False, 'this build is already in progress')
+ # we don't have a build for this yet, find a client to compile it
+ if self.run(info):
+ return (False, 'found a suitable client, going to build')
+ else:
+ self._queuelock.acquire()
+ try:
+ self._queued.append(info)
+ finally:
+ self._queuelock.release()
+ return (False, 'no suitable client found; your request is queued')
+
+ def run(self, info):
+ """find a suitable client and run the job if possible"""
+ # XXX shuffle should be replaced by something smarter obviously ;)
+ clients = self._clients[:]
+ random.shuffle(clients)
+ rev = info.pop('revision', 'trunk')
+ for client in clients:
+ # popping out revision here, going to add later... the client
+ # should be able to retrieve source code for any revision (so
+ # it doesn't need to match a revision field in client.sysinfo)
+ if client.busy_on or not issubdict(info, client.sysinfo):
+ continue
+ else:
+ info['revision'] = rev
+ self._channel.send(
+ 'going to send compile job with info %r to %s' % (
+ info, client
+ )
+ )
+ client.compile(info)
+ return True
+ info['revision'] = rev
+ self._channel.send(
+ 'no suitable client available for compilation with info %r' % (
+ info,
+ )
+ )
+
+ def serve_forever(self):
+ """this keeps the script from dying, and re-tries jobs"""
+ self._channel.send('going to serve')
+ while 1:
+ time.sleep(self.retry_interval)
+ self._cleanup_clients()
+ self._try_queued()
+
+ def get_new_buildpath(self, info):
+ path = BuildPath(str(self._buildpath / self._create_filename()))
+ path.info = info
+ return path
+
+ def compilation_done(self, info, path):
+ """client is done with compiling and sends data"""
+ self._channel.send('compilation done for %r, written to %s' % (
+ info, path))
+ emails = self._requeststorage.add_build(info, path)
+ for emailaddr in emails:
+ self._send_email(emailaddr, info, path)
+
+ def _cleanup_clients(self):
+ self._queuelock.acquire()
+ try:
+ clients = self._clients[:]
+ for client in clients:
+ if client.channel.isclosed():
+ if client.busy_on:
+ self._queued.append(client.busy_on)
+ self._clients.remove(client)
+ finally:
+ self._queuelock.release()
+
+ def _try_queued(self):
+ self._queuelock.acquire()
+ try:
+ toremove = []
+ for info in self._queued:
+ if self.run(info):
+ toremove.append(info)
+ for info in toremove:
+ self._queued.remove(info)
+ finally:
+ self._queuelock.release()
+
+ def _get_buildpaths(self, dirpath):
+ for p in py.path.local(dirpath).listdir():
+ yield BuildPath(str(p))
+
+ _i = 0
+ def _create_filename(self):
+ self._namelock.acquire()
+ try:
+ today = time.strftime('%Y%m%d')
+ buildnames = [p.basename for p in
+ py.path.local(self._buildpath).listdir()]
+ while True:
+ name = '%s-%s-%s' % (self._projname, today, self._i)
+ self._i += 1
+ if name not in buildnames:
+ return name
+ finally:
+ self._namelock.release()
+
+ def _send_email(self, addr, info, path):
+ self._channel.send('going to send email to %s' % (addr,))
+ if self._mailhost is not None:
+ msg = '\r\n'.join([
+ 'From: %s' % (self._mailfrom,),
+ 'To: %s' % (addr,),
+ 'Subject: %s compilation done' % (self._projname,),
+ '',
+ 'The compilation you requested is done. You can find it at',
+ str(path),
+ ])
+ server = smtplib.SMTP(self._mailhost, self._mailport)
+ server.set_debuglevel(0)
+ server.sendmail(self._mailfrom, addr, msg)
+ server.quit()
+
+initcode = """
+ import sys
+ sys.path += %r
+
+ try:
+ from pypy.tool.build.server import PPBServer
+ server = PPBServer(%r, channel, %r, %r, %r, %r)
+
+ # make the server available to clients as pypy.tool.build.ppbserver
+ from pypy.tool import build
+ build.ppbserver = server
+
+ server.serve_forever()
+ finally:
+ channel.close()
+"""
+def init(gw, port=12321, path=[], projectname='pypy', buildpath=None,
+ mailhost=None, mailport=25, mailfrom=None):
+ from pypy.tool.build import execnetconference
+ conference = execnetconference.conference(gw, port, True)
+ channel = conference.remote_exec(initcode % (path, projectname, buildpath,
+ mailhost, mailport,
+ mailfrom))
+ return channel
Added: pypy/dist/pypy/tool/build/test/fake.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/fake.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,54 @@
+from pypy.tool.build.server import BuildPath
+
+class FakeChannel(object):
+ def __init__(self):
+ self._buffer = []
+
+ def send(self, item):
+ self._buffer.append(item)
+
+ def receive(self):
+ return self._buffer.pop(0)
+
+ def close(self):
+ pass
+
+ def waitclose(self):
+ pass
+
+class FakeClient(object):
+ def __init__(self, info):
+ self.channel = FakeChannel()
+ self.sysinfo = info
+ self.busy_on = None
+
+ def compile(self, info):
+ info.pop('revision')
+ for k, v in info.items():
+ self.channel.send('%s: %r' % (k, v))
+ self.channel.send(None)
+ self.busy_on = info
+
+class FakeServer(object):
+ def __init__(self, builddirpath):
+ builddirpath.ensure(dir=True)
+ self._channel = FakeChannel()
+ self._builddirpath = builddirpath
+ self._clients = []
+ self._done = []
+
+ def register(self, client):
+ self._clients.append(client)
+
+ def compilation_done(self, info, data):
+ self._done.append((info, data))
+
+ i = 0
+ def get_new_buildpath(self, info):
+ name = 'build-%s' % (self.i,)
+ self.i += 1
+ bp = BuildPath(str(self._builddirpath / name))
+ bp.info = info
+ bp.ensure(dir=1)
+ return bp
+
Added: pypy/dist/pypy/tool/build/test/path.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/path.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,6 @@
+import py
+
+testpath = py.magic.autopath().dirpath()
+packagepath = testpath.dirpath()
+rootpath = packagepath.dirpath().dirpath().dirpath()
+py.std.sys.path.append(str(rootpath))
Added: pypy/dist/pypy/tool/build/test/test.zip
==============================================================================
Binary file. No diff available.
Added: pypy/dist/pypy/tool/build/test/test_client.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/test_client.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,43 @@
+import path
+from pypy.tool.build import client
+import py
+import time
+from fake import FakeChannel, FakeServer
+
+class ClientForTests(client.PPBClient):
+ def __init__(self, *args, **kwargs):
+ super(ClientForTests, self).__init__(*args, **kwargs)
+ self._done = []
+
+def setup_module(mod):
+ mod.temp = temp = py.test.ensuretemp('pypybuilder-client')
+ mod.svr = svr = FakeServer(temp)
+
+ import pypy.tool.build
+ pypy.tool.build.ppbserver = svr
+
+ mod.c1c = c1c = FakeChannel()
+ mod.c1 = c1 = ClientForTests(c1c, {'foo': 1, 'bar': [1,2]})
+ svr.register(c1)
+
+ mod.c2c = c2c = FakeChannel()
+ mod.c2 = c2 = ClientForTests(c2c, {'foo': 2, 'bar': [2,3]})
+ svr.register(c2)
+
+def test_compile():
+ info = {'foo': 1}
+ c1.compile(info)
+ c1.channel.receive()
+ c1.channel.send('foo bar')
+ c1.channel.send(None)
+
+ # meanwhile the client starts a thread that waits until there's data
+ # available on its own channel, with our FakeChannel it has data rightaway,
+ # though (the channel out and in are the same, and we just sent 'info'
+ # over the out one)
+ time.sleep(1)
+
+ done = svr._done.pop()
+
+ assert done[0] == info
+ assert done[1] == (temp / 'build-0')
Added: pypy/dist/pypy/tool/build/test/test_pypybuilder.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/test_pypybuilder.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,137 @@
+import path
+from pypy.tool.build import client, server, execnetconference
+from pypy.tool.build import config
+import py
+
+# some functional tests (although some of the rest aren't strictly
+# unit tests either), to run use --functional as an arg to py.test
+def test_functional_1():
+ if not py.test.pypybuilder_option.functional:
+ py.test.skip('skipping functional test, use --functional to run it')
+
+ # XXX this one is a bit messy, it's a quick functional test for the whole
+ # system, but for instance contains time.sleep()s to make sure all threads
+ # get the time to perform tasks and such...
+
+ sleep_interval = 0.3
+
+ # first initialize a server
+ sgw = py.execnet.PopenGateway()
+ temppath = py.test.ensuretemp('pypybuilder-functional')
+ sc = server.init(sgw, port=config.port, path=config.testpath,
+ buildpath=str(temppath))
+
+ # give the server some time to wake up
+ py.std.time.sleep(sleep_interval)
+
+ # then two clients, both with different system info
+ sysinfo1 = {
+ 'foo': 1,
+ 'bar': [1,2],
+ }
+ cgw1 = py.execnet.PopenGateway()
+ cc1 = client.init(cgw1, sysinfo1, port=config.port, testing=True)
+
+ sysinfo2 = {
+ 'foo': 2,
+ 'bar': [1],
+ }
+ cgw2 = py.execnet.PopenGateway()
+ cc2 = client.init(cgw2, sysinfo2, port=config.port, testing=True)
+
+ # give the clients some time to register themselves
+ py.std.time.sleep(sleep_interval)
+
+ # now we're going to send some compile jobs
+ code = """
+ import sys
+ sys.path += %r
+
+ from pypy.tool.build import ppbserver
+ channel.send(ppbserver.compile(%r, %r))
+ channel.close()
+ """
+ compgw = py.execnet.PopenGateway()
+ compconf = execnetconference.conference(compgw, config.port)
+
+ # this one should fail because there's no client found for foo = 3
+ compc = compconf.remote_exec(code % (config.testpath, 'foo1 at bar.com',
+ {'foo': 3}))
+
+ # sorry...
+ py.std.time.sleep(sleep_interval)
+
+ ret = compc.receive()
+ assert not ret[0]
+ assert ret[1].find('no suitable client found') > -1
+
+ # this one should be handled by client 1
+ compc = compconf.remote_exec(code % (config.testpath, 'foo2 at bar.com',
+ {'foo': 1, 'bar': [1]}))
+
+ # and another one
+ py.std.time.sleep(sleep_interval)
+
+ ret = compc.receive()
+ assert not ret[0]
+ assert ret[1].find('found a suitable client') > -1
+
+ # the messages may take a bit to arrive, too
+ py.std.time.sleep(sleep_interval)
+
+ # client 1 should by now have received the info to build for
+ cc1.receive() # 'welcome'
+ ret = cc1.receive()
+ assert ret == {'foo': 1, 'bar': [1], 'revision': 'trunk'}
+
+ # this should have created a package in the temp dir
+ assert len(temppath.listdir()) == 1
+
+ # now we're going to satisfy the first request by adding a new client
+ sysinfo3 = {'foo': 3}
+ cgw3 = py.execnet.PopenGateway()
+ cc3 = client.init(cgw3, sysinfo3, port=config.port, testing=True)
+
+ # again a bit of waiting may be desired
+ py.std.time.sleep(sleep_interval)
+
+ # _try_queued() should check whether there are new clients available for
+ # queued jobs
+ code = """
+ import sys, time
+ sys.path += %r
+
+ from pypy.tool.build import ppbserver
+ ppbserver._try_queued()
+ # give the server some time, the clients 'compile' in threads
+ time.sleep(%s)
+ channel.send(ppbserver._requeststorage._id_to_emails)
+ channel.close()
+ """
+ compgw2 = py.execnet.PopenGateway()
+ compconf2 = execnetconference.conference(compgw2, config.port)
+
+ compc2 = compconf2.remote_exec(code % (config.testpath, sleep_interval))
+
+
+ # we check whether all emails are now sent, since after adding the third
+ # client, and calling _try_queued(), both jobs should have been processed
+ ret = compc2.receive()
+ assert ret.values() == []
+
+ # this should also have created another package in the temp dir
+ assert len(temppath.listdir()) == 2
+
+ # some cleanup (this should all be in nested try/finallys, blegh)
+ cc1.close()
+ cc2.close()
+ cc3.close()
+ compc.close()
+ compc2.close()
+ sc.close()
+
+ cgw1.exit()
+ cgw2.exit()
+ compgw.exit()
+ compgw2.exit()
+ sgw.exit()
Added: pypy/dist/pypy/tool/build/test/test_request_storage.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/test_request_storage.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,64 @@
+import path
+import py
+from pypy.tool.build.server import RequestStorage
+
+def test_request_storage():
+ s = RequestStorage()
+
+ assert s._id_to_info == {}
+ assert s._id_to_emails == {}
+ assert s._id_to_path == {}
+
+ info = {'foo': 1}
+ infoid = s.get_info_id(info)
+
+ path = s.request('foo at bar.com', info)
+ assert path is None
+ assert s._id_to_info == {infoid: info}
+ assert s._id_to_emails == {infoid: ['foo at bar.com']}
+ assert s._id_to_path == {}
+
+ path = s.request('bar at bar.com', info)
+ assert path is None
+ assert s._id_to_info == {infoid: info}
+ assert s._id_to_emails == {infoid: ['foo at bar.com', 'bar at bar.com']}
+ assert s._id_to_path == {}
+
+ emails = s.add_build(info, 'foobar')
+ assert emails == ['foo at bar.com', 'bar at bar.com']
+ assert s._id_to_info == {infoid: info}
+ assert s._id_to_emails == {}
+ assert s._id_to_path == {infoid: 'foobar'}
+
+ info2 = {'foo': 2, 'bar': [1,2]}
+ infoid2 = s.get_info_id(info2)
+
+ path = s.request('foo at baz.com', info2)
+ assert path is None
+ assert s._id_to_info == {infoid: info, infoid2: info2}
+ assert s._id_to_emails == {infoid2: ['foo at baz.com']}
+ assert s._id_to_path == {infoid: 'foobar'}
+
+ emails = s.add_build(info2, 'foobaz')
+ assert emails == ['foo at baz.com']
+ assert s._id_to_info == {infoid: info, infoid2: info2}
+ assert s._id_to_emails == {}
+ assert s._id_to_path == {infoid: 'foobar', infoid2: 'foobaz'}
+
+ path = s.request('foo at qux.com', info)
+ assert path == 'foobar'
+
+def test__build_initial():
+ s = RequestStorage([({'foo': 1}, 'foo'), ({'foo': 2}, 'bar'),])
+
+ id1 = s.get_info_id({'foo': 1})
+ id2 = s.get_info_id({'foo': 2})
+
+ assert s._id_to_info == {id1: {'foo': 1}, id2: {'foo': 2}}
+ assert s._id_to_emails == {}
+ assert s._id_to_path == {id1: 'foo', id2: 'bar'}
+
+def test__normalize():
+ s = RequestStorage()
+ assert (s._normalize({'foo': ['bar', 'baz']}) ==
+ s._normalize({'foo': ['baz', 'bar']}))
Added: pypy/dist/pypy/tool/build/test/test_server.py
==============================================================================
--- (empty file)
+++ pypy/dist/pypy/tool/build/test/test_server.py Wed Jul 26 13:18:25 2006
@@ -0,0 +1,132 @@
+import path
+from pypy.tool.build import server
+import py
+from fake import FakeChannel, FakeClient
+from pypy.tool.build.server import RequestStorage
+from pypy.tool.build.server import BuildPath
+import time
+
+def setup_module(mod):
+ mod.temppath = temppath = py.test.ensuretemp('pypybuilder-server')
+ mod.svr = server.PPBServer('pypytest', FakeChannel(), str(temppath))
+
+ mod.c1 = FakeClient({'foo': 1, 'bar': [1,2]})
+ mod.svr.register(mod.c1)
+
+ mod.c2 = FakeClient({'foo': 2, 'bar': [2,3]})
+ mod.svr.register(mod.c2)
+
+def test_server_issubdict():
+ from pypy.tool.build.server import issubdict
+ assert issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2, 'baz': 3})
+ assert not issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'baz': 3})
+ assert not issubdict({'foo': 1, 'bar': 3}, {'foo': 1, 'bar': 2, 'baz': 3})
+ assert issubdict({'foo': [1,2]}, {'foo': [1,2,3]})
+ assert not issubdict({'foo': [1,2,3]}, {'foo': [1,2]})
+ assert issubdict({'foo': 1L}, {'foo': 1})
+ assert issubdict({}, {'foo': 1})
+ assert issubdict({'foo': [1,2]}, {'foo': [1,2,3,4], 'bar': [1,2]})
+
+# XXX: note that the order of the tests matters! the first test reads the
+# information from the channels that was set by the setup_module() function,
+# the rest assumes this information is already read...
+
+def test_register():
+ assert len(svr._clients) == 2
+ assert svr._clients[0] == c1
+ assert svr._clients[1] == c2
+
+ assert c1.channel.receive() == 'welcome'
+ assert c2.channel.receive() == 'welcome'
+ py.test.raises(IndexError, "c1.channel.receive()")
+
+ assert svr._channel.receive().find('registered') > -1
+ assert svr._channel.receive().find('registered') > -1
+ py.test.raises(IndexError, 'svr._channel.receive()')
+
+def test_compile():
+ # XXX this relies on the output not changing... quite scary
+ info = {'foo': 1}
+ ret = svr.compile('test at domain.com', info)
+ assert not ret[0]
+ assert ret[1].find('found a suitable client') > -1
+ assert svr._channel.receive().find('going to send compile job') > -1
+ assert c1.channel.receive() == 'foo: 1'
+ assert c1.channel.receive() is None
+ py.test.raises(IndexError, "c2.channel.receive()")
+
+ svr.compile('test at domain.com', {'foo': 3})
+ assert svr._channel.receive().find('no suitable client available') > -1
+
+ info = {'bar': [3]}
+ ret = svr.compile('test at domain.com', info)
+ assert svr._channel.receive().find('going to send') > -1
+ assert c2.channel.receive() == 'bar: [3]'
+ assert c2.channel.receive() is None
+ py.test.raises(IndexError, "c1.channel.receive()")
+
+ info = {'foo': 1}
+ ret = svr.compile('test at domain.com', info)
+ assert not ret[0]
+ assert ret[1].find('this build is already') > -1
+ assert svr._channel.receive().find('currently in progress') > -1
+
+ c1.busy_on = None
+ bp = BuildPath(str(temppath / 'foo'))
+ svr.compilation_done(info, bp)
+ ret = svr.compile('test at domain.com', info)
+ assert ret[0]
+ assert isinstance(ret[1], BuildPath)
+ assert ret[1] == bp
+ assert svr._channel.receive().find('compilation done for') > -1
+ for i in range(2):
+ assert svr._channel.receive().find('going to send email to') > -1
+ assert svr._channel.receive().find('already a build for this info') > -1
+
+def test_buildpath():
+ tempdir = py.test.ensuretemp('pypybuilder-buildpath')
+ # grmbl... local.__new__ checks for class equality :(
+ bp = BuildPath(str(tempdir / 'test1'))
+ assert not bp.check()
+ assert bp.info == {}
+
+ bp.info = {'foo': 1, 'bar': [1,2]}
+ assert bp.info == {'foo': 1, 'bar': [1,2]}
+ assert (sorted((bp / 'info.txt').readlines()) ==
+ ['bar: [1, 2]\n', 'foo: 1\n'])
+
+ assert isinstance(bp.zipfile, py.path.local)
+ bp.zipfile = ['foo', 'bar', 'baz']
+ assert bp.zipfile.read() == 'foobarbaz'
+
+def test__create_filename():
+ svr._i = 0 # reset counter
+ today = time.strftime('%Y%m%d')
+ name1 = svr._create_filename()
+ assert name1 == 'pypytest-%s-0' % (today,)
+ assert svr._create_filename() == ('pypytest-%s-1' % (today,))
+ bp = BuildPath(str(temppath / ('pypytest-%s-2' % (today,))))
+ try:
+ bp.ensure()
+ assert svr._create_filename() == 'pypytest-%s-3'% (today,)
+ finally:
+ bp.remove()
+
+def test_get_new_buildpath():
+ svr._i = 0
+ today = time.strftime('%Y%m%d')
+
+ path1 = svr.get_new_buildpath({'foo': 'bar'})
+ try:
+ assert isinstance(path1, BuildPath)
+ assert path1.info == {'foo': 'bar'}
+ assert path1.basename == 'pypytest-%s-0' % (today,)
+
+ try:
+ path2 = svr.get_new_buildpath({'foo': 'baz'})
+ assert path2.info == {'foo': 'baz'}
+ assert path2.basename == 'pypytest-%s-1' % (today,)
+ finally:
+ path2.remove()
+ finally:
+ path1.remove()
More information about the Pypy-commit
mailing list