Interfacing the buildbot with bug tracking

Since the Plone project switched from CMFCollector to trac I’ve been
jealous of their capability of closing issues by way of commit
messages. Well, not that much people use it. I believe it’s highly
underused these days except by a few brave folks.

Nonetheless, I had this in the back of my mind, and thought that
perhaps abusing our buildbot to perform similar stuff would be an
interesting task for one of those friday evenings.

And indeed, in a friday evening it happened. After reading the
buildbot interface definitions for a little while I realized that
writing a Scheduler that didn’t schedule anything but instead just
made a XML-RPC call to Zope performing the desired action would be
the easiest win.

For those interested in the code, I’m reproducing part of it
below. It’s really simple stuff, and I was surprised by how easy and
how little I had to learn about Twisted to get it to do what I wanted.

Here’s the Scheduler bit, with the nasty regex that parses the
commit message:

import re
from buildbot.scheduler import BaseScheduler
from buildbot.changes.changes import Change
from twisted.web.xmlrpc import Proxy

COMMAND = re.compile(
    r'(?P<action>[A-Za-z]*)(?:[ ]?(?:issue|issues)[ ]?)*(?:[: ]*).?'
    r'(?P<ticket>#[0-9]+(?:(?:[, &]*|[ ]?and[ ]?)#[0-9]+)*)'
    r'(?:[ ]?to[ ]?(?P<who>[A-Za-z0-9]*(?:(?:[, &]*|[ ]?and[ ]?)[A-Za-z0-9]+)*))*'
    r'(?:[ ]?(?:in|on|of|at)+[ ]?(?P<where>internal|public|test)(?:[ ]?(?P<what>desktop|server|proxy)))+')

TICKET = re.compile(r'#([0-9]*)')
WHO = re.compile(r'([A-Za-z0-9]+)')
KNOWN_COMMANDS = {'close'   : 'resolve',
                  'closes'  : 'resolve',
                  'closed'  : 'resolve',
                  'fix'     : 'resolve',
                  'fixed'   : 'resolve',
                  'fixes'   : 'resolve',
                  'resolve' : 'resolve',
                  'assign'  : 'assign',
                  'assigned': 'assign',
                  'review'  : 'assign',
                  're'      : 'comment',

def parse_action_bugs(msg):
    >>> from pprint import pprint

    >>> pprint(parse_action_bugs("""
    ...  - Fixes #330 and #339 in public server
    ...  - Fixed #331 at internal proxy
    ...  - Fixed issue #339 on internal desktop
    ...  - Fixes issues #330 and #339 of internal server
    ...  - Closes #320, #321 on public proxy
    ...  - Re: #334, #321 of internal proxy
    ...  - Assign #312 to cjohnson on internal desktop
    ...  - Assign #322 to cjohnson, sidnei and markh on public server
    ... """))
    [('resolve', ('330', '339'), (), 'public', 'server'),
     ('resolve', ('331',), (), 'internal', 'proxy'),
     ('resolve', ('339',), (), 'internal', 'desktop'),
     ('resolve', ('330', '339'), (), 'internal', 'server'),
     ('resolve', ('320', '321'), (), 'public', 'proxy'),
     ('comment', ('334', '321'), (), 'internal', 'proxy'),
     ('assign', ('312',), ('cjohnson',), 'internal', 'desktop'),
     ('assign', ('322',), ('cjohnson', 'sidnei', 'markh'), 'public', 'server')]
    actions = []
    for action, ticket, who, where, what in COMMAND.findall(msg):
        bugs = tuple(TICKET.findall(ticket))
        to = tuple([name for name in WHO.findall(who) if not name == 'and'])
        cmd = KNOWN_COMMANDS.get(action.lower())
        if cmd is None:
        actions.append((cmd, bugs, to, where, what))
    return actions

class CollectorNotifyScheduler(BaseScheduler):

    builderNames = ()

    def __init__(self, name, collector_mapping):
        self.collector_mapping = collector_mapping
        BaseScheduler.__init__(self, name)

    def listBuilderNames(self):
        return self.builderNames

    def addChange(self, change):
        print 'Adding change %r' % change
        actions = parse_action_bugs(change.comments)
        print actions
        for action in actions:
            self.processCommand(change, action)

    def processCommand(self, change, action):
        cmd, bugs, to, where, what = action
        loc = self.collector_mapping.get(where)
        if loc is None:
            # Doesn't map to a known collector location
        collector_url = loc.get(what)
        if collector_url is None:
            # Doesn't map to a known product collector
        changed_files = '\n'.join([' - %s' % fname for fname in change.files])
        msg = '%s\n\nIn revision: %s.\n\nChanged files:\n\n%s' % (

        print msg

        if '@' in collector_url:
            proxy = AuthProxy(collector_url)
            proxy = Proxy(collector_url)

        defer = proxy.callRemote('collector_issue_notify',
                                 change.who, cmd, bugs, to, msg)
        defer.addCallbacks(self._commandCallback, self._commandErrorCallback)

    def _commandErrorCallback(self, error):
        print error

    def _commandCallback(self, value):
        print value

I’ve used the AuthProxy and URI recipes from ASPN to allow usage of
authentication with XML-RPC.

Hooking this into the buildbot is very simple, here are the relevant bits:

from collector_notify import CollectorNotifyScheduler

collector_mapping = {'internal': {
  'server': 'http://intranet.enfoldsystems.local:8080/Plone/products/server/collector'}}
c = BuildmasterConfig = {}
    CollectorNotifyScheduler('collector_notify', collector_mapping)

And finally, on the Zope side, a not that simple yet not too terribly
complex method, that is attached to the Collector object itself
and called via XML-RPC. An interesting note about this method is
that it ‘impersonates’ the user that made the checkin by switching the
SecurityManager in Zope for a short period. Think of the sudo
command, if you like:

from StringIO import StringIO

from AccessControl import Unauthorized
from AccessControl.SecurityManagement import getSecurityManager
from AccessControl.SecurityManagement import setSecurityManager
from AccessControl.SecurityManagement import newSecurityManager

class Auth:

    def __init__(self, user_id):
        self.user_id = user_id

    def acquire(self, context):
        self.manager = getSecurityManager()
        user = context.acl_users.getUserById(self.user_id)
        if user is None:
        wrapped = user.__of__(context.acl_users)
        newSecurityManager(None, wrapped)
        return wrapped

    def validate(self, object, name):
            return getSecurityManager().validate(
                None, object, name, getattr(object, name))
        except Unauthorized:
            return False

    def release(self):

def collector_issue_notify(self, who, cmd, bugs, to, msg):
    """Handle a command for a contained issue.
    out = StringIO()
    auth = Auth(who)
    user = auth.acquire(self)
        url = self.absolute_url()
        if user is None:
            print >> out, 'Could not find user %s at %s' % (who, url)
            for bug in bugs:

                    issue = self[bug]
                except (KeyError, AttributeError):
                    print >> out, 'Could not find bug %s at %s' % (bug, url)

                if not auth.validate(issue, 'do_action'):
                    print >> out, ('User %s is not allowed to perform action '
                                   'on bug %s at %s' % (who, bug, url))

                issue.do_action(cmd, msg, assignees=to)
                print >> out, ('Executed action %s on issue %s '
                               'at %s' % (cmd, bug, url))
        return out.getvalue()

I hope that is of use to anyone. I know there are several people out
there using the buildbot yet very few still using
CMFCollector. One way or another, it shows the ease of extending
the buildbot to perform other tasks than it was intended for, and also
the ease of extending Zope to be called externally via XML-RPC.

All in all, the hardest part of this was writing the regex to parse
the commit message. :)

If this example is of any use to you and if you use it and make any
improvement please post a comment here, thank you!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s