Since the Plone project switched from CMFCollector to trac I’ve been
jealous of their capability of closing issues by way of commit
messages. Well, not that much people use it. I believe it’s highly
underused these days except by a few brave folks.
Nonetheless, I had this in the back of my mind, and thought that
perhaps abusing our buildbot to perform similar stuff would be an
interesting task for one of those friday evenings.
And indeed, in a friday evening it happened. After reading the
buildbot interface definitions for a little while I realized that
writing a Scheduler that didn’t schedule anything but instead just
made a XML-RPC call to Zope performing the desired action would be
the easiest win.
For those interested in the code, I’m reproducing part of it
below. It’s really simple stuff, and I was surprised by how easy and
how little I had to learn about Twisted to get it to do what I wanted.
Here’s the Scheduler bit, with the nasty regex that parses the
commit message:
import re
from buildbot.scheduler import BaseScheduler
from buildbot.changes.changes import Change
from twisted.web.xmlrpc import Proxy
COMMAND = re.compile(
r'(?P<action>[A-Za-z]*)(?:[ ]?(?:issue|issues)[ ]?)*(?:[: ]*).?'
r'(?P<ticket>#[0-9]+(?:(?:[, &]*|[ ]?and[ ]?)#[0-9]+)*)'
r'(?:[ ]?to[ ]?(?P<who>[A-Za-z0-9]*(?:(?:[, &]*|[ ]?and[ ]?)[A-Za-z0-9]+)*))*'
r'(?:[ ]?(?:in|on|of|at)+[ ]?(?P<where>internal|public|test)(?:[ ]?(?P<what>desktop|server|proxy)))+')
TICKET = re.compile(r'#([0-9]*)')
WHO = re.compile(r'([A-Za-z0-9]+)')
KNOWN_COMMANDS = {'close' : 'resolve',
'closes' : 'resolve',
'closed' : 'resolve',
'fix' : 'resolve',
'fixed' : 'resolve',
'fixes' : 'resolve',
'resolve' : 'resolve',
'assign' : 'assign',
'assigned': 'assign',
'review' : 'assign',
're' : 'comment',
}
def parse_action_bugs(msg):
'''
>>> from pprint import pprint
>>> pprint(parse_action_bugs("""
... - Fixes #330 and #339 in public server
... - Fixed #331 at internal proxy
... - Fixed issue #339 on internal desktop
... - Fixes issues #330 and #339 of internal server
... - Closes #320, #321 on public proxy
... - Re: #334, #321 of internal proxy
... - Assign #312 to cjohnson on internal desktop
... - Assign #322 to cjohnson, sidnei and markh on public server
... """))
[('resolve', ('330', '339'), (), 'public', 'server'),
('resolve', ('331',), (), 'internal', 'proxy'),
('resolve', ('339',), (), 'internal', 'desktop'),
('resolve', ('330', '339'), (), 'internal', 'server'),
('resolve', ('320', '321'), (), 'public', 'proxy'),
('comment', ('334', '321'), (), 'internal', 'proxy'),
('assign', ('312',), ('cjohnson',), 'internal', 'desktop'),
('assign', ('322',), ('cjohnson', 'sidnei', 'markh'), 'public', 'server')]
'''
actions = []
for action, ticket, who, where, what in COMMAND.findall(msg):
bugs = tuple(TICKET.findall(ticket))
to = tuple([name for name in WHO.findall(who) if not name == 'and'])
cmd = KNOWN_COMMANDS.get(action.lower())
if cmd is None:
continue
actions.append((cmd, bugs, to, where, what))
return actions
class CollectorNotifyScheduler(BaseScheduler):
builderNames = ()
def __init__(self, name, collector_mapping):
self.collector_mapping = collector_mapping
BaseScheduler.__init__(self, name)
def listBuilderNames(self):
return self.builderNames
def addChange(self, change):
print 'Adding change %r' % change
actions = parse_action_bugs(change.comments)
print actions
for action in actions:
self.processCommand(change, action)
def processCommand(self, change, action):
cmd, bugs, to, where, what = action
loc = self.collector_mapping.get(where)
if loc is None:
# Doesn't map to a known collector location
return
collector_url = loc.get(what)
if collector_url is None:
# Doesn't map to a known product collector
return
changed_files = '\n'.join([' - %s' % fname for fname in change.files])
msg = '%s\n\nIn revision: %s.\n\nChanged files:\n\n%s' % (
change.comments.strip('\n'),
change.revision,
changed_files)
print msg
if '@' in collector_url:
proxy = AuthProxy(collector_url)
else:
proxy = Proxy(collector_url)
defer = proxy.callRemote('collector_issue_notify',
change.who, cmd, bugs, to, msg)
defer.addCallbacks(self._commandCallback, self._commandErrorCallback)
def _commandErrorCallback(self, error):
print error
def _commandCallback(self, value):
print value
I’ve used the AuthProxy and URI recipes from ASPN to allow usage of
authentication with XML-RPC.
Hooking this into the buildbot is very simple, here are the relevant bits:
from collector_notify import CollectorNotifyScheduler
collector_mapping = {'internal': {
'server': 'http://intranet.enfoldsystems.local:8080/Plone/products/server/collector'}}
c = BuildmasterConfig = {}
c['schedulers'].append(
CollectorNotifyScheduler('collector_notify', collector_mapping)
)
And finally, on the Zope side, a not that simple yet not too terribly
complex method, that is attached to the Collector object itself
and called via XML-RPC. An interesting note about this method is
that it ‘impersonates’ the user that made the checkin by switching the
SecurityManager in Zope for a short period. Think of the sudo
command, if you like:
from StringIO import StringIO
from AccessControl import Unauthorized
from AccessControl.SecurityManagement import getSecurityManager
from AccessControl.SecurityManagement import setSecurityManager
from AccessControl.SecurityManagement import newSecurityManager
class Auth:
def __init__(self, user_id):
self.user_id = user_id
def acquire(self, context):
self.manager = getSecurityManager()
user = context.acl_users.getUserById(self.user_id)
if user is None:
return
wrapped = user.__of__(context.acl_users)
newSecurityManager(None, wrapped)
return wrapped
def validate(self, object, name):
try:
return getSecurityManager().validate(
None, object, name, getattr(object, name))
except Unauthorized:
return False
def release(self):
setSecurityManager(self.manager)
def collector_issue_notify(self, who, cmd, bugs, to, msg):
"""Handle a command for a contained issue.
"""
out = StringIO()
auth = Auth(who)
user = auth.acquire(self)
try:
url = self.absolute_url()
if user is None:
print >> out, 'Could not find user %s at %s' % (who, url)
else:
for bug in bugs:
try:
issue = self[bug]
except (KeyError, AttributeError):
print >> out, 'Could not find bug %s at %s' % (bug, url)
continue
if not auth.validate(issue, 'do_action'):
print >> out, ('User %s is not allowed to perform action '
'on bug %s at %s' % (who, bug, url))
continue
issue.do_action(cmd, msg, assignees=to)
print >> out, ('Executed action %s on issue %s '
'at %s' % (cmd, bug, url))
finally:
auth.release()
return out.getvalue()
I hope that is of use to anyone. I know there are several people out
there using the buildbot yet very few still using
CMFCollector. One way or another, it shows the ease of extending
the buildbot to perform other tasks than it was intended for, and also
the ease of extending Zope to be called externally via XML-RPC.
All in all, the hardest part of this was writing the regex to parse
the commit message. :)
If this example is of any use to you and if you use it and make any
improvement please post a comment here, thank you!
Sidnei Silva 