Made some good progress on my memcached Cache Manager for Zope
2. It is available from the CacheFu project in the collective.
While staring at the code for the memcached python wrapper, I wondered
if there was anything that could be done to make it faster, but sadly
anything that I tried just made it slower.
One of the things I tried was to read and write directly to the socket
instead of using loads and dumps from the cPickle
module. The win was unnoticeable for small objects, and sadly it
seemed to be slower for large objects.
Another thing that I tried was to optimize the recv method of the
_Host class, which looks like this:
def recv(self, rlen): buf = '' recv = self.socket.recv while len(buf) < rlen: buf = buf + recv(rlen - len(buf)) return buf
I’ve tried five or six variations that would do the same, and yet they
were still slower than this one. I’m particularly suprised that:
buf = buf + recv(rlen - len(buf))
seems to be faster than:
buf += recv(rlen - len(buf))
And that the two calls to len(buf) when replaced by a single call
also make the code slower. My bet is that’s because recv() always
finds a filled buffer so the loop only happens once.
I’m sure the fine folks at the Python Brasil list would love to
discuss these results.
All in all, I’ve only got some benchmarking done and not much anything
else. Here’s some results from the benchmark. Each result is for 10
iterations (using the timeit module). The object stored in
memcached is an instance of a simple class with a string attribute,
where the string length is what is indicated here as Size:
Benchmarking... Size: 0 time for set() 0.00602293014526 time for get() 0.00676512718201 Size: 131072 time for set() 0.441082000732 time for get() 0.0506250858307 Size: 262144 time for set() 0.571678161621 time for get() 0.138868808746 Size: 393216 time for set() 0.657498121262 time for get() 0.221096038818 Size: 524288 time for set() 0.702223062515 time for get() 0.381999015808
For completeness, this is the code used for the benchmark:
if __name__ == "__main__": import timeit servers = ["127.0.0.1:11211"] mc = Client(servers, debug=1) class FooStruct: def __init__(self, value="baz"): self.bar = value print "Benchmarking..." for i in xrange(0, (1<<19)+1, 1<<17): print 'Size: %d' % i f = FooStruct('x' * i) t = timeit.Timer("mc.set('foo_timer', f)", 'from __main__ import mc, f') print 'time for set()', t.timeit(10) t = timeit.Timer("mc.get('foo_timer')", 'from __main__ import mc, f') print 'time for get()', t.timeit(10)