Made some good progress on my memcached Cache Manager for Zope
2. It is available from the CacheFu project in the collective.

While staring at the code for the memcached python wrapper, I wondered
if there was anything that could be done to make it faster, but sadly
anything that I tried just made it slower.

One of the things I tried was to read and write directly to the socket
instead of using loads and dumps from the cPickle
module. The win was unnoticeable for small objects, and sadly it
seemed to be slower for large objects.

Another thing that I tried was to optimize the recv method of the
_Host class, which looks like this:

def recv(self, rlen):
  buf = ''
  recv = self.socket.recv
  while len(buf) < rlen:
      buf = buf + recv(rlen - len(buf))
  return buf

I’ve tried five or six variations that would do the same, and yet they
were still slower than this one. I’m particularly suprised that:

buf = buf + recv(rlen - len(buf))

seems to be faster than:

buf += recv(rlen - len(buf))

And that the two calls to len(buf) when replaced by a single call
also make the code slower. My bet is that’s because recv() always
finds a filled buffer so the loop only happens once.

I’m sure the fine folks at the Python Brasil list would love to
discuss these results.

All in all, I’ve only got some benchmarking done and not much anything
else. Here’s some results from the benchmark. Each result is for 10
iterations (using the timeit module). The object stored in
memcached is an instance of a simple class with a string attribute,
where the string length is what is indicated here as Size:

Benchmarking...
Size: 0
time for set() 0.00602293014526
time for get() 0.00676512718201
Size: 131072
time for set() 0.441082000732
time for get() 0.0506250858307
Size: 262144
time for set() 0.571678161621
time for get() 0.138868808746
Size: 393216
time for set() 0.657498121262
time for get() 0.221096038818
Size: 524288
time for set() 0.702223062515
time for get() 0.381999015808

For completeness, this is the code used for the benchmark:

if __name__ == "__main__":
    import timeit
    servers = ["127.0.0.1:11211"]
    mc = Client(servers, debug=1)
    class FooStruct:
        def __init__(self, value="baz"):
            self.bar = value
    print "Benchmarking..."

    for i in xrange(0, (1<<19)+1, 1<<17):
        print 'Size: %d' % i
        f = FooStruct('x' * i)

        t = timeit.Timer("mc.set('foo_timer', f)",
                         'from __main__ import mc, f')
        print 'time for set()', t.timeit(10)

        t = timeit.Timer("mc.get('foo_timer')",
                         'from __main__ import mc, f')
        print 'time for get()', t.timeit(10)