Coming back to (social) life
Due to some unplanned traveling I ended up near the Bay Area last week, more specifically Canonical was holding an internal Cloud Sprint in Oakland, CA, and Martin asked me to participate and push our agenda for the upcoming click packages upload and download services, which need to be live by October at least on its simplest form. But I’ll tell you more about that in a separate post.
What I want to share with you today is the joy of being able to connect with old friends and recollect memories, as I mentioned I was longing for in my last post. In those few days I was in California I managed to catch up with Limi and Philipp, said an en passant hi to Rob Miller at the Mozilla SF office, had dinner with Gustavo, walked around the city with Fernando, Alberto, and Geoff, ending up at an amazing Chinese restaurant pretty much by accident, paid a visit to Marlon, who took me on a guided tour of the Facebook HQ followed by lunch at The Cheesecake Factory which I couldn’t refuse. It was exausting, but really great catching up with everyone!
A recurring topic between all of us was the general issues that all of our companies (Mozilla, Canonical, Facebook) have with general public perception. Most interestingly perhaps is the similarity between Canonical and Facebook when it comes down to privacy matters, how there seems to be a disconnect between the internal and external messaging on those matters, and how much the public perception is biased by the media and the very loud minority of privacy tinfoil hat zealots. I wish I could do more to help with solving that. Perhaps pushing for more transparency, better communication at least from the technical side of things could be a way to improve that.
Tech talks aside, I was simply overwhelmed by how much my kids’ pictures and videos are popular amongst friends. Every single person that I talked to was quick to mention that as the very first thing. Oddly, that generally does not reflect in likes and comments on those Facebook posts, which is an interesting observation. Are people generally afraid of clicking that Like link or is it too much effort for them? I’m sure it would do for a great usability study.
I hope to explore a bit more on the outcome of the sprint on a later post. Suffice to say that I was really glad to be present and contribute some feedback to all the planning that’s going into the next cycle, and the opportunity to meet some old friends while at it was invaluable. Looking forward to be doing more of that in the coming months, at FISL and PythonBrasil.
As an article I’ve read yesterday mentioned, we tech heads seem to live on a bubble that mostly bounces between social networks and having post work hours drinks with colleagues, usually from the same company. I wish we could all be more social in the physical world, and talk more about things that are not so tech-related. About life, and family, and non-work things, and enjoy ourselves more.
Filed under: canonical, facebook, mozilla, python | 1 Comment
Tags: canonical, facebook, mozilla, nical, python, social
Extended Mourning
It’s been more than two years now since I’ve written anything on this blog. The last thing I wrote was my attempt at summing up all that Dorneles meant to me as a friend and co-worker. It still hurts. Every. Day.
I’ve been carrying on with life. Heads down into work. Enjoying my kids, which just started at daycare (or is afternooncare a word, even?) this week. But it’s going to hurt for quite some time. I can tell.
I feel like I should be writing more. I feel the need to hang out with old friends again. I might end up going to this year’s PythonBrasil/PloneConf one-two combo just for that. Hopefully some of the old-timers like Alan and Limi will be there. It would sure be great to see Paul again, which is apparently confirmed for one of the keynotes.
I actually started this post feeling very nostalgic, and the paragraph above shows. This was after stumbling on Gustavo’s 12 years ago post. Then I looked at the archive of this blog and found that my oldest post here is from July 20th 2003. Reading forward from there I’ve revived some really happy memories, and tons of now-broken links. One of the broken links was to one of the oldest traces of me out there, asking for help with Including TCL scripts in the CSLU Toolkit, back in 1999. Here’s another, from 2005, where I mention meeting Gustavo in person for the first time. And, the irony. I linked to his LiveJournal blog. :)
I feel much better now.
I should be writing more, and soon. I have lots of things to share, both personally and professionally. I really wish to become more like Dorneles. Not only fun to work with, but also fun to be around. I want to share more, not only about the things that I do, but also the things that I am. I hope to become a better person in the process, and I hope you can share and enjoy that journey with me.
Filed under: Uncategorized | 4 Comments
Tags: dorneles, life, plone, python
Unfinished Life of a Sushi Lover
In the last couple days since our beloved friend Dorneles died in a tragic car accident much has been said about him, both on and offline. Many people remember him from his technical prowess, though he never thought of himself as someone with exceptional programming skills. I know a lot of really brilliant people, both in the Python and Plone communities and elsewhere, people that Dorneles himself considered actual heroes, but if there’s one thing that made him stand out from the rest was his willingness to share knowledge and collaboratively solve problems.
The realization of that had a profound impact in my life. As programmers, technically minded people that we are, we tend to focus on ourselves and rarely share our knowledge, except to show it off at conferences here and there. We do it mostly to please our ego. Dorneles was nothing like that. He didn’t present at many conferences, though he was always there as either a participant or an organizer. He didn’t write lengthy blog posts full of technical details. In fact, he didn’t blog at all since 2007. He didn’t create brilliant frameworks. Instead, his contributions were mostly invisible, through IRC, IM or Skype, directly between him and the person seeking his help. It was very common for him to carry about five parallel conversations over IM, helping people all over the world without breaking a sweat. He would happily pull out his laptop or phone at a table in a blink to teach someone a trick. The endless stream of messages on twitter following his death should serve as proof that his way was much more effective in causing an impact in the world than a horde of highly skilled programmers ever could.
As other people well put it, Dorneles could never lose that smile on his face. Nothing could make him curse or complain. He was a mixture of cheerfulness, innocence, tranquility and awkwardness that would immediately conquer people’s hearts with no effort. He was like that young little brother that makes you blush at a family dinner for being excessively goofy, but that you love so much you would never reprehend.
To me and my wife, Dorneles was closer than family. My wife suffered more from his loss than for the loss of her lovely grandma, or from her uncle that died a slow death from brain cancer a couple years ago.
In retrospect, I can now see that he did treat me like an older brother, always eager to take on my advice. When he was alone in a big empty office in Caxias do Sul, following the break with the other partners at X3ng I told him to move to Garibaldi because we could rent a similar office for much cheaper. I suggested that he should reunite with his wife and two kids, which were living with her mom, and that raising them in Garibaldi would be much better for their education and overall quality of life. I said it would be nice if he had a car so that he didn’t need to call a cab at 3am when his kids were sick (he refused to buy a car for years, since he had cars stolen several times). At a time he was having trouble getting regular pay, I suggested he should work for Enfold. I told him to go to more conferences and sprints, so that he could put faces to the names on #plone. He was easily convinced. And all of these things had a great impact in his life.
I regret not spending as much time with him as I could in the last year. But many are the good memories that we’ve shared. Back in November 2009, when my backpack with laptop and cameras containing all the pictures from my vacation in New York got stolen on our first day in Houston he calmly told me: “Don’t worry, you can buy all those things again. They didn’t take from you the most precious thing: your life.”
I regret that he didn’t find the time to visit my 3mo old twins. But I’m glad that after my visit last Saturday, he went to visit his 2 weeks old nephew and his dad on the Sunday. I’m glad that we spent a good 2h last Saturday sitting around, chatting about the most diverse subjects while I was at his place waiting for the rain to stop.
Dorneles was a sushi lover. He once walked for tens of blocks in Houston on a cracking hot summer day just to eat sushi, and lived to tell the tale. That’s perhaps the most un-American thing he could do. The first thing we did when we met in Houston in November 2009 was to go out for sushi. When Nate came to Brazil, he took him out for sushi, and that’s perhaps the most un-Brazilian thing he could do. His love for sushi passed on to his kids. For several weeks I told my wife that we should invite his family out for sushi. We were waiting for the right occasion, and it never happened. In his memory, I think we should do it anyway.
Filed under: enfold, houston, plone, python | 2 Comments
Tags: plone, python
I’ve registered the sidneidasilva.com domain slighly over a year ago, and let my old awkly.org domain expire. Seems like a few places (like planet.plone.org) were still using the old URL. Sorry for the noise.
Filed under: Uncategorized | 1 Comment
Even Faster Web Sites, with YUI
Ever since I got my copy of Steve Souders’ Even Faster Web Sites I’ve became obsessed with speed. During my day job I’m constantly looking for things that can be improved to make the user experience smoother, specially for first-time visitors. I’m fairly happy with what we’ve achieved in the last year, though there are always things to be improved.
Today I’m going to share with you one of the tricks that we’ve discovered almost by accident and that can help with making your website faster, if YUI is your Javascript framework of choice.
We’ve started using YUI in Landscape around March of 2009, by rewriting all of our ExtJS-based scripts (which weren’t that many) to use YUI. Since our application requires HTTPS due to potentially sensitive information, we had to self-host YUI, to avoid mixed-content warnings. Initially we had little experience with it, so we went with the easiest option available: loading the YUI seed file and letting everything else be loaded on-demand by the YUI Loader, without a combo handler (more on that later).
Soon enough we noticed that it wasn’t such a great idea. First-time access was terrible, with load times over 30 seconds in some cases, due to the combination of HTTPS connection overhead, no static resource caching and large number of requests to fetch all the resources. It was worse than that even, because most browsers either don’t cache HTTPS content by default, or only cache it on memory, unless you set the proper caching headers. To give you an idea of what it looked like, here’s what a trivial page using the ‘overlay’ module looks like in terms of loading.
No combo loading at all
It was pretty clear that this kind of performance would not be acceptable, so I started looking for options to reduce the number of requests. It was around that time that I heard about Steve Souders work on web performance, through a long time friend from the Plone community that happened to be working at Google and shared a link to Souders’ blog post on ‘@import’.
Turns out that the YUI Loader supports using a combo handler for reducing the number of requests, when loading modules on demand. What this ‘combo handler’ does is basically take a GET request with a bunch of filenames as parameters and return a single response with the respective files concatenated. Here’s an example using the combo server from YUI’s CDN.
Combo loading with a single 'Y.use()'
The problem was that we couldn’t use YUI’s combo server on the CDN because it doesn’t support HTTPS, which was requirement for us.
In November of 2009, at UDS-Lucid in Dallas, we had our first YUI-only cross-team sprint, totalling about 20 people from Launchpad, Landscape, ISD and Ubuntu One. The main goals were to spread knowledge on YUI and improve support for IE on our shared codebase (lazr-js), but I managed to sneak in two things for the Landscape team: implementing a Python-based combo loader (served by Twisted, but it’s just a plain WSGI app) and extracting YUI module metadata, so that we could make it easier to load extension modules on-demand, the code for which ended up as part of the lazr-js project.
At this point however we realized that there’s a significant issue with using the combo loader on an application as diverse as Launchpad or Landscape: depending on how you group your ‘Y.use()’ calls, the combo URL generated by the YUI Loader can vary significantly, which means that re-use from the browser cache is less than optimal. Here’s an example of loading both the ‘anim-color’ module and the ‘overlay’ module within a page. Compare it to the previous example and notice how the parameters passed to the combo are completely different, since the second request (to load the ‘overlay’ module) only requests modules which haven’t been loaded by the first request (to load the ‘anim-color’ module).
Combo loader can be a caching problem
In a hurry to get the situation solved ASAP, and taking a clue from our fellow developers on the Launchpad team, we went with a simpler alternative: manually combining all of YUI, including the seed file, and loading all of that in a ‘script’ tag in the document ‘head’. It turns out that the situation improved quite dramatically, but there was still more to be done. Although the number of requests was much smaller, the total page size was actually larger, since we were loading a lot of stuff that wasn’t even being used. Worse, since we were putting that ‘script’ tag in the ‘head’, it was blocking the whole page from loading until the script was finished downloading, which I after following Souders’ blog for a while I realized wasn’t such a great idea either. Here’s an example of what that looked like.
Combined modules in the head
At this point we had a significant codebase of YUI-based modules, and so did the Launchpad team, but we had gone from solely relying on the YUI Loader to not using it at all (by just loading all the code in one big honking file).
Once we got the module metadata extraction figured out though, I resumed working on improving the first-view load time, by reducing the number of modules manually combined to the bare minimum shared by all of our pages. Also, by having a single file with the common modules being loaded on all the pages, we can be sure that it is highly cacheable. And then we let the YUI Loader kick in and load the missing modules on less-used pages, which was a really nice improvement in page weight and reducing the time to the ‘onload’ event.
At this point Souders started talking about asynchronous loading of scripts and the ‘defer’ attribute, and I got my copy of Even Faster Web Sites. I started pondering if there was something that could be done to defer the loading of the combined modules while still letting the YUI Loader do it’s job.
The problem lies in the fact that YUI asynchronously loads modules that are not on the page yet, calling a callback function (the last argument to ‘Y.use()’) once all the scripts containing the required modules have finished loading. If all the modules are already on the page, it just calls the callback right away. So by just tacking a ‘defer’ attribute on the ‘script’ tag for the combined modules there would be a race condition between the loader checking if the modules were loaded and the script with the combined modules being loaded.
Then, it struck me. If the last argument to ‘Y.use()’ is just a callback function, could we queue those functions to be called at a later time and load the combined modules ourselves, before letting the YUI loader proceed? After a little back and forth with Dav Glass and some pseudocode exchange, I got an implementation of this idea(which I’m calling the ‘Prefetch YUI Loader Hack‘). And thus we managed to move the loading of all the combined modules from a ‘script’ in the ‘head’ to use asynchronous script loading, shortening even further our time to the ‘onload’ event. Here’s an example of what it looks like.
Prefetch with a single request
But wait, there’s more! Souders then started talking about parallel loading, tickling my speed bug again. After thinking for a while about the problem, enlightenment came again: when an YUI module is loaded, it is not executed right away, but added to an internal registry of modules. What that means is that regardless of what other modules it depends on, they can all be loaded independently from each other! That means you can break up a manually combined file into N smaller files and have them be loaded in up to N parallel connections (where N varies by browser and by browser version) and when they are all done you can let the YUI Loader kick in. Here’s an example with two parallel downloads plus an extra module not prefetched.
Prefetch, with parallel download
So that’s where things stand now. There are still optimizations that can be done in Landscape, like using the combo loader to reduce the number of requests on less common pages, or even using fixed URLs to the combo loader with the prefetch hack, to simplify the build process. We also need to start using Caridy’s Event Binder module, since now the pages load so fast that the users start clicking around before the event handlers are in place (ha!).
I am also pushing to get those kinds of tips documented and passed around Canonical, through a project codenamed Dare2BFast, which has the goal of coming up with a set of Web Performance Guidelines that all Canonical web sites should follow, both on the frontend and backend. Stay tuned!
Filed under: canonical, landscape, yui | 6 Comments
Tags: canonical, landscape, speed, yui
Get it like you like it
This morning I woke up with this song in my head. It reminds me quite a bit of the Rolling Stones, and it’s very appropriate for the mood I’m in today.
I’m sitting here at Canonical’s office in Montreal, where we are having another Landscape sprint. There’s a couple things that make this sprint special, one of them being that Frank Wierzbicki has joined us! Welcome to the team, Frank, and I hope you enjoy!
It has also coincided with the end of our 1.5.5 milestone, which rolled out yesterday, with some interesting highlights:
Speed!
Contrary to what other people might tell you, YES speed does matter. We’ve been slowly (ha!) rolling out improvements to page load speed, and I’m happy to say that the improvements are very noticeable. Things we’ve done to improve this range from setting proper caching headers for static resources to heavily abusing the YUI Loader. I should write more about that on a more appropriate occasion. There’s always things to improve on, so keep an eye on it!
Faster Trials
Over the last four years, the trial registration for Landscape hasn’t been… enjoyable to say the least. This week we’ve finally rolled out the first iteration of our simplified registration process, which removes a manual approval step which used to take several weeks to be done. That means you can now register for a trial account in Landscape and start using it right away! Quite a novel concept, isn’t it? :) We’ll planning some more updates to it, so that when starting the registration process you have a better sense of where you are in the process and what are steps involved.
We have more exciting updates coming up before the end of the year, so keep an eye on it. But in the meantime, hit that ‘Registration’ link in Landscape, get it like you like it, and enjoy!
Filed under: Announcement, canonical | Leave a Comment
Tags: canonical, landscape, speed, ubuntu
Hi everyone,
After several months of silence, I finally managed to get all the needed pieces back together to build lxml again. More than once people asked over on the lxml mailing list about what happened to the said eggs. The good news is that an unnoficial release of lxml 2.2.6 for Python 2.6 on Windows is now available for testing, in 32 and 64 bit flavors. The 64 bits version was compiled without iconv support because that’s the only thing I haven’t figured out how to compile.
In case you didn’t hear about what happened, this delay was caused by a robbery that occurred on the same day I arrived in Houston, November of last year. I was with my wife and we left for a couple hours for dinner, and when we came back the place we were staying at (Robin’s Nest) had been broken into my backpack was taken, with my laptop, two cameras (with all the pictures we had taken in New York) and two cellphones. We have already replaced everything that was robbed and moved on. Thankfully Robin had insurance, which covered about half of our losses.
That was the bad news. The good news is completely unrelated to all of this: my wife is pregnant, and the babies are due in somewhere between late November/early December. Note I said babies. Yes. Twins. Apparently two girls (last ecography was a little bit to early to say, but the doctor gave us about 90% confidence). We’ve tentatively picked Laura and Rafaela for her names.
So if you’re as happy as we are about the news and want to show some gratitude, we’ve set up a Baby Registry at Babies’R'Us, and the shipping address is set to Enfold System’s office in Houston. I will be visiting my old friends there around September 10, on my way back from another Landscape sprint in Montreal to pick up the gifts and to say Hi.
Thanks everyone for your patience!
Filed under: Announcement, enfold, houston, landscape, twins | 2 Comments
