Maintaining Simplicity

KISS Me

Most people have heard the mantra "Keep It Simple, Stupid" applied to many things, including software. The French writer Antoine de Saint-Exupéry said "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." Albert Einstein's quote "The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience" is often paraphrased as "Everything should be made as simple as possible, but not one bit simpler."

Why is simplicity in software important? One reason is that simpler designs tend to have fewer bugs - bugs that are often easier to find and fix. Simpler code is easier to maintain because you don't have to remember as many pieces. In Open Source software simplicity is particularly important if you want other people to be able to read and make contributions to your code.

I'm certainly a proponent of simplicity in both software architecture and code. Of course, I don't think anyone intentionally makes their software overly complex, but things often (d)evolve that way. In fact it can take considerable effort to maintain simplicity in a software project as you add features. The tendency toward complexity almost seems like some kind of inevitable software entropy. As Paul Graham says, "Simplicity takes effort-- genius, even." He was referring to user interface design but the principle is true for software in general.

To help developers stem the tide of complexity, various programming principles and guidelines have popped up. The Extreme Programming (XP) methodology coined the term YAGNI (an acronym for "You aren't gonna need it") to keep developers focused on the current required features and to avoid overdesigning. "Don't optimize prematurely" is a rule that can avoid complex optimization in situations where a simple solution might appear to be slow but is not actually a performance bottleneck. Another XP saying is do the simplest thing that could possibly work . But to keep the overall simplicity of the system you will have to keep refactoring. A mishmash of "simplest things possible" thrown together will most likely not produce a simple final product.

Developers of libraries and frameworks have a particularly difficult task. No one likes an overly complicated API, but the system must be powerful and flexible enough to allow users to accomplish their tasks. Sometimes the tasks are things the framework designers hadn't thought of.

An Example

The Content Management Framework (CMF) is a collection of Zope Products that provide a framework for building content management applications. CPS , Plone and by extension Rhaptos are applications built on CMF. I'll picked on CMF a bit here, mainly to illustrate the simplicity/flexibility.

CMF stores each member's data in an object in a BTree on the portal_memberdata tool. Searching is done via a linear scan of the data objects. Now a linear search is certainly not the most efficient way of finding members but for sites where there are few members or member searching is rare, it may be good enough. Perhaps some CMF developer said "YAGNI" to cataloged member searches and that was that. After all, Zope allows you to add objects called ZCatalogs where you can index objects for fast lookups. So everything is good, right? The CMF implementation is simple and folks who need speed can just add a catalog? Unfortunately, no.

The problem is that the member data objects stored in portal_memberdata are uncatalogable due to an implementation quirk. And this is where "do the simplest thing" doesn't tell the whole story. I can't really fault them for not optimizing member search, but I do fault them for not at least anticipating that someone else using CMF might want to and making it easier for them. This is why I believe framework designers have such a challenging proposition, balancing simplicity with power.

Fortunately, the CMF is rather flexible in that individual tools can be swapped out easily for alternative implementations.

CatalogMemberDataTool

My goal was to do the least invasive and, of course, simplest thing that would speed up our member searches. With close to 5000 registered users, the linear member search had gotten a bit slow. Couple that with the fact that asking a GRUF group for its members also does a linear search and our workgroup members page was unusable. So I designed a replacement memberdata tool called CatalogMemberDataTool that stores the data objects in a BTreeFolder and makes them catalogable[1].

Once members objects were cataloged, I used Dieter Maurer's AdvancedQuery (wow, that's a useful Product!) that expands ZCatalog's query capabilities[2].

In order to speed up the workgroup members page I indexed the getGroups() method on user objects. This worked just fine, but I needed to modify the GroupDataTool slightly so that adding or removing a member from a group would call member.notifyModified() to recatalog the object. You must upgrade to GroupUserFolder-2.0.2 (or trunk) for this to work.

Closing Thoughts

Was is that hard for me to create a simple drop-in MemberDataTool replacement? No, of course, not. But I hold that I shouldn't have had to. A slight refactoring on the part of CMF would have retained simplicity on their end, and increased it on ours.

[1] So why not use CMFMember which also provides a member catalog? That was my original intent, but when we tried migrating to CMFMember, it had bugs with GRUF. Also, CMFMember is rather complicated and introduces things like Archetypes-based members and Member workflow that we just don't need right now. Remember, the goal is simplicity. For the sake of compatibility I did use the same catalog ID so Rhaptos site maintainers can switch to CMFMember if they wish.

[2] That's another area where I think the Zope folks erred on the side of oversimplification. Why is there no simple way to perform an OR query against the ZCatalog?