eigenblog

All Roads Lead to Zope3

daedalus — Sun, 29 Jun 2008 23:10:31

It seems I have come almost full circle in my search for a new web application server framework. While attempting to find an alternative to Zope2, and Zope3, I stumbled across Grok by way of Pylons. I'll explain.

Spot the Difference

TurboGears, which I gave my impressions of in my last post, is being rebuilt on top of Pylons in TurboGears 2.0. I started to look at Pylons because in playing with TurboGears, I realised that it was moving to use almost everything that Pylons uses. They were moving to SQLAlchemy from SQLObject as the default ORM, which is what Pylons uses. They were moving to Genshi as the default templating language, just like Pylons. You can use Routes to define the URI->controller/action mappings, which Pylons uses. The only real difference appeared to be Pylons' use of Paste as the webserver, vs. TurboGears' CherryPy.

A Snag

However, TurboGears 2.0 is a fair way off from release, and it looks like it still won't do what I want. The major flaw with TurboGears, IMHO, is that it doesn't let you combine multiple applications in the one server, making the combination of a Identity provider module with a Blog module and a Threaded Comments module quite a challenge. So I looked more closely at Pylons, since they seemed to be morphing into one another, and alas, Pylons is also quite immature. Both frameworks are only about a year old, so they are lacking features. Worse, Pylons doesn't appear to have the same community built around it, so it doesn't have even the scant applications that TurboGears does in their CogBin.

With either framework, I was looking at a lot of custom development to get the components I was lacking in Zope2, and losing a lot of the ones I already have in the process. Lose/Lose. Not good.

Back to the Drawing Board

So, the words of Marvin the Martian, "Oh well, back to the old drawing board." I needed to cast around for another possibility. Maybe I'd have to venture into Django land, or worse, into something with, erk, PHP. Somehow, I stumbled across Grok, and I am quietly hopeful.

Zope3, Only Usable

Grok appears to be an effort to build a rapid development web application framework on top of Zope3. It builds on the powerful and extensible, yet complex and unwieldy, Zope 3 by layering some useful shortcuts on top. You get the benefit of all the power of Zope3 under the hood, but the agility of a modern web app framework. It looks like you can do most of your development in the world of Grok, but you could drop down into ubergeek Zope3 mode if you need to do something more complex.

The best bit would be that you could also use a lot of the components that already exist for Zope3, and with some modifications, Zope2. Zope has a lot of heritage (not all of it good, hence Zope3) that means you don't have to reinvent a bunch of stuff. Zope3 is also based on the Twisted async engine, which is several flavours of awesome. Twisted also has a steep learning curve, as does Zope, but I've already climbed a lot of the way, so I can get a lot done in both of them. Perhaps I've been a bit hard on ol' Zope?

Learn More

In learning more about Grok and Zope3, it appears that all is not as bad as I had feared. Grok makes it easy to get going with Zope3. It lets you use Python eggs, which are awesome. You can plug in different templating languages, so I don't have to suffer with ZPT all the time if I don't want to. Grok has a scheduler built in. Grok seems to place a lot of emphasis on components and reuse, which is part of the strength of Zope3. Zope3 has nabbed the schemas concept from Archetypes, so web forms are still easy, and the set of Widgets available has gotten better. You can do ORM if you want to, but you can also just use the ZODB, which is actually really awesome now that I've done a bit of ORM based webapp development. Grok has a bunch of example code. Zope3 has a strong emphasis on automated testing, as does Twisted.

So it appears we have a new leader in the "Who will replace Zope2 as the eigenmagic preferred web framework?" stakes. Now I just need to Grok it.

TurboGears: First Impressions

daedalus — Tue, 24 Jun 2008 19:30:11

TurboGears is my first choice for a new webapp platform, as both it and Django seem to have a reasonable following. TurboGears seems to suit my ethos for a webapp platform more than Django for a variety of not particularly conscious reasons. It just feels more like what I want. I've started to play with it, and here are my first impressions.

The Good

After some initial hiccups (discussed below), installing the codebase is pretty easy. Getting a demo app running is quite simple, via tg-admin quickstart, though I couldn't find the 'login' link at first, because it was aaaaalllll the way over on the top right, and quite small. The graphics are quite shiny, and it's a good ad for the product. It made me feel good about my choice, so far.

The tg-admin toolbox is awesome. The CatWalk tool enables you to browse the database via the web, updating data if necessary. This is great when you're prototyping your app.

The web based, AJAX enabled schema browser/editor ModelDesigner is very, very handy. Again, it helps you to rapidly prototype your app by interactively tweaking your schema. Power users will probably just do this via the code, but for a novice like me, this is a great way to learn how the model stuff works. This is a big advantage over Zope 3, which seems to have a very steep learning curve and few tools to help you learn the basics.

Similarly, the Widget Browser lets you take a peek at the widgets that are available and how they work. This is a great way to learn what UI features you can use out of the box, and the range of widgets is impressive. There are AJAXified widgets that look quite good. To get similar stuff with Zope 2 you need Archetypes, which does some magical stuff under the covers that you may not want, and it doesn't seem to be available in Zope 3 yet, and has many deprecation warnings.

TurboGears is also quite lightweight, which makes it faster than Zope by a long way. But then, Zope has a lot of extra stuff in there designed to give you a lot of functionality for free.

The Bad

It's a little early to say that anything is really bad, as such. It appears the TurboGears ethos is to do-it-yourself, rather than have everyone adhere to a single, common thing. This is not necessarily a bad thing, and is a similar criticism I have for Zope. It may mean that you need to write a lot of boilerplate style code, rather than simply stand on the shoulders of other TurboGears coders. This will probably change as TurboGears gains wider acceptance and more people write stuff using it.

You don't get a whole bunch of features for free with your classes. In terms of Zope equivalents, there's no Catalog, but then, all your objects exist in a database accessible via an ORM, which implements search functions, so this is kinda moot. There's no base or mixin class that implements Dublin Core attributes, which is a shame, but again, this may not suit all models.

You can't simply add new subclasses of Objects into your app via the web the way Zope 2 does, via ZClasses and through-the-web class definitions. You have to define objects via classes and your database model. To be honest, I actually think this is a feature, not a bug. It removes a lot of the overhead and weirdness that some of the Zope 2 magic creates.

It looks like traversal, in the way that Zope 2 gives you a hierarchical inheritance in your web structure, isn't there. I'm not sure if this will be a problem or not.

Not having a Plone based layout designed for free means I'll need to do it all myself from scratch. Again, this may actually be a feature, since the Plone stuff has loads of guff in there to cater for a variety of people's needs, which don't necessarily match 100% with mine.

The Ugly

Install is relatively easy, though the Ubuntu version I'm on installs a Python 2.4 compiled version, while I actually run Python 2.5 for most things. Zope also needed Python 2.4, I guess because that's deemed 'stable' in the distro. This breaks the apt-get installed TurboGears. If you try to run tg-admin (the TurboGears admin program), you get a weird "int out of range" error. The fix: Don't use the apt-get version of TurboGears. apt-get remove the APT installed TurboGears and download the tgsetup.py file from the TurboGears website, and run that with your Python 2.5.

In Sum

My initial impressions lead me to want to do more with TurboGears and really take it for a test drive. There more I play with it, the more I think Zope 2 has become unwieldy bargeware that gets in my way more often than it helps me. It could just be that Zope 2 was ultimately a poor choice for what I wanted to do, and I shouldn't blame the tool for that.

I certainly haven't been turned off TurboGears at first usage, which bodes well. I'm hoping that as I learn more about it, the reasons for things being done a certain way become more apparent. I haven't had that 'Aha!' moment, so I don't really grok TurboGears yet. But, I'm hoping that I will.

So that's a good thing.

Bold Steps Ahead

daedalus — Tue, 17 Jun 2008 02:55:29

I've been contemplating moving away from Zope 2 as a web framework for a little while now. Once upon a time, I had high hopes for Zope. I thought that it might provide me with all the Role Based Authentication and template based stuff I might need, and with Python, since that's my preferred language at the moment. However, after several years of development using Zope, and running this personal website with it, I am.. frustrated.

Smart, And Gets Things Done

That's something Joel Spolsky has said about hiring the right people. You want someone smart, but they have to also get things done. You don't want an Architecture Astronaut, and increasingly, I'm feeling that Zope is a bit like this. There's too much *framework* getting in my way when all I want to do is render web pages. Fast. I want to be able to install my web-based application easily, without requiring users to jump through too many hoops. None would be good.

Now Zope 2 does a lot of things well, but it also does a lot of things.. half-assed. There's no single, robust, good blog Product that implements a bunch of blog type features that have been around for years. Like tags. I mean, come on. I have no idea if there's a decent PhotoAlbum implementation yet. ORM is hard. The security framework is very flexible, but doesn't have good shortcuts to do things with sensible defaults. Plone is a whole bunch of other code designed to try to make Zope more useful, and that adds a bunch of other issues. Widgets exist inside Archetypes, and you need Plone for that.. and to install it you need to install a PloneSite and the install your Product inside the PloneSite. DTML was a nasty hack of an idea that still hasn't been taken out and shot. ZCML is more of the same idiocy. ZPT can't do if/then/else clauses, only if/then clauses. The timezone code has been broken for years.

Sorry. FAIL.

Nice Try, But

So yeah, Zope has its issues, as I'm sure any other framework does. These particular Zope issues are really hurting me now, though. I've got a lot of sunk cost in all the development I've put into Zope related stuff. I want to add functionality to this Zope site, but I can't figure out how to do it. Syndication (RSS feeds) was a long time coming, and that's a really simple thing to implement. I'm generally getting pretty fed up with having to wait 5 years for Zope to catch up with what everyone else has long moved on from. AJAX? JSON? Where are these standard tools? What the hell are you guys working on that is more important?

What The Cool Kids Are Using

There just isn't that much out there using Zope. Lots of big sites are built out of ugly hacks with MySQL and PHP in a LAMP stack. And yet, they all work, and they're fast. As they say: "First make it work. Then make it fast." I have trouble doing either with Zope. I want what they're having. I want to be able to quickly roll out a web app, or add features to an existing one, without having to spend 3 weeks working out what the boilerplate 'model' code needs to look like. Oh, and running unit tests without having to load an entire virtual server would be nice, too.

The Plan

So, I have a plan. I am going to try out a couple of other 'web-framework' type things. I have 3 decent candidates so far: Django, TurboGears and Twisted.Web. Twisted is probably too much like hard work to do what I want to do, but I may be pleasantly surprised. I already speak Twisted, and I'm a founding sponsor (via seafelt.com), so I feel I should at least have a look at it. Django has been blessed by Guido, but TurboGears looks more like what I'm after, on the surface, at least.

So, I'm going to try to do something simple in each of them, and compare. I'll write up my experiences here at some point in the (hopefully near) future, so check back for the results of the Great Eigenmagic Web Framework Bakeoff.

Who knows, maybe I'll be able to tag it so you can find it easily.

NetApp + VMware = Impressive

daedalus — Tue, 17 Jun 2008 02:04:19

Check out this video of how you can use VMware with NetApp storage to provision 100 virtual Windows machines in under 15 minutes. Wow.

Imagine linking this stuff with a change management framework like ModiPy instead of just writing some custom shell scripts.

New Digs

daedalus — Sun, 15 Jun 2008 20:55:51

I made the big move this weekend. I'm now living in sin with my gorgeous other half out in the scary suburbs. The fool! Muahahaha!

Now I have a 25-30km ride in to work each day, instead of 7km. I'll be really fit in no time, no doubt. It was quite invigorating this morning, and only took a little over an hour, despite a few wrong turns due to not quite remembering the right way to go. It'll be second nature in a week or so.

Now the fun of selling the old place kicks in, along with trying to figure out where to put all the stuff we now have together. We worked out that between us, we have 6 can openers, and 3 irons. Neither of us likes to do ironing either, so I wonder if this a case of extreme irony. (Ha! Irony. I'll be here all week, folks. Try the veal!). Yes, she groaned at that one on the weekend, too. I don't care. It makes me laugh.

How to Fix IT

daedalus — Fri, 02 May 2008 22:16:20

Ok, so you're a business person who wants to fix the broken IT in your company. This is an admirable goal indeed, and one worth pursuing, however, before you get stuck in, there are a couple of things you need to know:

It's hard.
It's really hard.
If it was easy, everyone would be doing it.

The fact that there are so many broken IT shops around is a testament to just how hard it is to fix this problem. But hey, you're different, right? You'll succeed where everyone else has failed, right? Hang on to that positive attitude, because you're going to need it.

Why Is It So Bad?

IT is a really new thing. Accounting? Thousands of years old. Carpentry? Ditto. Changing ones into zeros, and vice versa, in order to make this box full of electrons do something useful? About 60 years old. All these other jobs have lots and lots of history about how to do things well, and they're not perfect. Far from it.

Enron and HIH could be called accounting problems, and the accounting standards are constantly evolving to deal with new issues. Sarbanes-Oxley, for example. Speak to someone who's built a house to learn how carpentry and other building trades are still fraught with issues. And these are areas that have lots and lots of examples of how to do things well, or badly. IT, compared to these disciplines, was invented five seconds ago.

This newness means that people haven't figured out how to do a lot of things well, so there's a lot of discovery and testing going on. Lots of mistakes are being made as everyone learns how to do things better by first doing them badly, and hopefully learning from the experience. IT is filled with people learning how to do things for pretty much the first time in history.

The Root Cause

And yet there is something that IT has in common with all of these other pursuits, something that causes all of the problems. It's people. It was people who made bad decisions about how to run Enron, and HIH. People designed the Tacoma Narrows bridge. And people are inside your IT department making the mistakes that frustrate you so very much.

Information Technology, whatever that actually means, doesn't actually *do* anything. The people who use it do. And just like every other problem in your company, it's the people that are the root cause.

Everything is a People Problem

Ok, so now we know what the problem is, the battle's half over, right? Well, not exactly. While it's true that a lot of people spend a lot of time and money trying to fix the technology, or the processes, or the Frameworks and Methodologies that are making their companies sick, precious few have figured out that the people problem is the most important one. So, you're already one step ahead of a lot of the other people out there. Good for you.

The bad news is that you've solved about 20% of the problem. The easy 20%. A lot of the people who figure out they have a people problem, immediately give up, because people are *hard*. They don't follow rules a lot of the time, they're unpredictable. They do things for reasons you don't understand. They're not *organised*. Makes you wonder why it's called an organisation, doesn't it?

If you want to fix the organisation, you'll be spending 80% of your time dealing with people. And they're a special kind of people, for the most part: they love technology, and largely aren't that good with people. The stereotype is there for a reason. Most of the people in IT are male, love computers, and have poor people skills.

How Do I Fix It?

Many, many books have been written about this topic. To condense it all into a blog post would be an enormous task, and one that I'm not a good enough writer to attempt. Worse, I'd probably lull you into a false sense of security about the whole thing, and do you a great disservice in the process.

So instead, I'll give you a series of pointers on the big things to look out for: the big mistakes most people make, and some of the best things you could do to make some quick progress. I'll put them into my next article, but remember: this is hard. There are no quick fixes. But if you're up to it, well, you'll have an incredible edge.

Issues of Scale

daedalus — Fri, 02 May 2008 22:21:51

Over the years, I've had some frustrating times at work due to the... shortcomings... of some people who purport to be IT professionals. Once upon a time, a project team was in a dither about a 'performance problem' with the storage their application/system was using. I was working in the storage team at the time, and we checked it out. The storage was fine, and we demonstrated to the project that this was so, suggesting that perhaps the problem was elsewhere. It often is, but people are quick to blame either the storage, or the network, as the root cause of their woe. In my experience, badly written applications are more often to blame.

The project people wandered off to look elsewhere and that was the last I heard about it, until a couple of weeks later. Someone external to the project had been attempting to help them fix the myriad of issues they had with their application, mere weeks before GoLive, and they noticed a fundamental design flaw that would cause massive scaling problems; problems that would likely bring the application offline mere months after going live. Not good. It turned out that the 'performance problem' some weeks earlier had, in fact, been a symptom of this design flaw. It was a doozy.

For some reason, the application programmers had decided that their application would store all its images on the filesystem, and would store the path to the images in its database, rather than just storing the images in the database. They anticipated that there would be a couple of million or so of these images every month. So where do you think they put the images?

All in one directory.

Yeah. Millions of files in a single directory. And they were apparently surprised that this didn't work terribly well (a storage problem, of course), so once they figured out that this was the cause of their 'performance problem', they changed the way the application worked. By having it create directories to contain similar image files. So instead of millions of image files, they had millions of directories, each containing 5 or so image files. And where did they put these millions of directories?

All in one directory.

The chief architect, lead programmers and everyone else on this project had somehow failed to understand that putting millions of things into a single directory will not perform well, even after this fact was pointed out to them by the application not working. Truly EPIC FAIL.

What amazes me is that only months earlier, we'd had to explain to a completely different project (with their own chief architect, lead programmers, etc.), slowly and painfully, this same concept: putting lots and lots of things into a single directory isn't going to scale well, and is a Bad Idea. Two totally different groups of people, the same, basic misunderstanding of scalability. We actually had to argue with them about why this wouldn't work well.

This isn't a new problem, and it has been solved many times before. Yet none of these people appeared to have any grasp of why they should think about the implications of scale.

Here's what really bothers me about this situation: The people who made these really obvious and very, very stupid mistakes, the Chief Architects, Lead Designers, or whatever, were charging lots and lots of money for their 'services'. Many times I've been in the situation where I, or my team, have been called upon to work miracles to somehow get these fundamentally broken designs to function. Always in incredibly short timeframes, always with the expectation that stopping to figure out what went wrong isn't acceptable because there are Deadlines. We don't have time to do it right, we only have time to do it fast.

This is why businesses hate IT. These projects cost millions of dollars, run late, don't deliver what they're supposed to, and then cost even more money while you try to fix the mistakes. And no one has the time to understand why things went badly, because they've all lurched off to another project that's in crisis for the same stupid reasons.

This is happening in lots and lots of very big companies right now. And it's not going to get better any time soon, because these same people are going to put 'Chief Architect of Big-Project-X at ImpressiveCorp' on their CV and get a job somewhere else, where they can make the same stupid mistakes, leaving you to clean up the mess they've left behind. You probably won't hear about these people, either, because a project has to fail really badly and publicly for anyone outside the company to notice, since companies don't tend to advertise just how bad they are at running IT projects to their competitors or customers. That, and if you knew you had one of these clowns working at your company, would you want to hinder their rapid departure? Very Dilbert-esque.

If you're a business person who's sick of this happening in your company, stay tuned for my next article on what you can do to fix it.

The Wisdom of Ages Past

daedalus — Mon, 21 Apr 2008 09:56:44

Last post for tonight. Really. I have to work tomorrow.

So last night I was reading Plato. No really. It's a book I borrowed from my gorgeous other half, A. It's called The Final Days of Socrates and is apparently a collection of Plato's writings about his teacher, Socrates, and his trial for heresy, and corrupting the youth of the day. Socrates was sentenced to death for his crimes, proving that politics, and religion, have changed very little in 2000 years.

I never did philosophy at school, or uni, and this is pretty basic stuff, apparently, but I read through the Euthyphro and was heartened to read that the concept of the Rule of Law was not a new thing in about 400BC. Plato explicitly mentions the idea that there should not be one rule for gods and another for men, nor for one group of men and not for another. It is sad that there are those, who, more than 2000 years later, still don't seem to be able to grasp this concept. Mr. Bush, so called 'Leader of the Free World' I'm looking at you. And don't think you're off the hook Mr. Mugabe.

This just further reinforces the idea to me that the humans of 2000 years ago were not so very different from ourselves, and it is nothing but sheer hubris that convinces us otherwise. That, and Archimedes, Euclid, Heron, Socrates, Plato and Pythagoras were amazing geniuses, and it would have been amazing to meet them and chat for a while. After I'd learned ancient Greek, of course.

Writing things down is quite possibly the most amazing thing we mere shaved apes have ever managed to accomplish in our short, brutish existence.