Assembling The Platform  

By Ryan Tomayko under Then they laugh at you... on 28. August 2005

This is a continuation of my last post, which talked a bit about the platform selection process I’ve been involved with from a fairly high level. I concluded that choosing between .NET, Java, and LAMP/friends for me came down to making two important decisions: where do I fall on the safety/freedom scale and in what manner would I like to proceed with my platform of choice? The first decision makes the second (or vise versa) by dictating whether you will be proceeding primarily by trying to make the chosen platform more safe or by trying to make the chosen platform more free.

While we haven’t decided to throw out our .NET based application (we have no reason compelling enough to justify a rewrite at present), I think it’s safe to say we’ll be moving forward with a different platform. The road here has been kind of weird so I’d like to talk about my activity over the past few months with regards to assembling a basic platform.

I’m a Python guy and have a tremendous interest in seeing Python grow. It’s my language of choice and it’s where I’m most productive. I also enjoy the community a great deal and am constantly humbled by the amount of competence some of these people are toting around. But there are a few gaps that I felt needed to be bridged (or at least needed a plan and visible progress) before I felt comfortable making a recommendation for Python as our general purpose language for building future web applications.

There’s pretty much constant banter in the Python community on which of the following two items is most important: refine the language or restructure / enhance the standard library. For someone in my situation, neither of these issues even blip the radar. Here’s what I see as the gaps in Python as a language for basic web development on top of some variation of LAMP. I consider these to be essential pieces of a language like Python’s overall feature-set (e.g. they should be considered batteries and should be included):

A Good Project Documentation Utility

pydoc is amazing from the terminal but is sorely lacking in its web interface. There are also considerable issues with the basic model of runtime introspection, such as not being able to determine (easily or reliably) whether a certain module/class attribute belongs to a certain module or was imported from somewhere else.

Beyond basic pydoc functionality, I’d like the ability to incorporate external documentation files (in text, Restructured Text, or HTML formats) into the doc build process such that building the documentation results in a static local copy of the project web-site. Some basic templating would go a long way here as well.

The Python Doc-SIG has enjoyed very little activity lately that I’m aware of outside of progress on restructured text. David Goodger and the rest of the folks working on the Docutils project have a plan for an enhanced source-based documentation utility with enhanced docstring formatting rules but progress seems to have slowed or halted.

A standard, Python-based build system

Something simple and task based like make, Ant, or Rake. I think distutils is a fine model but needs to be considered from a more generic build/make style perspective. Ideally this would have its own SIG and reach general acceptance in the community on the level of distutils.

Some of the things I’d like to be able to distill down into a single build script for a project include:

  • Initializing databases with SQL files or CSV dumps.
  • Build and package all versions of distributables with options for generating signed checksum files or directory indexes.
  • Push distributables to other environments and deploy.
  • Publish documentation built with the imaginary documentation utility to the project website.
  • Send out announcements of new releases.

Normal make type stuff really but simpler, cross platform, and removing the need for some to learn a new syntax (as much as you can actually “learn” make). If modules were easily pluggable that would be great to but I’d like a ton of varied functionality out of the box.

I know of no serious activity around this. There are a few decent build tools that are Python-based (SCons) but they seem oriented toward building non-Python projects.

A standard, Python-based distribution mechanism

This one is seeing some real progress with setuptools, Eggs, and EasyInstall thanks in no small part to Phillip J. Eby, Ian Bicking, and a bunch of other people working as part of the Distutils SIG.

I’ve had a chance to watch progress on this closely and contribute a little and it has just reaffirmed everything I love about the Python community when it decides to agree on something and dig in. In fact, this isn’t really a concern because these technologies are already enjoying wide-spread adoption and have reached a basic level of stability very quickly. That wasn’t the case even two months ago when I was doing this whole platform evaluation/assembly thing.

A Basically Good, Basically Complete Web Framework with a Huge Community

I know, I know, we’ve talked about this forever, during which time I’ve taken every possible position. The “Basically Good” part is met and exceeded with every Python web framework I’ve ever worked under, no doubt. The “Basically Complete” part means I want a full stack with project management capabilities. I don’t care if it’s pieced together from other components or built from the ground up. I’ve tried three of these: Django, Subway, and Paste/Webkit/SQLObject and I’ve liked them all fine enough for me to get work done.

I want a Huge Community.

I do very specific types of work over and over building web applications. I’d estimate that 80% of my code is very specific to building the same old normal web app and the remaining 20% of code is spread out in support libraries, patches, other general application development, or some new and cool style of web programming :) Given that ratio, I’ve made it a priority to find ways of maximizing my productivity through that 80% of code because it provides the biggest bang for the buck.

I’m fairly certain there are a large number of developers working on those same very specific types of tasks with 80% of their code and that if I would just talk to them, we would be able to simplify and enhance the basic model further, faster.

In some ways, the size of the community around the web framework I use is more important than the size of the community around language fundamentals and lower level libraries.

Again, I’m a couple of months behind in following everything closely and it looks like Django is functioning really well as a community. Ian’s Paste has some interesting qualities, though, and I wish Django would push more discussion into the Web-SIG. This is my problem; I sit here and do this and it takes up a piece of my 80% instead of relieving it. Now you’re doing it!

Some Assembly Required

Having established a basic LAMP environment and identified a few pieces I’m missing, I do what I’m supposed to do: I start building it as part of the community:

  • buildutils - The build tool.
  • pudge - The documentation tool.
  • lesscode.org - A weblog where I will attempt to convince people that they should be working on what I’m working on at various levels :)

I also contributed various patches and testing to setuptools when it was in very early stages of development and continued evaluating progress on the web frameworks. If these components could reach maturity and something shakes out on the web framework front, I feel I could just kick ass and take names. Hard problems will still be hard but I would have a simple, manageable, cost effective, community backed, agile, and free-as-in-Kevin system for web applications.

And then I began evaluating Rails for a series of Django comparison articles. I knew that Rails had the Basically Good, Basically Complete Web Framework with a Huge Community - Python will have one too. But I lost it when I found that Ruby has a standard build tool in Rake, a standard documentation tool that meets my needs in RDoc, and that Gems were everywhere. I had assumed that these would be no further developed than their analogs in the Python world.

The long and short of it is that my evaluation turned into what looks to be a long term relationship. I’m committed to my responsibilities on the Python projects I have going right now and I have a ton of existing applications and utilities written in Python that aren’t going anywhere but Rails ganked 80% of my future code somehow. What’s cool is that because I’m dealing with freedom languages, I don’t feel like I have to make a single choice here because they work equally well in my environment (and most any environment). What’s odd is that it wasn’t Rails that finally pushed me over so much as Ruby and its mysterious perfectly placed support libraries and utilities.

14 Responses to “Assembling The Platform”

  1. Douglas:

    You might be interested in SwitchTower for Rails deployment.

    comment at 28. August 2005

  2. Reinout van Rees:

    For a pythonic makefile variant, look at aap. It works pretty well, but the documentation is a bit unstructured.

    comment at 28. August 2005

  3. John:

    For me, Ruby has always seemd to stand out as the language that allows me to write code the way I want to with the minimum of noise (hosekeeping code; necessary but tending to obscure the main purpose). It has fundamentallly changed the way I write other languages (e.g. Java).

    Rails has acquired a lot of momentum, but ActiveRecord is basically suitable for relatively simple data models. Take a peek at Nitro/Og. Og (Object Graph) looks promising, though there’s still heavy development still going on. If George gets it all together, I think the Nitro/Og will open up a whole new set of applications.

    comment at 28. August 2005

  4. Swaroop C H:

    You just echoed my thoughts completely.

    What Python is lacking is the “platform” although it has all the pieces. It’s the same reason that I got attracted to Ruby but somehow I never grokked that language.

    I love what you’re thinking here w.r.t. Pudge and buildutils. Suddenly, I wish I didn’t have a day job…. ;-)

    comment at 28. August 2005

  5. Ryan Tomayko:

    SwitchTower does look interesting although I’ve only scanned the documentation. I should be looking at it with a bit more depth in the next few weeks.

    comment at 28. August 2005

  6. Kevin Dangoor:

    Well, we hate to see you go, Ryan. I know I’ve gotten a lot out of your contributions in the Python world.

    I can certainly understand why you would choose the toolset you did. Back when I started Zesty News, I was strongly tempted by Rails. I chose Python instead, largely because I felt my choices for packaging a desktop-installed webapp were better.

    I do have a couple of comments: RDoc, to my eyes, does not look any better than epydoc, both of which are based on Javadoc. RDoc, epydoc and Javadoc are all fine tools, but I don’t think you’d generally want to create a whole website with them

    Personally, I haven’t missed Rake. The first thing I did when I started Zesty News was create an automated build script (not distutils-based). Between the os, shutil and the occasional other module, I was able to easily automate every aspect of my build, using standard, procedural Python. That said, I do agree that a tool like Rake can help newcomers (and new projects) get up to speed more quickly. The work being done in setuptools and what you’ve done with buildutils does solve some of the problem there.

    Gems are terrific. For me, there’s a kernel of stuff going on in setuptools that is cooler than what Gems offer (from what I’ve read), but I have unusual requirements.

    I think you already have an idea where I stand on web frameworks, and I hope to write more about that late this week or early next.

    Thanks for sharing all of this with all of us. Having a spotlight aimed at cool stuff that’s happening elsewhere is always invigorating (to me, at least).

    comment at 28. August 2005

  7. Ryan Tomayko:

    Well, we hate to see you go, Ryan. I know I’ve gotten a lot out of your contributions in the Python world.

    Awww, shucks…

    No, but seriously, I’m not really going anywhere. You can take Python from me when you pry it from my cold, dead hands :)

    I’m going with Ruby/Rails as an intrinsic piece of a basic platform for web apps built by the company I’m currently involved with. But I’m not dissing on Python and I have no plans of abandoning it.

    RDoc, to my eyes, does not look any better than epydoc

    epydoc has the same issues as pydoc with regards to using runtime introspection. I also don’t like the idea of adopting a comment format that isn’t accepted by the wider community but I’d like to have a bit more than plain text. Lastly, I’ve found it overly complex to extend epydoc’s core theme and layout.

    RDoc uses source scanning as opposed to runtime introspection and I personally feel this yields better results for HTML reference documentation. (As a side note, Pudge actually uses both. It scans the source to find out what’s where, in what order things are defined, and tries to pick up non-docstring comments.) RDoc is also accepted by the Ruby community as a standard and will be (or maybe already is) included with Ruby core. I don’t have to worry about my formatted comments being unparsable next year when a new doc tool is invented. Finally, RDoc has a basically good and complete customization mechanism which I’m currently using to get rid of the frames, among other things.

    RDoc, epydoc and Javadoc are all fine tools, but I don’t think you’d generally want to create a whole website with them.

    I’ve had good luck so far combining RDoc and reusable Rake tasks to
    generate both source extracted documentation and website style content. It’s not quite what I had in kid/pudge/buildutils yet but its not far and I haven’t spent that much time on it.

    comment at 29. August 2005

  8. curthibbs:

    Just two quick notes…

    To John: I have to respectfully disagree with your assertion that ActiveRecord is suitable for simple data models. However, I equally agree that Og/Nitro is, in itself, also very interesting.

    To Ryan: Yes, RDoc is included in the core Ruby distribution.

    comment at 29. August 2005

  9. curthibbs:

    Let me repost with a minor correction (its amazing how omitting one word can change the meaning of a sentence):

    To John: I have to respectfully disagree with your assertion that ActiveRecord is only suitable for simple data models. However, I equally agree that Og/Nitro is, in itself, also very interesting.

    comment at 29. August 2005

  10. Aaron Bentley:

    I also don’t like the idea of adopting a comment format that isn’t accepted by the wider community but I’d like to have a bit more than plain text.

    If you mean the default text format, that’s true. But epydoc does support restructuredtext, too. And AIUI, there’s no accepted standard way of documenting parameters, return values, etc.

    comment at 29. August 2005

  11. alexbunardzic:

    Just a general comment, Ryan: being a lesscode advocate, it does not really behoove you to nurse a morecode product (.NET)

    Your dilemma should be easy to resolve — port your .NET app to some lesscode platform. I know that it’s an expensive proposition, but no matter how you slice it, lesscode always makes more sense than morecode.

    comment at 29. August 2005

  12. rough book »:

    […] Nachdem ich das Posting “Assembling The Platform” von Ryan auf lesscode.org gelesen habe, denke ich darüber nach ob Ruby on Rails nicht doch mehr als einen Blick wert ist für ein Projekt, das ich derzeit plane. Vielleicht lerne ich RoR dann auch so lieben wie einige andere. […]

    pingback at 30. August 2005

  13. Dan Hatfield:

    A very timely and well thought out set of posts, Ryan (for me at least), as I’m faced with a very similar decision with the small company I work for.
    The part I’m struggling with the most is letting go of the safety factor for several reasons:
    1) The components we are building are being deployed into a web application first and foremost - but may be embedded into an EDI/EAI application as well. And, that application may be at a customer site (i.e. an application infrastructure that I don’t control - and that infrastructure is most likely going to be .NET or Java)
    2) As a small company one of the option on the table is eventually selling your software to some larger company - in this case, it seems the platform could play a signficant factor in these negotiations?

    Did you consider either of these situations in your analysis?

    comment at 03. September 2005

  14. Zero Zero:

    Fred Drake does some excellent work with zpkgtools. May be well worth a look.

    comment at 04. September 2005