Tuesday, January 29, 2008

RailsConf 2008 Registration Is Open

Get ready, get set, go!! The annual frenzy of registration for RailsConf 2008 has begun.

Yes, I have already received my registration confirmation. No, I have not received my speaker confirmation...yet? Oh well, looks like RejectConf again!

See you all there... ahem, not all of you, unless you register right now!

The Patent Apocalypse Begins?

Software patents... as, Obi-Wan warned, "You will never find a more wretched hive of scum and villainy. We must be cautious." Well, the dark side has struck back, people. A massive salvo has been launched, this time aimed directly at the open source community in the form of Trend Micro vs. Barracuda Networks.

I was thinking about a cute little pictograph that would describe the public awareness and attitudes about software patents. Yes, this is highly unscientific. Also, I was too lazy to do it, so I settled for pie charts instead:




75% - Huh? Pass the beer...
24% - I know ALL about software patents... shoes, right?
1% - Programmers & Patent Lawyers



Within the programmers and patent lawyers, we could break it down this way:
75% - I don't mess with them, they don't mess with me... please don't hurt me...
24% - Ban the patent!
1% - Sue me? Sue you!

Is that last group now large enough to start the patent apocalypse? Just like any conflagration, some small sparks can result in a firestorm with devastating consequences.

Just ask yourself... really... are the ideas of free open source software and software patents compatible in any way? Are these two not overdue for a showdown? And what if we lose?

Be afraid... be very afraid...

Monday, January 28, 2008

I Want My API

I just ran into a really great resource for open source API documentation. Called gotAPI, it links in a delightfully searchable way all of the APIs we love, but turn out to not actually know. Well, we know them, just not well enough to have their numbers setup for speed dial on our cell phones. Or our code able to call them directly without looking up a reference first.

Here is a small sample of ones I will actually use regularly:

  • HTML

  • CSS

  • JavaScript

  • Prototype

  • jQuery

  • Ruby Standard Library

  • Ruby on Rails

  • Erlang

  • MySQL

  • PostgreSQL

  • Selenium


Plus gotAPI is a domain name I can probably remember. That should keep me from several pointless button mashing Google searches per hour just to end up back on the same sites that gotAPI is already linking to.

Monday, January 14, 2008

Beware Of The Merb

Apparently the Merb team was not yet satisfied with just having created a better Rails than Rails itself. Now, they are going after even more conceptual high ground, and creating an even more flexible web development paradigm that incorporates many of the great ideas percolating around the mongrel community.

According to Yahuda Katz, the next version of Merb will completely revise the existing Merb framework, in order to provide for many more ways to utilize it. He has an excellent writeup about what they are up to, and it sounds really cool.

The upcoming Merb 0.9 will break Merb into two parts: merb-core, and merb-more. merb-core will be a separate gem based on Rack, and as such will be able to plug into any web server like mongrel, evented_mogrel, thin, or even the venerable WebBrick. With this stripped down merb-core, you could create mini one file applications like those that you might been thinking of doing with Sinatra or Camping. Back to the Merb roots, if you will.

The other part of the new Merb, is the merb-more gem, which supports a more complete application structure like Rails or the current Merb. Combine this with a bunch of plugins, and the new Merb sounds to be an excellent way to really recombine elements to create an application.

I will be checking it out as soon as practical... stay tuned for more Merb.

Saturday, January 12, 2008

On The Grid With EC2

Lately, I have had a chance to get seriously into Amazon's Elastic Compute Cloud (EC2) service. I doubt any reader of this blog is unfamiliar with the existence of EC2, but many of you have probably not even gotten to the tire kicking stage, like I had been until recently.

Even after attending several sessions about Amazon's EC2 at RailsConf 2007, I still had only a very general idea about the service. However, it is true that there is no substitute for experience, and now that I have finally been getting hands-on, I am pretty excited about what you can do with it.

A client with a pretty complex Ruby on Rails application needed to create a processing farm for doing video conversion, as well as serving up the resulting video files. Sounds like the perfect application for Amazon's utility computing services. I dug in to the myriad of online sources of information available...

I decided to base my Amazon Machine Image (AMI) on Ubuntu 7.1. Oh, Ubuntu, such a modern operating system you are! That turned out to be a very good decision, and apt-get was quite co-operative getting the myriad of prerequisites my open source video processing farm would require.

I also decided to base my upload processor on Sinatra. My application's entrypoint is a single stateless call, but there are lots of these calls, and each requires a bunch of intensive back-end processing. Something nice and multithreaded, like some kind of mongrel handler, was called for. In this case, even Merb seemed like it was too heavyweight!

Luckily, I had colleague Ari Lerner available to help me strap Sinatra into my processing farm, who proved to be indispensable. We ran into some issues with Rack, which I am sure Ari will be blogging about. Thanks, Ari, you are the man.


Some lessons learned about working with EC2 and S3:

  • When you can launch a new instance whenever you want, you can really work on your machine configurations in an iterative way. Treat your AMI like source code, and don't be afraid to bundle and upload iterations of the AMI as you go along. If you mess up something, you will be glad you did. And there is no bandwidth charge between EC2 and S3 so go ahead, pile it into S3! At $0.15 per GB-Month of storage used, you cannot afford not to.


  • Amazon provides very complete instructions on how to create an image. When you are "developing" an AMI, by iteratively modifying a running instance, and re-bundling and re-uploading, you can skip the step of uploading your keys. However, you will need to delete /mnt/image before you can re-run the "ec2-bundle-vol" utility, or else you get an "the specified image file /mnt/image already exists" error.


  • You can get DRY with your AMIs. You can use a single AMI, and pass in launch parameters to set various configuration settings. For example, when I launch a processor instance, I use my main AMI and just pass in the required launch parameters to tell it if is staging or production, and which pool of processors to join itself to. And how do you retrieve this individual instance data at runtime? You use a REST call to get it. Neat!


  • At some point, I did something wrong with the code that generated keys for each file I was saving to S3, which in S3 corresponds to the path and file name. For whatever reason, it messed up, and I could not even delete the S3 objects using the S3 Firefox Organizer! Fortunately, the somewhat ugly, but much more functional Bucket Explorer was able to get rid of the offending files.


  • To really take advantage of the distributed nature of the solution, I decided to use a client-side load balancing scheme like that described by Lei Zhu in this article.

    The solution described does load balancing without a separate load balancer. It accomplishes this as follows: when a new processing node comes online, it registers itself by saving its name into a special bucket. Any server listed in this special bucket is available to serve requests. Add in regular peer to peer checking to verify that a node has not gone offline, and the peer removes it from the bucket. If that node, or a replacement, comes online, it will register itself. And so on... Zhu's solution is both effective and minimalist.

    As I said, the solution proposed at the end of the article was really cool, but no actual code was provided, since it is part of what appears to be the codebase for the author's startup company. So, I decided to implement it myself, which proved to be really fun and easy. A fully Ruby-based client side load balancer for EC2 is a pretty useful thing to have around. If anyone else is interested, I will publish as a gem or something.



We are not quite finished with our video processing farm, but it is almost there, and the results are a surprisingly small amount of code. I am been extremely impressed by the relative ease with which we have put together a pretty sophisticated solution.

Grid computing... it's not just the future, it's the present.

Friday, January 11, 2008

Making S3 Folders In Ruby

Amazon's S3 storage service is a really cool way to store and serve up massive quantities of data. Interestingly, it is more like a database than like a file system. Despite the convenience of referring to an object like "/path/to/thing.jpg" S3 really does not have any separate objects for "/path" or "/path/to". In fact, neither "/path" nor "/path/to" even exist.

This is why, when you store an S3 object, the name it is given is called "key" instead of "filename". It functions like a database key, returning a file. However, as very "ultra hipster Web 2.0" as this may be, it is still convenient to browse a large collection of files by using a familiar directory/sub-directory/etc.

In fact, all of the S3 utilities that I use allow you to create a folder inside of a bucket, and sub-folders inside of that etc. But I did not see any programmatic facility for creating a folder independent of any object inside of S3. If you have been paying attention, you remember that in S3 there is no such thing as a directory! So how do these nice GUI browsers do it?

I did some Googling, but no luck, beyond "it couldn't be done". Since I had two different programs that already did it, I knew THAT wasn't the case. It took a little old-fashioned investigation... I looked directly at the output from a query to a bucket that had the little folders in them already.

Lo and behold, it turns out, they cheat, by creating a specially named object for each "directory". For a directory named "/path", you would create an object with the key "path_$folder$", and for a directory named "/path/to", you create an object with the key "path/to_$folder$". Then to get a directory listing for "/path" you just do a query on S3 for all object whose key starts with "/path". Ignore any objects that end with "_$folder$" and there you have it: S3 folders.

I decided that it would be nice if the aws/s3 gem would support this foldering the same way that copying a file within a file system does: if the enclosing directories do not exist, they are created before the file copy.

Thanks to the beauty of modern dynamic languages, I was easily able to put together a little monkeypatch for aws/S3 to the S3Object class, that handles this.

# This is an extension to S3Object that supports the emerging 'standard' for virtual folders on S3.
# For example:
# S3Object.store('/folder/to/greeting.txt', 'hello world!', 'ron', :use_virtual_directories => true)
#
# This will create an object in S3 that mimics a folder, as far as the S3 GUI browsers like
# the S3 Firefox Extension or Bucket Explorer are concerned.
module AWS
module S3
class S3Object
class << self

alias :original_store :store
def store(key, data, bucket = nil, options = {})
store_folders(key, bucket, options) if options[:use_virtual_directories]
original_store(key, data, bucket, options)
end

def store_folders(key, bucket = nil, options = {})
folders = key.split("/")
folders.slice!(0)
folders.pop
current_folder = "/"
folders.each {|folder|
current_folder += folder
store_folder(current_folder, bucket, options)
current_folder += "/"
}
end

def store_folder(key, bucket = nil, options = {})
original_store(key + "_$folder$", "", bucket, options) # store the magic entry that emulates a folder
end
end
end
end
end


Sure, you can have the best of both worlds: massive virtual storage, and a convenient directory-like structure. And now you can have it with your favorite Ruby S3 library.

Happy storage!

And Here Comes The Money...

I read with great excitement that Engine Yard, my favorite of the Ruby-centric hosting companies, was taking in a round of $3.5 million from Benchmark Capital. I am a big fan of Ezra, and the rest of the crew, and the whole waiting list thing for getting a new slice setup has been a big inhibitor of their growth.

Funny, however, the press coverage has called EY a "Ruby on Rails hosting company". Yeah, sure, they are expert in RoR, no question. But the REAL stories there are Merb and Rubinius, and the press has not even caught a clue yet. Obviously Benchmark got it.

Congratulations, guys!

Thursday, January 03, 2008

Money In The Ghetto

I keep thinking about Zed Shaw's vitriolic (and newly updated) posting, and one particular aspect about it that seems to bother me. One way of looking at a ghetto is that is keeps people in. Another, is that is keeps outsiders away. In either case, it is an economy, like most any other. And like any economy, there are winners and losers.

What exactly is Zed complaining about? That the overall level of technical competence is very low, even among many self-proclaimed developers? That the industry has lots of hustlers and scammers? That the MAN is keeping us down and holding back our progress? I am just curious, what was the ideal environment that Zed was waxing sentimentally about? Oh yeah, developing software for the NYC Department of Corrections. Hmmm, time for a gratuitous link about the prison-industrial complex.

Ahem, back to whatever point I was trying to make. Some people look at the folks who are collecting recyclables out of the trash as disgusting. Myself, I see them as micro-entrepreneurs. It is a desirable occupation? Of course not. But it is an alternative to the corporate/governmental control structures.

I can certainly understand Zed's frustration with the relatively small amount of influence developers have in most development projects, and how many shysters are looking to exploit someone. But this is not something unique to any particular industry. It has to do with human nature, and let's not think for a minute that anyone is immune.

The "ghetto startup" mentality has certain advantages than the entrenched corporate interests, because in the creative chaos there is opportunity for the individual entrepreneur to do something independently from the established control systems. Is the Ruby on Rails community a ghetto? A shantytown? Or is it a successfully growing colony of ingenious social anarchists, and the inevitable hangers-on, sycophants, and moochers that follow success?

So, let me complete this metaphor. I, for one, am not going to complain too much about the stupid A & R execs and greedy managers that are hanging around the clubs signing anyone in sight. Nor am I going to complain about how so many artists are stealing my sound. Instead, I am going to sell mix tapes out of the trunk of my car, and just try to get paid.

Tuesday, January 01, 2008

Zed Is Mad

Zed Shaw has always been a rather curmudgeonly figure. I find him smart and hilarious, but of course many are rather insulted by his "tell it like he $%#@! well sees it" style. To me, Zed is like the Hunter S. Thompson of our community. The more your believe your own BS, the more he pisses you off.

Like Hunter S., Zed is a put up or shut up kind of person. Admit it, if you are reading this blog, you probably are running your vast empire on top of his code, while gleefully taking all credit yourself. But hey, Zed is just that kind of guy.

He also reminds me of a guy I used to work with long ago, who among other things had a masterfully inventive style of invective that could, like Zed, "blister the paint off of a bulkhead". They both have sharp minds, with a very low tolerance for mental laxity. They also both play guitar.

But one would be greatly mistaken to ignore the rant of such a mind, too easy to dismiss it as mere ravings. Zed chose to end his year with a wide-ranging assault against the "Rails Ghetto". It is a pretty interesting read... Are you afraid now to look? Perhaps you should be. But look anyhow. Then decide what you think.