Desperate to get sleep, but I’m terrified to wake up to 1,000 new bug reports. It’s like streaking the quad, but with code. #openstack @jmckenty

There was a video up from an interview I had given the week before.

There were over 100 developers camped out in the #openstack channel, asking probing questions. Most of them were politely worded versions of: “What the hell were you thinking?”

Patches started coming in. Then a New York Times blog post. More bug reports. More patches. More press.

Original Nebula Logo

I’d love to say that we shared a moment of solidarity as a team, toasted ourselves, and wrote more code. But truthfully, half of the NASA Nebula team were on a red-eye flight to DC, to run another set of training workshops for new Nebula users. Some were holed up with the legal teams, trying to figure out how to take the Nova example (brute-force software release) and use it to reshape NASA’s open source software release policy. Vish was bravely manning IRC, answering the flood of questions.

And in a classic case of nostalgia, I started thinking back over the crazy history of this “thing” that has had more names, than developers.

The code name of the “compute” component in Open Stack is called “Nova”. While, if you had the appropriate access to NASA internal systems, you could find early references to the “Novae” task in Nebula development (which was originally the PXE and Puppet-based configuration management system we used for deploying Nebula to physical hardware), the earliest public record of nova is this:

“NASA Nebula Bonding Lunch – We Are the Nova”

Which is possibly one of the worst photos of me on the internet. C’est la via.

While in Austin, Texas last week, at the first Open Stack Developer’s Summit, I had an opportunity to give a brief presentation about NASA Nebula – why we were in the Cloud business, some of the history that led us to this point, etc. While I never prepare slide decks to be standalone (I prefer to use them as audio-visual aids to the substance of the event, which is obviously paying attention to me), you can have a look at the whole thing:

It was based, loosely, on an internal memo I sent a while back to some members of the Nebula team (excerpts below):

There are, literally, a ton of people on the Nebula team now. Which is awesome. But it also mean there are dozens of you that I’ve never had a chance to interact with personally, who may have missed a lot of our early (ancient) discussions about WHY we’re all working so damn hard.

It’s not about the money.

Which is to say, Nebula is not, primarily, about cost recovery. It’s not (mostly) about saving money, for missions, projects, or the Agency at large.

And on the flip side, we’re not DOING this just because it’s our job. Nebula is not the place for “work to rule” – if you don’t wake up every morning with a burning desire to get to your keyboard and roll the Nebula ball forward, then there are honestly better places and projects for you to be working on.

It’s not about being on the “cutting edge“.

We get a ton of press on Nebula, most of it excited over how innovative NASAhas become, how we’re embracing new technologies, how we’re leading the government forward. And all of that is great – but that’s not why we’re doing it. This is not innovation, for the sake of innovation.

It’s about the Science.

I haven’t really been at NASA long enough to have a good grasp of its long and illustrious history. So in the same fashion that I get away with practicing Computer Science without having formally studied it, I have to work from first principles. In this case, the first principles are a happy little document titled, the “Space Act” (Specifically the Declaration of Policy and Purpose, section 102):

Sec. 102.  (a) The Congress hereby declares that it is the policy of the United States that activities in space should be devoted to peaceful purposes for the benefit of all mankind.

(7) Cooperation by the United States with other nations and groups of nations in work done pursuant to this Act and in the peaceful application of the results thereof;

(8) The most effective utilization of the scientific and engineering resources of the United States, with close cooperation among all interested agencies of the United States in order to avoid unnecessary duplication of effort, facilities, and equipment;

So I’d just like to point out that the mission statement of the Agency, includes specifically section 102(d)(8) – is to cooperate with other agencies in order to make effective utilization of scientific and engineering resources. AKA… Cloud Computing.

Open Source, by which I mean participation in an open source process, as well as the release of source code under an open license, is an activity NASA can partake of under section 102(a). It is way more meaningful than most of the rest of what we’re doing.

http://novacc.org

From the website:

Nova is a cloud computing fabric controller (the main part of an IaaS system) built to match the popular AWS EC2 and S3 APIs. It is written in Python, using the Tornadoand Twisted frameworks, and relies on the standard AMQP messaging protocol, and the Redis distributed KVS.

Nova is intended to be easy to extend, and adapt.

Jeff as a webhook

PROBLEM: Help people understand how cool webhooks are.
PROBLEM: Find out when there’s new content on a site.

SOLUTION: Put a webhook into the site’s search indexing tool – SOLR.
SOLUTION: Custom SOLR Token Filter.

The code, while JAVA and therefore evil, is trivial.

WHY?

Every cool web app (sooner or later) has some cool content in it. So you add a search engine. When your users want to build mashups with that content, you add webhooks to that engine. And you’re done.

Starting point: http://e-mats.org/2008/06/writing-a-solr-analysis-filter-plugin/

GOTCHAS:
JAVA – Evil and Stupid
Java classpath (on Mac) – also Evil and Stupid
Java permissions (to connect to remote servers) – http://forums.sun.com/thread.jspa?threadID=5309619

WHERE IS THIS GOING TO LIVE?

Webhooks coming to NASA soon...

FINAL TAKEAWAY: Fitz was right – everything should be HTTP.

As is often the case with the large, complex issues facing humanity, most of our effort is (dare I say it?) wasted on trying to fix the wrong things. Like changing our lightbulbs. Even changing our cars is, most of the time, the wrong thing. Why?

Making a thing, almost ANY thing, takes an enormous amount of energy input. These days, most of that energy input is petroleum, generating huge amounts of waste carbon. So regardless of how much BETTER the end product is, it needs to fulfill two criteria to justify the switch:

1. It needs to last forever (or some reasonable facsimile).
2. It needs to use so much less carbon, that it offsets the output of buying something NEW in the first place.

Our problem, as a culture, is less an OIL addiction, as it is an addiction to CONSUMING. Endlessly.

Let’s take the principles of the “Tucker” automobile, and apply them to all consumer goods. Do away with “designed obsolescence“, and try building things to LAST, for a change.

My favorite example of this? A Stanley #2 plane.

(‘But wait!’ you cry. ‘What about the Rabbits?’)

Ah yes. The Rabbits.

Even in “Peak Everything”, when the author discusses how America may need to shift back to a nation with 20-40% of our workforce involved directly in agriculture (in a move called re-ruralization), which mimics much of what Cuba did after the trade embargo was put into place – there’s still too much emphasis on simple vegetables and grain. As a passionate carnivore, I have to ask – Where’s the beef?!?

What’s wrong with cows, anyway? A lot, as it turns out.

1. They produce tons of methane, a greenhouse gas that’s hundreds of times worse than simple carbon dioxide.
2. They’re horribly inefficient at converting feed or pasture to meat.
3. They’re grazed at great distances from where they’re processed, and then transported great distances AGAIN to where they’re consumed.

4. They require relatively high-quality rangeland, and human-edible feed – which contributes directly to the worldwide shortage of arable land.

What’s better? Locally-grown, locally-processed rabbits. Why?

1. Rabbits are incredibly efficient at converting vegetable scraps, and simple non-human-edible vegetation (dandelions, for instance) into meat.
2. Their meat is lean, perhaps cutting down on the obesity endemic.
3. They’re fast and simple to process, with very little toxic byproduct from slaughtering.
4. They breed like, well, like rabbits.
5. They’re simple to breed on a highly-localized scale, reducing transport and attendant fuel costs, pollution, etc.

So here’s my pitch, Mr. Cuban – strike some direct-to-restaurant business deals with major fast-food chains (I’m thinking Wendy’s or McD-scale) for the “Bunny Burger”. Work with these guys on some compelling branding to take the stigma away from breaking fast on the bunny scramble, and put together massively distributed production of bunnies (and their attendant feed). Compost and sell the manure. Feed them kitchen scraps from the same restaurants that are buying your bunny burgers.

Hire the homeless as Neo-shepherds, to keep stray dogs and eagles away from your flock. And where do you put them? How bout the parking garages that will be abandoned as our global economy (and our ability to make ridiculous monthly payments) hits the toilet?

Rabbits. FTW.

How to use VPN to solve VoIP problems

As some of you may be aware, when I’m not busy with my day job at NASA, I run a small VoIP company. The why and when of this company is a story for another day, with intrigue, betrayal, ex-CIA agents, con artists, lawsuits and utter, utter sleep deprivation. But, as I said, that will have to wait.

It’s a simple enough business – multi-line phone service for homes and offices, complete with distinctive ring, cool automated phone trees, time-of-day dialing, etc. And it has always had one nasty technology problem in it – NAT.

Anyone who has ever dealt with VoIP has run into all the various ways that NAT can go wrong. And I thought I had worked out ways to deal with most of them – at least, until I started working with some new hardware.

Let’s skip the gory details, and summarize by saying that I’ve been forced to deal with this problem in a decisive, and final, fashion.

No more NAT. Instead – OpenVPN.

There are usually two different ways to handle getting a VoIP phone connected to a remote VPN server – either connect the phone directly (if it supports that), or put in some sort of VPN concentrator. There are even routers that support VPN tunneling – if you’re willing to pay for it.

There’s also another way – a happy story called DD-WRT. This is an alternative, open-source firmware for many of today’s most common hardware routers – including the WRT54G2 that I’ve been dealing with. Along with being just really cool, it offers both a VoIP SIP Proxy (which I opted against), and an OpenVPN implementation.

So this is what I’ve ended up with – an office LAN full of happy telephones, connected through a VPN tunnel that originates in their gateway device, and terminates in my Asterisk Server. No muss, no fuss.

And a few happy side effects, to boot:

• Because the VPN traffic is encrypted (and prioritized by the ISP), I’m no longer vulnerable to any nasty stateful packet inspection and downgrading of third-party VoIP (see the lawsuit between Vonage and Shaw for details).
• This VPN link works both ways – I can remotely configure all those phones without having to expose them to the public internet. Goodbye, site visits.

(Some credit for this solution goes to Chiral Software – although I didn’t find this post until after I had decided on an OpenVPN solution, it still had some helpful hints. But I believe using DD-WRT makes the whole thing more elegant.)

Turning 404 Errors into “I’m Feeling Lucky” Searches

Jesse Andrews and I often take turns copying each other. (Of course, since he’s ridiculously famous, and I’m only mildly annoying, people usually notice me copying him. Alas). Anyway, Jesse was the first one to point out Humanized Messages to me. After which, I used (and modified) a jQuery plugin to put Humanized Messages into a Firefox Extension.

Jesse had to go one better, and built almost the same effect into Searchy, one of HIS Firefox Extensions. But he wrote it from scratch, and made it actually much nicer.

Anyway, I had been hacking on various ways to monetize browser extensions at the time, and harking back to the days when we wrote the “HP Browser Booster” at Mercurial. One of the many things it did, was to redirect 404 errors to the AOL Search Page. I figured I could do better than that – and I wrote “Four-Oh”.

This happy extension detects a 404 error (well before the 404 page is displayed, which was a happy little piece of coding), strips the requested path off the base domain name, converts slashes into spaces, and then uses that as an “I’m feeling lucky” query for the site:<domainname.com> of the original request.

Lost?

It means that trying to go to http://overstimulate.com/taboo – will automatically land you on http://overstimulate.com/projects/taboo.

Not satisfied with simply a useful little utility, I also parse an RSS feed of random jokes, and pop one of them up on top of the resulting page – in a Humanized Message window that looks remarkably like Searchy.

Jesse, your turn.

Making money on the internet, at least at a small scale, is exactly the opposite of making money in the real world. Trying something is so cheap, you should just keep trying things until something works. Unfortunately, just because it works a little bit – doesn’t mean it’ll work much more than that.

I started writing software when I was seven years old. Most of the years I clung to the idea that, someday, I would write a piece of software that people would pay money to OWN. When I began working, first as a coder and then later as a software architect, I still held out dreams of building the next Google – a gigantic, brilliant piece of engineering that would make me the richest man in the world.

And then I discovered Toyota.

This was also just about the time I was going broke, chasing one of these big, complicated visions.

To take my mind off things, I decided to start copying Jesse Andrews (the godfather of UserScripts.org). Build simple, clean, simple, tiny, simple… really SIMPLE pieces of software that do only one thing.

But I took a little twist on Jesse’s way of doing it – I would only work on things that I could code, test, and ship – in one sitting.

And we called it “Tiny Apps“. And it was good.

But not that good. A great little app, thrown together in a few hours and tossed out into the surf of the Intarwebs, is a bit of a sitting duck. People started shooting them – by which, I mean ripping our ideas off EXACTLY – and then getting tons of press.

US: Whoissocial.com. Four days later: usernamecheck.com

US: Gastracker.cognition.ca. Several months later: A whole bunch of them.

My original tiny app, a grid-computer-powered price-watching service, remains the only Tiny App to make much money. Okay, any money. Really. Freddy Mangum and I worked on developing this concept further, and prototyped a few variants – GoogAzon, which adds Amazon and eBay search results to google pages; and “The Kraken“, which was a tongue-in-cheek response to the “Pirates of the Amazon” extension that came out late last year.

The Kraken was notable enough to get us some substantial press (and a fair number of downloads, too) – but it’s never made a cent.

Here’s how I’ve made money on teh internet:

• Google Adsense. But only a little. ($46 in three years, roughly).
• Amazon Affiliate revenue – both embedded in software, and in blog posts. (Some many bunches of dollars).
• eBay Affiliate revenue – About $3.

Now, if you’ve bothered to read all the way down here, this is where I’m going to give away a little secret. And by secret, I don’t mean something really clever that I dreamed up on my own – I mean a strange and lucky coincidence that happened to lead to something cool.

As of right now, I make more money from my blog, than from anything else.

Seriously.

Actually, to be fair – I make more money off of a single post on my blog.

This post.

Which brings me back to Seth Godin, and Jesse Andrews. Just build cool stuff, solve your own problems – and share.

Oh, and get an Amazon Affiliate account.

Google knows everything about me – and they’ve shown a few times that they’re not afraid to use it. (Ask Chris Campbell about the time Google emailed us when we were working on Netscape 8.)

Microsoft knows very little about me.

As a developer of software, I respect and emulate their business model – they write software, and then they sell it to people. It’s old-fashioned, but it works. And it’s straightforward.

And Google? Google, who makes a living by pimping out a spot in my attention span; Google, who has the moral backbone of a Tadpole – is considered to be the good guy.

Why?

I’m a geek. If you’re reading my blog, you’re probably a geek as well. And one of the defining characteristics of geeks, is that we like to think we’re different from “ordinary” people. One of the places we’re the MOST likely to feel special, is in our belief that advertising doesn’t affect us.

In the strange and emerging economies of ‘freemium‘ business models, commercial open source software, and crowdsourced marketplaces – in a world so backwards we’re giving money away to the producers of the most inefficient vehicles on the planet, while far away our siblings, children and cousins are dying for the oil needed to run those vehicles – well, any company that makes a fortune off of the weaknesses of the gullible Average Joe, is okay in our book. Our digital ‘Robin Hood’ to the Sheriff of Rampant Consumerism.

Except that it’s folly.

Geeks are no more immune to advertising than anyone else. Less, really. Ever heard of Gizmodo? Engadget? Apple Computer?

The long term, nearly-recession-proof success of advertising-based business models simply serves to confirm a simple truth – one that helps to explain (should you wonder) how we’ve ended up in this mess:

People are sheep.

Even geeks.

PS – Please buy something from Amazon for Valentines Days (using my referral code.) Ktnxbye.

It’s now 11pm on Wednesday night. Tomorrow morning, at 10am, I will be presenting my Project Plan to execute $6M worth of custom software development over the next 36 months.

That Project Plan doesn’t really exist yet.

It’s been a busy week. LAST night, at 11pm (roughly), I filed a Notice Of Intent, to bid on a DIFFERENT multi-million dollar, multi-year contract. Oh, yesterday was also my oldest daughter’s 6-year-old birthday.

There’s a point, in here, somewhere. We’ll wind our way towards it.

Technically, these days I’m an “Information Worker”. What I think that means, is that I get paid for thinking about things. At least, that’s how I choose to interpret it. My clients probably prefer to think I get paid for the OUTPUT of my thinking – but I’m all too keenly aware of how directly the quality of my output, is related to the quality of my thinking.

I drink a lot of coffee most days. It brings a certain crisp, painful clarity to my thoughts – great for coding, decent for hacking up a schedule, horrible for writing proposals.

Some days I drink beer, partly to fuzz those crisp, certain edges – partly to counteract the effects of the coffee. When I write strategy or policy, I usually combine the two. It produces documents with a certain bizarre, compelling lucidity – and leaves me useless afterwards.

Information workers, such as we are, can be noted for their ability to force their thoughts to follow a linear progression – to march through the gates of logic, as it were. This is, after all, what software is about – making rigidly explicit the implicit desires of the user.

And yet athletes often talk about being “In the Zone” – where every motion seems effortless, and the outcome so certain as to be written in time. Is there a place for such sentiments in the realm of informatica?

I like to think so.

And this belief is what leads me to my daily routine:

1. I never start (or finish) at the same time.
2. I rarely work in the same place. Anywhere with coffee and Wifi is a candidate – on writing days, the WiFi is optional.
3. My roles, while sweeping, are ill-defined – and I prefer it that way. When I’m in the mood to write, I write. When I’m in the mood to code, I code. If the phone calls, I dial it.

Such freedom comes at a price, however – the day I fail to meet a deadline, is the day my freedom ends. (NASA, after all, can only overlook my eccentricities while I’m impeccable). So occasionally, I push. But I don’t push the doing – I push the feeling. If I need to write, I’ll push at feeling a writing mood. If I need to code, I’ll push at feeling the thrill of execution and interpretation.

What about you? Do you believe in multitasking? Do you drink the Kool-Aid, that the A.D.D. we suffered with as children, was actually the early manifestations of a better, more intuitive and transcendent way of thinking?

Or really, are we truly the over-indulged, lazy and self-centered Generation-Me that “Suits” would have us think?

1. Numbers

Regardless of the language they use, or even the character set they use for writing it, most countries on the planet now use the “decimal positional notation” for all numbers and mathematics. This public, open standard for notation has allowed the development of relatively friction-free international commerce, and was the successful basis for…

2. Weights and Measures

What if, every time you bought a gallon of milk, a five-cent license fee was paid to the owner of the patent – on the Gallon?

Aside from underpinning all advances in commerce, engineering, scientific research, cartography, medical sciences, etc… just look at a few of the many examples of what goes wrong when we don’t follow established and open standards of weight and measure.

3. The Telephone System (SS7)

Ubiquitous enough that most people don’t realize there’s a standard at work here, the details of the Signaling System Number Seven protocol are the magic glue that makes global and local telephony possible. It’s worth noting that individual countries are able to implement and revise this base standard in significant ways, and that they’ve done so without abandoning the SS7 system. (This would be a great example of an extensible standard).

When was the last time you plugged a telephone into a wall jack – anywhere in the world – and couldn’t get it to work?

Other non-technical, non-software standards:

1. Shoe Sizes
2. The Dewey Decimal System
3. ISBN
4. Bullet Calibres
5. Screw sizes and screw heads, bolt, nut and nail sizes – (although not ALL screw heads)
6. Dimensional lumber
7. K-12 Grades
8. Zip Codes

The most obvious technology standards are, of course, electrical power – of the 110V or 240V, 50 or 60Hz variety. Can you imagine what limits on the innovation of electrical equipment we would have suffered under if GE collected a license fee every time you used a wall jack?

Now here’s an interesting question – why has it taken so many years for “Hi-Def” television to become ubiquitous? The underlying standard (broadcast or cable television) wasn’t extensible and, in fact, (similar to the Y2K bug) had been designed with very short-sighted considerations.

In a level playing field, (such as the emergent technology market of Internet servers, databases, scripting languages, frameworks, etc) open source software is a sure contender. However, when dealing with vendor lock-in (which is ALWAYS based on closed and/or proprietary standards) it doesn’t have a hope. This is a simple (perhaps simplistic) explanation for the failure of Linux to make the gains in desktop market share that everyone has expected.

In the long run, adoption of the right open standards becomes the trojan horse within proprietary software. It levels the playing field – let the best app win.

As a self-taught engineer, I tend to notice the glaringly-obvious – perhaps more than many of my well-educated peers. And there’s one obvious lesson in this – if solid state electronics are getting HOT, they’re wasting using a fair amount of power.

In a nuclear reactor somewhere out there, an atom died for the pixels on my screen. Another few drops of precious oil, or a few tons more gasified coal, were spilt for those extra minutes of Microsoft Word (or perhaps “Grand Theft Auto 4″).

Moore’s Law has shown us how the steady change of computing SPEED (doubling), and COST (halving), has reliably powered our advancing Information Age. Yet nothing in Moore’s Law has halted the seemingly inexorable increase in ENERGY requirements, of these most devious of machines.

This is not a problem that we’ve address head on – in our subsidized energy economy, there has been no real motivation to do so. In fact, as our dependency on computing infrastructure has deepened, we’ve made it WORSE. Here’s how it works:

“Hey Jim, did you realize that these hard drives fail as they get older? And really reliable hard drives are WAY more expensive.”

“Wow! That SUCKS! Why don’t we just put TWO hard drives in every computer, and copy all the data to BOTH of them?”

No joke. We’ve also done the same thing with (redundant) power supplies, and since all these extra disks and transformers are putting out more HEAT – we’ve had to install more AIR CONDITIONING, too. Oh, and in the REALLY high-end data centers, we’ve got at least one (sometimes two or more) Diesel Generators running at all times – just in CASE the power goes out.

As is typical of the world’s larger problems, the smartest folks haven’t done better – they’ve just done WORSE… in a more complicated way. Let’s take a look at programmers.

Writing software has changed a lot since we started this business. We’ve drifted from the “low-level” languages, up through the “high-level” languages, to things now rightly called “frameworks” that are too abstracted to be considered a language at all.

But along the way (with all this gratuitous computing power sitting around), we’ve gotten UNBELIEVABLY lazy. Here’s an example – it is, by no means, the only one:

Ruby on Rails. A very popular framework, with some fairly typical problems.

Ruby is not, to start with, an efficient language. This means it doesn’t do a very good job of translating “high-level” code, into “low-level” machine code.

Rails, as a framework, is intended to be really easy to use. Which means it does a LOT of stuff for you – much of which you don’t actually need done, in any given case.

Rails uses a database access pattern called “Active Record”. It’s pretty cool, actually – it encapsulates about 10 layers of abstraction, into a few easy-to-use commands.

Unfortunately, that encapsulation ignores most performance impacts. Also unfortunately, the implementation of that pattern – is not thread-safe.

Here’s where the laziness kicks in – rather than fixing the ActiveRecord implementation, or making ruby more efficient in general, someone wrote Mongrel.

What’s Mongrel? An extra web-serving layer, it simply manages a bunch of ruby instances, and hands http requests back and forth. It’s a workaround for the thread-safety problems we were mentioning above.

In software development, it’s called a “Kludge”. In human society, it’s called a “Rat Race”. (We all drive SUVs, even when we object to them environmentally, because we need to be safe – safe in case someone ELSE driving an SUV crashes INTO us. Are you starting to appreciate the irony?)

“So what,” I can hear you saying, “Buying more hardware is cheaper than fixing a whole language, right?”

Yes, it is. And that’s the problem.

Because there are other, more insidious, global impacts of this wasteful computing philosophy. If you need a wickedly fast CPU and a few gigs of Ram to run a Ruby on Rails server, might as well turf out the old servers, right? Those old servers end up as e-waste in developing nations. (HEADLINE: “Lazy coding kills children in the third world”)…

Now, I’m not espousing a return to the good-ol-days of hand-coded assembler, or even web apps written in pure C. But I’ve managed 10,000 http requests per second on a single server (yes, including DB) – simply by taking the extra day or so to tune the database, install some byte-code cacheing – oh, and not running RoR.

Even if you don’t care about the 12 year old kids dying in Thailand while they bake your discarded motherboard over a camel-dung campfire – at least think about the abuse you’re putting your electrons through.

(Editor’s note: Many thousands of electrons were tortured in the writing of this blog post. And yes, this is deliberately provocative link-bait for my overly-zealous RoR-fanatic friends. Which doesn’t make it any less true.)