(Guest Post by Todd Khozein:)

I am proud to say that I have written my first, yes my first, web app ever in the form of a tool to proliferate the growing and widespread impulsive want-it-NOW shopper’s need to be the first to know in the form of a Twitter Bot that will, indeed, tell you what the latest DVD releases are with a link to buy it NOW! Thanks to Joshua for holding my tender little virgin hacker’s hand through the intricacies of building something that actually works in a world where run-on sentences are acceptable, nay encouraged.

I think I have officially taken a significant step towards geekdom and feel that I should commemorate this occasion of the alpha release by inviting you, the reader, to make this app wildly successful and dedicating the remainder of your waking hours tonight and possibly tomorrow in telling all of your people how sexy this app really is.

By now you’ve all heard about BuyLater, my happy little firefox extension that (thanks to an unexpected LifeHacker.com article) is rapidly climbing towards 1000 users and world domination. Without getting TOO technical, I thought I would share with you how I saved BuyLater from becoming an infrastructure nightmare – one that would have either killed the value of the application (real-time updates), or sucked tons of money and hardware into a technology backwash.

This will be a little controversial, I think – simply because the technique I used, (grid computing), is most often used for less… legitimate… purposes. So much so, that it is almost synonymous with “Bot Nets”.

But let’s go back to the beginning.

When I approached Jesse about WiiMe, and suggested that he ought to generalize it beyond a single product (Wii), and a single interface (Twitter), he told me it would be too hard. The key problem, he pointed out, was keeping a large number of items up-to-date. “Don’t you realize Amazon has a limit on their API?”

Well, I did realize that. But I also had a few ideas.

Step 1 – Do more with Less

Amazon limits API requests to 1 request per second, a totally reasonable limit for most purposes. However, they enforce that limit based on IP address, NOT based on API key. So getting a bunch of extra API keys and round-robining through them was not going to work. (That was a trick I had used on Google’s Search API, some years ago.)

However, most people don’t realize that you can poll for more than one item, per API call. In fact, you can pack 10 items into a single request. Doing this gave me a theoretical maximum of 600 items per minute. When I broke the 300 user mark (somewhere in the first two days), and the total number of items exceeded 600, I had to drop the refresh interval back to 2 minutes. Uh-oh – I could see where this was going.

Sure enough, over the course of the next week, I gradually reduced the update interval to 6 minutes – which meant that BuyLater became essentially useless for tracking Wiis and other scarce items, where the time in-stock is typically 5 minutes or less.

I needed a BUNCH more IP addresses, and quick.

Step 2 – With a little help from my friends…

Rather than start buying additional servers (which I couldn’t afford), or additional IP addresses (which I couldn’t get), I did what any sensible child of the digital age would do – I made it someone else’s problem. I simply added a small service to the BuyLater extension – that fetches a given URL every 60 seconds, and returns the resulting XML data to the BuyLater server. In essence, I distributed the task of polling amazon to the end-users.

Why 60 seconds? Simple math, really. I’ve always wanted to maintain a 60-second refresh interval for the BuyLater service; most users are following 2 unique items, and as a basic assumption, I assume people have their browser open 20% of the time. (Having users in the UK and, hopefully soon, Asia, helps to spread out the polling). Remember, each query to Amazon fetches 10 items – which means, hopefully, that the cluster will be able to maintain my target refresh rate… indefinitely.

Now, obviously I’ll still need to add some more servers at some point, since all this data is still going back to one place. But at least I’ll be adding them for the right reasons.

Step 3 – ???????

There are two questions that people have asked me, so far:

• What do your users think about that?
• What does Amazon think about that?

To the first one, I have no idea. That’s really what this post is about – what DO you think about it? Is it alright for Larry to be fetching data from Amazon, that helps Sally get a deal? Should I have made the whole thing opt-in, or opt-out? From a technical standpoint, BuyLater users were already visiting both Amazon, and the BuyLater site (albeit not once every 60 seconds), and there’s no personal info in any of this data, so what’s the difference?

On to the second question – again, I have no idea. But since there were a couple of @amazon.com email addresses in yesterday’s batch of users, I imagine if they have a problem with it… I’ll hear about it pretty quick.

They decided to run a story on my happy little bot this morning. I didn’t know about it, came back from lunch – and I had 100 users. (For the last week, that number has been stubbornly stuck at 8).

I poked a little further, and realized that only 20 of those 100 users had twitter accounts. Hmm – I guess I better get email notifications working, eh?

In the five to six minutes that it took me to whip up an email solution, I added another 15 users. Wow. It suddenly occurred to me that I didn’t know where this traffic was coming from – I hadn’t gotten around to adding google analytics code yet.

tail -f /var/log/httpd/buylater-access.log…

Ah. Lifehacker.

Here’s what happened to my user counts (new users per hour, and per day):

1      12      2008-03-28
20     13     2008-03-28
45     14     2008-03-28
57     15     2008-03-28
35     16     2008-03-28
20     17     2008-03-28
20     18     2008-03-28
22     19     2008-03-28
12     20     2008-03-28
12     21     2008-03-28
12     22     2008-03-28
15     23     2008-03-28
8       0     2008-03-29
7       1     2008-03-29
1       2     2008-03-29
2       3     2008-03-29
3       4     2008-03-29

I have to admit the 57-new-user peak was pretty exciting – nothing like a new user every minute to blow your socks off.

I quickly started having problems, of course – as the user count climbed, so too did the item count – which meant I was hammering Amazon’s API almost continuously. They didn’t like that, and started bouncing some of my replies. Luckily, I found this article that explained how to query up to 10 items per API request, which means I’ve been able to maintain an update interval of less than 3 minutes. (I’ll tune that for the most popular / rapidly changing items soon.)

For about an hour, a bug in my email function (the one I whipped up in the first 5 minutes) caused everyone to receive blank emails (sorry about that one). Oh, and I stopped echoing ALL the item updates to the twitter stream after the first 20 minutes or so.

I tried to jump into the comments on lifehacker, the consumerist.com, and a couple of other blog posts, but since I didn’t have a preapproved account, my comments don’t appear to have shown up. Ah well.

After the first bug report got filed by way of Jesse Andrews (who was kind enough to route it back to me), I threw a form up on the front page of the website to collect this priceless feedback. My favorite so far – apparently the beautiful web2.0 autogenerated logo I was using is “worse than nothing.” I wonder what they would think of my Web 2.0 Merit Badge (which I made for the Twitter Color Wars.)

So there isn’t really any summary statement just yet – tomorrow I’ll be able to start combing through the feedback, see how many of the users “stuck”, and go from there.

If you get a strange, visceral pleasure in wasting hours, even days, of your life, waiting for your local WalMart to get more Beanie Babies in stock – then you should stop reading right now.

If you like to revisit your local grocery store every night, just to see if they’ve dropped the price on those great donuts in aisle 4… then hit the Back Button, and read something else.

But – if you have a life, and you still want to try and buy something online – I might have something that can help.

It’s called BuyLater, and that’s exactly what it’s for – buying Amazon products, later on.

Later can be: When it’s back “In Stock” (can someone say Wii?), or simply when it’s a little cheaper (or even on sale).

Unlike many of my ideas (which are unique, innovative, and incomprehensible), this one actually isn’t mine. My buddy Jesse Andrews did it first, with a Wii-only bot called WiiMe. I just took the idea, and strreeettched it a little.