Moving on from the success of BuyLatr, I’ve been playing around with other ways to make bargain hunting and online shopping easier. Today I’m launching the beta of “GoogAzon” – an extension for the Flock and Firefox browsers that adds related Amazon search results to the same Google results page.

It starts with an unobtrusive orange bar (yes, I love orange):

And expands into a listing of the top 10 items (complete with Image Preview tooltips):

Planned enhancements include price comparison (showing the lowest prices from Amazon and other sites), support for Yahoo/MSN search results… and whatever else the users ask for. We’re using UserVoice again for feedback, after our great experience on the BuyLatr.com website.

Some of you may note that I’ve stolen the “SpandexFox” brand from myself (previously the name of an expanded ElasticFox extension). I’ve been invited by Amazon to contribute patches to ElasticFox directly, and frankly the name seemed much cooler than a “Dashboard for managing EC2 instances”. This way, it can be all about making your browsing experience… tighter. Oh, and yes – pictures of “Megan Fox” in Spandex are definitely a possibility.

By now you’ve all heard about BuyLater, my happy little firefox extension that (thanks to an unexpected LifeHacker.com article) is rapidly climbing towards 1000 users and world domination. Without getting TOO technical, I thought I would share with you how I saved BuyLater from becoming an infrastructure nightmare – one that would have either killed the value of the application (real-time updates), or sucked tons of money and hardware into a technology backwash.

This will be a little controversial, I think – simply because the technique I used, (grid computing), is most often used for less… legitimate… purposes. So much so, that it is almost synonymous with “Bot Nets”.

But let’s go back to the beginning.

When I approached Jesse about WiiMe, and suggested that he ought to generalize it beyond a single product (Wii), and a single interface (Twitter), he told me it would be too hard. The key problem, he pointed out, was keeping a large number of items up-to-date. “Don’t you realize Amazon has a limit on their API?”

Well, I did realize that. But I also had a few ideas.

Step 1 – Do more with Less

Amazon limits API requests to 1 request per second, a totally reasonable limit for most purposes. However, they enforce that limit based on IP address, NOT based on API key. So getting a bunch of extra API keys and round-robining through them was not going to work. (That was a trick I had used on Google’s Search API, some years ago.)

However, most people don’t realize that you can poll for more than one item, per API call. In fact, you can pack 10 items into a single request. Doing this gave me a theoretical maximum of 600 items per minute. When I broke the 300 user mark (somewhere in the first two days), and the total number of items exceeded 600, I had to drop the refresh interval back to 2 minutes. Uh-oh – I could see where this was going.

Sure enough, over the course of the next week, I gradually reduced the update interval to 6 minutes – which meant that BuyLater became essentially useless for tracking Wiis and other scarce items, where the time in-stock is typically 5 minutes or less.

I needed a BUNCH more IP addresses, and quick.

Step 2 – With a little help from my friends…

Rather than start buying additional servers (which I couldn’t afford), or additional IP addresses (which I couldn’t get), I did what any sensible child of the digital age would do – I made it someone else’s problem. I simply added a small service to the BuyLater extension – that fetches a given URL every 60 seconds, and returns the resulting XML data to the BuyLater server. In essence, I distributed the task of polling amazon to the end-users.

Why 60 seconds? Simple math, really. I’ve always wanted to maintain a 60-second refresh interval for the BuyLater service; most users are following 2 unique items, and as a basic assumption, I assume people have their browser open 20% of the time. (Having users in the UK and, hopefully soon, Asia, helps to spread out the polling). Remember, each query to Amazon fetches 10 items – which means, hopefully, that the cluster will be able to maintain my target refresh rate… indefinitely.

Now, obviously I’ll still need to add some more servers at some point, since all this data is still going back to one place. But at least I’ll be adding them for the right reasons.

Step 3 – ???????

There are two questions that people have asked me, so far:

• What do your users think about that?
• What does Amazon think about that?

To the first one, I have no idea. That’s really what this post is about – what DO you think about it? Is it alright for Larry to be fetching data from Amazon, that helps Sally get a deal? Should I have made the whole thing opt-in, or opt-out? From a technical standpoint, BuyLater users were already visiting both Amazon, and the BuyLater site (albeit not once every 60 seconds), and there’s no personal info in any of this data, so what’s the difference?

On to the second question – again, I have no idea. But since there were a couple of @amazon.com email addresses in yesterday’s batch of users, I imagine if they have a problem with it… I’ll hear about it pretty quick.

I’ve been trying to get this new feature online over the past two days – let users elect to watch just Amazon-supplied price changes, rather than all vendors. This is handy because, when combined with free shipping, Amazon is often cheaper overall – even when they have a higher price.

So I thought I had it working, by about 1 am last night. Pushed it into production, and let it rip.

Ooops.

Every. Single. Item. In the database. Began sending announcements of “Initial Price: $X” to every user.

I caught it, of course – after about 60 seconds. Which was enough time to send announcements for the first 600 items or so.

If you got one in an email, you can probably safely discard it. If your cell phone woke you up in the middle of the night with a text message (yes, I’ve already got a few bug reports about that) – what can I say? I promise I won’t do it again.

1. It doesn’t work in Amazon.ca, or Amazon.co.uk
2. Can I control what product condition (Used/New) or Vendor (Amazon/Others) it watches?

While the second one is proving a little difficult, I’m happy to announce that the new version of BuyLater has full support for Canada and the UK! If you haven’t tried it out yet, you should. And if you’re an extension developer yourself, feel free to grab the source code from github.com and have a look.

A big thank you to everyone who sent in a bug report – if you keep ‘em comin, I’ll keep fixin them. Other planned enhancements for the next version include:

1. Product condition and vendors (as mentioned above)
2. Fine-grained notification controls (including direct, non-twitter SMS and IM delivery)
3. Being able to delete your account (probably by providing notification when the plugin is uninstalled)
4. More countries
5. More shopping portals

On that last note, what would you like to see next? EBay? More book sellers? Alienware? What?

They decided to run a story on my happy little bot this morning. I didn’t know about it, came back from lunch – and I had 100 users. (For the last week, that number has been stubbornly stuck at 8).

I poked a little further, and realized that only 20 of those 100 users had twitter accounts. Hmm – I guess I better get email notifications working, eh?

In the five to six minutes that it took me to whip up an email solution, I added another 15 users. Wow. It suddenly occurred to me that I didn’t know where this traffic was coming from – I hadn’t gotten around to adding google analytics code yet.

tail -f /var/log/httpd/buylater-access.log…

Ah. Lifehacker.

Here’s what happened to my user counts (new users per hour, and per day):

1      12      2008-03-28
20     13     2008-03-28
45     14     2008-03-28
57     15     2008-03-28
35     16     2008-03-28
20     17     2008-03-28
20     18     2008-03-28
22     19     2008-03-28
12     20     2008-03-28
12     21     2008-03-28
12     22     2008-03-28
15     23     2008-03-28
8       0     2008-03-29
7       1     2008-03-29
1       2     2008-03-29
2       3     2008-03-29
3       4     2008-03-29

I have to admit the 57-new-user peak was pretty exciting – nothing like a new user every minute to blow your socks off.

I quickly started having problems, of course – as the user count climbed, so too did the item count – which meant I was hammering Amazon’s API almost continuously. They didn’t like that, and started bouncing some of my replies. Luckily, I found this article that explained how to query up to 10 items per API request, which means I’ve been able to maintain an update interval of less than 3 minutes. (I’ll tune that for the most popular / rapidly changing items soon.)

For about an hour, a bug in my email function (the one I whipped up in the first 5 minutes) caused everyone to receive blank emails (sorry about that one). Oh, and I stopped echoing ALL the item updates to the twitter stream after the first 20 minutes or so.

I tried to jump into the comments on lifehacker, the consumerist.com, and a couple of other blog posts, but since I didn’t have a preapproved account, my comments don’t appear to have shown up. Ah well.

After the first bug report got filed by way of Jesse Andrews (who was kind enough to route it back to me), I threw a form up on the front page of the website to collect this priceless feedback. My favorite so far – apparently the beautiful web2.0 autogenerated logo I was using is “worse than nothing.” I wonder what they would think of my Web 2.0 Merit Badge (which I made for the Twitter Color Wars.)

So there isn’t really any summary statement just yet – tomorrow I’ll be able to start combing through the feedback, see how many of the users “stuck”, and go from there.

If you get a strange, visceral pleasure in wasting hours, even days, of your life, waiting for your local WalMart to get more Beanie Babies in stock – then you should stop reading right now.

If you like to revisit your local grocery store every night, just to see if they’ve dropped the price on those great donuts in aisle 4… then hit the Back Button, and read something else.

But – if you have a life, and you still want to try and buy something online – I might have something that can help.

It’s called BuyLater, and that’s exactly what it’s for – buying Amazon products, later on.

Later can be: When it’s back “In Stock” (can someone say Wii?), or simply when it’s a little cheaper (or even on sale).

Unlike many of my ideas (which are unique, innovative, and incomprehensible), this one actually isn’t mine. My buddy Jesse Andrews did it first, with a Wii-only bot called WiiMe. I just took the idea, and strreeettched it a little.