I appeared in my second College Humor video today (a hardly working). You can see the back of my head around :48. If you want to hire me to be in any films or videos, please get in touch with my agent.

The last few days have been database/server optimization and scaling day! Before optimizing any of the default mySQL options we decided to test performance against a number of different database engines to see what would happen. As a side note mySQL wasn’t optimized from the beginning, we decided last minute our other box wasn’t going to handle the load well.

Lets try every reasonable mySQL engine…
Since our tables are read only, we tried packing the keys. This had no noticeable effect. With the packed keys we then compressed our core table. We managed to drop the table size from 1.4GB to 1.1GB, which is a pretty significant decrease. This also had no worthwhile effect. Next move was converting the tables over to innoDB. (note: you cant convert compressed tables to innoDB, so un-pack them first :P). My original theory was queries were slow because of table locking, so innoDB would offer us row level locking. That was a wrong assumption and innoDB caused a performance decrease by about 4 fold. The next and last option we tried was MEMORY (old Heap format). A 1.4GB table didn’t fit in the 1.6GB of allocated memory we gave to the engine. It got about 40% of the rows in and tapped out. We ran some queries on the 2.x million rows that actually made it in and the results were decently quick, but this wasn’t an optimal solution on our 2 gigs of RAM.

We finally gave in and decided to really fine tune and optimize mySQL and Apache. It took about 4 hours to get everything exactly how we wanted it. Right off the bat queries responded infinitely faster. Tomorrow will be judgment day, but I don’t expect to see anymore database issues until traffic increases again. Right now our traffic has stabilized and we have a nice core user base for our beta program. We will now begin phase two of operation save you lots of money, and do a full launch soon…

The lesson here is optimizing mySQL should be your number one configuration priority in database applications. Apache only needed a few tweaks, but the mySQL defaults are designed for small sites on low end servers. If the configuration ends up being successful I will post the file.

I’m three days late to this post, but Roddy and I launched the beta of Priceadvance. Just five weeks after our Ycombinator interview (we didn’t get accepted). Priceadvance is a Firefox extension that finds better prices for products you are already viewing. Our objective is to cut out the process of doing price comparison research. Instead you should focus on finding products you want while we find the best prices.

The beta version is only a fraction of what we plan to accomplish with Priceadvance. We did a lot of internal testing and found our software to have a relative low number of bugs, now with over 3,000 installs we haven’t received a single bug report yet. The reason we released early was so we could determine if market for this type of product existed and if it did, what were the expectations. If 90% of the feedback we received was negative we would most likely be moving on to the next project.

When the idea for a Firefox extension first presented itself, we predicted the number one barrier to use would be the download/install. Our analytics show that around 40% of the visitors that went to Priceadvance.com installed the application.

One of the big unknown’s for us was what to expect as far as traffic. All we knew was we needed a dedicated box, mainly for customization reasons. We leased one from Layeredtech with decent specs (something like 1.8 Core Duo, 2 gigs of RAM and a good hard drive). In the first 24 hours after launch we landed on the front page of Lifehacker, Mashable and The Consumerist. Over the course of three days our traffic went from zero to over 12,000 page views. A Google search for “Priceadvance” on Monday returned 120 results, almost all of them unrelated to us. A search now returns over 3,000 results.

Working full time on CollegeHumor and Bustedtees has given me a substantial amount of experience in building applications that scale. Priceadvance is running a custom PHP framework, a “somewhat” optimized mySQL database using MyISAM tables and a Memcached cache layer. Even with my experience at CollegeHumor, we very rarely query tables with rows in the millions. The Priceadvance database has several million rows and appears to be our biggest bottleneck. Thankfully our server hasn’t been totally killed at any point, but slow response times are common. As I’m writing this, I’m killing slow queries.

We’re not positive what the answer to our database problem is. We are in the process of converting all our tables over to innoDB to test for the next day or two. Our queries are pretty simple and one theory is that the table level locking is what’s holding us back. If the innoDB experiment doesn’t work we may try Heap tables. If all of this fails, we will split the tables into several smaller tables and devise a mapping system for requests. We are also looking into adopting the Google server methodology and dumping our expensive machine in exchange for 2 or 3 lower end boxes for the same price. With all that said, I’m sort of glad we are having scaling issues only three days in.

So far this has been an incredible experience. Everything from the Ycombinator interview (I will write about that someday) to a friend telling me Priceadvance was the talk of his office. Take a look at the site; we appreciate all feedback, suggestions and comments.

I wrote a class to deal with preference handling in the PriceAdvance Firefox extension. I found the Mozilla documentation to be difficult to understand. So, I decided to share the class. Theres a little bug in Firefox that took me two days and borderline suicide to figure out… Extensions can have a default preference file that you can query using the preference interface. These preference files are “magically” loaded from a very specific file folder/location - “defaults\preferences\xxxxx.js”. To set a preference you simply put “pref(”branch_title.variable_name”, value);” in the file. If you use the class I made these preferences can be accessed like this: “Preference.getPref(”variable_name”)”. Now the bug I found is with the way you configure your extension. If the extension is bundled and installed “properly” it will work without a hitch. For development purposes I have my extension using a pointer from the extension folder to another folder location where I do all the work. Apparently when its setup as a pointer it wont load the preference files. Whats really annoying is theres no way to verify if the magical files have been loaded or not. Because of this I have avoided preference defaults and use my own variables class.

Preferences = {
initialize: function() {
  this.prefs = Components.classes['@mozilla.org/preferences-service;1']
  .getService(Components.interfaces.nsIPrefService)
  .getBranch('extensions.branch_title.');
 
  this.prefs.QueryInterface(Components.interfaces.nsIPrefBranch2);
},
 
getPref: function(name) {
  var type = this.prefs.getPrefType(name);
 
  if (type == this.prefs.PREF_STRING)
    return this.prefs.getCharPref(name);
  else if (type == this.prefs.PREF_INT)
    return this.prefs.getIntPref(name);
  else if (type == this.prefs.PREF_BOOL)
   return this.prefs.getBoolPref(name);
},
 
setPref: function(name, value) {
  var type = this.prefs.getPrefType(name);
 
  if(type == 0) {
    if(typeof(value) == 'number')
      type = 64;
    else if(typeof(value) == 'string')
      type = 32;
    else if(typeof(value) == 'boolean')
      type = 128;
  }
 
  if (type == this.prefs.PREF_STRING)
    this.prefs.setCharPref(name, value);
  else if (type == this.prefs.PREF_INT)
    this.prefs.setIntPref(name, value);
  else if (type == this.prefs.PREF_BOOL)
    this.prefs.setBoolPref(name, value);
}
};
 
// USAGE:
Preferences.initialize();
Preference.setPref('this_works', true);
alert(Preference.getPref('this_works'));
/******************/

To view preference variables that have been set, type “about:config” into the address bar and filter for your branch.

Earlier this week I was out to lunch with a College Humor colleague. He happens to be the only Connected Ventures employee with a kid. We got talking about toys. He told me his son plays with Ghostbusters and Thomas the train. Has the toy industry really not done any innovation in the last 20 years to capture kid’s attention?

I was on the subway heading uptown to meet a friend and I saw a billboard for this new TV series about the Wizard of oz. From the billboard I gathered it would be about the other side (not Pink Floyd) of Oz. Maybe it’s about what the tin man’s favorite drink is, or that the lion was nailing Dorothy on the side? Either way, it’s interesting that so many years later that’s what people are still into.

Year after year the top rated rock and roll songs are from the 80’s. The coolest cars are from the 40’s to 70’s. People collect just about anything that’s old, it obvious that is has some value to someone. What it comes down to, is a lot of the things we grew up with are still awesome today. My question is, in 20 years will we look back at things from this time and think the same way? Because there a very few things that I have recently discovered that are leaving a lasting impression. But I will never forget the first time I got a plasma screen TV, it must have been a similar experience to what it was like to get a color TV.