Growth, performance and madness
Phew! been a busy time in code land for me. Mostly Entrecard. We've got a lot to do, things have picked up hugely and lots of people are having a good time which is great, but we've got a long way to go to where we want to be.
Various fascinating reads on the site and forum recently, obviously we had the interview with Graham on problogger where we got a ton of positive comments from new and old alike (yes, I really do read pretty much everything anyone says if I can find it, although that became a lot harder since google started returning 178,000+ results for "entrecard").
We had an interesting scrap about the economy that ran for a bit. I don't really agree with anything Slam has to say on the subject of credit value but then I'm biased ;).
People have talked about buying and selling of credits as if it were some black and white thing where either credits are entirely trapped within the system or credits are rushing out the door like free icecream, but when it comes down to it there are a wide variety of alternatives. The majority of the major credit-based economies out there complete almost all of their transactions in-house, only a relatively small percentage of credits bought or sold are traded for regular currency despite the relative freedom to do so. Even if Entrecard was a totally open market with "BUY CREDITS NOW" on the homepage, I'm willing to bet we wouldn't generate more than 5% of the total credits per day in currency conversion, and we're not going to do anything like that.
Remember, we hit half a million drops the other day, that's a million credits into the econ in a little over two months, that's 17k/day even if you're going by the averages rather than the numbers we do per day now. Any alternate methods of credit creation are highly unlikely to make any real dent in that number.
I'd tell you the exact numbers but our admin economy page is currently out of order, it can't handle the volume of data anymore. Recently we've been pretty much doubling the number of visits every 10 days, a level of growth that poses problems no matter how good your planning is.
It's fascinating in a sense, our architecture is designed to scale up to a million blogs in theory, but as you bring more and more users on board you hit milestones where certain parts of the system just hit rough edges you didn't see coming. We've run load testing on entrecard.com that says it should be able to push over 50 dynamic page views per second on our current platform, and yet it started struggling recently well short of that.
The reason is pretty simple. Load testing systems are not real users. You can construct load testing setups that do a much better job of simulating them than the one I used, but even then people simply do unexpected things. In addition, the planned patterns of use often turn out to be bogus. In the case of Entrecard it was almost a product of a change in our focus as we got more familiar with the product. Initially it was thought that users would turn up to the main site only to fiddle with their control panel, there was nothing else really planned, we had the categories page and that was about it (those who can remember it at release are laughing).
As a consequence I was heavily focused on making sure we could deliver enough widgets - the amazon S3 system we use for example - rather than making the "control panel" website quick.
Then came chain dropping, the category tabs, the forum, the shop, the blog, the message system...and suddenly Entrecard.com itself became a primary destination, and our visits-per-user shot up...and thus our load testing profile was suddenly worthless.
So this week I got a bit sidetracked, delayed a few new things for a couple of days and started doing some proper profiling and taking some serious performance action. Much of it won't be visible until the next code drop but we did do some rearchitecting of the hosting systems tonight, I'm going to be interested to see the results in the morning.
For those wondering what kind of effort it takes to host Entrecard, consider this, we have to serve a widget for every page view that every one of over 2,500 blogs get. Some of those blogs are pretty damn popular all by themselves - and more sign up every day.
It's a challenge, and I love it. Just occasionally I wonder why I'm not working at a standard corporate fast asleep at my desk tho ;)
Various fascinating reads on the site and forum recently, obviously we had the interview with Graham on problogger where we got a ton of positive comments from new and old alike (yes, I really do read pretty much everything anyone says if I can find it, although that became a lot harder since google started returning 178,000+ results for "entrecard").
We had an interesting scrap about the economy that ran for a bit. I don't really agree with anything Slam has to say on the subject of credit value but then I'm biased ;).
People have talked about buying and selling of credits as if it were some black and white thing where either credits are entirely trapped within the system or credits are rushing out the door like free icecream, but when it comes down to it there are a wide variety of alternatives. The majority of the major credit-based economies out there complete almost all of their transactions in-house, only a relatively small percentage of credits bought or sold are traded for regular currency despite the relative freedom to do so. Even if Entrecard was a totally open market with "BUY CREDITS NOW" on the homepage, I'm willing to bet we wouldn't generate more than 5% of the total credits per day in currency conversion, and we're not going to do anything like that.
Remember, we hit half a million drops the other day, that's a million credits into the econ in a little over two months, that's 17k/day even if you're going by the averages rather than the numbers we do per day now. Any alternate methods of credit creation are highly unlikely to make any real dent in that number.
I'd tell you the exact numbers but our admin economy page is currently out of order, it can't handle the volume of data anymore. Recently we've been pretty much doubling the number of visits every 10 days, a level of growth that poses problems no matter how good your planning is.
It's fascinating in a sense, our architecture is designed to scale up to a million blogs in theory, but as you bring more and more users on board you hit milestones where certain parts of the system just hit rough edges you didn't see coming. We've run load testing on entrecard.com that says it should be able to push over 50 dynamic page views per second on our current platform, and yet it started struggling recently well short of that.
The reason is pretty simple. Load testing systems are not real users. You can construct load testing setups that do a much better job of simulating them than the one I used, but even then people simply do unexpected things. In addition, the planned patterns of use often turn out to be bogus. In the case of Entrecard it was almost a product of a change in our focus as we got more familiar with the product. Initially it was thought that users would turn up to the main site only to fiddle with their control panel, there was nothing else really planned, we had the categories page and that was about it (those who can remember it at release are laughing).
As a consequence I was heavily focused on making sure we could deliver enough widgets - the amazon S3 system we use for example - rather than making the "control panel" website quick.
Then came chain dropping, the category tabs, the forum, the shop, the blog, the message system...and suddenly Entrecard.com itself became a primary destination, and our visits-per-user shot up...and thus our load testing profile was suddenly worthless.
So this week I got a bit sidetracked, delayed a few new things for a couple of days and started doing some proper profiling and taking some serious performance action. Much of it won't be visible until the next code drop but we did do some rearchitecting of the hosting systems tonight, I'm going to be interested to see the results in the morning.
For those wondering what kind of effort it takes to host Entrecard, consider this, we have to serve a widget for every page view that every one of over 2,500 blogs get. Some of those blogs are pretty damn popular all by themselves - and more sign up every day.
It's a challenge, and I love it. Just occasionally I wonder why I'm not working at a standard corporate fast asleep at my desk tho ;)
Labels: entrecard, madness, performance
posted by PhiRatE at
1:12 AM
14 Comments
