Growth, performance and madness
Phew! been a busy time in code land for me. Mostly Entrecard. We've got a lot to do, things have picked up hugely and lots of people are having a good time which is great, but we've got a long way to go to where we want to be.
Various fascinating reads on the site and forum recently, obviously we had the interview with Graham on problogger where we got a ton of positive comments from new and old alike (yes, I really do read pretty much everything anyone says if I can find it, although that became a lot harder since google started returning 178,000+ results for "entrecard").
We had an interesting scrap about the economy that ran for a bit. I don't really agree with anything Slam has to say on the subject of credit value but then I'm biased ;).
People have talked about buying and selling of credits as if it were some black and white thing where either credits are entirely trapped within the system or credits are rushing out the door like free icecream, but when it comes down to it there are a wide variety of alternatives. The majority of the major credit-based economies out there complete almost all of their transactions in-house, only a relatively small percentage of credits bought or sold are traded for regular currency despite the relative freedom to do so. Even if Entrecard was a totally open market with "BUY CREDITS NOW" on the homepage, I'm willing to bet we wouldn't generate more than 5% of the total credits per day in currency conversion, and we're not going to do anything like that.
Remember, we hit half a million drops the other day, that's a million credits into the econ in a little over two months, that's 17k/day even if you're going by the averages rather than the numbers we do per day now. Any alternate methods of credit creation are highly unlikely to make any real dent in that number.
I'd tell you the exact numbers but our admin economy page is currently out of order, it can't handle the volume of data anymore. Recently we've been pretty much doubling the number of visits every 10 days, a level of growth that poses problems no matter how good your planning is.
It's fascinating in a sense, our architecture is designed to scale up to a million blogs in theory, but as you bring more and more users on board you hit milestones where certain parts of the system just hit rough edges you didn't see coming. We've run load testing on entrecard.com that says it should be able to push over 50 dynamic page views per second on our current platform, and yet it started struggling recently well short of that.
The reason is pretty simple. Load testing systems are not real users. You can construct load testing setups that do a much better job of simulating them than the one I used, but even then people simply do unexpected things. In addition, the planned patterns of use often turn out to be bogus. In the case of Entrecard it was almost a product of a change in our focus as we got more familiar with the product. Initially it was thought that users would turn up to the main site only to fiddle with their control panel, there was nothing else really planned, we had the categories page and that was about it (those who can remember it at release are laughing).
As a consequence I was heavily focused on making sure we could deliver enough widgets - the amazon S3 system we use for example - rather than making the "control panel" website quick.
Then came chain dropping, the category tabs, the forum, the shop, the blog, the message system...and suddenly Entrecard.com itself became a primary destination, and our visits-per-user shot up...and thus our load testing profile was suddenly worthless.
So this week I got a bit sidetracked, delayed a few new things for a couple of days and started doing some proper profiling and taking some serious performance action. Much of it won't be visible until the next code drop but we did do some rearchitecting of the hosting systems tonight, I'm going to be interested to see the results in the morning.
For those wondering what kind of effort it takes to host Entrecard, consider this, we have to serve a widget for every page view that every one of over 2,500 blogs get. Some of those blogs are pretty damn popular all by themselves - and more sign up every day.
It's a challenge, and I love it. Just occasionally I wonder why I'm not working at a standard corporate fast asleep at my desk tho ;)
Various fascinating reads on the site and forum recently, obviously we had the interview with Graham on problogger where we got a ton of positive comments from new and old alike (yes, I really do read pretty much everything anyone says if I can find it, although that became a lot harder since google started returning 178,000+ results for "entrecard").
We had an interesting scrap about the economy that ran for a bit. I don't really agree with anything Slam has to say on the subject of credit value but then I'm biased ;).
People have talked about buying and selling of credits as if it were some black and white thing where either credits are entirely trapped within the system or credits are rushing out the door like free icecream, but when it comes down to it there are a wide variety of alternatives. The majority of the major credit-based economies out there complete almost all of their transactions in-house, only a relatively small percentage of credits bought or sold are traded for regular currency despite the relative freedom to do so. Even if Entrecard was a totally open market with "BUY CREDITS NOW" on the homepage, I'm willing to bet we wouldn't generate more than 5% of the total credits per day in currency conversion, and we're not going to do anything like that.
Remember, we hit half a million drops the other day, that's a million credits into the econ in a little over two months, that's 17k/day even if you're going by the averages rather than the numbers we do per day now. Any alternate methods of credit creation are highly unlikely to make any real dent in that number.
I'd tell you the exact numbers but our admin economy page is currently out of order, it can't handle the volume of data anymore. Recently we've been pretty much doubling the number of visits every 10 days, a level of growth that poses problems no matter how good your planning is.
It's fascinating in a sense, our architecture is designed to scale up to a million blogs in theory, but as you bring more and more users on board you hit milestones where certain parts of the system just hit rough edges you didn't see coming. We've run load testing on entrecard.com that says it should be able to push over 50 dynamic page views per second on our current platform, and yet it started struggling recently well short of that.
The reason is pretty simple. Load testing systems are not real users. You can construct load testing setups that do a much better job of simulating them than the one I used, but even then people simply do unexpected things. In addition, the planned patterns of use often turn out to be bogus. In the case of Entrecard it was almost a product of a change in our focus as we got more familiar with the product. Initially it was thought that users would turn up to the main site only to fiddle with their control panel, there was nothing else really planned, we had the categories page and that was about it (those who can remember it at release are laughing).
As a consequence I was heavily focused on making sure we could deliver enough widgets - the amazon S3 system we use for example - rather than making the "control panel" website quick.
Then came chain dropping, the category tabs, the forum, the shop, the blog, the message system...and suddenly Entrecard.com itself became a primary destination, and our visits-per-user shot up...and thus our load testing profile was suddenly worthless.
So this week I got a bit sidetracked, delayed a few new things for a couple of days and started doing some proper profiling and taking some serious performance action. Much of it won't be visible until the next code drop but we did do some rearchitecting of the hosting systems tonight, I'm going to be interested to see the results in the morning.
For those wondering what kind of effort it takes to host Entrecard, consider this, we have to serve a widget for every page view that every one of over 2,500 blogs get. Some of those blogs are pretty damn popular all by themselves - and more sign up every day.
It's a challenge, and I love it. Just occasionally I wonder why I'm not working at a standard corporate fast asleep at my desk tho ;)
Labels: entrecard, madness, performance
posted by PhiRatE at
1:12 AM
14 Comments:
Ahhhh.... the joy of growing pains, huh? ;)
Excellent post! Yeah, I'm really out here supposedly chain-dropping and I keep getting sidetracked.
Excellent post, and great to read what's happening behind the scenes.
I'm found Entrecard to be a great way to boost traffic, and also to discover some great blogs. I know there have been some debates about the value of Entrecard visits, but I know from my own experiences that you can retain some of the droppers...if your content is good enough.
Certainly I've subscribed to 10 or more blogs I've discovered...
Thanks for all the help. Hopefully, we figure this situation out with the Firefox browser! I love firefox and so far I've only been able to drop via Internet Explorer.
So when will you start bringing in ads and make real money from this thing? You need a vacation and more help, dude!
Glad to know that you guys are working hard to ensure the success of Entrecard. Those BlogExplosion guys could learn a thing or two from you.
I've come across 6 sources of people selling credits including my own site. 3 of those sales offers were 1 time only. I make more at work than I do clicking for an hour, but it a rewarding way to get rid of extra credits.
i add another thing to do as your homework. it is, entrecard ad appeare not properly while browse using opera mobile on my se k510. the Edropcard bar always raise at the middle of card hence break it. hope you could fix it. thanks in advance.
i mean the proble occured on the 125 entrecard format widget. fyi, on the dashboard, email icon and ad price is good looking but the card image resized smaller.
My non-Blog job is in software quality. Enterprise-class software on the net is rife with possible bottlenecks. Load tests, stress tests, and trying to account for rapidly aggressive growth is tough to test for, by any stretch of the imagination ...
Even though you sense problems and scalability is beginning to worry you, the site is great and the responsiveness of your widget to my site is awesome.
So, you go!!! :)
Thanks, and take care,
Steve
Just curious, but are you the only person doing all this work? I've only ever seen you and Entrecard Support on the forums...
I bet oneday you will look back at this post an say "I thought 2,500 blogs was a lot!"
It is truly amazing to see how this website has taken off. I am just coming out of my Newbie cocoon right now and to me this site feels like its been around a long time.
Sadly I was not there at the start (I thought this was some scam or something that everyone was doing) but even now I see tons of changes.
I have found some of my most read blogs here, I have gained viewers, and had tons of fun that I never knew I could have using a blog.
Nice read,
lee
The speed of growth of Entrecard has amazed me. One week you can buy an ad on someone for 11 credits/day. Two weeks later, same person is 256 credits/day. I should have joined your forums and blog sooner. Would have found out the secret to so many blogger's success with Entrecard bringing more traffic to their site.
The Entrecard is still only showing part of the time on my blog, Sweet Resistance and have to refresh the screen two to three times to get it to show. Hope you can get this fixed soon.
Entrecard ROCKS!!!! I've been using EC for about 2 months and it brought more than 2000 visitors to my 3 months old blog. I will support EC all the way, LONG LIVES ENTRECARD!!!!
Post a Comment
<< Home