Dev day, rants etc

For those who asked what I recommend as a web platform, the basic answer is I don't. I personally use Turbogears, a python-based framework - it works really well for me, but the key thing is that pretty much any of the next-gen platforms (or should I say "current-gen") are a huge step up. Much of it comes down to the way in which you like to code - some of the frameworks work better for people who like working in text and coding, others work better for people who like wizard-based construction and auto-templates. Go experiment.

Dev day at Entrecard went really amazingly well, while the 8am session was dead that was perhaps to be expected, the rest of the sessions had plenty of people, lively discussion, and I managed to implement most of the features people suggested on the spot.

I've got to say, just before the first session I thought to myself "You're mad, why are you doing this? nobody implements features on the spot, it's just stupid", but damn if I didn't manage to pull it off.

In the end I see it as a fundamental statement about what we are - other places try and tack a shiny web2.0 appearance onto an outdated technical approach, this was as bold a statement as I knew how to make for saying "we live and breathe agile". Also, it was great fun, and we got some fantastic ideas.

Also, full credit to Mibbit for their ajax IRC client, the embedded version performed perfectly and let us provide an easy, robust chat platform without putting any load on our server.

Those users are crazy

Well, to be quite honest I am now officially overwhelmed. The Entrecard community, the sheer volume of stuff going on, is beyond my ability to keep up. I'm still trying, for what it's worth, but there's just so much to do!

We have 4,700+ live widgets at this point and it's climbing fast, and the feature ideas list is getting longer faster than I can implement them, even for the "holy crap that's a great idea" mini-list.

Anyway, enough of my complaining, I guess it's a vastly better set of worries than sitting in the middle of a big empty site going "meep?".

One of the consequences of having so much to do is that I don't have quite as good a touch for what everyone is doing anymore. I used to have a good idea about the kind of advertising levels people were generally running. I implemented a new type of calendar display for the ad system today, then went to check a few dashboards to make sure it came out ok, and ran into this:



That's just crazy, but after doing a check around, that kind of volume of ads doesn't seem that uncommon. I may have to rethink the displays for high-level advertisers like that.

We've got some fun stuff coming up soon, I'm really looking forward to the coding day, it'll be cool to chat with people and deliver stuff "live".

Time to go to bed, 2am is too late for me these days. zzz.

Growth, performance and madness

Phew! been a busy time in code land for me. Mostly Entrecard. We've got a lot to do, things have picked up hugely and lots of people are having a good time which is great, but we've got a long way to go to where we want to be.

Various fascinating reads on the site and forum recently, obviously we had the interview with Graham on problogger where we got a ton of positive comments from new and old alike (yes, I really do read pretty much everything anyone says if I can find it, although that became a lot harder since google started returning 178,000+ results for "entrecard").

We had an interesting scrap about the economy that ran for a bit. I don't really agree with anything Slam has to say on the subject of credit value but then I'm biased ;).

People have talked about buying and selling of credits as if it were some black and white thing where either credits are entirely trapped within the system or credits are rushing out the door like free icecream, but when it comes down to it there are a wide variety of alternatives. The majority of the major credit-based economies out there complete almost all of their transactions in-house, only a relatively small percentage of credits bought or sold are traded for regular currency despite the relative freedom to do so. Even if Entrecard was a totally open market with "BUY CREDITS NOW" on the homepage, I'm willing to bet we wouldn't generate more than 5% of the total credits per day in currency conversion, and we're not going to do anything like that.

Remember, we hit half a million drops the other day, that's a million credits into the econ in a little over two months, that's 17k/day even if you're going by the averages rather than the numbers we do per day now. Any alternate methods of credit creation are highly unlikely to make any real dent in that number.

I'd tell you the exact numbers but our admin economy page is currently out of order, it can't handle the volume of data anymore. Recently we've been pretty much doubling the number of visits every 10 days, a level of growth that poses problems no matter how good your planning is.

It's fascinating in a sense, our architecture is designed to scale up to a million blogs in theory, but as you bring more and more users on board you hit milestones where certain parts of the system just hit rough edges you didn't see coming. We've run load testing on entrecard.com that says it should be able to push over 50 dynamic page views per second on our current platform, and yet it started struggling recently well short of that.

The reason is pretty simple. Load testing systems are not real users. You can construct load testing setups that do a much better job of simulating them than the one I used, but even then people simply do unexpected things. In addition, the planned patterns of use often turn out to be bogus. In the case of Entrecard it was almost a product of a change in our focus as we got more familiar with the product. Initially it was thought that users would turn up to the main site only to fiddle with their control panel, there was nothing else really planned, we had the categories page and that was about it (those who can remember it at release are laughing).

As a consequence I was heavily focused on making sure we could deliver enough widgets - the amazon S3 system we use for example - rather than making the "control panel" website quick.

Then came chain dropping, the category tabs, the forum, the shop, the blog, the message system...and suddenly Entrecard.com itself became a primary destination, and our visits-per-user shot up...and thus our load testing profile was suddenly worthless.

So this week I got a bit sidetracked, delayed a few new things for a couple of days and started doing some proper profiling and taking some serious performance action. Much of it won't be visible until the next code drop but we did do some rearchitecting of the hosting systems tonight, I'm going to be interested to see the results in the morning.

For those wondering what kind of effort it takes to host Entrecard, consider this, we have to serve a widget for every page view that every one of over 2,500 blogs get. Some of those blogs are pretty damn popular all by themselves - and more sign up every day.

It's a challenge, and I love it. Just occasionally I wonder why I'm not working at a standard corporate fast asleep at my desk tho ;)

Wasting my time

Well, today we had our first real attempt to abuse the system, fake accounts and all. it's bizarre, people don't seem to realise it's a closed system - no matter what you do we can follow the credits, there is no way to "launder" them in Entrecard, no matter what tricks you pull it's a matter of a few clicks in the admin system to remove all the ads you bought, lock/delete your account, penalise you credits or whatever.

In this case the optimistic abuser thought that perhaps by sending credits to another account using the coupon system that he'd be able to avoid detection. it's pointless, we know exactly where they came from and where they went and all you do is end up with no credits, no ads and a locked account.

Instead, all he really did was waste time I could have spent writing actual features that would have helped him, and everyone else, actually get some value. Makes no sense.

In other news, another user decided to publish their own idea of how to bypass the Entrecard drop security system. The post was thinly washed over with "I hope they don't penalise me" as if it wasn't common sense not to post that kind of stuff. Really bizarre. If you wanna contribute materially to the security of Entrecard, then send us private feedback. We're ok with you taking credit, but only if you post about it *after* we've had a reasonable chance to fix it. If you just go off and post straight off, you go straight into the system abuse category, and we hit the penalty button - what other option do we have?

A day in the life off..

..an Entrecard programmer.

This post inspired by this forum thread

Ok, I'm going to outline todayish (it isn't even finished yet), but it's not my typical day, honestly, my life is crazy but not quite this crazy.

Ok, we'll start at 00:00...erm, ok, I'm still awake at 00:00, busy coding the latest Entrecard feature. I'm really excited about it but there are just so many small bits and pieces to cover with it. Wait, I'm getting ahead of myself. Welcome to December. In december, every year, the following 2 things happen:

1. Every single one of my clients, plus a couple of new ones, suddenly realise that they need something done urgently.
2. Every single year, I forget that this is the case.

So what happens is that by the end of November I have my December all nicely planned out, there will be holidays, a few decent projects, everything just so.

By Dec 2nd, things are going downhill rapidly. Emergency stuff is turning up like crazy and my regular project work is getting pushed back or squeezed.

So, back to my day, I spend the next few hours, until around 2:30am, getting the latest feature pretty much ready to go. Done with that, I can't release it at this point, it still needs a bit of layout polish plus I can't release big stuff just before I go to bed, gotta be around in case things blow up :)

So, at 2:30, I get to start on my next job, reconfiguring a netscreen router at another clients place. This rapidly descends into the usual exercise in frustration that networking stuff always is - it's much easier to debug code - but I nail the problems one by one and make some good headway on the main problem. At 4:30am I get stuck, I'm trying to manage a straight-through src/dst NAT on a VPN tunnel (it's even more complex than it sounds) and the docs say "If you'd like to know more about this, see page 156". This, in a 100 page document. I fight with it for a while but eventually it hits 5am and I've gotta be up fairly early so I hit the sack and stare at the ceiling for a bit while my brain winds down.

At 9:08am I stagger out of bed, have a shower, check a few things on the computer, and then walk across town to my dentist, where they give me some anesthetic and then drill out the center of my tooth (it's called a Root Canal). it didn't hurt, bu t at one point the dentist accidentally let the "long thin file" she was using into my line of sight. Nothing like knowing your dentist is cleaning the *inside* of your tooth with what is basically a small ice-pick.

11:45am, $350 poorer, with a numb mouth and 2 more sessions to go, I wander back out onto the street and off home. Most of the walk passes in a daze, I stop to pick up some yogurt because I know from experience it's great post-dentist food. I then get woken from my daze by loud horn blaring. Slight diversion: you may or may not be aware that Japan likes to hunt down and kill whales. By "likes" I mean, they appear to have no regard for what the rest of the world thinks about it, they'll do anything, including the laughably thin-veiled cover of "science", to hunt and kill whales. It's really weird, the Japanese I know seem like nice people, I'm not certain how a bunch of nice people end up living in a nation that does that.

Anyway, New Zealanders really like whales, and they get quite upset about this time of year every year then the whaling boats go out once again and the inevitable greenpeace stand-offs etc. Anyway, there were a bunch of girls standing on the sidewalk with huge signs saying "HONK FOR WHALES". Needless to say, in NZ, this basically resulted in infinite amounts of honking. I'm sure the people working in the offices above were delighted.

I finally made it home, sat down, ate some yogurt, and then got down to trying to get the latest feature stuff out for Entrecard. This is the ideal time to do it because all the hard thikning had already been done "last night", so all I really needed to do was clean up some styles and push it out to the live platform. A few hiccups later (big upgrades invariably lead to some problems, especially when you're releasing as fast and furious as Entrecard does) and it was all live. I went and bought myself a featured status, and emailed a friend who helps out quite a bit with Entrecard debugging to get him to get one too. Fixed a bug he hit immediately, and we're away.

All during this I'm also managing about 3 conversations with clients via IM, which can be occasionally distracting. Oh, and I've run out of yogurt, and my tooth hurts.

Now I'm thinking I should write something about the new feature, but in fact I don't want to. One of the things I quite like doing is letting people discover new stuff themselves. Yes Graham will do a post about it soon enough no doubt, but initially I like to reward people who are active and paying attention. In this case, that could be quite a big win but since I didn't add a button to the top bar nobody seems to have noticed it yet - probably just as well :)

Anyway, so, no big announcement, but I did wanna write something, and then I remembered the writing thing we were going to do that people have been talking about on the forums, so I figured why not and jumped into it. That brings us pretty much up to date.

From here, I've got more client work to do, some bug fixing and new features from a release the other day. I need to go pick up some Goon Show episodes from a friend so I can take them on holiday, I need to go renew our video hire of "Bones" (I love her office. I love it. It's just..awesome) which we'll probably watch some of later tonight, I have to work out whether I'm willing to take the risk of playing Twilight soccer with my tooth the way it is, then more work later in the evening trying to get the rest of that router config stuff working.

yeah. they don't call it the Silly Season for nothing. Bring on the holidays!

What is a futex and why does Phi hate it?

A futex is a mechanism for dealing with resource contention and synchronisation between (semi)independent threads or processes. It is used to build locks and various other tools for managing threads.

What? didn't understand a word of that? no big deal. it's just a coding concept that means hundreds of different user requests can all access the database at the same time without getting in each others way.

I hate it, because right now, I'm having real trouble with one on Entrecard. The problem is simply a situation called a "deadlock". Imagine the following scenario (simplification):

Piece of code A uses Lock 1, then Lock 2
Piece of code B uses Lock 2, then Lock 1

These will always work, except in the instance when A and B initiate at precisely the same time. When this occurs, code A gets Lock 1, but can't get Lock 2, and code B gets Lock 2, but can't get Lock 1. As a result both of them sit there waiting for the other, and neither of them will release their lock. Immediate result: horrible nasty application hang.

The evil bit is that you'll never notice this scenario when you're developing your application, because a single user almost never triggers this kind of event. Nor are you likely to notice it when your application has a low volume of users, for the same reason. It appears out of nowhere just when you have a bunch of users, and just when you really really don't want things like horrible nasty application hangs.

I have one of these somewhere. The problem is not fixing it, the problem is finding it. Entrecard is a lot of code, with a very high level of abstraction, and the nature of a deadlock makes it very difficult to debug after it has happened in a complex environment. Worse, it happens totally randomly, so I'm unable to narrow down where in the application it is happening. I'm confident I'll get it but HOLY CRAP is it frustrating, especially when I have to drag myself out of bed at 3am because it hung the app server and I have to restart it.

Entrecard Tricks and Tips 1: Card design

Ok, this may or may not be a regular series but hey, we can hope right?

Good design itself isn't my province, I can't help you there, I'm a coder :) however I can give a few Entrecard-specific suggestions:

• Definitely go for the 125x125 pixel image based card. The text stuff is ok, but it doesn't stand out since we only have 5 designs, so in a big card listing you get lost with all the other black + silver cards.


• Upload your image in PNG format. For security reasons we have to re-encode all the images uploaded, and JPEG doesn't re-encode very well so you end up with some compression artifacts. PNG on the other hand should work perfectly.


• When on display in a widget, your card will always be surrounded by a border of #444444 (dark gray), in other spaces the border can be black or white. Make sure you test against these to be sure it doesn't look weird.


• It is possible, with a bit of work, to design a card specifically for a given advert spot. To do so, do your design (say, in pink and white for display on galadarling.com), upload it (Edit on your dashboard), and then apply to the advertiser. The design then "sticks" with them, and you can upload a new image to use as your regular card or apply at a new site without changing what will end up displaying on the first spot.

Stats, settings, bits and pieces

A fair few changes dropping on a regular basis. it's likely nobody besides me even realises how many micro-changes hit every day, I probably commit roughly 6-12 times a day to the production tree, although only a few of those result in user-visible changes.

Anyway, the latest major one to drop in yesterday was the new stats screen:



We introduced some new numbers, specifically the organic clicks, and fixed the term at 30 days - the 7 day chart was just silly. More importantly however was the explanation I added next to each chart, explaining what it is and how you can improve your numbers. I think this is something that's often missed - places provide you with piles of numbers but no good information on what you can reasonably do about them.

I've added another control to the Advert settings page, granting people control over whether they want to auto-accept previously approved entrecards. The system is pretty much flawless, in the sense that if you've approved it before, there's little reason not to again, however part of our mission is to ensure control remains in the hands of the site owner so having the switch there is important, and we defaulted it to "off" so it won't auto-accept unless you ask it to.

A few other bits and pieces around the place, fixes, nailed one of the cookie bugs which sorted the last two no-drop cases I'm aware of which is great.

In case nobody noticed, I also managed to finally stab that weird firefox rendering error to death. For those who were curious, it turns out that despite happily displaying block-level elements inside an "a href", firefox occasionally fails to render the contents properly. However, if you use span tags instead, but use CSS to change their display mode to block..it works fine. Nutty.

Queue limits, multiple advert block

Well, when someone like ProBlogger arrives it inevitably highlights any bugs in the management of advert requests. In this case the poor guy got bombed with ad requests, including multiple requests from single users.

We hadn't really decided whether multiple ad requests were a problem before. It was one of the reasons people were rejecting ads, but at the same time it seemed managable and nobody was complaining. But 5 requests from the same person is just..a little silly.

So that's gone now, if you have an ad queued, approved or running on a given blog you can't submit further adverts. Later I hope to provide people with finer grain control of whether they accept multiple requests but for now we needed a solid solution. Feedback welcome.

In addition, we now have a "queued" limit, you can't have more than 15 entrecards queued at a time. This prevents someones mailbox getting too stuffed on their first day or if they're suddenly popular for some reason :)

I hope to make the blocks a little clearer soon - at the moment they're an error displayed if you try to advertise rather than preventing you from clicking the button in the first place.

Recommendations, advert options

New shiny things today.

We got Recommendations, which let you post a public recommendation of another Entrecard site. The recommendations are approval-only, so they're not exactly going to paint an unbiased picture of a given blog, but at the same time they do help people get a feel for what the blog is, whether it does a good job and it lets people who like a given blog show some appreciation in a meaningful way.

That's basically why I did it anyway :)

In other news, we have the option to switch off advertising for your entrecard. When you do this (under Advert settings) your card will no longer accept adverts (the ones you have queued will continue to display as normal). The objective is to prevent people who are, say, going on holiday or otherwise too occupied to avoid ending up with a massive pending queue and then irritating everyone with a string of rejects.

So much to do!

In general in software development, things go roughly according to this formula:

tight timelines x level of innovation = level of crappy architecture

The basic issue is that the faster you have to build something, the less time you have to "do it properly" and the more inclined you are to take shortcuts. Similarly, the more "innovative" or - in other words - off the edge of the map your application is the less chance you have of picking the right architectural elements to support what it will eventually become.

Thus when the two combine you normally end up with architecture that makes a dogs breakfast sound well organised.

That said, I'm delighted to say it wasn't what happened with Entrecard. I'd like to say it was all hard work and skill, and to be fair I'm going to take some credit because it's my blog and I'm vain like that. But the fact is there's a fair chunk of luck involved as well.

Unfortunately, the natural consequence of this is that we now have a nice solid extensible architecture with no actual chronic outstanding issues. And we have Ideas. So Many Ideas. Now I have a feature-list a million miles long, all of which are awesome and I really really want to have them in Right Now Damnit.

Ah well. there are worse fates than having a bunch of cool stuff to write for an appreciative audience :)

Forums, new dashboard

ok, we have some experimental forums live. I know I know I reinvented this wheel and it's somewhat octagonal and has no suspension, however it does integrate well with our main application and we have plans to integrate other aspects further as well.

It's not really central to the business plan but at the very least we needed somewhere we could get users to engage with each other - the feedback system has been really good but that's very much a 1-to-us thing.

We also have a modified dashboard, including a "buzz" slot where we can push some 1-liner announcements, and I've removed a lot of the stats stuff and put more explanation in on some things. The main reasoning behind the buzz slot is that we're changing things so rapidly that we really need a good way of keeping people in the loop.

The removal of the stats was..well, that's just more appropriate for the statistics section anyway, it was taking up space. I'm not so sure about taking away the balance (it's on every page in the top right anyway), but I couldn't figure out a good way of keeping it in play without it looking out of place, so I took the hint. We can always put it back somewhere :)

Go play on the forum!

Security

Lets be honest, while I'm sure all of our users are impeccably honest and would never do anything as sneaky as trying to use a script to do the drops for them, it's important that these, and other areas, are properly secured.

Unfortunately, I can't really discuss the measures we use to protect the integrity of the drop system - while security through obscurity is hardly ideal, the principle of defense in depth holds. The idea basically being that you maintain a number of secure layers, information control being one of them, in order to prevent any single flaw from resulting in a complete breach.

I can say, however, that today someone decided to test that out. Needless to say I wouldn't be smiling about it if I hadn't already anticipated the attempt and had countermeasures installed. Suffice to say I'm pretty pleased with how it all worked out :)

Organic click tracking

We haven't been tracking what I (probably incorrectly) term "organic clicks", that is, clicks that go from our details page to your blog as opposed to from the widgets. It wasn't an oversight so much as simply not a priority since most people who really care have analytics or something which tells them how much comes from Entrecard anyway.

However it has reached the point where the information is fundamentally interesting for us, and for those users who are using the Entrecard provided statistics as a measure of how valuable the system is.

As a result, we're now tracking organic clicks from the details page via a redirector. It should make no visible difference to anything, it's simply useful. once we have a few days history in the database we'll add a chart onto the statistics page with the details.

Economy update

For those wondering where the economy is standing right now, basically its..on fire.

As at this posting, the total earned is 35,510 credits roughly. I say roughly because it's tearing upwards so fast the number changes every time I reload the page.

Of those, 12,317 is currently sitting in accounts, 10,243 credits have been invested in adverts currently running or queued, and the remaining 12,950 credits have been removed from the system via tax on adverts that have already run.

Active users who have spent any credits at all spent an average of 59% of their total earnings, which is much better than it was prior to the price double-up.

There is currently 1,120 days (26,880 hours) of advertising queued on 199..err..201..shit..202 widget-displaying blogs, an average of 5.6 days per blog.

We have recorded an epic 13,620 card drops in total. Seriously, people seem to find it rather addictive. I'm a little worried for some of you.

I've had 26 credits removed from my personal account by the bug bounty program :(

We've had a total of 3,359 advert requests, 444 of which were rejected.

That's all I can think of right now, gotta go grab dinner. leave a comment or feedback if you'd like any other specific info (that doesn't invade privacy obviously).

Internet Explorer 7, P3P and Entrecard

You have no idea how angry I am right now. My anger towards Microsoft and IE7 is a towering inferno of geek fury. I can barely restrain my urge to unleash a rant.

I am going to go play computer games until I calm down.

Long story short, I believe I have resolved the issue that prevented IE7 users from being able to drop on widgets. It came at the expense of one area of particularly cool traffic offloading that we had, and at the expense of my sanity. What was left of it.

The essence of the problem was identified by Wavumi earlier. Basically, Internet Explorer went off on this one-browser missing to "make the world safer" by refusing "third party cookies" (that is, cookies being delivered in an iframe, which is how we make the widget work). Except, of course, that doubleclick etc needed 3rd party cookies in order to work, so bowing to commercial pressure they decided to allow them if the website includes a machine-readable privacy policy, called P3P.

Now, at this point everything basically sinks into the sea. You see, the number of people who want to do 3rd party cookie stuff in order to put their forum in an iframe or whatever, is N. The number of people who can be bothered reading the P3P spec is N/100, and the number of people who can fight their way throught he commercial smackfest that is P3P generators, make any kind of reasonable judgement about all the vague terminology in the spec, and actually implement a rational, intelligent P3P file is N/10000.

That means, 9999 out of every 10000 sites with P3P settings have complete crap in them, rendering the entire exercise worthless. However, I have this retarded perfectionist gene which make me wade through all that stuff to Do It Properly.

Only to discover that IE7 *doesn't care* that I did all that work, because the only part of the whole thing it'll read, is the "compact" form, and the only place it'll read it is in the HTTP headers, which means the really spectacularly geeky-cool trick I played with javascript redirectors in order to set cookies on the amazon s3 service is completely useless for IE7 users.

So in the end I had to rip all of that stuff out so that we could serve the "button" iframe from our primary server instead, just so we could set an HTTP header, just so Microsoft could pat themselves on the back for protecting "the users" from..nothing at all.

Anyway, in the final analysis it shouldn't change things much. Our ability to handle future traffic has probably just dropped from millions of blogs to hundreds of thousands or something like that but you won't see me losing sleep over it any time soon, and it won't slow any user sites down since the latency (if there's any) is contained within the drop button frame and thus won't delay the page load at all.

Oh the pain. Time to go play some more supreme commander.

Modified widget

Ok, the widget code has been modified (new? improved?). We were having some layout issues with the way we were doing it before, and a couple of people with Nortons AV firewall and adzapper were ending up with partially non-functional widgets (completely black/missing cards etc).

So, new version is out. We've checked a good sample of the existing blogs with the new version and it seems to all be ok. Something to note, the previous version auto-centered. This caused some problems in places so we've reverted that. If you want to center the widget, just stick it in some center tags.

Please hit the feedback box if you see anything odd, we are working hard to make sure the widget stuff is as compatible as we can. We've already solved a couple of real head-scratchers so I'm up to solving anything else you guys might run into :)

Rejection :(

One of the more interesting problems we've had is the Reject button. I mentioned earlier that I might possibly have chosen the wrong word for it, rejecting an advertiser is probably more often to do with making space for other things than it is a direct rejection of the user themselves.

That said, the actual fact is that we don't know. The only thing we do know is that just over 1 in 5 advert requests get rejected. And people were getting a bit unhappy about that, and fair enough.

So, we've added a "reject reason" feature. When you go to reject an ad, you can enter a short explanation which will be relayed to the applicant. Normally I would expect this to be used to take the sting out of it a bit, ie "I'm making space for a friend who wants to advertise tomorrow". However, I guess it's entirely possible that it will also be used to *add* sting. We'll be keeping track of the reasons why people are rejecting, and hopefully we'll be able to address some of the more common complaints before it really becomes a problem.

I suspect that a few of the rejections are basically "You're not in any way related to my blog, why are you even applying?". It'll be interesting to see - watch this space.

Feedback, as always, welcome.

Random thoughts

Well the price hike seems to have gone without a hitch..probably didn't hurt that we dropped shiny features in there at the same time. note to self :)

The Bug We Won't Mention not withstanding, everything seems to be moving fairly smoothly. I was only out 2 credits overnight, nothing too serious.

Quite a few new users joining up today. The rate of joining appears to be smoothly climbing upwards. Quick aside:

If you ever find yourself in our seat trying to start something like this up, you'll find yourself in a rather bizarre position as far as signups go. The whole system really depends on picking up sufficient participants that it becomes self-sustaining, as do many apps of this nature. That's fine, you know that when you go into it.

What you don't know is:

1. You don't realise you have *no idea* what number "self-sustaining" actually is. It's not like a business where it's basically $x/month or you're sunk, we're trying to predict "motivation", "curiosity", "novelty" and a vast array of other intangible variables in judging what precisely will work. It's just too early to tell.

2. You desperately want lots of people. Every "shiny new" application has a "honeymoon period", that is a period during which users are patient with faults and generally enthusiastic about the concept. We're definitely in that period, people have been saying positive things everywhere we're looking. What we don't know is how long our honeymoon is, and what will happen when it's over - will we still look good? we're racing to bring in features we know should be there in the hopes that as people settle down to us as simply part of their lives, that we'll still be valuable enough to hang out with.

3. You not entirely sure you really want lots of people. We engineered from the beginning to deal with popular blogs, our use of services like Amazon S3, nettica and other high availability, high performance services are a direct result of our understanding that nothing we do should ever, *ever* slow down our users sites. Many of our users make their livelihood from their blogs, there's no chance we're going to do anything that risks that for them.

At the same time, there's this nervous sense that it's simply not possible to predict what 10,000 people might do. We're not anywhere near that yet, but I've worked on popular systems before, I know exactly what an exponential curve looks like, and they're sneaky like a ninja.

I very much doubt it'll ever cause problems with the widgets, those things have been optimised to hell, but the more dynamic parts of the system like the statistics and analysis stuff...well, it doesn't keep me awake at night but I can't help but be conscious of it whenever our user numbers jump suddenly :)

Oh dear. Price calculation bug

Ok, so, we had a somewhat embarrassing bug. In technical terms, the price you were paying for an ad-spot was actually based on your own spot price, not that of the person you were buying it from.

The discovery of this bug happened while I was adding the transaction list noted in a previous post, and in concert Graham had also noticed a symptom of this bug earlier today while buying some ads on the site.

I was left facing a somewhat difficult situation. I had fixed the bug, but some users were left out of pocket as a result. I knew who they were and with the information available I could make amends (somewhat, it wasn't possible to perfectly refund the difference but it was possible to get reasonably close).

However, I wasn't entirely sure I wanted to tell anyone about it. it was terribly embarrassing and not the kind of thing that inspires confidence in a "financial" system, even one as shiny and new as ours.

After some reflection, I decided that we weren't going to start building a good relationship with our users by lying to them. It wasn't *likely* that anyone was going to notice the bug as a result of looking at the transaction log, but it was possible, and besides - there's principle involved.

I constantly tell my clients that honesty is key to building a good user relationship and I believe it. It was remarkably difficult to convince myself to translate that belief into reality though.

Anyway, instead of ignoring it, I figured out how to generate the appropriate refunds and pushed them into the transaction system, and then sat down to write this post.

It's plausible that people will read this and think "I can't trust these jokers because of this", but I'd like to think (perhaps I'm naive) that many more will come to trust our work more because we're willing to admit our mistakes and make amends as best we can.

New..err..feature?

Entrecard will now reward anyone who manages to generate an exception by taking a credit from my account (if I have any) and giving it to the bug finder (maximum of 1 find per day). If you see a 1 credit reward entitled "Bug bounty" in your transactions list, that was my credit :(

I'm doomed to be credit-poor :(

Entrecard changes

Well, we dropped in some changes today.

The one that everyone will notice immediately is that the spot prices have all doubled. We considered more drastic moves but our current analysis of the spending trends suggests 2x looks like about right for the economy as it stands.

For those people who have been sitting on their credits, we're sorry about the sudden devaluation, however it's not yet a stable currency so if you see a spot you'd like, my current advice is don't wait around :)

Please note that the tax rate as been increased so that sellers still receive the same amount as they did before after the buy. In other words, it hasn't got any easier to get credits (would have been somewhat self-defeating).

Other changes, some improvements on the dashboard, including a transaction history (someone requested it, and it wasn't much trouble. I hope to bring more detailed information into it at a later date), and a more advanced layout for the "Your ads" stack which helps you see when it'll show.

We also have an experimental "advertising history" on the details display page. This shows your previous adverts on the spot, what you paid, and what the result was (ie, if you were rejected). This helps you figure out where you've already displayed and whether it's worth applying again.

Feedback welcome, even if it's a line of burning torches and placards titled "Down with price rises!".

Also, we had a bug with datetime updates which means dates earlier than today on the transaction list are kinda busted. Oh well, life goes on :/

Bug fixed for people having trouble editing their entrecard, unicode error got by me, all fixed.

Entrecard economy

So. Economy. As suggested, we're going to have to make some changes. There's just too much credit around, slots are filling up fast. It seems that people really quite like dropping cards - we've recorded 2,119 card drops so far, and each one puts 1 credit in the dropper and 1 in the dropped user, resulting in..a lot of credits floating around.

We have in excess of 10,000 hours of advertising loaded currently, and that isn't dropping, with 17 spots currently full.

Some other interesting(?) statistics:

Type	Max/user	Avg/user	Total (all users)
Earned	227	54	5050
Spent	228	24	2286
Holding	199	30	2764

In essence this means that there's a bit too much credit lying around, and not enough to spend it on. Obviously as more and more people join up the amount of space that can be advertised on will continue to increase, so in a sense we're trying to predict a moving target in terms of precisely what the appropriate income vs outgoing is.

We have plans to introduce additional things to spend credit on, but we're also looking at increasing the cost per ad slot with less return. Ie, we could boost the ad spot cost by a factor of 4 (so a starting slot is 4 credits) but the owner of the slot only gets 1 credit for the ad (75% tax rate! :).

We're not certain yet exactly what we're going to do. Feedback welcome, as are requests for additional aggregate data.

Entrecard..nerves!

Well, after 3 weeks of crazy work, Entrecard is going live at 7am. Well, technically we're already live but I'm still fiddling around a bit.

Bit nervous, not really worried about the systems etc, but the whole system is so very different from a regular web application that it's hard to tell where things will end up.

There were a number of fascinating things about this project. For a start, we had these chunky 125x125 things flying around everywhere, and we needed a good way to be able to work with them from the dashboard. The dashboard concept of 4 vertical columns was my solution to this, since the chunks were time-related - less interesting stuff goes *down* and eventually falls off the end of the page, so you only really end up scrolling if you're bored.

Other areas included about 3 attempts at determining how the advertising system worked. Previous to the queue system we've got now we had a day-by-day concept which eventually suffered ideological collapse under the weight of timezones and the interfaces necessary to permit day selection etc.

And, of course, there's Amazon S3. While most of dev we were using a local version of the card management, it was clear that the last thing we wanted to do was screw with our customers' blogs. It was utterly vital that the widgets appearing on the blogs were always available, and plenty fast regardless of load - so, S3.

Pulling this off, given the widget is dynamic, not static, took a fair bit of doing. I'm pretty proud of the results however - clever javascript FTW.

Mostly, the crazy thing was that for much of the project we were just making it up. Graham had this idea, but in its original form it was just this concept..we had no pre-conceptions, we weren't copying anyone else, we basically had to try and discard idea after idea - sometimes as late as an hour or two ago when I redid the front page entirely - in an attempt to find methods that worked well.

On top of that part of the work of course, there were deadlines (blogger conference in a few hours), video people needed source material (feel free to play Spot the Error in the video), we had to build the hosting system as well from the ground up (right up to deciding where to host), I believe we had people working on this thing around the globe, from Serbia to New Zealand, with no contact except the internet. It was great fun, really challenging and..now..nerves-inducing.

So, err, go sign up and get a card eh? admire my vertical inbox. Oh, and the charts, click on "Statistics", pretty charts. Lemme know if you have any ideas, or just dump 'em in the feedback box on the dashboard.

Oh, one last thing, we have discovered the truly spookiest firefox rendering bug you have ever seen. I have to go try and make that go away now, when I can only make it happen one in every 10 or so page loads.