So much to do!

In general in software development, things go roughly according to this formula:

tight timelines x level of innovation = level of crappy architecture

The basic issue is that the faster you have to build something, the less time you have to "do it properly" and the more inclined you are to take shortcuts. Similarly, the more "innovative" or - in other words - off the edge of the map your application is the less chance you have of picking the right architectural elements to support what it will eventually become.

Thus when the two combine you normally end up with architecture that makes a dogs breakfast sound well organised.

That said, I'm delighted to say it wasn't what happened with Entrecard. I'd like to say it was all hard work and skill, and to be fair I'm going to take some credit because it's my blog and I'm vain like that. But the fact is there's a fair chunk of luck involved as well.

Unfortunately, the natural consequence of this is that we now have a nice solid extensible architecture with no actual chronic outstanding issues. And we have Ideas. So Many Ideas. Now I have a feature-list a million miles long, all of which are awesome and I really really want to have them in Right Now Damnit.

Ah well. there are worse fates than having a bunch of cool stuff to write for an appreciative audience :)

Forums, new dashboard

ok, we have some experimental forums live. I know I know I reinvented this wheel and it's somewhat octagonal and has no suspension, however it does integrate well with our main application and we have plans to integrate other aspects further as well.

It's not really central to the business plan but at the very least we needed somewhere we could get users to engage with each other - the feedback system has been really good but that's very much a 1-to-us thing.

We also have a modified dashboard, including a "buzz" slot where we can push some 1-liner announcements, and I've removed a lot of the stats stuff and put more explanation in on some things. The main reasoning behind the buzz slot is that we're changing things so rapidly that we really need a good way of keeping people in the loop.

The removal of the stats was..well, that's just more appropriate for the statistics section anyway, it was taking up space. I'm not so sure about taking away the balance (it's on every page in the top right anyway), but I couldn't figure out a good way of keeping it in play without it looking out of place, so I took the hint. We can always put it back somewhere :)

Go play on the forum!

50 drop limit raised

In response to feedback and our own concerns, the daily drop limit has been raised to 300. Nobody has hit this yet so we're confident it should be ok. Further feedback welcome.

Drop limit

After a bit of discussion, I convinced Graham that we needed to put in a daily drop limit. This went in today, you cannot drop more than 50 times in one day.

The reasoning is that the traffic that we're generating, we'd like to be "quality" traffic. That is, there's a decent chance that they're willing to read a bit of your site and see if it's something they'd like. This doesn't happen if people are busy rushing from blog to blog in a desperate attempt to get credits, so while we want to reward people for breaking out of their normal routine and going and having a look around, we don't want to particularly reward the "rushers".

We understand, however, that card dropping is *fun*. So we picked a number (50) that seemed reasonable after looking at the average number of drops people did. We're open to changing this, and there's also the option of allowing the drop but simply not giving a credit for it. Leave a comment or enter feedback on your dashboard if you have preferences.

Security

Lets be honest, while I'm sure all of our users are impeccably honest and would never do anything as sneaky as trying to use a script to do the drops for them, it's important that these, and other areas, are properly secured.

Unfortunately, I can't really discuss the measures we use to protect the integrity of the drop system - while security through obscurity is hardly ideal, the principle of defense in depth holds. The idea basically being that you maintain a number of secure layers, information control being one of them, in order to prevent any single flaw from resulting in a complete breach.

I can say, however, that today someone decided to test that out. Needless to say I wouldn't be smiling about it if I hadn't already anticipated the attempt and had countermeasures installed. Suffice to say I'm pretty pleased with how it all worked out :)

Organic click tracking

We haven't been tracking what I (probably incorrectly) term "organic clicks", that is, clicks that go from our details page to your blog as opposed to from the widgets. It wasn't an oversight so much as simply not a priority since most people who really care have analytics or something which tells them how much comes from Entrecard anyway.

However it has reached the point where the information is fundamentally interesting for us, and for those users who are using the Entrecard provided statistics as a measure of how valuable the system is.

As a result, we're now tracking organic clicks from the details page via a redirector. It should make no visible difference to anything, it's simply useful. once we have a few days history in the database we'll add a chart onto the statistics page with the details.

Economy update

For those wondering where the economy is standing right now, basically its..on fire.

As at this posting, the total earned is 35,510 credits roughly. I say roughly because it's tearing upwards so fast the number changes every time I reload the page.

Of those, 12,317 is currently sitting in accounts, 10,243 credits have been invested in adverts currently running or queued, and the remaining 12,950 credits have been removed from the system via tax on adverts that have already run.

Active users who have spent any credits at all spent an average of 59% of their total earnings, which is much better than it was prior to the price double-up.

There is currently 1,120 days (26,880 hours) of advertising queued on 199..err..201..shit..202 widget-displaying blogs, an average of 5.6 days per blog.

We have recorded an epic 13,620 card drops in total. Seriously, people seem to find it rather addictive. I'm a little worried for some of you.

I've had 26 credits removed from my personal account by the bug bounty program :(

We've had a total of 3,359 advert requests, 444 of which were rejected.

That's all I can think of right now, gotta go grab dinner. leave a comment or feedback if you'd like any other specific info (that doesn't invade privacy obviously).

Email notifications

Some people are just too popular :) it turns out that, unlike me, some people are getting piles of advert request emails and it's just filling up their email box and making them sad.

I don't like having sad users, so I whipped up a couple of email settings checkboxes that should let you switch off email notifications for adverts or rejections. it's shiny new so let me know if you have any problems with it, but it should help those people who are struggling under a large enough email load as it is without us adding to it.

You can get to it off the dashboard.

Internet Explorer 7, P3P and Entrecard

You have no idea how angry I am right now. My anger towards Microsoft and IE7 is a towering inferno of geek fury. I can barely restrain my urge to unleash a rant.

I am going to go play computer games until I calm down.

Long story short, I believe I have resolved the issue that prevented IE7 users from being able to drop on widgets. It came at the expense of one area of particularly cool traffic offloading that we had, and at the expense of my sanity. What was left of it.

The essence of the problem was identified by Wavumi earlier. Basically, Internet Explorer went off on this one-browser missing to "make the world safer" by refusing "third party cookies" (that is, cookies being delivered in an iframe, which is how we make the widget work). Except, of course, that doubleclick etc needed 3rd party cookies in order to work, so bowing to commercial pressure they decided to allow them if the website includes a machine-readable privacy policy, called P3P.

Now, at this point everything basically sinks into the sea. You see, the number of people who want to do 3rd party cookie stuff in order to put their forum in an iframe or whatever, is N. The number of people who can be bothered reading the P3P spec is N/100, and the number of people who can fight their way throught he commercial smackfest that is P3P generators, make any kind of reasonable judgement about all the vague terminology in the spec, and actually implement a rational, intelligent P3P file is N/10000.

That means, 9999 out of every 10000 sites with P3P settings have complete crap in them, rendering the entire exercise worthless. However, I have this retarded perfectionist gene which make me wade through all that stuff to Do It Properly.

Only to discover that IE7 *doesn't care* that I did all that work, because the only part of the whole thing it'll read, is the "compact" form, and the only place it'll read it is in the HTTP headers, which means the really spectacularly geeky-cool trick I played with javascript redirectors in order to set cookies on the amazon s3 service is completely useless for IE7 users.

So in the end I had to rip all of that stuff out so that we could serve the "button" iframe from our primary server instead, just so we could set an HTTP header, just so Microsoft could pat themselves on the back for protecting "the users" from..nothing at all.

Anyway, in the final analysis it shouldn't change things much. Our ability to handle future traffic has probably just dropped from millions of blogs to hundreds of thousands or something like that but you won't see me losing sleep over it any time soon, and it won't slow any user sites down since the latency (if there's any) is contained within the drop button frame and thus won't delay the page load at all.

Oh the pain. Time to go play some more supreme commander.

Modified widget

Ok, the widget code has been modified (new? improved?). We were having some layout issues with the way we were doing it before, and a couple of people with Nortons AV firewall and adzapper were ending up with partially non-functional widgets (completely black/missing cards etc).

So, new version is out. We've checked a good sample of the existing blogs with the new version and it seems to all be ok. Something to note, the previous version auto-centered. This caused some problems in places so we've reverted that. If you want to center the widget, just stick it in some center tags.

Please hit the feedback box if you see anything odd, we are working hard to make sure the widget stuff is as compatible as we can. We've already solved a couple of real head-scratchers so I'm up to solving anything else you guys might run into :)

Rejection :(

One of the more interesting problems we've had is the Reject button. I mentioned earlier that I might possibly have chosen the wrong word for it, rejecting an advertiser is probably more often to do with making space for other things than it is a direct rejection of the user themselves.

That said, the actual fact is that we don't know. The only thing we do know is that just over 1 in 5 advert requests get rejected. And people were getting a bit unhappy about that, and fair enough.

So, we've added a "reject reason" feature. When you go to reject an ad, you can enter a short explanation which will be relayed to the applicant. Normally I would expect this to be used to take the sting out of it a bit, ie "I'm making space for a friend who wants to advertise tomorrow". However, I guess it's entirely possible that it will also be used to *add* sting. We'll be keeping track of the reasons why people are rejecting, and hopefully we'll be able to address some of the more common complaints before it really becomes a problem.

I suspect that a few of the rejections are basically "You're not in any way related to my blog, why are you even applying?". It'll be interesting to see - watch this space.

Feedback, as always, welcome.

Letting your users help you help them

One of the big bonuses of an application like Entrecard is the userbase. In general, if you're writing an in-house app for a regular corporate, you're going to have to do all the heavy-lifting of diagnostics etc by yourself. While users in corporates follow the usual bell-curve of skills, corporate environments rarely encourage people to engage in other departments activities, and many of them just have better things to do or don't care about your app that much in the first place.

Apps like Entrecard on the other hand attract a distinct interested group of users, especially in the early days when few people are just "following the crowd". As a consequence we've been lucky to have feedback from some users who have gone to truly extraordinary lengths to help us fix problems.

In one case, Wavumi actually went so far as to give us the solution to a 3rd party cookie problem we ran into with IE7. In another, two of our users have been doing a marvelous job ruling out various browsers, settings and configs for the "missing widget" bug (the widget seems to display only once in about 10 sites, and the card space is blank, if you experience this, please use the feedback form to let us know).

The great thing has been the patience, and willingness to help, that has come from all our users - which makes my life easier and means I can spend more time adding the features that make the experience better, rather than scratching my head trying to figure out really obscure bugs.

So, shout outs etc to everyone helping.

Couple of new things in

Some basic things. The category page was getting a bit stale..people who get popular tend to stay popular, so I added in one for the newbies, a set of tabs, one of which lists the most recent 30 cards to join. And to balance it a bit, top 30 most popular regardless of category.

Also have mouseovers for titles, some people liked to just have an image with no title, hopefully the mouseover will give a bit more detail (doesn't work in the widget, only in browsing).

Pretty chart

We're slowly building up better and better instrumentation on the system as a whole. Thought I'd share one of the charts:



Each bar represents a user, the chart is ordered left to right from lowest total income to highest (the blue "area"), each of the columns is that users total spend.

As you can see, there are a few people who are really having fun at the far right, but the bit which I'm pleased about is that they're by no means the only people engaging. About 30% of the users are over the 100 credit earning mark and while the chart does curve up for the last 20% or so, it's still fairly flat, suggesting that it's not just popular bloggers who have opportunities to gain visibility.

Random thoughts

Well the price hike seems to have gone without a hitch..probably didn't hurt that we dropped shiny features in there at the same time. note to self :)

The Bug We Won't Mention not withstanding, everything seems to be moving fairly smoothly. I was only out 2 credits overnight, nothing too serious.

Quite a few new users joining up today. The rate of joining appears to be smoothly climbing upwards. Quick aside:

If you ever find yourself in our seat trying to start something like this up, you'll find yourself in a rather bizarre position as far as signups go. The whole system really depends on picking up sufficient participants that it becomes self-sustaining, as do many apps of this nature. That's fine, you know that when you go into it.

What you don't know is:

1. You don't realise you have *no idea* what number "self-sustaining" actually is. It's not like a business where it's basically $x/month or you're sunk, we're trying to predict "motivation", "curiosity", "novelty" and a vast array of other intangible variables in judging what precisely will work. It's just too early to tell.

2. You desperately want lots of people. Every "shiny new" application has a "honeymoon period", that is a period during which users are patient with faults and generally enthusiastic about the concept. We're definitely in that period, people have been saying positive things everywhere we're looking. What we don't know is how long our honeymoon is, and what will happen when it's over - will we still look good? we're racing to bring in features we know should be there in the hopes that as people settle down to us as simply part of their lives, that we'll still be valuable enough to hang out with.

3. You not entirely sure you really want lots of people. We engineered from the beginning to deal with popular blogs, our use of services like Amazon S3, nettica and other high availability, high performance services are a direct result of our understanding that nothing we do should ever, *ever* slow down our users sites. Many of our users make their livelihood from their blogs, there's no chance we're going to do anything that risks that for them.

At the same time, there's this nervous sense that it's simply not possible to predict what 10,000 people might do. We're not anywhere near that yet, but I've worked on popular systems before, I know exactly what an exponential curve looks like, and they're sneaky like a ninja.

I very much doubt it'll ever cause problems with the widgets, those things have been optimised to hell, but the more dynamic parts of the system like the statistics and analysis stuff...well, it doesn't keep me awake at night but I can't help but be conscious of it whenever our user numbers jump suddenly :)

Oh dear. Price calculation bug

Ok, so, we had a somewhat embarrassing bug. In technical terms, the price you were paying for an ad-spot was actually based on your own spot price, not that of the person you were buying it from.

The discovery of this bug happened while I was adding the transaction list noted in a previous post, and in concert Graham had also noticed a symptom of this bug earlier today while buying some ads on the site.

I was left facing a somewhat difficult situation. I had fixed the bug, but some users were left out of pocket as a result. I knew who they were and with the information available I could make amends (somewhat, it wasn't possible to perfectly refund the difference but it was possible to get reasonably close).

However, I wasn't entirely sure I wanted to tell anyone about it. it was terribly embarrassing and not the kind of thing that inspires confidence in a "financial" system, even one as shiny and new as ours.

After some reflection, I decided that we weren't going to start building a good relationship with our users by lying to them. It wasn't *likely* that anyone was going to notice the bug as a result of looking at the transaction log, but it was possible, and besides - there's principle involved.

I constantly tell my clients that honesty is key to building a good user relationship and I believe it. It was remarkably difficult to convince myself to translate that belief into reality though.

Anyway, instead of ignoring it, I figured out how to generate the appropriate refunds and pushed them into the transaction system, and then sat down to write this post.

It's plausible that people will read this and think "I can't trust these jokers because of this", but I'd like to think (perhaps I'm naive) that many more will come to trust our work more because we're willing to admit our mistakes and make amends as best we can.

New..err..feature?

Entrecard will now reward anyone who manages to generate an exception by taking a credit from my account (if I have any) and giving it to the bug finder (maximum of 1 find per day). If you see a 1 credit reward entitled "Bug bounty" in your transactions list, that was my credit :(

I'm doomed to be credit-poor :(

Entrecard changes

Well, we dropped in some changes today.

The one that everyone will notice immediately is that the spot prices have all doubled. We considered more drastic moves but our current analysis of the spending trends suggests 2x looks like about right for the economy as it stands.

For those people who have been sitting on their credits, we're sorry about the sudden devaluation, however it's not yet a stable currency so if you see a spot you'd like, my current advice is don't wait around :)

Please note that the tax rate as been increased so that sellers still receive the same amount as they did before after the buy. In other words, it hasn't got any easier to get credits (would have been somewhat self-defeating).

Other changes, some improvements on the dashboard, including a transaction history (someone requested it, and it wasn't much trouble. I hope to bring more detailed information into it at a later date), and a more advanced layout for the "Your ads" stack which helps you see when it'll show.

We also have an experimental "advertising history" on the details display page. This shows your previous adverts on the spot, what you paid, and what the result was (ie, if you were rejected). This helps you figure out where you've already displayed and whether it's worth applying again.

Feedback welcome, even if it's a line of burning torches and placards titled "Down with price rises!".

Also, we had a bug with datetime updates which means dates earlier than today on the transaction list are kinda busted. Oh well, life goes on :/

Bug fixed for people having trouble editing their entrecard, unicode error got by me, all fixed.

Entrecard economy

So. Economy. As suggested, we're going to have to make some changes. There's just too much credit around, slots are filling up fast. It seems that people really quite like dropping cards - we've recorded 2,119 card drops so far, and each one puts 1 credit in the dropper and 1 in the dropped user, resulting in..a lot of credits floating around.

We have in excess of 10,000 hours of advertising loaded currently, and that isn't dropping, with 17 spots currently full.

Some other interesting(?) statistics:

Type	Max/user	Avg/user	Total (all users)
Earned	227	54	5050
Spent	228	24	2286
Holding	199	30	2764

In essence this means that there's a bit too much credit lying around, and not enough to spend it on. Obviously as more and more people join up the amount of space that can be advertised on will continue to increase, so in a sense we're trying to predict a moving target in terms of precisely what the appropriate income vs outgoing is.

We have plans to introduce additional things to spend credit on, but we're also looking at increasing the cost per ad slot with less return. Ie, we could boost the ad spot cost by a factor of 4 (so a starting slot is 4 credits) but the owner of the slot only gets 1 credit for the ad (75% tax rate! :).

We're not certain yet exactly what we're going to do. Feedback welcome, as are requests for additional aggregate data.

Widget scanning

Well, I added widget scanning on Entrecard today. Basically it's a system that verifies that the user is actually displaying the widget. It was fun to code, but it was a bit of a shame it became necessary.

At first I thought we could get away with something that simply gently reminded the user that they didn't have the widget up but it seems that people didn't really respond to that.

The main reason why it's a problem is that without the widget on display, other people might still spend credits to get an ad on your widget..and get nothing for it.

So, scanner. All it does is prevent users with no widget on their site from showing up in the category browser, so nobody will accidentally run an ad on them. Hopefully the interface is fairly friendly and you should be able to restore category visibility within about 2 minutes of putting the widget back up.

It'll be interesting to see how it affects our widget stats.

It's also interesting to see that the economy may be a little..liquid..ie, people who've been working at it even a little bit have plenty of credits and advertising slots are going like crazy, a few of the more popular people are maxed out already. We have some ideas about how to deal with that - will be looking into it next week. Thanks to everyone who has dropped on the widget, nice to see you all :)

Life, Entrecard, etc

Well, life has been getting pretty crazy recently. In order of importance:

My grandfather died yesterday night (14th NZST), we had all rushed up there to see him after the hospital rang to say he wasn't good. Thankfully it was peaceful and everyone who could get there was there. The end of a long struggle with dementia, then lithium poisoning and a long list of subsequent complications. Thankyou to all those who conveyed their condolences.

They're building a great big bloody 15 story building outside my window, probably starting Jan. Busy trying to see if we can get out of our lease but to be honest I'm doing that a bit half-heartedly, the fact is I have so much on at the moment that adding "moving" on the top of it just doesn't feel great.

Entrecard appears to be taking off, with lots of people joining, and lots of people finding new and interesting bugs. In order of spookiness:

• Firefox 2 (only) will occasionally spaz out and recompute the tag order for an entrecard on the site, causing a long bizarre render. We have (using firebug and a few other tricks) determined that there's a bug in the actual browser itself, rarely triggered, which does odd things with the DOM tree within that situation. We don't know what's causing it, I thought i had a solution the other day but it just seems to come and go randomly. Spooky rating: 11/10


• IE6/7 shows all the widgets the same on the get code page. Really bizarre, it appears to fail to pass the right params or something to the subsequent script tags. Spooky rating: 7/10


• PNG has an interlace mode (who knew?), and the Python Imaging Lib can't load it (bet you didn't know that), so when we were saying "PNGs are fine" and then trying to process them, it was screwing up and giving people our Serious Error page. A different approach for interlace PNGs has been implemented. Spooky rating: 4/10


• Disappearing card images are happening for some users. As far as I can tell, this is related to the fact that the validator doesn't do quite enough checking to verify an image has been uploaded before it lets a post succeed and I think that has bitten a few people. The primary reason is the following scenario:
  
  
  
  1. User fills out form mostly correctly, including image to upload
  
  
  2. user submits, hits a validation error with, say, passwords not matching
  
  
  3. Firefox helpfully (for security reasons! because all websites are terrorists!) allows every field to be pre-filled *except* for the file upload field.
  
  
  4. User doesn't notice that the file field got blanked, corrects their validation error and hits submit, only to end up with an empty image
  
  
  
  Insanely, this scenario was *never encountered* in our whole time testing. We have some better validation now although it is still possible to cause a blank image under another scenario I'm trying to clean up as well. If this situation bit you, just go to settings and upload your image there.
  

On the plus side, the scalable architecture is totally, utterly kicking butt. This is our server, delivering widgets on 60+ blogs, some really quite popular:

Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

I'd say we have some room there :)

Entrecard..nerves!

Well, after 3 weeks of crazy work, Entrecard is going live at 7am. Well, technically we're already live but I'm still fiddling around a bit.

Bit nervous, not really worried about the systems etc, but the whole system is so very different from a regular web application that it's hard to tell where things will end up.

There were a number of fascinating things about this project. For a start, we had these chunky 125x125 things flying around everywhere, and we needed a good way to be able to work with them from the dashboard. The dashboard concept of 4 vertical columns was my solution to this, since the chunks were time-related - less interesting stuff goes *down* and eventually falls off the end of the page, so you only really end up scrolling if you're bored.

Other areas included about 3 attempts at determining how the advertising system worked. Previous to the queue system we've got now we had a day-by-day concept which eventually suffered ideological collapse under the weight of timezones and the interfaces necessary to permit day selection etc.

And, of course, there's Amazon S3. While most of dev we were using a local version of the card management, it was clear that the last thing we wanted to do was screw with our customers' blogs. It was utterly vital that the widgets appearing on the blogs were always available, and plenty fast regardless of load - so, S3.

Pulling this off, given the widget is dynamic, not static, took a fair bit of doing. I'm pretty proud of the results however - clever javascript FTW.

Mostly, the crazy thing was that for much of the project we were just making it up. Graham had this idea, but in its original form it was just this concept..we had no pre-conceptions, we weren't copying anyone else, we basically had to try and discard idea after idea - sometimes as late as an hour or two ago when I redid the front page entirely - in an attempt to find methods that worked well.

On top of that part of the work of course, there were deadlines (blogger conference in a few hours), video people needed source material (feel free to play Spot the Error in the video), we had to build the hosting system as well from the ground up (right up to deciding where to host), I believe we had people working on this thing around the globe, from Serbia to New Zealand, with no contact except the internet. It was great fun, really challenging and..now..nerves-inducing.

So, err, go sign up and get a card eh? admire my vertical inbox. Oh, and the charts, click on "Statistics", pretty charts. Lemme know if you have any ideas, or just dump 'em in the feedback box on the dashboard.

Oh, one last thing, we have discovered the truly spookiest firefox rendering bug you have ever seen. I have to go try and make that go away now, when I can only make it happen one in every 10 or so page loads.