OlinData are awesome

I received an unsolicited email from Walter Heck and his team at OlinData. Before I’d even read it, I had another email apologising for the mistake. Then I actually got a second mistaken email, but at this point, I’m so grateful for the apology and the responsible attitude, I don’t much care. To make matters even better, Walter posted on their blog explaining the mistake.

I decided in contrast to previous rants, it was time for a praising post. Thanks Walter for your responsible email behaviour, I salute you sir.

Processing incoming emails

I’ve been looking into how to programatically process incoming emails. For example, to create an email address where somebody can send a CSV file and then have that data parsed and automatically inserted into a database.

There are some interesting tools in this space. The easiest, at least in principle, appears to be Email Yak. They expose a JSON API which will trigger a POST or GET request for incoming messages, and can send message likewise. However, upon signup I got a 500 error and likewise after logging in. So I can’t currently test the service. In principle though, it looks interesting. 500 emails a month on their domain for free, 1000 on any domain for $5/month. Then it kicks up to $40 or $150 for 20k or 100k emails.

Another interesting tool is Context.IO. This is essentially a web friendly API in front of IMAP mailboxes. Their pricing model is also interesting, starting at $1.50 per mailbox per month, with a $15 minimum. Currently the service is read-only, but the option to move messages around is coming in api v2. They also have a free account which includes 3 mailboxes, and they charge 85c/GB for attachment transfer over 100MB.

This is really about extracting knowledge from email inboxes and focuses Google’s mail hosting (gmail/google apps), but will work with others. There seems to be a strong focus on attachments and conversations. Could be a useful component in quickly building another service, but I’d guess I’d want to build my own version eventually.

Google App Engine provides a mechanism to handle incoming emails and pipe them to a script. Sounds very sensible, and it would probably be possible to build a mail routing system on top of this by having the python script send the mail onwards via an API call or POST request.

I also read a few articles about having postfix send mail to a script. This one is useful. This article talks about configuring custom reply-to addresses to know which emails bounce, something called VERP apparently.


It seems to me like Yak Mail (I’ve had a reply from them about the 500 error while typing this blog post, impressed!) and Context.IO are useful pieces. Google’s mail API is also smart, and Amazon will probably add something similar to AWS before too long. If this was core to an application, I’d probably set up postfix to forward mail to a script eventually. But in the early days, I think Yak Mail is probably the way to go.


In my roundup, appeared to have completely missed the best offering of all, mailgun. Free tier includes 200 messages a day on a shared IP, pro tier is $19 minimum per month on a shared ip, $59 minimum on a dedicated IP. APIs to send and receive mail, mailboxes accessible via IMAP and POP3, charges only for storage and message counts, not mailoxes. Plus open source helper libraries available. Looks like the slickest of the lot.

Free creates social noise

The proliferation of “free” is creating a society in which we are exposed to ever more content. It may seem at a macro level like just more crap on YouTube, but at a micro level it’s more crap our friends are creating, sending, forwarding, spamming. I’m calling this social noise. Crap we are exposed to by our friends.

In exploring this topic, let me clarify some of the terms I’ll use.

Social utility. I’ll talk about lots of social utilities, a term borrowed from Facebook’s self description. I consider a social utility to be any technology that allows people to connect. That’s very broad. It includes the telephone, email, blogs, blog comments, facebook, twitter, and a myriad of others.

Free. What do I mean by free? When I talk about free, I mean generally free to use. Email is largely free because it costs me virtually nothing to send you an email. My typing time is actually the most valuable component of the email, so a single email is not really “free”. But when I email 100 of my friends at once, then it becomes free because I typed once, spammed a hundred times.

Noise. When a Facebook friend sends me an invite to an event on another continent, that’s noise. It was free to send me the invite yet it’s totally unrelated to me. Noise. I chose the word noise because of the signal to noise concept. How much stuff do I get through a social utility which I don’t want versus the stuff I do want. That’s signal to noise. Noise is what I don’t want.

Why does free equal noise?

This is a fascinating question. Why does a friend send me an invite to an event in Edinburgh, Scotland even though I’m in Vancouver, Canada? Why do people post rubbish on Twitter that nobody wants to read? Why do friends email me crap that I have no interest in?

Free. It costs them nothing. If I can easily and freely send rubbish to my friends, I will, simply because I can. It’s human nature. Email started out as useful. Imagine sending a message to the other side of the world, for free, instantly. Amazing. As time goes on, that freeness is the very aspect of email that destroys its usefulness. Estimates suggest that more than 90% of all email sent today is spam. Noise.

Now email has a cost. Not to send, but to receive. There is a cost in sorting 100 messages received to find the 3 you actually want to read. We employ spam and virus filtering in a bid to make the task easier. But by automated means, or otherwise, email now has a cost. It is not free to receive email. One must spend resource distinguishing between what we want to receive and what we do not.

Facebook is the same way. At the time of writing, I have some 409 friends on Facebook. That means any one of 409 people can generate content which will flow to me. When I first joined Facebook, I had few friends, there was little cost. I could log in, see what was happening with my friends quickly, move on. Now I need to wade through 409 people’s garbage to find anything useful.

Two Facebook friends recently got engaged. This was posted on their pages as a message. Amongst the diatribe of other nonsense on Facebook I missed that message. Instead I was reading about hair cuts, parties, weather and other useless crap. The noise drowned out the signal.

Now Facebook has a cost. I need to construct friend lists and filters to separate the signal from the noise, the wheat from the chaff. Free to send, it costs to receive. Much like email before it and no doubt Twitter after it. Free creates noise.

What you resist, you strengthen / Change / Resistance / Conclusions

Before I continue, let me state a few assumptions.

You cannot stop progress. Progress currently means more noise. There’s no stopping it. If you try to resist progress, only you will suffer as a result. This is a principle of life. To nerdily quote the borg, resistance is futile.

As the cost of producing content decreases, the volume of content increases. When it’s free to upload videos to YouTube, millions of people do so. As the cost of video editing, photography and other forms of creation comes down, more and more people create.

As an aside, I fully support this democratisation of creation. Everybody becomes an artist, a source of media. Let the masses create I say. Fill YouTube with a trillion videos of teenage angst, crappy comedy and boring music. Let the content flow.

A torrential storm of content is breaking. This is just the beginning. Resisting is futile.

What are we doing? / What do we do about it? / Response / Result

What does it mean? Where will it all go? Will it ever stop? Great questions. Most likely unanswerable except in the fullness of time. Any answers are predictions at best.

How are people responding? This is a question we can answer today. The trend seems to be filtering, sorting, processing. Automated means of distinguishing what we want to see from what we do not want to see. As purely anecdotal evidence, I’ll quote some examples.

Almost every email service now employs spam filtering and virus filtering. These are automated methods to try and separate content from noise. Blog comments are similarly processed. This automated spam filtering is big business in itself.
Guy Kawasaki talks about how to use twitter as a twool [sic]. He says “Get as many followers as you can.” and then “Monitor what people are saying about you”. If you’re following 20’000 people on Twitter it’s impossible to read all those tweets. Instead the advice goes, be selective in what you read, filter.

The Huffington Post apparently employs at least two comment moderators. These are real human people whose job is presumably to read posted comments and sort the content from the noise. Real people employed as human noise filters.

Seth Godin is notorious for taking the opposite approach, he doesn’t take comments on his blog. He says he made the decision because the cost (responding, curating) is too high.

One way or another, the rise of freeness creates a cost. In the age of free content, it costs not to produce content but to consume it. The cost is borne by the recipients, the readers, the moderators, not by the senders, the writers, the creators.

What can we do?

Given the current landscape of freeness and noise, what can we do about it? I’d like to get specific, so here are the challenges I personally face.

  • I get too much email, most of it I don’t want.
  • It’s hard, if not impossible, to get useful information out of Facebook.
  • Reading Twitter is of virtually no benefit to me.

In addition, the subject of blog comments is close to my heart. As I embark on a journey of creating content and sharing it online, I hope to soon have a noise problem.

As I see it, responses to these challenges are all about cost. There is inherent cost in communication. Free communication moves the cost to the recipient. Non free communication shares the cost between sender and recipient. As I see it, these are the two ways of responding. We can either charge the sender and therefore split the cost, or create systems as recipients to better filter content.

In this next section I will outline some ideas to meet these challenges. These ideas might be copied from existing sources, downright wacky, utterly unconventional and potentially complete nonsense. I leave those judgements up to you.

Professional or Personal Assistant

I believe that most companies are filled with too many managers and not nearly enough assistants. I see, time and time again, managers doing tasks which could easily and practically be delegated to assistants.

In this vein, I am considering hiring a personal assistant to perform a few functions. The first will be read all of my email. Personal and otherwise. Read it all, deal with the noise based on a predefined set of rules, and then give me only the content I want. It will take some time and effort to perfect this system, but once in place, I think the payoff will likely be worth it.

I’m also considering what other social utilities I can have filtered by a real person. Could I have my assistant log into Facebook regularly and write a summary of the most significant activity? Would this be cost effective? Can I do the same with Twitter?

This raises other questions. How much training does a personal assistant need? How much of the value of this setup will be contained within the person’s head and how much in the rules that we establish? Lots to consider, this option is an involved one.

Pay per comment

I have considered charging for blog comments. For $20 you can comment on 20 blog posts. I think that if a reader was to spend $1 to respond to a blog post, they would think more carefully before writing.

This raises additional challenges. I believe in offering those who cannot pay a non monetary alternative. That in itself is challenging. Could users also choose to answer 12 (or 25 or 50) captchas instead of paying $1 to post a comment? Some other time consuming task?

How about allowing users to post a selection of pre-defined messages for free. So a simple “Thank you” comment is free, but a comment that requires a follow up has a cost. This might also allow for visual separation of comments. All the “Thank you” comments could be grouped, listing only the names of the people who said “Thank you”.

Automated trust / Feedback / Trust

Crowdsource the cost. As content is posted, allow users to “rate” the content. As positive ratings flow, the content is highlighted. As negative ratings flow the content is hidden.

This model could be applied to many areas from blog comments to Facebook status updates to Twitter tweets. It’s possible to use global or local models. In a global model all positive / negative feedbacks are grouped. In a local model only my friends and friends of my friends feedback matters to me.

There is an inherent challenge in this. It requires users to see some raw output in order to split that output into content and noise. This leaves an opening for spam, even if fewer people see the spam because it is quickly hidden, the cost to the sender is free, so the economy of scale will allow spam to continue.


I’m seriously considering resigning from Twitter. I gain almost no value whatsoever from my Twitter account. It’s virtually worthless, yet it consumes my time. Facebook is a less obvious example. I gain some value from Facebook, but primarily only from direct, personal content. When a friend sends me a message, replies to my status, these are messages I want to see. When somebody spams me about an event on another continent, that’s noise.

I could try to reduce my Facebook noise by ignoring all invitations and all group messages. I don’t think Facebook currently allows me to do this automatically, so it may require some work on my part. Perhaps this ties into the Professional / Personal Assistant option.


There is a site dedicated to spreading the word about BCC. Send emails to multiple people, put them in BCC not TO. You could consider this to be a campaign. Try to educate people on polite, sensible use of the new technologies available to us. Unfortunately, these campaigns are largely ineffective. They face a couple of challenges.

Firstly, only true friends actually care. The people who send commercial spam really don’t care. They know what they’re doing, they’re doing it anyway, so trying to urge them to do otherwise is not likely to be effective. At least not in a gentle campaign type way.

Secondly, these campaigns take such a long time and require so much effort, the problem has likely moved to a new social utility before any real impact is felt. Personally, I see fewer and fewer emails where a hundred people are in the TO field. Mostly, people in my circles are learning to use BCC. However, that does not appear to have any effect on the level of Facebook crap I receive. So the BCC campaign may be successful, but it doesn’t seem to help the next place the same problem arises.

Having said all that, I think campaigns are the ultimate, long term solution. Only through education can we change habits. In time, as people become increasingly comfortable with new social utilities (not some new service, but the idea that there are forever “new” utilities), the norms of politeness and manners will adapt. But we’ll all have drowned in a quagmire of crap by then unless we find solutions in the mean time.

Google Wave

This might be the most exciting technological development since email. I’m truly impressed at Google’s approach to this project. It gives me newfound faith in Google.

The guys behind Google Maps set out to answer the question “What would email look like if it were invented today?”

Their answer is truly outstanding. Wave is a collaborative communication tool. Something like email crossed with a wiki, instant messaging client, and much, much more. As I watched the video I was thinking, all well and good, but when I got to around 1 hour 8 minutes, I got really excited. In a truly genius move, Google has made the whole protocol behind this new platform open source. That allows independent organisations to build their own Wave servers, and privacy is tightly coded within the system. No Google snooping. Wow.

If you’re technically minded, watch the video here. I’m not embedding the video because it’s 1 hour 20 minutes long and you probably want to watch it in high def on YouTube directly. See more info and sign up for a demo account on Google Wave here.

I was clapping with the audience as the video ends. Truly amazing. Thanks to Pete Mall for the tip. 🙂

FuseMail is Go

After my bitching about FuseMail, two of their guys chimed in and saved the day. My faith in FuseMail was restored. I’ve gone ahead and moved my personal email over to them. It’s live. 🙂

A few things I like:

  • Mail is filtered on the server. I no longer have to download spam if I don’t want to.
  • Virus checking and spam filtering are automatic, I don’t have to worry about it.
  • My mail should be relatively safe, automatic backups, etc.

Thanks to Pat and Henry for responding to my blog post and dealing with the issues. I feel reassured.