I have a signal overload problem
I have a signal overload problem.
At Filtrbox, we spent the better part of the last three years solving the information overload problem by separating the signal from the noise. With that problem yet to be completely solved, I am already dead smack in the middle of a signal overload problem.
Needless to say, I am used to a great signal because I developed and used Filtrbox for my content filtering. However, recently my “Folksabox” (human version of Filtrbox powered by folks) , which is Twitter, has also started to produce ever-increasing large quantities of quality signal that is absolutely overwhelming and is now almost impossible to keep up with. I credit my signal overload to the quality of content shared by people I follow on Twitter. The majority of the people I follow are sharing thoughts, opinions and notions that they have put a lot of thought into, they are also sharing very valuable data/content (this may not reflect everyone’s experience on Twitter, I tend to follow people and organizations on Twitter who have an inherent understanding and appreciation of its value as an information dissemination/opinion sharing tool).
Too much signal has become a problem for me. As I write this post, I lament as I watch a steady stream of great content that I would love to read but I know is going to go unread because there is just too much of it and I don’t have enough time in the day to do so. On a good day, I manage to read just 1% of the total amount of content that I would love to read. For me, this is creating an information poverty of a different kind. I am not well informed despite too much good information (try to figure that one out). Given the increase in the zettabytes of content being generated, this problem is bound to get worse – this is just the beginning.
How do we solve the problem of signal overload on our way to becoming SWIFs (super well-informed information freaks)? Until we, ordinary humans, have the means and capacity to absorb large amounts of data and information in a short time, we’ll settle for good ‘ol algorithmic solutions. The solution lies not in filtering out signal further to get another uber-signal. The solution lies in the extraction of important data, facts and opinion from the signal in order to reduce the amount of information so that it can be human-consumable within a reasonable amount of time. Today, there are a few products that perform content summarization but like other NLP-related products (e.g. sentiment), none of them do it well and there is still a long way to go. Given that we are still a long ways, in my opinion, this one of the major “whats next” areas in information processing (are you listening future Techstars 2011 applicant). I hope we can start working on signal overload solutions soon because the problem is bound to get worse.
I agree with your sentiments that you and many others, myself included, are experiencing an information poverty. However, separating signal from noise or intelligent knowledge representation still is tricky giving the level of noise. I think an artificial system would have to have a basis in order to achieve separating signal from noise. For example, a forum given a topic as the basis could index knowledge of the topic and use formalized reasoning to organize posts by importance.
You are right, there are still some challenges separating the signal from the noise. That is what we were doing at Filtrbox and continue to work on it at Jive. However, the point that I am making is that tools like Twitter have helped me separate the signal and now my problem is too much signal, that is, signal overload. Some of the techniques for signal separation can be applied to the signal overload problem, however in my opinion the first step is NLP-based summarization tools that summarize the signal for me.
NLP summarization would be an interesting approach to solving the signal overload you speak of, and I agree that it may be a first step. I’ll have to research it more to get a better context of the practical conveniency. But after further reading of NLP, I agree that this could be the “whats next” area in information processing. From your perspective what are some other examples, beside twitter, that have or could have potent signals?
The interesting thing about Twitter is that it has become a live index of all the other “potent signals”. By this, I mean that when content is published on a blog, a video site or in mainstream media for example it is echoed on Twitter. This means that by merely looking at my Twitter stream, I have an index that acts as a reference me the original content. I could go to those sources directly but then I would have to self filter the content whereas using Twitter, my Twitter social graph filters the content for me and my signal is comprehensively potent.