Thursday, February 24, 2005

Way too much data

Let's say your aggregator gets 4000 items every day. Then, on average, you cannot spend more than 21.6 seconds per item.

It's impossible to read all that, much less sample it properly. Developing tools to make that mass manageable is a solution as useful as driving ten cars to work to shorten your commute time.

Occam's razor, anyone?


ade said...

It all depends on what you mean by "managing that data." I happen to mean that a good tool is not one which makes it easy or possible for you to read all 4000 but one which gives you an easy way to identify the 4 entries you ought to read.

The entire rationale for building tools to handle that kind of data-flood is precisely because human's can neither sample nor reason about that much data on a daily basis. Tools like Aggrevator, Findor, etc are trying to follow in the footsteps of Doug Engelbart's work on intelligence augmentation and they're as misunderstood as Engelbart's work.

Andres said...

My point was more along the lines of "if everybody is talking, nobody listens".

In my opinion, nothing can determine usefulness like our brain. I would not relinquish my ability to determine what is useful to any program or technique that I am familiar with.

So far, I am concerned that what we will accomplish is to create an automatic fashion trend spotter. Indeed: how could such a program determine what is utterly useless from a genius idea we don't even know about?

And let's not forget that when we filter articles, we're really filtering people.

Would we trust such a program to distinguish the subtle difference between Einstein and Lloyd from Dumb and Dumber? Would we trust it to determine whether our kid would succeed in life or not? Would we let it plan our kid's education accordingly?

I don't think so.