More Signal, Less Noise: The Power of RSS Mashups

First, an Overview of Services

Yahoo! Pipes (Beta)

When Yahoo! Pipes launched, there was much buzz in the blogosphere about a new, powerful tool that let users express ultimate control over how they used their feeds. Tim O’Reilly called it “a milestone in the history of the internet”. And yet, since then, there have been few public proclamations of the power of Pipes. It’s an extremely flexible tool, one with great potential. But it suffers from two key flaws:

1. It’s technically intimidating, too much so for a layman. It may be “drag and drop”, but remember that even the most basic aspect of Pipes, the sources(in the form of RSS, XML, or other syndicated data), have not yet been adopted by the mainstream. Even older terms like “Operators” and the various functions they contain, are technical jargon to non-programmers.
2. Pipes is completely invisible in its implementation, ironically much like the actual hardware it takes its name from. If I use Pipes to remix the feeds I’m subscribed to, my readers will never be aware of it. If I build an application that receives its data through a Pipes mashup or filter, there isn’t a requirement that I label my application, “Powered by Yahoo! Pipes”. Only if you browse pipes can you see some kind of usage data.

Pipes isn’t the only technology that’s had to overcome this problem. RSS and Atom’s effects are largely invisible, though they at least have the benefit of the iconic orange buttons so prevalent today. But if you publish a feed, did you have to install the software to output XML yourself? Unless your name is Dave Winer, the answer is probably no. Modern blogging tools automatically implement syndication for us, and modern browsers know what to do when you click one of those little orange buttons. Pipes and other feed-filters need to integrate with feed reader services the same way RSS and Atom have been integrated into blogging tools. More on this in the second part of my entry.

Microsoft Popfly (Alpha)

While Microsoft may not always do a great job of pleasing communities of developers, it does considerably better in keeping the mainstream three steps behind(several fewer than they might be otherwise). In 2005, when even more people didn’t ‘get’ blogs, MSN Spaces was doing a better job of mainstreaming the concept than Yahoo! 360, which to this day I know of no reason to use unless you’re somehow unable to leave the Yahoo! site.

With Popfly, MS has removed intimidating categories and put everything on a list of “blocks”, which includes both sources of data and things you can do with them. From the beginning a tutorial panel floats on the right, offering to help guide you through creating a mashup, show you a project example, or even play a demonstration video. Projects can be saved via familiar desktop functions, “Save”, and “Save As”. Advanced features? The ability to add custom HTML to your mashup output. Interface extras? A pencil tool, presumably for drawing explanatory lines, hopefully not for writing. A text tool would be a welcome addition.

Popfly is much more about images than it is information, as can be seen in functions like Carousel, Image Scraper, Image Comment, PhotoFlip, PhotoSphere, PhotoStack, PhotoTiles, you get it already. A large number of possible sources for images are included by default, and Virtual Earth’s inclusion is emphasized in the tutorials and examples as a place to add data on top of. Pipes is more powerful when dealing with data and information, but Popfly will gain more mainstream popularity if the results of using it are more visual.

The key deal-breaker: Popfly doesn’t offer any true output function. Once you’ve determined your inputs and your operations, you save the project. If you share that project, you can embed it on a page, download it as a gadget, or add to Windows Live Spaces. But even if you’re only combining RSS feeds, you can’t turn that into a new feed, as best I can tell thus far. Which means that not only does Popfly control the way you’re mixing your data, it also controls the presentation of your data. Unless this changes, PopFly will remain a tool for photo mash-ups and visualizations(still useful!), but won’t be a competitor with Pipes and Google Mashup Editor.

Google Mashup Editor (Beta)

Not having received an invite to try this service yet, much information can still be gleaned from the help pages and Brady Forrest’s comments from when it was announced. Like Pipes and Popfly, data can be brought in from any RSS or Atom feed. Working with Google’s own service brings some particular Atom feeds to mind more prominently, namely those provided by GData. It’s entirely possible Google will further integrate data from its other services into GMashEd.

GMashEd has none of Popfly’s user-friendliness, but all of Pipe’s power, and then some. If MS considers adding custom HTML an “advanced feature”, it would be interesting to see how they describe a tool that lets users use “JavaScript, CSS, and HTML along with GME’s declarative XML tags” plus a JavaScript API that further allows a user to customize presentation of GMashEd’s output. While at this stage GMashEd is very much a techie’s tool, it’s not so great a leap to add a graphical user interface onto this.

Second, How To Use Them

Blog feeds let you send information. Feed readers let you receive it. Pipes, PopFly, and GMashEd let you:

• filter it.
• visualize it.
• combine it.
• correlate it.
• advertise it.

Filter It

I’m currently subscribed to 267 feeds. Robert Scoble subscribes to 622 feeds. We act as human aggregators and filters for our readers, a useful service that applies human insight and creativity to increase the amount of meaningful information(signal) you receive, and reduce the amount of repetitive, uninteresting information(noise) you have to sort through to get it. RSS mashups like Pipes can aid this service by pre-filtering information for us, reducing noise by taking unrelated or uninteresting entries from our feeds.

For instance, if I were a better programmer, I could use Pipes to compare the content of my feeds to the official press releases of Google, Yahoo!, Microsoft, Amazon, and other major companies. Then anything that matches the content of the press release by too great of a percentage(not enough original content in the post), gets filtered out. Or I could create a set of rules that ranks the credibility of sources, and filters certain feeds of mine so that I only hear about an event from the posts of the most credible sources. If there’s only person covering an event, he gets through. If 50 people cover an event, I get the highlights from the top 5 most credible people.

Visualize It

Currently PopFly’s most distinguishing filter, the ability to attach any sort of information to a map is invaluable. Facebook could determine how college networks affect the spread of information by mashing together information on users’ colleges, interests, and where they go after graduation. Addresses from Craigslist could be matched with pictures from Google Maps Streetview. Mentions of a stock ticker symbol could show a chart of the past year’s price.

Combine It

I use Pipes to combine the Twitter feeds of a few A-listers and other interesting people, so that I only have to skim through 1 feed to find interesting links and commentary rather than 15 feeds with 3 posts each. Yahoo! Pipes includes translation service BabelFish as an operator, letting a user input a feed in one language and output it translated into a different language. If you have several sources that update infrequently, you can combine them into one feed, reducing the space they take up on your feed list.

Correlate It

Display information from the Better Business Bureau with mentions of local companies. Show counter-points to U.S. news with stories on the same subject from foreign news services. Compare every service written about by Mashable to a list of other Web 2.0 services, and list links to the ones that are most similar, ordered chronologically by which ones started first. Compare statements of politicians to previous statements they’ve made about the same topic. Correlating your feeds with data from other sources is one of the most powerful methods of using RSS mashups.

Advertise It

Update 07/02/07: There’s an interesting service that plans on opening in August 2007, which will be helping developers insert advertisements specifically for map mashups. Lat49’s ads will be contextual, un-intrusive, and update the location-based results as a user moves the map. Generally, I’m not a fan of advertising pervading every new medium that gets created, but there is a time and a place for it. Since map mashups are often used to help people find local services, this is an instance where advertising would actually be useful. GIS User has a great first-look at Lat49.

Finally, What They Mean

For Companies

As a strategist, seeing how a company takes disparate services and links them together into something greater is one of the most exciting parts of learning about these innovations. Google is the only company I see that currently has a clear strategy in this area, no surprise considering their goal: “to organize the world’s information and make it universally accessible and useful”. Google is the only company with an asset at every link of the information chain. Feedburner publishes, GMashEd remixes, and Google Reader receives. Google also provides some of the most useful data in creating mashups, provides the most-used tool for finding information, and even tracks site statistics without feeds, now enhanced by Feedburner’s SiteStats.

On a more individual level, Google can track how information is spread person-to-person via IM and e-mail. If that’s not a near-complete platform for understanding the flow of information, I don’t know what is. Maybe understanding what information people act on, commercially, as well?

It’s important to understand that seeing the big picture helps you understand the small details even better. Google will use this knowledge to improve each service seperately, making them even better as a whole. Companies trying to compete with a product of Google’s one-on-one won’t have the same understanding of the big picture, and the advantages that entails. The Search Engine Roundtable points out that Google Gears can make Google Reader work offline, which as Nick O’Neill points out, makes it a competitor to desktop feed readers. Are many deaktop feed readers better than Google Reader? In some ways, I think so. But can they stay that way as Google gains new insight into the full syndication process? Maybe, but doubtful, especially considering that Google Reader is free, and integration with other Google services is more often seen as a pro than a con.

For Users

The above uses are only a few examples of how RSS mashups can improve the way we process information on the web, no matter what your goal is. If you want to spend less time reading to find information you’re interested in, filtering and combining feeds according to your personal preferences is the best thing to find online until personalized newspapers come out. If you want to be a better informed reader, you don’t just want less noise, you want better signal as well. Combining and correlating feeds will let you wrap your mind around a subject in a way no single blogger’s entry can hope to do.

The next step is for other services to integrate these tools, so that it’s easier for users to make these improvements and personalizations. Bloglines could use their service to ask users if they want entries too similar to another filtered out. Blog software and feed publishers like Feedburner can offer a default list of services(translation, citations, etc.) bloggers can add to their feed before a user receives it. By removing extra steps, it’s easier for everyone to add value for the end-user.

RSS mashup tools still have many ways to develop. The future isn’t here yet, but I hope I’ve given you some small glimpse of it. I encourage you to leave your thoughts in the comments below, or you can reach me personally at jay dot neely at socialstrategist dot com.