Using Yahoo! Pipes to download a web page given the URL isolated from an RSS feed

Question

The RSS feed of a website gives only the link to the particular entry, to read which I have to open my web browser and go to the link. My aim is to use Yahoo! Pipes to retrieve this link from the feed, download the page and read it in my RSS reader newsbeuter.

I use the "Fetch Feed" module and link it to the "Sub-element" module and then isolate the link using "item.link"

The problem I now face is that how to pipe the output of the "sub-element" module to any module which will allow me to download the associated webpage. I tried the "URL builder", "string builder" among others but I couldn't find a module which would accept the output of the "sub-element" module and then by itself or via an other module download the associated webpage.

How can I accomplish this?

Answer 1

My solution for a similar problem was to use the "String Builder" module to construct a YQL (Yahoo Query Language) query like this:

select * from html where url="BUILDER PUTS THE URL HERE" and xpath='Replace this with an XPath expression for the content you want' 

...and then use the "YQL" module to execute it, producing an HTML document fragment suitable for feeding to the Description field of the "Create RSS" module.

If you want to, you can take a look at mine to get a better idea of how it's accomplished.

Update: As I understand it, the gray and blue connections are color-coding for "single item" and "list of items", respectively.

I'm not sure why they don't have any kind of "reduce" operator (takes a list, returns one) other than "count", given that they have so many "map" (take a list, return a list) operators, but they do have a "Loop" operator. With "Loop" you can snap in something like String Builder and have it called once for each item in the input list.

The only solution I was ever able to find for what you want was to do that and be clever in how I structured my pipes so they didn't get too hairy. It's actually a technique I use in the pipe I linked, but I forgot to draw attention to it.