Take the 2-minute tour ×
Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

What is the recommended way to extract the relevant (non-ad, non-navigation) text pieces of an arbitrary website (say that beside the URL nothing is known) and what tools or APIs are available?

I know of the Google operators inurl:example.com and info:example.com, but would like to get a bit more information, optimally a text abstract of 3-4 sentences.

For a good example of the extracted text that is a nice preview: http://www.dzone.com/links/interview_with_cedric_beust_about_testng_android.html

share|improve this question
    
Does tldr.it interest you? –  Jaywalker Oct 25 '10 at 12:06
    
The problem with that is that the feeds aren't updating on a regular basis. I tried it with TechCrunch and the feed was over a week old. Plus it simply stripped most of the articles to the first couple sentences of the first paragraph. If bloggers followed the professional journalist's way of writing articles that would be helpful, but almost none do. –  Agent_9191 Oct 25 '10 at 14:28
    
Thanks, Jaywalker, yes, in general that is the kind of tool I am looking for. I tried it for some URLs and wasn't really convinced of the summary quality. –  ngeek Oct 26 '10 at 5:59
add comment

3 Answers 3

I think you are looking for something like Readability, a bookmarklet which will (try to) present only the stuff you want to read on a webpage. That is, it tries to present only the important content of a page.

share|improve this answer
    
Even I thought so at first, but I think what he wants is more of an abstract than a stripped off, text-only version of the site. Maybe a concise summary of the page in 3-4 lines. –  thunderror Oct 24 '10 at 12:19
    
If the first relevant 3-4 sentences are returned I am also fine, I think delivering an abstract without any domain knowledge is quite a difficult task. –  ngeek Oct 24 '10 at 15:52
    
Thanks, Senseful, Readability is quite a nice tool, but actually I am looking for a tool which I can embed into my webapp, so either a webservice or an smart text extraction tool would be preferred. @thunderror more ideas on what could be of help? –  ngeek Oct 24 '10 at 15:55
add comment

In the meantime I found an interesting article by Sujit Pal called "Extracting Useful Text from HTML" (based on a earlier published work by alexjc). This extraction method is based on the two following principles:

  • For each line of input HTML, compute the text density and discard the ones whose densities are below some predefined threshold value. This has the effect of removing heavily marked up text.
  • To the text that remains, train and apply a neural network to remove boiler-plate text such as disclaimers, etc

http://sujitpal.blogspot.com/2009/11/extracting-useful-text-from-html.html

share|improve this answer
add comment

If the website have RSS or other feed you can take the texts from there.

share|improve this answer
    
That's true, but I am truly looking for a more generic way for any imaginable URL, get back the first paragraph which contains relevant content (no navigation, no linkbar, no ads). –  ngeek Oct 27 '10 at 10:06
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.