Web Applications Stack Exchange is a question and answer site for power users of web applications. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've noticed that many paragraphs and article sections have been copied and pasted from one Wikipedia article to another, leading to excessive amounts of redundant text on Wikipedia. Do any tools, scripts, or APIs exist that would make it possible to automatically identify these duplicate sections and paragraphs (so that they can be removed)?

share|improve this question
I want to do this programmatically because doing it manually would be extremely tedious. For example, I once found that the article "Nutrition in animals" was mostly copied and pasted from the "Nutrition" article, and now they have both been modified, they will need to be merged together again. – Anderson Green Aug 18 '12 at 17:51
On Wikipedia, text duplication is not necessarily considered a bad thing by itself. – nic Sep 19 '13 at 3:10
up vote 3 down vote accepted

I'm afraid there's not any way to do this using the API or anything like that. However, you could probably do something with the Wikimedia dumps to find the sort of duplication you're looking for. The people already doing research might also be able to help you out.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.