-1

I'm looking for a webapp to convert a list of URLs into their respective page titles with links to the respective sites. The output could be either raw HTML or rich text. The tool would of course need to access the remote site and scrape the title of each URL.

For example, the following input:

https://www.nytimes.com/2020/02/07/health/cdc-coronavirus-china.html
https://www.nytimes.com/2020/02/07/us/politics/alexander-vindman-gordon-sondland-fired.html

would produce the following output: (or alternately its respective HTML source)

C.D.C. and W.H.O. Offers to Help China Have Been Ignored for Weeks

Trump Fires Impeachment Witnesses Gordon Sondland and Alexander Vindman in Post-Acquittal Purge

I modified harshvchwala's script to do this locally, but would like to find a webapp to do the same.

#!/bin/bash -e
echo "usage: $0 <txt file containing list of urls>"
inputFile=${1}
if [ $# -ne 1 ] || [ ! -f ${inputFile} ];then
    echo 'check usage'
    exit -1
fi

outputFile='bookmarks-scripted.html'
echo '<DL>' > $outputFile
NumLines=`wc -l ${inputFile} | cut -d' ' -f1`
currLineNum=0

for eachUrl in `cat ${inputFile}`
do
    let currLineNum=${currLineNum}+1
    echo "processing Line ${currLineNum} of ${NumLines}..."

#    urlTitle=`echo $eachUrl | rev | cut -d'/' -f1 | rev`
    urlTitle=`wget -qO- $eachUrl | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si'`

    if [ -z ${urlTitle} ];then
        urlTitle=`echo $eachUrl | cut -d'.' -f2`
    fi

    echo ${urlTitle}
    urlLine='<DT><A HREF="'
    urlLine+=${eachUrl}
    urlLine+='">'
    urlLine+=${urlTitle}
    urlLine+='</A>'
    echo ${urlLine} >> $outputFile
    echo
done

echo '</DL>' >> $outputFile
-1

What about Shaarli? It's an open source Webapp developed by its community. I personally use it for keeping an eye on valuable hypertexts around the Web, but I'm pretty sure it would match your expectations since you can easily export those informations using an RSS feed. If you're looking for a strict export to HTML, then the bookmark export may be interesting too.

What you still need to do though is to click on each resource Shaarli has to browse and index.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.