Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want a list of the Titles and URLs for the search results of a particular query. In my particular situation, there are only about 20 pages of results so it's not that much data.

share|improve this question

Lots of answers here, but YQL, although not google out of the box, the following query will help.

select title,url from search.web where query="pizza"

The RESTful URL for those results is:

http://query.yahooapis.com/v1/public/yql?q=select%20title%2Cabstract%2Curl%20from%20search.web%20where%20query%3D%22pizza%22&format=json&diagnostics=true&callback=cbfunc

YQL itself can be found here: http://developer.yahoo.com/yql/ It's a really useful bit of kit by Yahoo.

share|improve this answer
    
You can push further with this and just use yahoo pipes. – phwd Oct 1 '10 at 8:56

I don't know about google but I know bing has an option to export any search page as rss. You simply add format=rss to the query string. For instance

http://www.bing.com/search?q=london&format=rss

This will give 10 result per page so for the second page you add

http://www.bing.com/search?q=london&format=rss&first=11

and the third

http://www.bing.com/search?q=london&format=rss&first=21

and so on...

share|improve this answer

I have no experience myself. But I have found that someone suggests Seo FireFox Plugin.

For another option you can have a look at www.marketing2oh.com/scrape-serps-for-seo-analysis/. It describes the export with title, url and even description.

share|improve this answer

You could try using Google AJAX Search API. For example querying the URL http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Svick will give you results for my username in JSON.

See this blog post for more information.

share|improve this answer
    
I think it's deprecated code.google.com/apis/websearch – Jader Dias May 12 '11 at 12:51

I ended up cutting and pasting HTML from the search results and then using jQuery to pull out the titles and URLs.

share|improve this answer

There's a Firefox plugin available at: http://tools.seobook.com/firefox/seo-for-firefox.html

When you install the plugin (easy), and click on the little SEO logo so that it is in color (not gray), and then do a search in Google, you get a ton of extra data in the results.

Much of the extra information is useful if you are doing search engine optimization. But if you are just looking at how to export the results, there is a tiny 'CSV' link. That will save the results into a CSV format, which you can then open easily with Excel!

You can also try Send to excel option in Internet Explorer.

share|improve this answer

I would suggest a modification of Error on getting the URL result of a Google search for 200 results, using Excel.

Copy and paste the code in an Excel module (Alt+F11, right button on left panel VBAProject, insert module). Go to the tools menu, references, and select Microsoft Internet controls, Microsoft XML 6.0, Microsoft HTML object library.

Having your queries in the A column in a sheet of the document, click on the play button of the module window to run the code.

Sub XMLHTTP2()

Dim url As String, lastRow As Long
Dim XMLHTTP As Object, html As Object, objResultDiv As Object, link As Object, objH3 As Object
Dim start_time As Date
Dim end_time As Date

lastRow = Range("A" & Rows.Count).End(xlUp).Row

Dim cookie As String
Dim result_cookie As String

start_time = Time
Debug.Print "start_time:" & start_time


For i = 1 To lastRow
c = 0
d = 1
    url = "https://www.google.com.br/search?num=100&site=webhp&source=hp&q=" & Cells(i, 1) & "&rnd=" & WorksheetFunction.RandBetween(1, 10000)

    Set XMLHTTP = CreateObject("MSXML2.serverXMLHTTP")
    XMLHTTP.Open "GET", url, False
    XMLHTTP.setRequestHeader "Content-Type", "text/xml"
    XMLHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0"
    XMLHTTP.send

    Set html = CreateObject("htmlfile")
    html.body.innerHTML = XMLHTTP.ResponseText
    Set objResultDiv = html.getelementbyid("rso")

                Set objH3 = objResultDiv.getElementsByTagName("H3")
                     For Each objH3sep In objH3
                     On Error Resume Next
                     Set link = objH3sep.getElementsByTagName("a")(0)
                     str_text = Replace(link.innerHTML, "<EM>", "")
                     str_text = Replace(str_text, "</EM>", "")
                     c = c + 2
                     d = d + 2
                     Cells(i, c) = str_text
                     Cells(i, d) = link.href
                     Next
    DoEvents

    url = "https://www.google.com.br/search?num=100&start=100&site=webhp&source=hp&q=" & Cells(i, 1) & "&rnd=" & WorksheetFunction.RandBetween(1, 10000)

    XMLHTTP.Open "GET", url, False
    XMLHTTP.setRequestHeader "Content-Type", "text/xml"
    XMLHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0"
    XMLHTTP.send

    html.body.innerHTML = XMLHTTP.ResponseText
    Set objResultDiv = html.getelementbyid("rso")

                Set objH3 = objResultDiv.getElementsByTagName("H3")
                     For Each objH3sep In objH3
                     On Error Resume Next
                     Set link = objH3sep.getElementsByTagName("a")(0)
                     str_text = Replace(link.innerHTML, "<EM>", "")
                     str_text = Replace(str_text, "</EM>", "")
                     c = c + 2
                     d = d + 2
                     Cells(i, c) = str_text
                     Cells(i, d) = link.href
                     Next
    DoEvents

    Next

    With ActiveSheet
  Range("A1").Resize(Cells.Find(what:="*", SearchOrder:=xlRows, _
  SearchDirection:=xlPrevious, LookIn:=xlValues).row, _
  Cells.Find(what:="*", SearchOrder:=xlByColumns, _
  SearchDirection:=xlPrevious, LookIn:=xlValues).Column).Select
    Selection.Copy
    Cells(Application.Rows.Count, 1).End(xlUp).Offset(2, 0).Value = "Results"
    Cells(Application.Rows.Count, 1).End(xlUp).Offset(2, 0).Select
    Selection.PasteSpecial Paste:=xlPasteValues, transpose:=True

    End With

end_time = Time


Debug.Print "end_time:" & end_time

Debug.Print "done" & "Time taken : " & DateDiff("n", start_time, end_time)
MsgBox "done" & "Time taken : " & DateDiff("n", start_time, end_time)
End Sub
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.