0

I want to scrape a text within this website in google sheets with the IMPORTXML function. Through dev tools by inspecting the text in question, I get the corresponding HTML:

<li _ngcontent-udl-c4="">The outbound route is available for sale until 28/03/2020 and these are the last available dates.</li>

If I right-click > copy full XPath, I get /html/body/main/nas-fare-calendar/nas-info/div/div/ul/li[1]. If I input that in my sheet, as in (A1 has the URL linked to above):

=IMPORTXML(A1,"/html/body/main/nas-fare-calendar/nas-info/div/div/ul/li[1]")

I get an error 'Imported content is empty.'. What gives?

I've also tried Web Scrapper extension and the way that selects the text in question is with '.list--spaceless li:nth-of-type(1)'. Any way I could use that as inspiration for an XPath? Just putting that as input for IMPORTXML doesn't work.

What is the easiest way to select for the innerHTML of that HTML snippet above? Just trying with '//li' just won't include the li above. I don't know how to include _ngcontent-udl-c4 or li that has _ngcontent-udl-c4 as a style.

  • 1
    Related: webapps.stackexchange.com/q/115664/88163 – Rubén Jan 16 at 0:00
  • 1
    Thanks for linking that. For people looking for a quick answer, by following the steps of that answer, I realized the li text in question is generated by JS and therefore not accessible through ImportXML. Will probably implement something with google apps script to see if I manage to isolate/web scrape the text I want. – jlo Jan 16 at 10:14
0

As mentioned in the comment above, it's not possible to retrieve the text via importxml. I managed, however to scrape it successfully. Here's how:

  1. Open developer tools (Ctrl+Shift+i or File → Developer → Developer Tools)
  2. Go the site in question.
  3. Click on the network tab -> Filter icon -> select XHR to filter out some of the requests
  4. Find the relevant request (see screenshot)
  5. Right click on the request to copy its API URL
  6. Now in google script query that URL

d

function getXMLAndExtractDate() { var url = "some.url/request/"; var res = UrlFetchApp.fetch(url); if(res){ string = res.getContentText(); var regex =/(?<=The outbound route is available for sale until )(.*)(?= and these)/; var out = string.match(regex)[0];
return out; //Logger.log(out); //28/03/2020 } }
After that you can do with the date what you want (in my case I write it into a spreadsheet and track them daily. In case of any changes, it fires me an email.

Inspiration: https://beshaimakes.com/js-scrape-data

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.