ImportXML error: 'imported content is empty' when selecting for text within a list item

Question

I want to scrape a text within this website in google sheets with the IMPORTXML function. Through dev tools by inspecting the text in question, I get the corresponding HTML:

<li _ngcontent-udl-c4="">The outbound route is available for sale until 28/03/2020 and these are the last available dates.</li>

If I right-click > copy full XPath, I get /html/body/main/nas-fare-calendar/nas-info/div/div/ul/li[1]. If I input that in my sheet, as in (A1 has the URL linked to above):

=IMPORTXML(A1,"/html/body/main/nas-fare-calendar/nas-info/div/div/ul/li[1]")

I get an error 'Imported content is empty.'. What gives?

I've also tried Web Scrapper extension and the way that selects the text in question is with '.list--spaceless li:nth-of-type(1)'. Any way I could use that as inspiration for an XPath? Just putting that as input for IMPORTXML doesn't work.

What is the easiest way to select for the innerHTML of that HTML snippet above? Just trying with '//li' just won't include the li above. I don't know how to include _ngcontent-udl-c4 or li that has _ngcontent-udl-c4 as a style.

Answer 1

Thanks for linking to How to know if Google Sheets IMPORTDATA, IMPORTFEED, IMPORTHTML or IMPORTXML functions are able to get data from a resource hosted on a website?. For people looking for a quick answer, by following the steps of that answer, I realized the li text in question is generated by JS and therefore not accessible through ImportXML. Will probably implement something with google apps script to see if I manage to isolate/web scrape the text I want

---

As mentioned in the comment above, it's not possible to retrieve the text via importxml. I managed, however to scrape it successfully. Here's how:

1. Open developer tools (Ctrl+Shift+I or File → Developer → Developer Tools)
2. Go the site in question.
3. Click on the network tab -> Filter icon -> select XHR to filter out some of the requests
4. Find the relevant request (see screenshot)
5. Right click on the request to copy its API URL
6. Now in google script query that URL

function getXMLAndExtractDate() {   
  var url = "some.url/request/";   
  var res = UrlFetchApp.fetch(url);
  if(res){
    string = res.getContentText();
    var regex =/(?<=The outbound route is available for sale until )(.*)(?= and these)/; 
    var out = string.match(regex)[0];  
    return out;   //Logger.log(out); //28/03/2020   
  } 
}

After that you can do with the date what you want (in my case I write it into a spreadsheet and track them daily. In case of any changes, it fires me an email.

Inspiration: https://beshaimakes.com/js-scrape-data