0

I am trying to retrieve all of the company names & their symbols from the NYSE Arca (a subsidiary of NYSE). A site exists that lists all of these, but it only lists about 10 results at a time. It can be filtered, or tabbed through. There are over 600 tabs. I am curious if there is any method of manipulating the HTML through DevTools to get it to display more results.

The list is here: Lisitings Directory | NYSE

Edit: I can't comment below after this got moved to Web Apps. I'm looking in the Network tab of DevTools and found the filter request. I'm unable to find the part where it says maxResults. I've clicked on all the tabs (headers, preview, response, etc.).

  • Not an answer, but if the DevTools approach fails, you could (potentially) use browser automation to simply scrape the relevant pages for the names. A language like Python has tools such as Requests, BeautifulSoup and Selenium which could assist with this. – Anaksunaman Nov 21 at 11:19
1

As you suggested, using the Developer Tools of the browser you can view the requests and response when refreshing the page. You hope with this sort of request that the "data" part of a page comes from an API and is a structured response such as XML or JSON.

In this case, there is the query to https://www.nyse.com/api/quotes/filter. If you look at the body of the request, it has a maxResultsPerPage. This is set to 10 by default but it seems you can adjust it so the API returns more JSON. It's really just using your favourite tool to get and parse/present the data, knowing this API can be called.

For example, this one liner might help, having changed it to 100:

wget -Uri "https://www.nyse.com/api/quotes/filter" -Method "POST" -ContentType "application/json" -Body "{`"instrumentType`":`"EQUITY`",`"pageNumber`":1,`"sortColumn`":`"NORMALIZED_TICKER`",`"sortOrder`":`"ASC`",`"maxResultsPerPage`":100,`"filterToken`":`"`"}" | ConvertFrom-Json | % { $_ } | ogv
| improve this answer | |
New contributor
HelpingHand is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.