Easy screen scraping with the built-in DOM operations in EGL Rich UI
ChrisLaffra 060000KCEQ Visits (3825)
Research has shown that during the development of every single successful software project pizza was eaten by one or more of the developers. Therefore, we conclude pizza is essential to the success of your projects.
Now.... where to find those pizzas? Of course, the boring way would be to use Google and find those pizzas on a map in a few clicks:
Real computer scientists, however, don't surrender so easily to ready-to-use tools. No. They write a script. Especially in EGL Rich UI that is easy to do. First, you make a call to find the pizzas:
The search string and the zipcode are hard-coded, but everyone knows how to parameterize them into a function and do some string manipulations on the url string. Right?
When the service call comes back with an answer, something special happens. Normally, service calls in EGL return either SOAP or JSON, depending on what type of service is being used. However, when making a REST GET service call, the data that is being sent back is left up entirely to the implementer of the service. In this particular case, the result is pure HTML.
To handle the HTML we will inspect it inside this function:
First we create an instance of a Div widget; one of the basic EGL widgets. Then we find all the spans inside the entire HTML file. Then for each span, we see if it is a title or an address and print it out. The result:
This is just a very simple example of how to scrape a certain page. Always make sure you have the appropriate rights to scrape a certain site. The above example to scrape Google maps is for educational purposes only. This type of scraping probably violates Google's terms of service.
Have fun scraping!