Webscraping using python + chrome

In mac OS X, install chromedriver and selenium:

brew install chromedriver

Install selenium via pip:

pip install -U selenium

if required to install globally sudo pip install selenium

To test, open a terminal, type python and try:

Python 2.7.12 (default, Oct 11 2016, 05:20:59)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get("https://nbari.com")


The following example will do:

  1. Fill a form,
  2. Parse links of the results.
  3. Open each link.
  4. Get text data from a div.
  5. Go back to step 2 until all links are parsed.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()
inputElement = driver.find_element_by_id("ctl00_ContentPlaceHolder1_textCompanyName")
links = driver.find_elements_by_xpath("//*[@id='ctl00_ContentPlaceHolder1_GridView1']/tbody/tr/td/a")

data = {}
for i in xrange(0, len(links)):
    links = driver.find_elements_by_xpath("//*[@id='ctl00_ContentPlaceHolder1_GridView1']/tbody/tr/td/a")
    idx_name = links[i].text
    div = driver.find_element_by_id("companyDetails")
    data[idx_name] = div.text

for key, value in sorted(data.iteritems()):
    print "-" * 64
    print "%s\n%s" % (key, value)