Using Python selenium and ChromeDriver in a Shiny app

shiny shinyapps.io
,
1

I have a shiny application that looks like this:

image
image1290×386 20.7 KB

When the "Scrape" button is clicked, a Python function runs. This function uses selenium to navigate to the site of interest and proceeds to scrape its contents. To initialize the driver, I need to pass the path to the chrome driver to webdriver.Chrome(executable_path=path). Here is what this part of my Python function looks like:

from selenium import webdriver
import urllib3
import re
import time
import pandas as pd


def scrape(chromedriver="C:/Users/Robpr/OneDrive/Documents/chromedriver.exe"):
        
    # Create driver object. Opens browser
    driver = webdriver.Chrome(executable_path=chromedriver)

    # Rest of the function ...

I call this in my shiny server function like this:

server <- function(input, output, session) {
  
  # Scrape and store returned data frame from .py module in df()
  df = reactive({
    
    if (input$scrape) {
      dfii = sii$scrape(chromedriver="chromedriver.exe")
      dfii$`Filing Date` = as.Date(x=dfii$`Filing Date`, format="%B %d, %Y")
      write_sheet(dfii, ss=sheetId, sheet='filings')
      dfii
    } else if (input$load) {
      read_sheet(ss=sheetId, sheet='filings')
    }
  })

    # Rest of the server function ...

This works beautifully locally. When I publish my app and try to click "Scrape", I get an error: "Error: An error has occurred. Check your logs or contact the app author for clarification.". So, I check my logs:

2021-07-02T18:51:07.355310+00:00 shinyapps[4346040]: Detailed traceback:
2021-07-02T18:51:07.355307+00:00 shinyapps[4346040]: Warning: Error in py_call_impl: WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
2021-07-02T18:51:07.355310+00:00 shinyapps[4346040]: 
2021-07-02T18:51:07.355320+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
2021-07-02T18:51:07.355311+00:00 shinyapps[4346040]:     driver = webdriver.Chrome(executable_path=chromedriver)
2021-07-02T18:51:07.355311+00:00 shinyapps[4346040]:   File "/srv/connect/apps/StatCanWebScrapingApp/scrapeinsolvencyinsider.py", line 11, in scrape
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:     self.service.start()
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:     raise WebDriverException(
2021-07-02T18:51:07.355321+00:00 shinyapps[4346040]:   File "/home/shiny/.virtualenvs/statcanEnv/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 81, in start
2021-07-02T18:51:07.355322+00:00 shinyapps[4346040]: 
2021-07-02T18:51:07.362398+00:00 shinyapps[4346040]:   135: <Anonymous>

I've published the chromedriver.exe file along with my app and module: image

How can I get this to work on the server?

2

You can test if it actually is in the PATH, if you open a cmd and type in chromedriver and hit Enter. If Starting ChromeDriver 2.15.322448 is appearing, the PATH is set appropriately and there is something else going wrong. Then try the following:

Download ChromeDriver

Then you have multiple options:

  • Add it to your system path
  • Put it in the same directory as your python script
  • Specify the location directly via executable_path

driver = webdriver.Chrome(executable_path='C:/path/to/chromedriver.exe')

Before you add the chromedriver to your path, make sure it's the same version as your browser.

If not, you will need to match versions: either update/downgrade you chrome, and upgrade/downgrade your webdriver.