How to Automate any Browser Task with Selenium

Have you ever wondered how to automate repetitive tasks in a web browser, or how to test web applications efficiently? Enter Selenium, an open-source framework that allows you to automate web browsers using various programming languages like Python, Java, and C#.

Download and Install Selenium

If you do not already have python installed, install the latest version here!

Once python is installed, open up command prompt and enter the following command.

pip install selenium

This command will download the latest version of Selenium to your machine.

Now depending on the browser you want to automate, you will need the compatible WebDriver for it. Once you are aware of the version of your wanted browser, go to the following links to download the WebDriver:

For Chrome Browsers
For Firefox Browsers

Of course there are more than these two browsers, but these are the two that are the most popular. Once downloaded, take note of the path of the file you just downloaded. Below is the code to simply set your driver to the WebDriver that you just installed.

Use webdriver.Chrome for Chrome browsers
Use webdriver.Firefox for Firefox Browsers

from selenium import webdriver

# Specify the path to your WebDriver
driver = webdriver.Chrome(executable_path="/path/to/chromedriver")

However, if you want to turn on easy mode, you can have your script automatically download the WebDriver for you upon start up.

First, download the required package.

pip install webdriver-manager

Then, Use the following code in your script to do so:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

# Automatically download and set up the WebDriver
driver = webdriver.Chrome(ChromeDriverManager().install())

The Basics — Navigating/Interacting

All code from here on will use webdriver to automatically download the webdriver and will use the chrome browser.

Going to a website

Let’s first learn how to simply open a webpage and stay on that webpage.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

# Automatically download and set up the WebDriver
driver = webdriver.Chrome(ChromeDriverManager().install())

# Open a browser and navigate to a website
driver.get("https://www.google.com")

The above code snippet will open the browser window and go to the website google.com and will remain open until you manually close the browser.

Locating elements on the page

To click buttons or interact with controls, Selenium needs a way to identify each HTML element. Here are the five common locator methods:

By ID — driver.find_element(By.ID, "element_id")
By name — driver.find_element(By.NAME, "element_name")
By class name — driver.find_element(By.CLASS_NAME, "element_class")
By CSS selector — driver.find_element(By.CSS_SELECTOR, ".element_class")
By XPath — driver.find_element(By.XPATH, "//tagname[@attribute='value']")

The general best way to locate an element is by ID, but if the element does not have an ID, you can use the other above methods!

Interacting with elements on the page

Here are some common ways you can interact with elements on a page:

1. Clicking a button

button = driver.find_element(By.ID, "submit_button") # Locating element
button.click() # Interacting with element

2. Typing text into a field

input_field = driver.find_element(By.NAME, "input_name") # Locating element
input_field.send_keys("Your text here") # Interacting with element

3. Clearing a field

input_field.clear() # Using element name and .clear() method

Navigating Between Pages

Just like how you can go backwards to the last page you were on in your browser, you can do the same with Selenium!

driver.back()  # Go back to the previous page
driver.forward()  # Move forward in browser history

Handling Browser Tabs or Windows

# Get the current window handle
main_window = driver.current_window_handle

# Open a new tab or window
driver.execute_script("window.open('https://www.example.com', '_blank');")

# Switch to the new tab
driver.switch_to.window(driver.window_handles[1])

# Switch back to the original tab
driver.switch_to.window(main_window)

Basic — Managing Sessions

Ever imagined how that particular website knows you are still logged in even though you actually logged in a couple weeks ago? This is what sessions are for, websites use different methods for authentication but the most popular one is via browser cookies.

Cookies are small pieces of data stored by the browser and sent with every HTTP request to the server, often used for session management.

Here is the code to add any cookie(s) to your selenium browser.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

# Automatically download and set up the WebDriver
driver = webdriver.Chrome(ChromeDriverManager().install())

# Open a browser and navigate to a website
driver.get("https://www.example.com")

# Define the cookie with name and value
cookie = {'name': 'session_id', 'value': 'abc123'}

# Add the cookie to the browser
driver.add_cookie(cookie)

driver.delete_cookie('session_id')  # Delete a specific cookie
driver.delete_all_cookies()  # Delete all cookies

Another way websites store important data such as session tokens is by placing them in the browser’s local storage!

Local storage, on the other hand, is a web storage object that allows sites to store key-value pairs in a user’s browser with no expiration date.

Many Single Page Applications use this method. Here is how to do it in selenium!

Add a Value to Local Storage

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

# Automatically download and set up the WebDriver
driver = webdriver.Chrome(ChromeDriverManager().install())

# Open a browser and navigate to a website
driver.get("https://www.example.com")

# Add a key-value pair to local storage
driver.execute_script("window.localStorage.setItem('token', 'abc123');")

Removing a Value from Local Storage

# Remove an item from local storage
driver.execute_script("window.localStorage.removeItem('token');")

Advanced — Bot Detection

There are many websites that use various bot detection techniques ranging from a simple captcha solver such as Google’s captcha (Recaptcha) to highly sophisticated VM server-side algorithms. Let’s talk about how to get around them in this section!

Most CAPTCHA Services

There are multiple ways to solve captchas in selenium. You can use third-party captcha solvers, or house your own custom captcha solver (which will require much more effort). In this example, we will talk about using a third-party captcha solver such as Capsolver. Thankfully, Capsolver has an extension just for this use case!

Download Capsolver’s selenium extension here.
Unzip it into the ./CapSolver.Browser.Extension directory at the root of your project.
Insert your API key in the apiKey field in ./assets/config.json.
Use the code below!

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import os

def main():
    extension_path = os.path.abspath('./CapSolver.Browser.Extension')
    chrome_options = Options()
    chrome_options.add_argument(f'--load-extension={extension_path}')

    driver = webdriver.Chrome(options=chrome_options)
    driver.get('https://www.google.com/recaptcha/api2/demo') # website that has recaptcha

    # Wait for the captcha to be solved automatically by the extension
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, 'recaptcha-demo-submit')))

    # Here, you can proceed with your form submission or any other actions post-captcha solving

    driver.quit()

if __name__ == "__main__":
    main()

Where CAPTCHAs Are the Least of Your Worries

Most websites don’t rely on a CAPTCHA alone to prevent web crawlers. Some use sophisticated bot-detection services from companies such as Akamai and DataDome.

Because a basic Selenium browser is relatively easy to detect, you may need additional tooling. One possible approach is to use selenium-wire alongside undetected-chromedriver.

Install selenium-wire | pip install selenium-wire
Install undetected-chromedriver | pip install undetected-chromedriver

Use the following code to setup your selenium browser.

import seleniumwire.undetected_chromedriver as uc

chrome_options = uc.ChromeOptions()

driver = uc.Chrome(
    options=chrome_options,
    seleniumwire_options={}
)

Selenium-wire can also do some cool things such as access request data, mock responses, use your own certificates, and add headers! Read more about these features here!

Obviosuly just using these packages out of the box won’t solve all of your potential issues. Make sure to read the docs on these packages to learn all the different ways you can make your selenium browser look as realistic as possible!

Examples of Real World Cases

Here are some real world examples where we can leverage the power of selenium in day-to-day work!

1. Testing Web Applications & CI/CD Pipelines

Selenium can be used to automate end-to-end testing for web applications such as testing certain frontend features that a user may use.

2. Web Scraping for Competitive Data Analysis

Selenium can be used to web scrape valuable information that can be used in future projects!

Hope you enjoyed this how-to on selenium, a clap would be greatly appreciated! 😊

Links

My LinkedIn

My Website