Locating and selecting elements from the web page is the key to web scraping with Selenium. You can select elements using a tag name, ID, class name, XPath selector, CSS selector, etc. in Selenium. You can also select elements that have a specific text with Selenium. This is helpful for selecting links and buttons from the web page easily. Even if the page structure changes, as long as the text of the web page element remains the same, your selector should work just fine. This is the advantage of selecting links and buttons using text in Selenium.

In this article, I am going to show you how to locate and select elements from web pages using text in Selenium with the Selenium python library. So, let’s get started.

Prerequisites:

To try out the commands and examples of this article, you must have:

  1. A Linux distribution (preferably Ubuntu) installed on your computer.
  2. Python 3 installed on your computer.
  3. PIP 3 installed on your computer.
  4. Python virtualenv package installed on your computer.
  5. Mozilla Firefox or Google Chrome web browsers installed on your computer.
  6. Must know how to install the Firefox Gecko Driver or Chrome Web Driver.

For fulfilling the requirements 4, 5, and 6, read my article Introduction to Selenium in Python 3.

You can find many articles on the other topics on LinuxHint.com. Be sure to check them out if you need any assistance.

Setting Up a Project Directory:

To keep everything organized, create a new project directory selenium-text-select/ as follows:

$ mkdir -pv selenium-text-select/drivers

How to Find Element by Text with Selenium Web Scraping

Navigate to the selenium-text-select/ project directory as follows:

$ cd selenium-text-select/

How to Find Element by Text with Selenium Web Scraping

Create a Python virtual environment in the project directory as follows:

How to Find Element by Text with Selenium Web Scraping

Activate the virtual environment as follows:

$ source .venv/bin/activate

How to Find Element by Text with Selenium Web Scraping

Install Selenium Python library using PIP3 as follows:

How to Find Element by Text with Selenium Web Scraping

Download and install all the required web driver in the drivers/ directory of the project. I have explained the process of downloading and installing web drivers in my article Introduction to Selenium in Python 3.

How to Find Element by Text with Selenium Web Scraping

Finding Elements by Text:

In this section, I am going to show you some examples of finding and selecting web page elements by text with the Selenium Python library.

I am going to start with the simplest example of selecting web page elements by text, selecting links from the web page.

In the login page of facebook.com, we have a link Forgotten account? As you can see in the screenshot below. Let’s select this link with Selenium.

How to Find Element by Text with Selenium Web Scraping

Create a new Python script ex01.py and type in the following lines of codes in it.

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

from selenium.webdriver.common.by import By

from time import sleep


browser = webdriver.Chrome(executable_path=“./drivers/chromedriver”)


browser.get(“https://www.facebook.com/”)


forgotAccountLink = browser.find_element(By.XPATH, “<a id="post-66569-_Hlk47727696“>


//*[text()=’Forgotten account?’]”
)


forgotAccountLink.send_keys(Keys.ENTER)

Once you’re done, save the ex01.py Python script.

How to Find Element by Text with Selenium Web Scraping

Line 1-4 imports all the required components into the Python program.

How to Find Element by Text with Selenium Web Scraping

Line 6 creates a Chrome browser object using the chromedriver binary from the drivers/ directory of the project.

How to Find Element by Text with Selenium Web Scraping

Line 8 tells the browser to load the website facebook.com.

How to Find Element by Text with Selenium Web Scraping

Line 10 finds the link which has the text Forgotten account? Using XPath selector. For that, I have used the XPath selector //*[text()=’Forgotten account?’].

The XPath selector starts with //, which means the element can be anywhere on the page. The * symbol tells Selenium to select any tag (a or p or span, etc.) that matches the condition inside the square brackets []. Here, the condition is, the element text is equal to the Forgotten account?

How to Find Element by Text with Selenium Web Scraping

The text() XPath function is used to get the text of an element.

For example, text() returns Hello World if it selects the following HTML element.

<a href=“http://dummysite.com”>Hello World</a>

Line 11 sends the key press to the Forgotten account? Link.

How to Find Element by Text with Selenium Web Scraping

Run the Python script ex01.py with the following command:

How to Find Element by Text with Selenium Web Scraping

As you can see, the web browser finds, selects, and presses the key on the Forgotten account? Link.

How to Find Element by Text with Selenium Web Scraping

The Forgotten account? The link takes the browser to the following page.

How to Find Element by Text with Selenium Web Scraping

In the same way, you can easily search for elements that have your desired attribute value.

Here, the Log In button is an input element which has the value attribute Log In. Let’s see how to select this element by text.

How to Find Element by Text with Selenium Web Scraping

Create a new Python script ex02.py and type in the following lines of codes in it.

from selenium import webdriver

from selenium.webdriver.common.keys import Keys

from selenium.webdriver.common.by import By

from time import sleep


browser = webdriver.Chrome(executable_path=“./drivers/chromedriver”)


browser.get(“https://www.facebook.com/”)


sleep(5)


emailInput = browser.find_element(By.XPATH, “//input[@id=’email’]”)


passwordInput = browser.find_element(By.XPATH, “//input[@id=’pass’]”)


loginButton = browser.find_element(By.XPATH, “//*[@value=’Log In’]”)


emailInput.send_keys([email protected])


sleep(5)


passwordInput.send_keys(‘secret-pass’)


sleep(5)


loginButton.send_keys(Keys.ENTER)

Once you’re done, save the ex02.py Python script.

How to Find Element by Text with Selenium Web Scraping

Line 1-4 imports all the required components.

How to Find Element by Text with Selenium Web Scraping

Line 6 creates a Chrome browser object using the chromedriver binary from the drivers/ directory of the project.

How to Find Element by Text with Selenium Web Scraping

Line 8 tells the browser to load the website facebook.com.

Everything happens so fast once you run the script. So, I have used the sleep() function many times in ex02.py for delaying browser commands. This way, you can observe how everything works.

How to Find Element by Text with Selenium Web Scraping

Line 11 finds the email input text box and stores a reference of the element in the emailInput variable.

How to Find Element by Text with Selenium Web Scraping

Line 12 finds the email input text box and stores a reference of the element in the emailInput variable.

How to Find Element by Text with Selenium Web Scraping

Line 13 finds the input element which has the attribute value of Log In using XPath selector. For that, I have used the XPath selector //*[@value=’Log In’].

The XPath selector starts with //. It means the element can be anywhere on the page. The * symbol tells Selenium to select any tag (input or p or span, etc.) that matches the condition inside the square brackets []. Here, the condition is, the element attribute value is equal to Log In.

How to Find Element by Text with Selenium Web Scraping

Line 15 sends the input [email protected] to the email input text box, and line 16 delays the next operation.

How to Find Element by Text with Selenium Web Scraping

Line 18 sends the input secret-pass to the password input text box, and line 19 delays the next operation.

How to Find Element by Text with Selenium Web Scraping

Line 21 sends the key press to the login button.

How to Find Element by Text with Selenium Web Scraping

Run the ex02.py Python script with the following command:

How to Find Element by Text with Selenium Web Scraping

As you can see, the email and password text boxes are filled with our dummy values, and the Log In button is pressed.

How to Find Element by Text with Selenium Web Scraping

Then the page navigates to the following page.

How to Find Element by Text with Selenium Web Scraping

Finding Elements by Partial Text:

In the earlier section, I have shown you how to find elements by specific text. In this section, I am going to show you how to find elements from web pages using partial text.

In the example, ex01.py, I have searched for the link element which has the text Forgotten account?. You can search the same link element using partial text such as Forgotten acc. To do that, you can use the contains() XPath function, as shown in line 10 of ex03.py. The rest of the codes are the same as in ex01.py. The results will be the same.

How to Find Element by Text with Selenium Web Scraping

In line 10 of ex03.py, the selection condition used the contains(source, text) XPath function. This function takes 2 arguments, source, and text.

The contains() function checks whether the text given in the second argument partially matches the source value in the first argument.

The source can be the text of the element (text()) or the attribute value of the element (@attr_name).

In ex03.py, the text of the element is checked.

How to Find Element by Text with Selenium Web Scraping

Another useful XPath function to find elements from the web page using partial text is starts-with(source, text). This function has the same arguments as the contains() function and is used the same way. The only difference is that the starts-with() function checks whether the second argument text is the starting string of the first argument source.

I have rewritten the example ex03.py to search for the element for which the text starts with Forgotten, as you can see in line 10 of ex04.py. The result is the same as in ex02 and ex03.py.

How to Find Element by Text with Selenium Web Scraping

I have also rewritten ex02.py so that it searches for the input element for which the value attribute starts with Log, as you can see in line 13 of ex05.py. The result is the same as in ex02.py.

How to Find Element by Text with Selenium Web Scraping

Conclusion:

In this article, I have shown you how to find and select elements from web pages by text with the Selenium Python library. Now, you should be able to find elements from web pages by specific text or partial text with the Selenium Python library.

About the author

How to Find Element by Text with Selenium Web Scraping

Shahriar Shovon

Freelancer & Linux System Administrator. Also loves Web API development with Node.js and JavaScript. I was born in Bangladesh. I am currently studying Electronics and Communication Engineering at Khulna University of Engineering & Technology (KUET), one of the demanding public engineering universities of Bangladesh.