Importance of web scraping in e-commerce and e-marketing

6 min readJan 2, 2021

The new technological trend has caused us to change the way we do our business. The Internet is now the new place for business. Knowing how to use the internet and the various opportunities that could result from it could be one of the keys to success in e-commerce and e-marketing.

Web scraping

Web scraping, also known as data mining, is the process of collecting large amounts of data from the web and then placing it in databases for future analysis and later use. Web scraping offers insight into price data, market dynamics, prevailing trends, practices employed by your competitors and the challenges they face. This is readily available data if you know how to get it. What many marketers don’t know is how useful it can be to them.

In this article, we will explain the advantage of web scraping and provide a practical example based on jumia.com that could be beneficial for e-commerce businesses and online marketers.

What is E-commerce?

E-commerce, also known as electronic commerce or Internet commerce, refers to the buying and selling of goods or services over the Internet, and the transfer of money and data to complete these transactions.

E-commerce operates in the following four main market segments:

From consumer to consumer
Consumer to business
Business to business
Business to consumer

What is e marketing?

E-Marketing refers to the marketing conducted over the Internet.

Benefits Of Web Scraping

Web scraping has become an important strategy for e-commerce businesses, especially for delivering rich, data-driven information. For example, e-commerce web scrapers help identify consumer preferences and choices.

Price-monitoring and Product Research
Better Customer analysis
Increased productivity
Better advertisements
Influences Marketing and Sales Strategy
Helps in Future analysis

Why a business needs web scraping ?

An online presence can be tracked. It is also an important aspect of web scraping where company profiles and reviews on websites can be removed. This can be used to see product performance, user behavior, and reaction.
Personalized analysis and curation. This one is mainly for new websites / channels where the retrieved data can be useful for channels to know viewer behavior. This is done for the purpose of delivering targeted news to the public. So, what you watch online sets the pattern for the website to behave so that the business knows its audience and can deliver what the audience actually likes.
Online reputation. When the company plans its ORM strategy, the data extracted helps understand which audiences companies are most hoping to have an impact and which areas of responsibility may most expose the brand to reputation damage. By understanding these areas of vulnerability, the business can use them to its greatest advantage.
Provide better targeted advertising to customers. Scrapping not only gives numbers, but also sentiment and behavioral analysis, so the business can know what audience types and choice of ads they want to see.
To bring together public opinion. Monitor specific company pages on social media to gather updates on what people are saying about certain companies and their products. Data collection is always useful for product growth.
Search engine results for SEO tracking. By scraping off organic search results, you can quickly find your SEO competitors for a particular search term. You can determine the title tags and the keywords they target. So you get an idea of what keywords drive traffic to a website, what categories of content get links and user engagement, what kind of resources it takes to rank your site.
Scratch Leads: This is another important use for the sales-oriented organization in which lead generation is done. Sales teams are always hungry for data, and using web scraping technique, you can pull leads from many directories and then contact them to do an introduction to selling. The data can be extracted in any format you want and can be used for lead generation, brand development, or other purposes.
To create vertical specific search engines. Even though this is a new thing that is popular in the market, but also requires a lot of data, so web scraping is done for as much public data as possible because this volume of data is practically impossible to collect.

How do Web Scrapers Work?

First, the web scraper will receive one or more URLs to load before scraping. The scraper then loads the entire HTML code for the page in question. More advanced scrapers will render the entire website including CSS and JavaScript elements.
Then the scraper will extract all the data from the page or specific data selected by the user before running the project.
Ideally, the user will go through the process of selecting the specific data they want on the page. For example, you might want to browse an Amazon product page for pricing and models, but you’re not necessarily interested in product reviews.
Finally, the web scraper will display all the collected data in a format more useful to the user.

Technology and tools to use

You will find two categories of tools, free and paid. but in our article we will be using free tools.

Well-known free technologies and tools

R programing with rvest package
Python with Beautiful Soup

Practical example

Jumia is an online marketplace for electronics and fashion, among others, targeting several African countries, but headquartered and incorporated in Germany. The company is also a logistics service, which enables the shipment and delivery of packages from sellers to consumers, and a payment service, which facilitates transactions between active participants and the platform of Jumia in selected markets. It has established partnerships with more than 50,000 local African businesses and individuals and is a direct competitor of Konga in Nigeria.

In this exercise, we will collect details of phones sold on jumia kenya Platform.

First step get the url

To get the link, you can go to amazon.com and search for smartphone menu. Here is the link we use for our demonstration.

Second step Inspecting the Page and Find the data you want to extract

You need to have some knowledge of HTML to understand the structure of a web page.

Third step Write the code

We chose to use r programming if you want to know more about it you can click here.

library(tidyverse)
library(rvest)
library(stringi)
library(dplyr)
#Specifying the url
url_base <- ‘https://www.jumia.co.ke/smartphones'
#You need to get href and loop on hrefs
all_pages <- url_base %>% read_html() %>% html_nodes(“.-pvxl > a”) %>% html_attr(“href”)
all_pages[1] <- url_base
all_pages <- stri_remove_empty(stri_na2empty(all_pages))
#create an empty table to store results
result_table <- tibble()

# get all data first pagination
page_source <- read_html(all_pages[1])
title <- html_nodes(page_source,’.name’) %>% html_text()
price <- html_nodes(page_source,’.prc’) %>% html_text()
temp_table <- tibble(title = title, price = price)
result_table <- bind_rows(result_table,temp_table)
# Get all data start by the second pagination
#create an empty table to store results
result_table2 <- tibble()
all_pages2 <- paste(url_base ,all_pages[2:5], sep=””)
for(page in all_pages2 ){
page_source <- read_html(page)
title <- html_nodes(page_source,’.name’) %>% html_text()
price <- html_nodes(page_source,’.prc’) %>% html_text()
temp_table <- tibble(title = title, price = price)
result_table2 <- bind_rows(result_table,temp_table)
}
## Remove empty value
result_table <- result_table %>%
filter(price != “”)
result_table2 <- result_table2 %>%
filter(price != “”)
# Add datasets vertically
result <- rbind(result_table, result_table2)

You can change the url_base <- ‘https://www.jumia.co.ke/smartphones'

to :

jumia Egypt : https://www.jumia.com.eg/smartphones,

jumia Uganda : https://www.jumia.com.eg/smartphones

jumia ivory cost : https://www.jumia.ci/smartphones

All jumia can be scrape with the same code

Fourth step exporting data

Conclusion

With the ever increasing demand for web data, having web scraping skills makes your resume stand out immediately. Mastering these technologies can help you get all the data you want from the web, since you have the technological resources to back it up.