When people visit your site to view your products and make purchases, that’s human traffic. When automated software — also called a bot — visits your site, that’s bot traffic.
Bot traffic can be both good and bad. Google, for example, has bots catalog your site and its contents, so that it appears in search results. Here are some examples of good bots:
But bots aren’t all positive. Cybercriminals leverage bad bots to validate your consumers’ credentials, payment card data, and other personal information for use in credential stuffing, carding and account takeover (ATO) attacks. Malicious bots extract your pricing and product content in web scraping attacks, giving your competitors an edge. They buy up your inventory and resell at inflated prices in scalping and denial of inventory attacks.
Bot traffic is different from human traffic. Bots can execute tasks much faster than people, and bot traffic comes in much higher volumes than human traffic. You’ll see bots produce record-high numbers of page views in an instant. Human users usually do several different actions and clicks at a more moderate pace.
Bots spend both significantly more and less time on your site than human visitors do. If web sessions start and end very quickly, it’s typically a sign that a bot is crawling your site much faster than a human ever would. Exceptionally long sessions mean that a bot is browsing the site very slowly. Web sessions for humans are more consistent in duration than for bots.
Next is their behavior once they land on your site. Bots often visit a page, but do nothing once they get there. Instead of navigating predictably through your site or purchasing products, they might just visit a page and leave. This can be seen in unusually high bounce rates and, sometimes, lower conversion rates as well.
Another way to identify bots is through the use of honeypots. This might mean adding a hidden HTML input element to a page that legitimate human users are not able to see — so if a user accesses the element, you can be sure it’s a bot. Another technique is to stack two clickable elements in the same place on a page. A legitimate user can only click on the upper element, while a bot automatically will click on both.
In addition, you can consider the source of suspect traffic. Bots may invade your site from geographies where your customers don’t live, and the inhabitants don’t use the language on your site. This is an indication that visitors aren’t human.
Website owners can analyze server logs to identify malicious bot traffic hiding inside legitimate site traffic. Specific signals can help you to flag bot activity on your site. For example:
Bot traffic negatively affects your organization by driving flawed business decisions, diminished site performance and lower search engine rankings — which can all lead to competitive disadvantages.
Automated traffic taxes your infrastructure and increases your costs for bandwidth and compute cycles. Bot traffic can overwhelm your network and slow site performance, which frustrates customers and negatively impacts user experience.
Search engines consider your site’s speed as a factor in positioning your site among search results. Your site can appear lower in search due to the effects of bot traffic, which can keep consumers from finding you.
In addition, bot traffic contaminates your data and skews your analytics — for example, by appearing to show an increase in consumer demand. Flawed metrics on user behavior can lead you to make poor business decisions about pricing, stocking goods, and investing in marketing and advertising.
These outcomes are byproducts of bot traffic itself. But perhaps more damaging is what cybercriminals can use bots to do. Malicious bot traffic means bot attacks to validate stolen information on your site. These include:
You can do several things to prevent bots from crawling your web and mobile apps and APIs.
Monitor your site for malicious bot traffic
You can gain visibility by monitoring your site for malicious activity. Once you establish a baseline of typical human behavior, you can compare and contrast suspected bot activity. When visitors and accounts reach a threshold of likely bot behavior, you can trigger alerts and automated responses to block them.
Challenge the bots to prove they are human
You can challenge bots to prove they are human when you suspect malicious behavior using a problem that only real people can solve. A CAPTCHA is one example of this, though sophisticated bots can beat those tests. Furthermore, CAPTCHAs often frustrate human users, driving abandonment. Human Challenge, an alternative human verification system, keeps bots out while preserving the user experience.
Limit the rate of bots’ illicit behaviors
You can frustrate malicious bots by limiting the rate of their repetitive behaviors. Bots often make many login or payment attempts in a short amount of time. Rate limiting slows the process, so cybercriminals move to easier targets with quicker payoffs. However, using something like a web application firewall (WAF) to rate-limit traffic is not enough to block bots on its own. WAFs cannot detect bots that piggyback on real users’ identities and mirror their behavior, nor can they recognize botnets that rotate through thousands of different IP addresses to bypass IP-based rules.
Require proof of work
The proof of work (PoW) technique requires computational effort to be expended before logging in, verifying a payment method or executing another task. PoW consumes a large amount of energy and CPU cycles at scale, which places a cost burden on attackers trying to do many fraudulent activities at once. Because of this, PoW makes it expensive for hackers to complete their attacks and disincentivizes them from launching future attacks against your site.
Block the connection
Once you confirm malicious bot activity, you can block bot access using block pages, redirect the malicious traffic or block the internet address responsible for the bot traffic.
The Human Defense Platform offers a suite of bot management solutions that detect and mitigate bad bots with unparalleled accuracy. These include Account Takeover Defense, Transaction Abuse Defense, Scraping Defense, Account Fraud Defense, Programmatic Ad Fraud Defense, and Data Contamination Defense. The solution uses machine learning, behavioral analysis and predictive methods to protect against bot traffic to web and mobile apps and APIs. The algorithms evolve in real time, becoming more sophisticated as bots do.
By blocking unwanted bot traffic at the edge and optimizing the use and performance of your web infrastructure, HUMAN's bot management solutions preserve page load performance, and optimize security resources and infrastructure costs. The solution enables your team to focus on innovation and growth, instead of chasing down bad bot traffic.
What are Skewed Analytics and How to Avoid Them
What is Bot Detection? | How to Detect & Block Bad Bots
What is Bot Mitigation? | 4 Types of Bots & Botnets | How to Stop Bots
What is Fake Account Creation? | How to Prevent It
What is Scraping? Protection from Web Scraping & Data Scraping