I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Even when I'm just browsing manually just using chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I've checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal chrome browser.
When I browse to these sites in normal chrome everything works fine, but the moment I use Selenium I'm detected.
In theory chromedriver and chrome should look literally exactly the same to any webserver, but somehow they can detect it.
If you want some testcode try out this:
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--profile-directory=Default')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print 'arguments done'
driver.get('http://stubhub.com')
If you browse around stubhub you'll get redirected and 'blocked' within one or two requests. I've been investigating this and I can't figure out how they can tell that a user is using Selenium.
How do they do it?
EDIT UPDATE:
I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal firefox browser with only the additional plugin.
EDIT:
When I use Fiddler to view the HTTP requests being sent back and forth I've noticed that the 'fake browser\'s' requests often have 'no-cache' in the response header.
EDIT:
results like this Is there a way to detect that I'm in a Selenium Webdriver page from Javascript suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.
EDIT:
The site uploads a fingerprint to their servers, but I checked and the fingerprint of selenium is identical to the fingerprint when using chrome.
EDIT:
This is one of the fingerprint payloads that they send to their servers
{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionModule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContentDecryptionModuleapplication/x-ppapi-widevine-cdm","4":"NativeClientExecutableapplication/x-nacl","5":"PortableNativeClientExecutableapplication/x-pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"TrebuchetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationMono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}
Its identical in selenium and in chrome
EDIT:
VPNs work for a single use but get detected after I load the first page. Clearly some javascript is being run to detect Selenium.
http://stackoverflow.com/questions/3614472/is-there-a-way-to-detect-that-im-in-a-selenium-webdriver-page-from-javascrip - Ryan Weinstein 2015-10-21 19:43
distill
bot detection technology and delivers content using akamaitechnologies.com
CDN from diffrent ips e.g. 95.100.59.245
, 104.70.243.66
, 23.202.161.241
SIslam 2015-10-22 10:12
Replacing cdc_
variable using Vim or Perl
You can use vim
, or as @Vic Seedoubleyew has pointed out in the answer by @Erti-Chris Eelmaa, perl
, to replace the cdc_
variable in chromedriver
(See post by @Erti-Chris Eelmaa to learn more about that variable). Using vim
or perl
prevents you from having to recompile source code or use a hex-editor. Make sure to make a copy of the original chromedriver
before attempting to edit it. Also, the methods below were tested on chromedriver version 2.41.578706
.
vim /path/to/chromedriver
After running the line above, you'll probably see a bunch of gibberish. Do the following:
cdc_
by typing /cdc_
and pressing return
.a
.$cdc_lasutopfhvcZLmcfl
and replace what was deleted with an equal amount characters. If you don't, chromedriver
will fail.esc
.:wq!
and press return
.:q!
and press return
.Go to the altered chromedriver
and double click on it. A terminal
window should open up. If you don't see killed
in the output, you successfully altered the driver.
The line below replaces cdc_
with dog_
:
perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver
Make sure that the replacement string has the same number of characters as the search string, otherwise the chromedriver
will fail.
Perl Explanation
s///g
denotes that you want to search for a string and replace it globally with another string (replaces all occurrences).
e.g.,
s/string/replacment/g
So,
s///
denotes searching for and replacing a string.
cdc_
is the search string.
dog_
is the replacement string.
g
is the global key, which replaces every occurrence of the string.
How to check if the Perl replacement worked
The following line will print every occurrence of the search string cdc_
:
perl -ne 'while(/cdc_/g){print "$&\n";}' /path/to/chromedriver
If this returns nothing, then cdc_
has been replaced.
Conversely, you can use the this:
perl -ne 'while(/dog_/g){print "$&\n";}' /path/to/chromedriver
to see if your replacement string, dog_
, is now in the chromedriver
binary. If it is, the replacement string will be printed to the console.
Go to the altered chromedriver
and double click on it. A terminal
window should open up. If you don't see killed
in the output, you successfully altered the driver.
After altering the chromedriver
binary, make sure that the name of the altered chromedriver
binary is chromedriver
, and that the original binary is either moved from its original location or renamed.
I was previously being detected on a website while trying to log in, but after replacing cdc_
with an equal sized string, I was able to log in. Like others have said though, if you've already been detected, you might get blocked for a plethora of other reasons even after using this method. So you may have to try accessing the site that was detecting you using a VPN, different network, or what have you.
Basically the way the selenium detection works, is that they test for pre-defined javascript variables which appear when running with selenium. The bot detection scripts usually look anything containing word "selenium" / "webdriver" in any of the variables (on window object), and also document variables called $cdc_
and $wdc_
. Of course, all of this depends on which browser you are on. All the different browsers expose different things.
For me, I used chrome, so, all that I had to do was to ensure that $cdc_
didn't exist anymore as document variable, and voila (download chromedriver source code, modify chromedriver and re-compile $cdc_
under different name.)
this is the function I modified in chromedriver:
call_function.js:
function getPageCache(opt_doc) {
var doc = opt_doc || document;
//var key = '$cdc_asdjflasutopfhvcZLmcfl_';
var key = 'randomblabla_';
if (!(key in doc))
doc[key] = new Cache();
return doc[key];
}
(note the comment, all I did I turned $cdc_
to randomblabla_
.
Here is a pseudo-code which demonstrates some of the techniques that bot networks might use:
runBotDetection = function () {
var documentDetectionKeys = [
"__webdriver_evaluate",
"__selenium_evaluate",
"__webdriver_script_function",
"__webdriver_script_func",
"__webdriver_script_fn",
"__fxdriver_evaluate",
"__driver_unwrapped",
"__webdriver_unwrapped",
"__driver_evaluate",
"__selenium_unwrapped",
"__fxdriver_unwrapped",
];
var windowDetectionKeys = [
"_phantom",
"__nightmare",
"_selenium",
"callPhantom",
"callSelenium",
"_Selenium_IDE_Recorder",
];
for (const windowDetectionKey in windowDetectionKeys) {
const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
if (window[windowDetectionKeyValue]) {
return true;
}
};
for (const documentDetectionKey in documentDetectionKeys) {
const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
if (window['document'][documentDetectionKeyValue]) {
return true;
}
};
for (const documentKey in window['document']) {
if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
return true;
}
}
if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;
if (window['document']['documentElement']['getAttribute']('selenium')) return true;
if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
if (window['document']['documentElement']['getAttribute']('driver')) return true;
return false;
};
according to user @szx, it is also possible to simply open chromedriver.exe in hex editor, and just do the replacement manually, without actually doing any compiling.
cdc_
with a 4 character other string in the chromedriver binary. The way to do that is to use a similar perl command : perl -pi -e 's/cdc_/abc_/g' /path/to/chromedriver
. As soon as someone else confirms, feel free to edit the original answe - Vic Seedoubleyew 2018-01-30 21:24
$cdc
with xxxx
in chromedriver.exe
in a hex editor and it worked! I also noticed that if you maximize the browser window (rather than use a predefined size) it's detected less often - szx 2018-02-25 17:59
selenium.common.exceptions.WebDriverException: Message: Service /Users/ishandutta2007/Downloads/chromedriver2 unexpectedly exited. Status code was: -9
ishandutta2007 2018-08-23 03:38
Uncaught ReferenceError: $cdc_asdjflasutopfhvcZLmcfl_ is not defined
Nino Škopac 2019-01-31 05:36
false
every time (which is good I think) - Nino Škopac 2019-02-04 14:10
As we've already figured out in the question and the posted answers, there is an anti Web-scraping and a Bot detection service called "Distil Networks" in play here. And, according to the company CEO's interview:
Even though they can create new bots, we figured out a way to identify Selenium the a tool they’re using, so we’re blocking Selenium no matter how many times they iterate on that bot. We’re doing that now with Python and a lot of different technologies. Once we see a pattern emerge from one type of bot, then we work to reverse engineer the technology they use and identify it as malicious.
It'll take time and additional challenges to understand how exactly they are detecting Selenium, but what can we say for sure at the moment:
Decided to post it as an answer, since clearly:
Can a website detect when you are using selenium with chromedriver?
Yes.
Also, what I haven't experimented with is older selenium and older browser versions - in theory, there could be something implemented/added to selenium at a certain point that Distil Networks bot detector currently relies on. Then, if this is the case, we might detect (yeah, let's detect the detector) at what point/version a relevant change was made, look into changelog and changesets and, may be, this could give us more information on where to look and what is it they use to detect a webdriver-powered browser. It's just a theory that needs to be tested.
Example of how it's implemented on wellsfargo.com:
try {
if (window.document.documentElement.getAttribute("webdriver")) return !+[]
} catch (IDLMrxxel) {}
try {
if ("_Selenium_IDE_Recorder" in window) return !+""
} catch (KknKsUayS) {}
try {
if ("__webdriver_script_fn" in document) return !+""
Try to use selenium with a specific user profile of chrome, That way you can use it as specific user and define any thing you want, When doing so it will run as a 'real' user, look at chrome process with some process explorer and you'll see the difference with the tags.
For example:
username = os.getenv("USERNAME")
userProfile = "C:\\Users\\" + username + "\\AppData\\Local\\Google\\Chrome\\User Data\\Default"
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir={}".format(userProfile))
# add here any tag you want.
options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors", "safebrowsing-disable-download-protection", "safebrowsing-disable-auto-update", "disable-client-side-phishing-detection"])
chromedriver = "C:\Python27\chromedriver\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = chromedriver
browser = webdriver.Chrome(executable_path=chromedriver, chrome_options=options)
chrome tag list here
partial interface Navigator { readonly attribute boolean webdriver; };
The webdriver IDL attribute of the Navigator interface must return the value of the webdriver-active flag, which is initially false.
This property allows websites to determine that the user agent is under control by WebDriver, and can be used to help mitigate denial-of-service attacks.
Taken directly from the 2017 W3C Editor's Draft of WebDriver. This heavily implies that at the very least, future iterations of selenium's drivers will be identifiable to prevent misuse. Ultimately, it's hard to tell without the source code, what exactly causes chrome driver in specific to be detectable.
I have checked the chromedriver source code. That injects some javascript files to the browser.
Every javascript file on this link is injected to the web pages:
https://chromium.googlesource.com/chromium/src/+/master/chrome/test/chromedriver/js/
So I used reverse engineering and obfuscated the js files by Hex editing. Now i was sure that no more javascript variable, function names and fixed strings were used to uncover selenium activity. But still some sites and reCaptcha detect selenium!
Maybe they check the modifications that are caused by chromedriver js execution :)
Edit 1:
I discovered there are some parameters in 'navigator' that briefly uncover using of chromedriver. These are the parameters:
So what i needed was a chrome extension to run javascript on the web pages. I made an extension with the js code provided in the article and used another article to add the zipped extension to my project. I have successfully changed the values; But still nothing changed!
I didn't find other variables like these but it doesn't mean that they don't exist. Still reCaptcha detects chromedriver, So there should be more variables to change. The next step should be reverse engineering of the detector services that i don't want to do.
Now I'm not sure does it worth to spend more time on this automation process or search for alternative methods!
It sounds like they are behind a web application firewall. Take a look at modsecurity and owasp to see how those work. In reality, what you are asking is how to do bot detection evasion. That is not what selenium web driver is for. It is for testing your web application not hitting other web applications. It is possible, but basically, you'd have to look at what a WAF looks for in their rule set and specifically avoid it with selenium if you can. Even then, it might still not work because you don't know what WAF they are using. You did the right first step, that is faking the user agent. If that didn't work though, then a WAF is in place and you probably need to get more tricky.
Edit: Point taken from other answer. Make sure your user agent is actually being set correctly first. Maybe have it hit a local web server or sniff the traffic going out.
driver.set_preference("general.useragent.override","your_user_agent_string")
Nathan Bellowe 2015-10-25 21:40
Even if you are sending all the right data (e.g. Selenium doesn't show up as an extension, you have a reasonable resolution/bit-depth, &c), there are a number of services and tools which profile visitor behaviour to determine whether the actor is a user or an automated system.
For example, visiting a site then immediately going to perform some action by moving the mouse directly to the relevant button, in less than a second, is something no user would actually do.
It might also be useful as a debugging tool to use a site such as https://panopticlick.eff.org/ to check how unique your browser is; it'll also help you verify whether there are any specific parameters that indicate you're running in Selenium.
Firefox is said to set window.navigator.webdriver === true
if working with a webdriver. That was according to one of the older specs (e.g.: archive.org) but I couldn't find it in the new one except for some very vague wording in the appendices.
A test for it is in the selenium code in the file fingerprint_test.js where the comment at the end says "Currently only implemented in firefox" but I wasn't able to identify any code in that direction with some simple grep
ing, neither in the current (41.0.2) Firefox release-tree nor in the Chromium-tree.
I also found a comment for an older commit regarding fingerprinting in the firefox driver b82512999938 from January 2015. That code is still in the Selenium GIT-master downloaded yesterday at javascript/firefox-driver/extension/content/server.js
with a comment linking to the slightly differently worded appendix in the current w3c webdriver spec.
window.navigator.webdriver
is not defined - speedplane 2017-10-02 17:56
The bot detection I've seen seems more sophisticated or at least different than what I've read through in the answers below.
EXPERIMENT 1:
EXPERIMENT 2:
As before, I open a browser and the web page with Selenium from a Python console.
This time around, instead of clicking with the mouse, I use Selenium (in the Python console) to click the same element with a random offset.
The link doesn't open, but I am taken to a sign up page.
IMPLICATIONS:
Seems mysterious, but I guess they can just determine whether an action originates from Selenium or not, while they don't care whether the browser itself was opened via Selenium or not. Or can they determine if the window has focus? Would be interesting to hear if anyone has any insights.
Write an html page with the following code. You will see that in the DOM selenium applies a webdriver attribute in the outerHTML
<html>
<head>
<script type="text/javascript">
<!--
function showWindow(){
javascript:(alert(document.documentElement.outerHTML));
}
//-->
</script>
</head>
<body>
<form>
<input type="button" value="Show outerHTML" onclick="showWindow()">
</form>
</body>
</html>
Some sites are detecting this:
function d() {
try {
if (window.document.$cdc_asdjflasutopfhvcZLmcfl_.cache_)
return !0
} catch (e) {}
try {
//if (window.document.documentElement.getAttribute(decodeURIComponent("%77%65%62%64%72%69%76%65%72")))
if (window.document.documentElement.getAttribute("webdriver"))
return !0
} catch (e) {}
try {
//if (decodeURIComponent("%5F%53%65%6C%65%6E%69%75%6D%5F%49%44%45%5F%52%65%63%6F%72%64%65%72") in window)
if ("_Selenium_IDE_Recorder" in window)
return !0
} catch (e) {}
try {
//if (decodeURIComponent("%5F%5F%77%65%62%64%72%69%76%65%72%5F%73%63%72%69%70%74%5F%66%6E") in document)
if ("__webdriver_script_fn" in document)
return !0
} catch (e) {}
It seems to me the simplest way to do it with Selenium is to intercept the XHR that sends back the browser fingerprint.
But since this is a Selenium-only problem, its better just to use something else. Selenium is supposed to make things like this easier, not way harder.
Additionally to the great answer of @Erti-Chris Eelmaa - there's annoying window.navigator.webdriver
and it is read-only. Event if you change the value of it to false
it will still have true
. Thats why the browser driven by automated software can still be detected.
MDN
The variable is managed by the flag --enable-automation
in chrome. The chromedriver launches chrome with that flag and chrome sets the window.navigator.webdriver
to true
. You can find it here. You need to add to "exclude switches" the flag. For instance (golang):
package main
import (
"github.com/tebeka/selenium"
"github.com/tebeka/selenium/chrome"
)
func main() {
caps := selenium.Capabilities{
"browserName": "chrome",
}
chromeCaps := chrome.Capabilities{
Path: "/path/to/chrome-binary",
ExcludeSwitches: []string{"enable-automation"},
}
caps.AddChrome(chromeCaps)
wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", 4444))
}