Can a website detect when you are using selenium with chromedriver?

Go To StackoverFlow.com

225

I've been testing out Selenium with Chromedriver and I noticed that some pages can detect that you're using Selenium even though there's no automation at all. Even when I'm just browsing manually just using chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected. I've checked my user agent, and my browser fingerprint, and they are all exactly identical to the normal chrome browser.

When I browse to these sites in normal chrome everything works fine, but the moment I use Selenium I'm detected.

In theory chromedriver and chrome should look literally exactly the same to any webserver, but somehow they can detect it.

If you want some testcode try out this:

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=1, size=(1600, 902))
display.start()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--profile-directory=Default')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--disable-plugins-discovery");
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.delete_all_cookies()
driver.set_window_size(800,800)
driver.set_window_position(0,0)
print 'arguments done'
driver.get('http://stubhub.com')

If you browse around stubhub you'll get redirected and 'blocked' within one or two requests. I've been investigating this and I can't figure out how they can tell that a user is using Selenium.

How do they do it?

EDIT UPDATE:

I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub.com in the normal firefox browser with only the additional plugin.

EDIT:

When I use Fiddler to view the HTTP requests being sent back and forth I've noticed that the 'fake browser\'s' requests often have 'no-cache' in the response header.

EDIT:

results like this Is there a way to detect that I'm in a Selenium Webdriver page from Javascript suggest that there should be no way to detect when you are using a webdriver. But this evidence suggests otherwise.

EDIT:

The site uploads a fingerprint to their servers, but I checked and the fingerprint of selenium is identical to the fingerprint when using chrome.

EDIT:

This is one of the fingerprint payloads that they send to their servers

{"appName":"Netscape","platform":"Linuxx86_64","cookies":1,"syslang":"en-US","userlang":"en-US","cpu":"","productSub":"20030107","setTimeout":1,"setInterval":1,"plugins":{"0":"ChromePDFViewer","1":"ShockwaveFlash","2":"WidevineContentDecryptionModule","3":"NativeClient","4":"ChromePDFViewer"},"mimeTypes":{"0":"application/pdf","1":"ShockwaveFlashapplication/x-shockwave-flash","2":"FutureSplashPlayerapplication/futuresplash","3":"WidevineContentDecryptionModuleapplication/x-ppapi-widevine-cdm","4":"NativeClientExecutableapplication/x-nacl","5":"PortableNativeClientExecutableapplication/x-pnacl","6":"PortableDocumentFormatapplication/x-google-chrome-pdf"},"screen":{"width":1600,"height":900,"colorDepth":24},"fonts":{"0":"monospace","1":"DejaVuSerif","2":"Georgia","3":"DejaVuSans","4":"TrebuchetMS","5":"Verdana","6":"AndaleMono","7":"DejaVuSansMono","8":"LiberationMono","9":"NimbusMonoL","10":"CourierNew","11":"Courier"}}

Its identical in selenium and in chrome

EDIT:

VPNs work for a single use but get detected after I load the first page. Clearly some javascript is being run to detect Selenium.

2015-10-20 00:08
by Ryan Weinstein
I would suggest using a local proxy to take a look at the web traffic going from your request to the server and back. You should be able to tell from there. fiddler, burp, ZED proxy any of them will do the trick. I prefer Bur - iNoob 2015-10-20 00:43
I did this and there's no difference between the requests at all. How are they possibly doing this - Ryan Weinstein 2015-10-21 00:41
@RyanWeinstein: It is not traffic. My guess is that Selenium needs to expose some JavaScript hooks which can be detected on the client-side JavaScript - Mikko Ohtamaa 2015-10-21 19:04
Or if it is traffic then it is a traffic pattern.... you are browsing pages too fast - Mikko Ohtamaa 2015-10-21 19:06
I'm not browsing too fast. I only load a single page and I navigate through it normally using my mouse and keyboard. Also it doesn't make sense that Selenium needs to expose hooks, because its literally running chrome.exe. It just runs normal chrome and allows you to get data from it. Any other ideas? I was thinking maybe it has something to do with cookies. This is driving me crazy - Ryan Weinstein 2015-10-21 19:12
Mikko's idea sounds pretty plausible to me. Selenium needs to be able to detect events so that it can respond to them. So it must be injecting some javascript code which is detected even when you're not actually automating anything - ArtOfWarfare 2015-10-21 19:38
Its possible, but if that's the case then it would go against everything that anyone knows about Selenium. There is nowhere online where anyone knows a way to detect if a user is using chromedriver. So if there's a way to detect it using frontend javascript then I want to know what it is.

http://stackoverflow.com/questions/3614472/is-there-a-way-to-detect-that-im-in-a-selenium-webdriver-page-from-javascrip - Ryan Weinstein 2015-10-21 19:43

Can you post the request headers (except the sensitive info, of course)? That's one... There's also probably some user interaction that the site is expecting which is not happening - CKmum 2015-10-22 05:33
This site uses distill bot detection technology and delivers content using akamaitechnologies.com CDN from diffrent ips e.g. 95.100.59.245 , 104.70.243.66 , 23.202.161.241SIslam 2015-10-22 10:12
Same happens to http://www.411.com - SIslam 2015-10-22 13:31
I am able to hit stubhub.com and 411.com with chromedriver/selenium and your listed settings. Reading on distil from SIslam it appears as though they are keeping a fingerprint in memory, which is probably triggered based on behaviour - ryanmc 2015-10-23 20:18
The fingerprint contains some browser information mostly. You say you are able to go to stubhub and navigate around in chromedriver? Are you sure you're using exactly my settings? I can't navigate around using chromedriver so there must be some difference. @ryanm - Ryan Weinstein 2015-10-23 20:43
I am experiencing the same issue with Selenium and the firefox driver. The interesting thing to note is I am running Selenium in a VMWare Workstation Virtual Machine that is accessing the internet through a NAT. The host machine is able to access stubhub, while the VM is unable to access when using Selenium, or even the browser instance Selenium launched. I had the VM Browser instance Blocked and stubhub still recognizes the machine and has it blocked. So it must be performing a fingerprint of the browser and machine in some manner - Brian Cain 2015-10-23 21:34
@RyanWeinstein Yes fp is generated from about 40 variables that encompasses client and network info! What i read in distill - SIslam 2015-10-25 13:16
Without using pyvirtualdisplay, I can connect to stubhub.com and browse around just fine; I don't get blocke - cwa 2015-10-27 21:00
@cwa: Can you post your code? Noone else here seems to have gotten it to work - iceui2 2015-12-19 13:46
Did you try to connect via remote debugging instead of starting a new instance ? https://sites.google.com/a/chromium.org/chromedriver/help/operation-not-supported-when-using-remote-debugging The webdriver will be invisible for the website - Bebeoix 2016-09-28 15:14
I added an answer how to bypass disti - Erti-Chris Eelmaa 2016-12-19 10:15
Case of me it is working. I can open stubhub.com with Selenium and navigate items later - erm3nda 2017-06-13 00:13
You can use the 'webbot' library for much dynamic behaviour with lot more options than selenium - Natesh bhat 2018-09-21 07:31


10

For Mac Users

Replacing cdc_ variable using Vim or Perl

You can use vim, or as @Vic Seedoubleyew has pointed out in the answer by @Erti-Chris Eelmaa, perl, to replace the cdc_ variable in chromedriver(See post by @Erti-Chris Eelmaa to learn more about that variable). Using vim or perl prevents you from having to recompile source code or use a hex-editor. Make sure to make a copy of the original chromedriver before attempting to edit it. Also, the methods below were tested on chromedriver version 2.41.578706.


Using Vim

vim /path/to/chromedriver

After running the line above, you'll probably see a bunch of gibberish. Do the following:

  1. Search for cdc_ by typing /cdc_ and pressing return.
  2. Enable editing by pressing a.
  3. Delete any amount of $cdc_lasutopfhvcZLmcfl and replace what was deleted with an equal amount characters. If you don't, chromedriver will fail.
  4. After you're done editing, press esc.
  5. To save the changes and quit, type :wq! and press return.
  6. If you don't want to save the changes, but you want to quit, type :q! and press return.
  7. You're done.

Go to the altered chromedriver and double click on it. A terminal window should open up. If you don't see killed in the output, you successfully altered the driver.


Using Perl

The line below replaces cdc_ with dog_:

perl -pi -e 's/cdc_/dog_/g' /path/to/chromedriver

Make sure that the replacement string has the same number of characters as the search string, otherwise the chromedriver will fail.

Perl Explanation

s///g denotes that you want to search for a string and replace it globally with another string (replaces all occurrences).

e.g., s/string/replacment/g

So,

s/// denotes searching for and replacing a string.

cdc_ is the search string.

dog_ is the replacement string.

g is the global key, which replaces every occurrence of the string.

How to check if the Perl replacement worked

The following line will print every occurrence of the search string cdc_:

perl -ne 'while(/cdc_/g){print "$&\n";}' /path/to/chromedriver

If this returns nothing, then cdc_ has been replaced.

Conversely, you can use the this:

perl -ne 'while(/dog_/g){print "$&\n";}' /path/to/chromedriver

to see if your replacement string, dog_, is now in the chromedriver binary. If it is, the replacement string will be printed to the console.

Go to the altered chromedriver and double click on it. A terminal window should open up. If you don't see killed in the output, you successfully altered the driver.


Wrapping Up

After altering the chromedriver binary, make sure that the name of the altered chromedriver binary is chromedriver, and that the original binary is either moved from its original location or renamed.


My Experience With This Method

I was previously being detected on a website while trying to log in, but after replacing cdc_ with an equal sized string, I was able to log in. Like others have said though, if you've already been detected, you might get blocked for a plethora of other reasons even after using this method. So you may have to try accessing the site that was detecting you using a VPN, different network, or what have you.

2018-08-31 03:49
by colossatr0n
this doesn't work with current chromedriver versio - Leka Baper 2018-11-09 21:06
@LekaBaper Thanks for the heads up. The chromedriver version that I used was version 2.41.578706 - colossatr0n 2018-11-12 04:47


107

Basically the way the selenium detection works, is that they test for pre-defined javascript variables which appear when running with selenium. The bot detection scripts usually look anything containing word "selenium" / "webdriver" in any of the variables (on window object), and also document variables called $cdc_ and $wdc_. Of course, all of this depends on which browser you are on. All the different browsers expose different things.

For me, I used chrome, so, all that I had to do was to ensure that $cdc_ didn't exist anymore as document variable, and voila (download chromedriver source code, modify chromedriver and re-compile $cdc_ under different name.)

this is the function I modified in chromedriver:

call_function.js:

function getPageCache(opt_doc) {
  var doc = opt_doc || document;
  //var key = '$cdc_asdjflasutopfhvcZLmcfl_';
  var key = 'randomblabla_';
  if (!(key in doc))
    doc[key] = new Cache();
  return doc[key];
}

(note the comment, all I did I turned $cdc_ to randomblabla_.

Here is a pseudo-code which demonstrates some of the techniques that bot networks might use:

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window['document'][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window['document']) {
        if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
            return true;
        }
    }

    if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;

    if (window['document']['documentElement']['getAttribute']('selenium')) return true;
    if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
    if (window['document']['documentElement']['getAttribute']('driver')) return true;

    return false;
};

according to user @szx, it is also possible to simply open chromedriver.exe in hex editor, and just do the replacement manually, without actually doing any compiling.

2016-12-19 10:14
by Erti-Chris Eelmaa
This is interesting, you tested it and it worked even across multiple requests - Ryan Weinstein 2016-12-19 17:43
yes it worked without probs, note one problem is if you fell into the "blacklist" BEFORE this change, it's quite hard to get out. if you want to get out of the existing black list, you need to implement fake canvas fingerprinting, disable flash, change IP, and change request header order (swap language and Accept headers). Once you fell into the blacklist, they have very good measures to track you, even if you change IP, even if you open chrome in incognito, et - Erti-Chris Eelmaa 2016-12-19 19:23
This is very interesting, thanks for going to the trouble. Its been more than a year since I asked this question and its nice to finally have an answer. I'll see if I can pass you the check-mark once I've tested the solution myself - Ryan Weinstein 2016-12-19 22:41
Do you know if there is a similar function within geckodriver for Firefox - greedybuddha 2017-03-02 09:16
I only did it with chrome, however I would not be surprised - Erti-Chris Eelmaa 2017-03-03 12:34
Just tested and it worked. Be prepared that at least one hour is needed though, to clone chromium repository and build it.. - Tiexin Guo 2017-05-24 11:49
I cannot find call_function.js in selenium sources at all - Makalele 2017-06-08 21:12
Ok, looks like this file is in chromium webdriver source. But in my case it's var key = '$cdcasdjflasutopfhvcZLmcfl'; right on the start - Makalele 2017-06-09 20:34
Makalele: I clarified: change the $cdc_ part to something completely random, eg change it to 'MAKALELE_ - Erti-Chris Eelmaa 2017-06-09 21:14
What is the most straight forward way to compile chromedriver on Windows - Arya 2017-06-23 19:34
Hey there @Erti-ChrisEelmaa can you answer this question? https://stackoverflow.com/questions/44720193/how-to-build-chromedriver-from-source-in-ubuntu64-bi - hungryWolf 2017-06-24 10:06
@Arya https://chromium.googlesource.com/chromium/src/+/master/docs/windowsbuildinstructions.m - Abhigyan 2017-06-26 04:03
@Abhigyan Thanks. After I follow the instructions (until "fetch chromium") in the url, I can't find "/js" or "/js/callfunction.js" file from "chromium/src/" to remove such "$cdc". Do you know where it is located - JonghoKim 2017-07-01 01:06
I found the file "/Users/your_username/chromium/src/chrome/test/chromedriver/js - JonghoKim 2017-07-01 09:08
@JonghoKim are you compiling it for mac - Abhigyan 2017-07-01 17:29
Yes. I did it. But I guess, I am already in blacklist. So I can"t check whether it works or not. Currently, I am trying to use Tor and AWS Services to run my code - JonghoKim 2017-07-01 17:35
In my case the webdriver is apparently still detected, even though this $cdc_ variable had its name changed. Any other suggestions on what can be done to make Chrome look "100% natural" - Dennis Thrysøe 2017-11-18 14:00
@DennisThrysøe did you see the later part of my post - Erti-Chris Eelmaa 2017-11-21 08:24
@Erti-ChrisEelmaa: Yes, none of these seem to be present - Dennis Thrysøe 2017-11-22 20:17
compile older version of Chromium. See when my post was written and take chrome version that was about then roughly. A lot can change in a year - Erti-Chris Eelmaa 2017-11-23 11:47
How can I implement this solution in .Net - DarioN1 2018-01-05 15:31
There is a simple, and automatable way to do that, which is to replace cdc_ with a 4 character other string in the chromedriver binary. The way to do that is to use a similar perl command : perl -pi -e 's/cdc_/abc_/g' /path/to/chromedriver. As soon as someone else confirms, feel free to edit the original answe - Vic Seedoubleyew 2018-01-30 21:24
I also find that there is only "cdc" to replace on Mac and Linux 64, and on Linux 32 one should replace both "wdc" and "selenium". Again as soon as someone confirms I'll add that to the pos - Vic Seedoubleyew 2018-02-01 23:36
I simply replaced $cdc with xxxx in chromedriver.exe in a hex editor and it worked! I also noticed that if you maximize the browser window (rather than use a predefined size) it's detected less often - szx 2018-02-25 17:59
@szx nice job, much better way. compiling chromedriver is nightmare - Erti-Chris Eelmaa 2018-02-26 10:39
Hi! I read all of this. I have same problem, but some kind different: when I builded my application, errors happened: restart, rewrite, etc untill app cours was ok. So...(I have to mention that I am completelly 0 at cookies knowledges) the site gave me a worning notification about abusive deletting of cookies. I can't find any reference about webdrive deletting cookies, but I guess at the end of execution all of them are deletted, right? So a site can become angry about this...so...what should be done about this - Botea Florin 2018-03-02 09:39
@szx What version have you tried this on - Rafael Almeida 2018-06-12 10:21
@RafaelAlmeida ChromeDriver 2.35.52813 - szx 2018-06-15 17:21
was this on windows, osx, or linux? Hex editing on osx doesn't seem to work - Nish 2018-07-23 22:28
hex-editted with $zzzzzzzzzzzzzzzzzzzzzzzzz (same amount of characters) but didn't work - Aymon Fournier 2018-08-09 20:54
anyone having problems take my JS code that detects bots and upload it to somewhere, then visit that page and see what is being detected - Erti-Chris Eelmaa 2018-08-10 07:25
@Nish on mac I get selenium.common.exceptions.WebDriverException: Message: Service /Users/ishandutta2007/Downloads/chromedriver2 unexpectedly exited. Status code was: -9ishandutta2007 2018-08-23 03:38
@JonghoKim can you give the steps for OSX - ishandutta2007 2018-08-23 03:39
@ JonghoKim nevermind I got the step, can you share a modified chromedriver for mac. It's taking hours to buil - ishandutta2007 2018-08-24 08:28
Tried this with windows 2.44 exe version, only one place has 'cdc_', and replace it seems not working, still can be detected by websites - Tiw 2018-12-08 06:28
Replacing $cdc_ worked for a couple of times, then got blocked again. It happens only on Linux though, in windows my script works fine. any other solution alongside this one will be of great help - rajkaran singh 2019-01-24 01:47
I'm assuming that Selenium/Chrome71 no longer exposes these vars - I've just tried putting them into the console and they didn't exist: Uncaught ReferenceError: $cdc_asdjflasutopfhvcZLmcfl_ is not definedNino Škopac 2019-01-31 05:36
@NinoŠkopac you should be trying the example of "how bot networks might use to detect" script I gave at the bottom of the post. the suffix after cdc_ is not constant, and changes between builds. The CDC variable also doesn't appear immediately, it appears when you've been browsing a little on web - Erti-Chris Eelmaa 2019-01-31 22:47
I just tried executing your function in Chrome 72 via BrowserStack - it returned false every time (which is good I think) - Nino Škopac 2019-02-04 14:10


71

As we've already figured out in the question and the posted answers, there is an anti Web-scraping and a Bot detection service called "Distil Networks" in play here. And, according to the company CEO's interview:

Even though they can create new bots, we figured out a way to identify Selenium the a tool they’re using, so we’re blocking Selenium no matter how many times they iterate on that bot. We’re doing that now with Python and a lot of different technologies. Once we see a pattern emerge from one type of bot, then we work to reverse engineer the technology they use and identify it as malicious.

It'll take time and additional challenges to understand how exactly they are detecting Selenium, but what can we say for sure at the moment:

  • it's not related to the actions you take with selenium - once you navigate to the site, you get immediately detected and banned. I've tried to add artificial random delays between actions, take a pause after the page is loaded - nothing helped
  • it's not about browser fingerprint either - tried it in multiple browsers with clean profiles and not, incognito modes - nothing helped
  • since, according to the hint in the interview, this was "reverse engineering", I suspect this is done with some JS code being executed in the browser revealing that this is a browser automated via selenium webdriver

Decided to post it as an answer, since clearly:

Can a website detect when you are using selenium with chromedriver?

Yes.


Also, what I haven't experimented with is older selenium and older browser versions - in theory, there could be something implemented/added to selenium at a certain point that Distil Networks bot detector currently relies on. Then, if this is the case, we might detect (yeah, let's detect the detector) at what point/version a relevant change was made, look into changelog and changesets and, may be, this could give us more information on where to look and what is it they use to detect a webdriver-powered browser. It's just a theory that needs to be tested.

2015-10-28 23:39
by alecxe
This is crazy. So they really have a way of detecting it that no one else has. I really want to figure out how they're doing it. Can you provide any other information at all as to how they could possibly be doing it - Ryan Weinstein 2015-10-29 20:50
@RyanWeinstein well, we have no actual proof and we can only speculate and test. For now, I would say they have a way to detect us using selenium. Try experimenting with selenium versions - this may give you some clues - alecxe 2015-10-29 22:19
Could it have to do with how ephemeral ports are determined? The method stays away from well-known ranges. https://github.com/SeleniumHQ/selenium/blob/ab1e647d0fc8fc39e6b00ae94321ab228b6728f2/java/client/src/org/openqa/selenium/net/PortProber.java#L8 - Elliott 2016-01-12 22:12
Easyjet are using distilnetwork service, yeah it can block dummy bots but not the complicated ones because we have tested it with more than 2000 requests a day from different IPs (which we re-use again 'same' address) so basicly each IP go for a 5-10 requests a day and from this I can tell that all this bot detecting services are just there to develop and sell some 45% working algorithmes, the scrapper we used was easy to detect I can block it while destilnetworks, squareshield and others couldn't which pushed me to never use any of them - Jeffery ThaGintoki 2017-02-20 18:16
@alecxe Any updates on this? Still can't use Selenium with Distil - Utku 2018-05-21 15:17
I think they are detecting navigator.webdriver in chrome webdriver. I tried to make navigator.webdriver = false with the help of https://intoli.com/blog/not-possible-to-block-chrome-headless/ and https://stackoverflow.com/questions/47297877/to-set-mutationobserver-how-to-inject-javascript-before-page-loading-using-sele. It returns a bot detect page instead of https://www.distilnetworks.com/distilidentifycookie.htm - hoozecn 2018-09-13 03:57


18

Example of how it's implemented on wellsfargo.com:

try {
 if (window.document.documentElement.getAttribute("webdriver")) return !+[]
} catch (IDLMrxxel) {}
try {
 if ("_Selenium_IDE_Recorder" in window) return !+""
} catch (KknKsUayS) {}
try {
 if ("__webdriver_script_fn" in document) return !+""
2016-09-11 23:21
by aianitro
why is the last try not closed ? besides can u explain your answer a little - ishandutta2007 2018-08-22 16:42


9

Try to use selenium with a specific user profile of chrome, That way you can use it as specific user and define any thing you want, When doing so it will run as a 'real' user, look at chrome process with some process explorer and you'll see the difference with the tags.

For example:

username = os.getenv("USERNAME")
userProfile = "C:\\Users\\" + username + "\\AppData\\Local\\Google\\Chrome\\User Data\\Default"
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir={}".format(userProfile))
# add here any tag you want.
options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors", "safebrowsing-disable-download-protection", "safebrowsing-disable-auto-update", "disable-client-side-phishing-detection"])
chromedriver = "C:\Python27\chromedriver\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = chromedriver
browser = webdriver.Chrome(executable_path=chromedriver, chrome_options=options)

chrome tag list here

2015-10-28 16:39
by Kobi K
this is nonsens - Corey Goldberg 2019-02-27 01:31


9

partial interface Navigator { readonly attribute boolean webdriver; };

The webdriver IDL attribute of the Navigator interface must return the value of the webdriver-active flag, which is initially false.

This property allows websites to determine that the user agent is under control by WebDriver, and can be used to help mitigate denial-of-service attacks.

Taken directly from the 2017 W3C Editor's Draft of WebDriver. This heavily implies that at the very least, future iterations of selenium's drivers will be identifiable to prevent misuse. Ultimately, it's hard to tell without the source code, what exactly causes chrome driver in specific to be detectable.

2017-01-27 23:05
by bryce
"it's hard to tell without the source code" .. well the source code is freely availabl - Corey Goldberg 2017-11-27 16:08
I meant without the website in question's source code. It's hard to tell what they are checking against - bryce 2018-03-19 21:12


7

Obfuscating JavaScripts result

I have checked the chromedriver source code. That injects some javascript files to the browser.
Every javascript file on this link is injected to the web pages: https://chromium.googlesource.com/chromium/src/+/master/chrome/test/chromedriver/js/

So I used reverse engineering and obfuscated the js files by Hex editing. Now i was sure that no more javascript variable, function names and fixed strings were used to uncover selenium activity. But still some sites and reCaptcha detect selenium!
Maybe they check the modifications that are caused by chromedriver js execution :)


Edit 1:

Chrome 'navigator' parameters modification

I discovered there are some parameters in 'navigator' that briefly uncover using of chromedriver. These are the parameters:

  • "navigator.webdriver" On non-automated mode it is 'undefined'. On automated mode it's 'true'.
  • "navigator.plugins" On headless chrome has 0 length. So I added some fake elements to fool the plugin length checking process.
  • "navigator.languages" was set to default chrome value '["en-US", "en", "es"]' .

So what i needed was a chrome extension to run javascript on the web pages. I made an extension with the js code provided in the article and used another article to add the zipped extension to my project. I have successfully changed the values; But still nothing changed!

I didn't find other variables like these but it doesn't mean that they don't exist. Still reCaptcha detects chromedriver, So there should be more variables to change. The next step should be reverse engineering of the detector services that i don't want to do.

Now I'm not sure does it worth to spend more time on this automation process or search for alternative methods!

2018-12-05 12:56
by ShayanKM


5

It sounds like they are behind a web application firewall. Take a look at modsecurity and owasp to see how those work. In reality, what you are asking is how to do bot detection evasion. That is not what selenium web driver is for. It is for testing your web application not hitting other web applications. It is possible, but basically, you'd have to look at what a WAF looks for in their rule set and specifically avoid it with selenium if you can. Even then, it might still not work because you don't know what WAF they are using. You did the right first step, that is faking the user agent. If that didn't work though, then a WAF is in place and you probably need to get more tricky.

Edit: Point taken from other answer. Make sure your user agent is actually being set correctly first. Maybe have it hit a local web server or sniff the traffic going out.

2015-10-23 23:28
by Bassel Samman
I think you are on the correct path. I tested with my setup and replaced the User Agent with a valid user agent string that successfully went through and received the same result, stubhub blocked the request - Brian Cain 2015-10-23 23:36
Okay, if the user agent is fine, then they have attack detection in place for sure. WAF is a good place to start. Not that I'm condoning hitting other websites. I'm just answering in the name of science and advancement of human knowledge - Bassel Samman 2015-10-23 23:49
Try adding:

driver.set_preference("general.useragent.override","your_user_agent_string")Nathan Bellowe 2015-10-25 21:40

This is purely for science. I want to know how its possible they are detecting something that should theoretically be undetectable. Maybe they have a small backdoor in Selenium that does a small obscure js thing that they can use to detect it - Ryan Weinstein 2015-10-26 04:50
There are a lot of ways to detect if a user is a bot or is possibly malicious in general. It has nothing to do with selenium itself. In this case, selenium is acting suspiciously. One way to detect bad behavior or suspicious behavior is by checking behavior heuristics. For example, how fast a user is consuming http requests. If they are hitting the same url from the same ip over and over again he could be attempting a Denial of Service attack. Etc.. - Bassel Samman 2015-10-26 05:07
This topic is very vast, I would say if you don't understand it, and you want to understand it, here is not the right place. Start with owasp. Look into penetration testing and web security. Also, like I said before, look into modsecurity and WAF for specifically this topic - Bassel Samman 2015-10-26 05:13
Yeah but I want to know how they do it in this instance. I'm not flooding them with requests. There isn't any suspicious HTTP traffic. Its just me using a browser to go on their website, and somehow they know its Selenium. In this case, Selenium is acting suspiciously and I need to know why - Ryan Weinstein 2015-10-26 17:28
Okay, so again there is no simple and sweet answer to give, but checkout this article https://blogs.akamai.com/2013/06/identifying-and-mitigating-unwanted-bot-traffic.htm - Bassel Samman 2015-10-26 17:32
One thing to consider is are all your http headers being set correctly. Headers are a big thing that a browser takes care of for you. Are you also setting those. That is another type of low hanging fruit you can try. Since you don't know what is actually triggering the blocking, you have to do it by trial and error. They also sometimes check if javascript is active, how the client responds to http header changes, some even track click patterns. It really depends on the type of security they have in place - Bassel Samman 2015-10-26 17:40
Finally, if you want the literal rules you are probably looking at evading, go back to my original hint. Look at mod_security. Here are the base rules that pretty much any security company has to cover to be even respectable IMHO. They are the owasp core rule set and were the target of my original hint, but I think you didn't catch my drift: https://github.com/SpiderLabs/owasp-modsecurity-cr - Bassel Samman 2015-10-26 18:00
If it was an HTTP header issue then wouldn't the normal browser get blocked? The HTTP headers are exactly the same. Also what exactly am I looking at with that github link? Have you tried using selenium to go on stubhub? Something is very very off - Ryan Weinstein 2015-10-26 21:15
Look at http://builtwith.com/stubhub.com it tells you what they are using. They are using Akamai which uses the mod_security core rule set. Again, what I hinted at. If you don't know what you are looking at even and are not interested in learning, then you probably should just stop trying. Honestly, this is becoming border line spoon fooding. I don't know what you are trying to do, so it might even be borderline illegal. The github link literally tells you what rules they are using. If you don't know how to read the rules, then you probably can't figure out how to bypass them - Bassel Samman 2015-10-26 21:33
I will not find the exact rule that is blocking you and tell you how to avoid it. That is way beyond the ethical line. I can help you understand why something is happening and point you in the direction of greater understanding, but it is unethical for me to help you abuse a web application. Their security is there for a reason and you are trying to bypass it. If you were hitting your own application and you gave me access, I would probably be a lot more helpful - Bassel Samman 2015-10-26 21:53
I'm sorry for the confusion. I'll look into that and you don't have to help me anymore if you don't want to. Most of my experience is in programming systems applications, so I was not familiar with these modsecurity rules that you're talking about. I'll take a look and try to educate myself. I'm not trying to bypass anything, I was just interested in knowing how these websites detect a user using selenium - Ryan Weinstein 2015-10-27 18:49
I'm a developer too :). Learning is a cause I can get behind. I don't mind helping, I just wanted to make clear that I didn't know your intentions and could not exactly help you bypass their website security. To answer your question though, it is not selenium that they are detecting. The rules detected suspicious behavior and decided to take the appropriate measures against the offending client. They catch you by what you are not doing more than by what you are doing. In the repo link, you can checkout this file to get an idea baserules/modsecuritycrs20protocol_violations.con - Bassel Samman 2015-10-28 01:29


5

Even if you are sending all the right data (e.g. Selenium doesn't show up as an extension, you have a reasonable resolution/bit-depth, &c), there are a number of services and tools which profile visitor behaviour to determine whether the actor is a user or an automated system.

For example, visiting a site then immediately going to perform some action by moving the mouse directly to the relevant button, in less than a second, is something no user would actually do.

It might also be useful as a debugging tool to use a site such as https://panopticlick.eff.org/ to check how unique your browser is; it'll also help you verify whether there are any specific parameters that indicate you're running in Selenium.

2015-10-25 22:01
by lfaraone
I've already used that website and the fingerprint is identical to my normal browser. Also I'm not automating anything. I'm just browsing as normal - Ryan Weinstein 2015-10-26 04:46


5

Firefox is said to set window.navigator.webdriver === true if working with a webdriver. That was according to one of the older specs (e.g.: archive.org) but I couldn't find it in the new one except for some very vague wording in the appendices.

A test for it is in the selenium code in the file fingerprint_test.js where the comment at the end says "Currently only implemented in firefox" but I wasn't able to identify any code in that direction with some simple greping, neither in the current (41.0.2) Firefox release-tree nor in the Chromium-tree.

I also found a comment for an older commit regarding fingerprinting in the firefox driver b82512999938 from January 2015. That code is still in the Selenium GIT-master downloaded yesterday at javascript/firefox-driver/extension/content/server.js with a comment linking to the slightly differently worded appendix in the current w3c webdriver spec.

2015-10-27 23:44
by deamentiaemundi
I just tested webdriver with Firefox 55 and I can confirm this is not true. The variable window.navigator.webdriver is not defined - speedplane 2017-10-02 17:56


4

The bot detection I've seen seems more sophisticated or at least different than what I've read through in the answers below.

EXPERIMENT 1:

  1. I open a browser and web page with Selenium from a Python console.
  2. The mouse is already at a specific location where I know a link will appear once the page loads. I never move the mouse.
  3. I press the left mouse button once (this is necessary to take focus from the console where Python is running to the browser).
  4. I press the left mouse button again (remember, cursor is above a given link).
  5. The link opens normally, as it should.

EXPERIMENT 2:

  1. As before, I open a browser and the web page with Selenium from a Python console.

  2. This time around, instead of clicking with the mouse, I use Selenium (in the Python console) to click the same element with a random offset.

  3. The link doesn't open, but I am taken to a sign up page.

IMPLICATIONS:

  • opening a web browser via Selenium doesn't preclude me from appearing human
  • moving the mouse like a human is not necessary to be classified as human
  • clicking something via Selenium with an offset still raises the alarm

Seems mysterious, but I guess they can just determine whether an action originates from Selenium or not, while they don't care whether the browser itself was opened via Selenium or not. Or can they determine if the window has focus? Would be interesting to hear if anyone has any insights.

2018-04-11 18:41
by M3RS
My belief is that Selenium injects something into the page via javascript to find and access elements. This injection is what I believe they are detecting - zeusalmighty 2018-10-25 13:31


2

Write an html page with the following code. You will see that in the DOM selenium applies a webdriver attribute in the outerHTML

<html>
<head>
  <script type="text/javascript">
  <!--
    function showWindow(){
      javascript:(alert(document.documentElement.outerHTML));
    }
  //-->
  </script>
</head>
<body>
  <form>
    <input type="button" value="Show outerHTML" onclick="showWindow()">
  </form>
</body>
</html>

2015-10-28 04:10
by PC3TJ
The attribute is added only in Firefox - Louis 2015-10-28 09:22
And it is possible to remove it from the selenium extension that controlls browser. It will work anyway - erm3nda 2017-06-12 23:53


2

Some sites are detecting this:

function d() {
try {
    if (window.document.$cdc_asdjflasutopfhvcZLmcfl_.cache_)
        return !0
} catch (e) {}

try {
    //if (window.document.documentElement.getAttribute(decodeURIComponent("%77%65%62%64%72%69%76%65%72")))
    if (window.document.documentElement.getAttribute("webdriver"))
        return !0
} catch (e) {}

try {
    //if (decodeURIComponent("%5F%53%65%6C%65%6E%69%75%6D%5F%49%44%45%5F%52%65%63%6F%72%64%65%72") in window)
    if ("_Selenium_IDE_Recorder" in window)
        return !0
} catch (e) {}

try {
    //if (decodeURIComponent("%5F%5F%77%65%62%64%72%69%76%65%72%5F%73%63%72%69%70%74%5F%66%6E") in document)
    if ("__webdriver_script_fn" in document)
        return !0
} catch (e) {}
2017-08-22 09:52
by Néstor Lim
This doesn't work for Chrome and Firefox, selenium 3.5.0, ChromeDriver 2.31.488774, geckodriver 0.18. - jerrypy 2017-08-29 06:35


1

It seems to me the simplest way to do it with Selenium is to intercept the XHR that sends back the browser fingerprint.

But since this is a Selenium-only problem, its better just to use something else. Selenium is supposed to make things like this easier, not way harder.

2018-12-02 01:32
by pguardiario
What are other options to selenium - Tai 2018-12-03 17:38
And can they be detected as well - Tai 2018-12-03 17:45
I guess Requests would be the main python option. If you send the same exact requests that your browser sends, you will appear as a normal browser - pguardiario 2018-12-03 23:14


0

Additionally to the great answer of @Erti-Chris Eelmaa - there's annoying window.navigator.webdriver and it is read-only. Event if you change the value of it to false it will still have true. Thats why the browser driven by automated software can still be detected. MDN

The variable is managed by the flag --enable-automation in chrome. The chromedriver launches chrome with that flag and chrome sets the window.navigator.webdriver to true. You can find it here. You need to add to "exclude switches" the flag. For instance (golang):

package main

import (
    "github.com/tebeka/selenium"
    "github.com/tebeka/selenium/chrome"
)

func main() {

caps := selenium.Capabilities{
    "browserName": "chrome",
}

chromeCaps := chrome.Capabilities{
    Path:            "/path/to/chrome-binary",
    ExcludeSwitches: []string{"enable-automation"},
}
caps.AddChrome(chromeCaps)

wd, err := selenium.NewRemote(caps, fmt.Sprintf("http://localhost:%d/wd/hub", 4444))
}
2019-01-28 14:47
by FDG
Ads