Headless browser Hide & Seek

This presentation discusses methods to detect unwanted headless browsers interacting with your website

Transcript

1. AppSec USA 2014! Denver, Colorado Headless Browser Hide & Seek! Sergey Shekyan, Bei Zhang! Shape Security
2. • Bei  Zhang   Senior  Software  Engineer  at  Shape  Security,  focused  on  analysis  and   countermeasures  of  automated  web  attacks.  Previously,  he  worked  at   the  Chrome  team  at  Google  with  a  focus  on  the  Chrome  Apps  API.  His   interests  include  web  security,  source  code  analysis,  and  algorithms.   • Sergey  Shekyan   Principal  Engineer  at  Shape  Security,  focused  on  the  development  of   the  new  generation  web  security  product.  Prior  to  Shape  Security,  he   spent  4  years  at  Qualys  developing  their  on  demand  web  application   vulnerability  scanning  service.  Sergey  presented  research  at  security   conferences  around  the  world,  covering  various  information  security   topics. Who We Are
3. Scriptable  browser  environment  that  doesn’t  require  GUI   • Existing  browser  layout  engine  with  bells  and  whistles   (PhantomJS  -­‐  WebKit,  SlimerJS  -­‐  Gecko,  TrifleJS  -­‐  Trident)   • Custom  software  that  models  a  browser  (ZombieJS,   HtmlUnit)   • Selenium  (WebDriver  API) What Is a Headless Browser and How it Works
4. Discussion  will  focus  on  PhantomJS:   • Backed  by  WebKit  engine   • Cross-­‐platform   • Very  popular   • True  headless What Is a Headless Browser and How it Works
5. PhantomJS World PhantomJS   JavaScript   Context QWebFrame QtWebKit Web  Page   JavaScript   Context Control Callback Injection PageEvent Callbacks  are   serialized
6. PhantomJS World PhantomJS   JavaScript   Context QWebFrame QtWebKit Web  Page   JavaScript   Context Control Callback Injection PageEvent Callbacks  are   serialized var  page  =  require('webpage').create();   page.open(url,  function(status)  {      var  title  =  page.evaluate(function()  {          return  document.title;      });      console.log('Page  title  is  '  +  title);   });
7. • Web  Application  functional  and  performance  testing   • Crawler  that  can  provide  certain  amount  of  interaction   to  reveal  web  application  topology,  Automated  DOM   XSS,  CSRF  detection   • SEO  (render  dynamic  web  page  into  static  HTML  to   feed  to  search  engines)   • Reporting,  image  generation Legitimate uses and how you can benefit
8. • Fuzzing   • Botnet   • Content  scraping   • Login  brute  force  attacks   • Click  fraud   • Bidding  wars   Web  admins  tend  to  block  PhantomJS  in  production,  so   pretending  to  be  a  real  browser  is  healthy  choice Malicious Use of Headless Browser
9. Impersonating a legitimate browser
10. • Outdated  WebKit  engine  (close  to  Safari  5  engine,  4  y.o.)   • Uses  Qt  Framework’s  QtWebKit  wrapper  around  WebKit   • Qt  rendering  engine   • Qt  network  stack   • Qt  Cookie  Jar,  that  doesn’t  support  RFC  2965   • No  Media  Hardware  support  (no  video  and  audio)   • Exposes  window.callPhantom  and  window._phantom   • No  sandboxing How It Is Different From a Real Browser
11. • Look  at  user  agent  string Headless Browser Seek if  (/PhantomJS/.test(window.navigator.userAgent))  {      console.log(‘PhantomJS  environment  detected.’);   }
12. • Making  user-­‐agent  (and  navigator.userAgent)  a  “legitimate”  one:   ! ! ! ! Headless Browser Hide var  page  =  require(‘webpage').create();   ! page.settings.userAgent  =  ‘Mozilla/5.0  (Macintosh;  Intel  Mac   OS  X  10.9;  rv:30.0)  Gecko/20100101  Firefox/30.0';
13. ! ! That’s  all  we  got.  Thank  you!   ! ! ! ! Impersonating a legitimate browser
14. ! ! Sike!   ! ! ! ! Impersonating a legitimate browser
15. Score board Phantom Web  site User  Agent  String Win Lose
16. Headless Browser Seek • Sniff  for  PluginArray  content if  (!(navigator.plugins  instanceof  PluginArray)  ||   navigator.plugins.length  ==  0)  {          console.log("PhantomJS  environment  detected.");      }  else  {          console.log("PhantomJS  environment  not  detected.");      }
17. • Fake  navigator  object,  populate  PluginArray  with  whatever  values  you  need.   ! ! ! ! ! ! ! ! ! • Spoofing  Plugin  objects  inside  the  PluginArray  is  tedious  and  hard.   • Websites  can  actually  create  a  plugin  to  test  it.   • CONCLUSION:  Not  a  good  idea  to  spoof  plugins. Headless Browser Hide page.onInitialized  =  function  ()  {          page.evaluate(function  ()  {                  var  oldNavigator  =  navigator;                  var  oldPlugins  =  oldNavigator.plugins;                  var  plugins  =  {};                  plugins.length  =  1;                  plugins.__proto__  =  oldPlugins.__proto__;   !                window.navigator  =  {plugins:  plugins};                  window.navigator.__proto__  =  oldNavigator.__proto__;          });   };
18. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win
19. • Alert/prompt/confirm  popup  suppression  timing   detection Headless Browser Seek var  start  =  Date.now();      prompt('I`m  just  kidding');      var  elapse  =  Date.now()  -­‐  start;      if  (elapse  <  15)  {          console.log("PhantomJS  environment  detected.  #1");      }  else  {          console.log("PhantomJS  environment  not  detected.");      }
20. • Can’t  use  setTimeout,  but  blocking  the  callback  by  all   means  would  work Headless Browser Hide page.onAlert  =  page.onConfirm  =  page.onPrompt  =  function  ()  {          for  (var  i  =  0;  i  <  1e8;  i++)  {          }          return  "a";   };
21. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win
22. • Default  order  of  headers  is  consistently  different  in   PhantomJS.  Camel  case  in  some  header  values  is  also  a   good  point  to  look  at. Headless Browser Seek PhantomJS  1.9.7   GET  /  HTTP/1.1   User-­‐Agent:     Accept:   Connection:  Keep-­‐Alive   Accept-­‐Encoding:   Accept-­‐Language:   Host: Chrome  37   GET  /  HTTP/1.1   Host:   Connection:  keep-­‐alive   Accept:   User-­‐Agent:   Accept-­‐Encoding:     Accept-­‐Language:
23. • A  custom  proxy  server  in  front  of  PhantomJS   instance  that  makes  headers  look  consistent   with  user  agent  string Headless Browser Hide
24. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose
25. • PhantomJS  exposes  APIs:   • window.callPhantom   • window._phantom  //  not  documented. Headless Browser Seek if  (window.callPhantom  ||  window._phantom)  {      console.log("PhantomJS  environment  detected.");   }  else  {      console.log("PhantomJS  environment  not  detected.");   }
26. • store  references  to  original  callPhantom,  _phantom   • delete  window.callPhantom,  window._phantom Headless Browser Hide page.onInitialized  =  function  ()  {          page.evaluate(function  ()  {                  var  p  =  window.callPhantom;                  delete  window._phantom;                  delete  window.callPhantom;                  Object.defineProperty(window,  "myCallPhantom",  {                          get:  function  ()  {  return  p;},                          set:  function  ()  {},  enumerable:  false});                  setTimeout(function  ()  {  window.myCallPhantom();},  1000);          });   };   page.onCallback  =  function  (obj)  {  console.log(‘profit!');  }; Unguessable  name
27. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose window.callPhantom Win Lose
28. • Spoofing  DOM  API  properties  of  real  browsers:   • WebAudio   • WebRTC   • WebSocket   • Device  APIs   • FileAPI   • WebGL   • CSS3  -­‐  not  observable.  Defeats  printing.   • Our  research  on  WebSockets:  http://goo.gl/degwTr Headless Browser Seek
29. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose window.callPhantom Win Lose HTML5  features Lose Win
30. • Significant  difference  in  JavaScript  Engine:  bind()  is   not  defined  in  PhantomJS  prior  to  version  2 Headless Browser Seek (function  ()  {          if  (!Function.prototype.bind)  {              console.log("PhantomJS  environment  detected.");              return;          }          console.log("PhantomJS  environment  not  detected.");      })();
31. • Significant  difference  in  JavaScript  Engine:  bind()  is   not  defined  in  PhantomJS  prior  to  version  2 Headless Browser Seek (function  ()  {          if  (!Function.prototype.bind)  {              console.log("PhantomJS  environment  detected.");              return;          }          console.log("PhantomJS  environment  not  detected.");      })(); Function.prototype.bind  =  function  ()  {          var  func  =  this;          var  self  =  arguments[0];          var  rest  =  [].slice.call(arguments,  1);          return  function  ()  {                  var  args  =  [].slice.call(arguments,  0);                  return  func.apply(self,  rest.concat(args));          };   };
32. • Detecting  spoofed  Function.prototype.bind  for   PhantomJS  prior  to  version  2 Headless Browser Seek (function  ()  {          if  (!Function.prototype.bind)  {              console.log("PhantomJS  environment  detected.  #1");              return;          }          if  (Function.prototype.bind.toString().replace(/bind/g,  'Error')  !=     Error.toString())  {              console.log("PhantomJS  environment  detected.  #2");              return;          }          console.log("PhantomJS  environment  not  detected.");      })();
33. Headless Browser Hide • Spoofing  Function.prototype.toString function  functionToString()  {          if  (this  ===  bind)  {                  return  nativeFunctionString;          }          return  oldCall.call(oldToString,  this);   }
34. • Detecting  spoofed  Function.prototype.bind  for   PhantomJS  prior  to  version  2 Headless Browser Seek (function  ()  {          if  (!Function.prototype.bind)  {              console.log("PhantomJS  environment  detected.  #1");              return;          }          if  (Function.prototype.bind.toString().replace(/bind/g,  'Error')  !=     Error.toString())  {              console.log("PhantomJS  environment  detected.  #2");              return;          }          if  (Function.prototype.toString.toString().replace(/toString/g,  'Error')  !=   Error.toString())  {              console.log("PhantomJS  environment  detected.  #3");              return;          }          console.log("PhantomJS  environment  not  detected.");      })();
35. Headless Browser Hide • Spoofing  Function.prototype.toString.toString function  functionToString()  {          if  (this  ===  bind)  {                  return  nativeFunctionString;          }          if  (this  ===  functionToString)  {                  return  nativeToStringFunctionString;          }          if  (this  ===  call)  {                  return  nativeCallFunctionString;          }          if  (this  ===  apply)  {                  return  nativeApplyFunctionString;          }          var  idx  =  indexOfArray(bound,  this);          if  (idx  >=  0)  {                  return  nativeBoundFunctionString;          }          return  oldCall.call(oldToString,  this);   }
36. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose window.callPhantom Win Lose HTML5  features Lose Win Function.prototype.bind Win Lose
37. • PhantomJS2  is  very  EcmaScript5  spec   compliant,  so  checking  for  outstanding   JavaScript  features  is  not  going  to  work Headless Browser Seek
38. • Stack  trace  generated  by  PhantomJs   ! ! ! • Stack  trace  generated  by  SlimerJS   ! ! • Hmm,  there  is  something  common… Headless Browser Seek at  querySelectorAll  (phantomjs://webpage.evaluate():9:10)   at  phantomjs://webpage.evaluate():19:30   at  phantomjs://webpage.evaluate():20:7   at  global  code  (phantomjs://webpage.evaluate():20:13)   at  evaluateJavaScript  ([native  code])   at  global  code  (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14) Element.prototype.querySelectorAll@phantomjs://webpage.evaluate():9   @phantomjs://webpage.evaluate():19   @phantomjs://webpage.evaluate():20
39. var  err;   try  {      null[0]();   }  catch  (e)  {      err  =  e;   }   if  (indexOfString(err.stack,  'phantomjs')  >  -­‐1)  {      console.log("PhantomJS  environment  detected.");   }  else  {      console.log("PhantomJS  environment  is  not  detected.");   } Headless Browser Seek It  is  not  possible  to  override  the  thrown   TypeError generic  indexOf(),  can  be  spoofed,  define  your  own
40. Headless Browser Hide
41. • Modifying  webpage.cpp: Headless Browser Hide QVariant  WebPage::evaluateJavaScript(const  QString  &code)   {      QVariant  evalResult;      QString  function  =  "("  +  code  +  ")()";      evalResult  =  m_currentFrame-­‐>evaluateJavaScript(function,          QString("phantomjs://webpage.evaluate()"));   return  evalResult;   }
42. • Modifying  webpage.cpp: Headless Browser Hide QVariant  WebPage::evaluateJavaScript(const  QString  &code)   {      QVariant  evalResult;      QString  function  =  "("  +  code  +  ")()";      evalResult  =  m_currentFrame-­‐>evaluateJavaScript(function,          QString("phantomjs://webpage.evaluate()"));   return  evalResult;   } at     at     at     at     at  evaluateJavaScript  ([native  code])   at     at  global  code  (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14) • Produces
43. • Spoofed  PhantomJs  vs  Chrome:   ! ! ! ! ! ! Headless Browser Hide    at  querySelectorAll  (Object.InjectedScript:9:10)      at  Object.InjectedScript:19:30      at  Object.InjectedScript:20:7      at  global  code  (Object.InjectedScript:20:13)      at  evaluateJavaScript  ([native  code])      at        at  global  code  (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14) TypeError:  Cannot  read  property  '0'  of  null      at  HTMLDocument.Document.querySelectorAll.Element.querySelectorAll  [as   querySelectorAll]  (<anonymous>:21:5)      at  <anonymous>:2:10      at  Object.InjectedScript._evaluateOn  (<anonymous>:730:39)      at  Object.InjectedScript._evaluateAndWrap  (<anonymous>:669:52)      at  Object.InjectedScript.evaluate  (<anonymous>:581:21)  
44. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose window.callPhantom Win Lose HTML5  features Lose Win Function.prototype.bind Win Lose Stack  trace Lose Win
45. Score board Phantom Web  site User  Agent  String Win Lose Inspect  PluginArray Lose Win Timed  alert() Lose Win HTTP  Header  order Win Lose window.callPhantom Win Lose HTML5  features Lose Win Function.prototype.bind Win Lose Stack  trace Lose Win SCORE: 4 4
46. Attack on PhantomJs
47. Attack on PhantomJs What if you caught a malicious PhantomJs?
48. The  usual  picture:   • PhantomJS  running  as  root   • No  sandboxing  in  PhantomJS   • Blindly  executing  untrusted  JavaScript   • outdated  third  party  libs  (libpng,  libxml,  etc.)   Can  possibly  lead  to:   • abuse  of  PhantomJS   • abuse  of  OS  running  PhantomJS How to turn a headless browser against the attacker
49. var  html  =  document.querySelectorAll('html');   var  oldQSA  =  document.querySelectorAll;   Document.prototype.querySelectorAll  =   Element.prototype.querySelectorAll  =  function  ()  {      var  err;      try  {          null[0]();      }  catch  (e)  {          err  =  e;      }      if  (indexOfString(err.stack,  'phantomjs')  >  -­‐1)  {          return  html;      }  else  {          return  oldQSA.apply(this,  arguments);      }   }; Headless Browser Seek It  is  not  possible  to  override  the  thrown  TypeError generic  indexOf(),  can  be  spoofed,  define  your  own
50. • In  a  lot  of  cases  -­‐-­‐web-­‐security=false  is  used  in   PhantomJS Headless Browser Seek    var  xhr  =  new  XMLHttpRequest();      xhr.open('GET',  'file:/etc/hosts',  false);      xhr.onload  =  function  ()  {          console.log(xhr.responseText);      };      xhr.onerror  =  function  (e)  {          console.log('Error:  '  +  JSON.stringify(e));      };      xhr.send();
51. Headless Browser Seek • Obfuscate,  randomize  the  output,  randomize  the  modified  API  call var  _0x34c7=["\x68\x74\x6D \x6C","\x71\x75\x65\x72\x79\x53\x65\x6C\x65\x63\x74\x6F \x72\x41\x6C\x6C","\x70\x72\x6F\x74\x6F \x74\x79\x70\x65","\x73\x74\x61\x63\x6B","\x70\x68\x61\x6E \x74\x6F\x6D\x6A\x73","\x61\x70\x70\x6C\x79"];var   html=document[_0x34c7[1]](_0x34c7[0]);var   apsdk=document[_0x34c7[1]];Document[_0x34c7[2]] [_0x34c7[1]]=Element[_0x34c7[2]][_0x34c7[1]]=function  (){var   _0xad6dx3;try{null[0]();}  catch(e) {_0xad6dx3=e;}  ;if(indexOfString(_0xad6dx3[_0x34c7[3]],_0x34c7[4 ])>-­‐1){return  html;}  else  {return  apsdk[_0x34c7[5]] (this,arguments);};};
52. • If  you  don’t  need  a  full  blown  browser  engine,  don’t   use  it   • Do  not  run  with  ‘-­‐  -­‐web-­‐security=false’  in  production,   try  not  to  do  that  in  tests  as  well.   • Avoid  opening  arbitrary  page  from  the  Internet   • No  root  or  use  chroot   • Use  child  processes  to  run  webpages   • If  security  is  a  requirement,  use  alternatives  or  on  a   controlled  environment,  use  Selenium Tips for Using Headless Browsers Safely
53. • It  is  not  easy  to  unveil  a  user  agent.  Make  sure  you   want  to  do  it.   • Do  sniff  for  headless  browsers  (some  will  bail  out)   • Combine  several  detection  techniques   • Reject  known  unwanted  user  agents  (5G  Blacklist  2013   is  a  good  start)   • Alter  DOM  API  if  headless  browser  is  detected   • DoS   • Pwn Tips For Web Admins
54. Start Detecting Already Ready  to  use  examples:   github.com/ikarienator/phantomjs_hide_and_seek
55. • http://www.sba-­‐research.org/wp-­‐content/uploads/publications/jsfingerprinting.pdf   • https://kangax.github.io/compat-­‐table/es5/   • https://media.blackhat.com/us-­‐13/us-­‐13-­‐Grossman-­‐Million-­‐Browser-­‐Botnet.pdf   • http://ariya.ofilabs.com/2011/10/detecting-­‐browser-­‐sniffing-­‐2.html   • http://www.darkreading.com/attacks-­‐breaches/ddos-­‐attack-­‐used-­‐headless-­‐ browsers-­‐in-­‐150-­‐hour-­‐siege/d/d-­‐id/1140696?   • http://vamsoft.com/downloads/articles/vamsoft-­‐headless-­‐browsers-­‐in-­‐forum-­‐ spam.pdf   • http://blog.spiderlabs.com/2013/02/server-­‐site-­‐xss-­‐attack-­‐detection-­‐with-­‐ modsecurity-­‐and-­‐phantomjs.html   • http://googleprojectzero.blogspot.com/2014/07/pwn4fun-­‐spring-­‐2014-­‐safari-­‐part-­‐ i_24.html References
56. @ikarienator   ! @sshekyan Thank you!