Digital Cat-and-Mouse: Strategies to Outsmart Scrapers, Phishers, and Thieves

As app developers, defending our digital assets goes far beyond firewalls and passwords. With scrapers, phishers, and content thieves at every turn, creative defences are often our best tool. Whether it’s trying to sell 10 Downing Street or spotting the Eiffel Tower in Dublin, this talk will explore inventive techniques to outsmart threats in real-world scenarios.

We'll share stories from the trenches and discuss technical strategies to stay one step ahead of competitors and bad actors alike. As with all good stories, plenty of mistakes were made along the way! You’ll leave with actionable tactics to protect your assets, outwit your competition, and stay ahead in this ever-evolving landscape.

Transcript

1. Digital Cat & Mouse: Strategies to Outsmart Scrapers, Phishers and Thieves Paul Conroy / @conroyp
2. From Dublin, Ireland Started playing with the web 30+ years ago (Notepad, Frontpage & Geocities!) CTO at Square1 conroyp.com / @conroyp Paul Conroy 👴 🌍 🇮🇪
3. Threat Landscape • Infrastructure security issues • 0-day exploits • Database exfiltration
4. https://xkcd.com/538/ Threat Landscape • Infrastructure security issues • 0-day exploits • Database exfiltration
5. https://xkcd.com/538/ Threat Landscape • Infrastructure security issues • 0-day exploits • Database exfiltration
6. Creative Sabotage
7. Boom Time • Popular property site in Ireland • 2 main players in the market, we recently became #1 • Height of property boom - lots of new competitors • Most fail - difficult to get agents to upload to new site!
8. http://dublincoastaldevelopment.com/
9. http://dublincoastaldevelopment.com/
10. http://dublincoastaldevelopment.com/ 🇳🇱
11. http://dublincoastaldevelopment.com/ 🇳🇱 💰💰
12. 👨
13. 👨
14. 👨
15. 👨 👨💼
16. 👨 👨💼 💸
17. 👨 👨💼 Them Us
18. 👨💼 👨 👩💼 👴 👩🦰 Them Us
19. 👨💼 👨 👩💼 👴 👩🦰 Them Them
20. What are our options?
21. What are our options? 🙏 Ask them nicely to stop
22. What are our options? 🙏 Ask them nicely to stop 🧑⚖ Go to court… ⏳💰💰💰
23. What are our options? 🙏 Ask them nicely to stop 🧑⚖ Go to court… ⏳💰💰💰 👩💻 Technical countermeasures
25. 🇮🇪 🇮🇪 🇮🇪 🇳🇱 🇳🇱 🇳🇱 🇳🇱 🇮🇪 🇮🇪 🇮🇪
31. Them Us
32. 👨💼 👨 👩💼 👴 👩🦰 Them Us
33. Them 😡 😡 😡 😡 🤬 Us
39. The Result?
40. The Result? 🌅 Results updated next morning
41. The Result? 🌅 Results updated next morning 🗑 Filled with obvious nonsense
42. The Result? 🌅 Results updated next morning 🗑 Filled with obvious nonsense 🤣 Relentlessly mocked on social media
43. The Result? 🌅 Results updated next morning 🗑 Filled with obvious nonsense 🤣 Relentlessly mocked on social media ✋ Stopped scraping us - flawless victory!
44. The Result? 🌅 Results updated next morning 🗑 Filled with obvious nonsense 🤣 Relentlessly mocked on social media ✋ Stopped scraping us - flawless victory! 🥳 🎉 🎊
45. Flawless Victory! (almost)
46. Flawless Victory! (almost)
47. 🦾 Developer doing local testing 🙈 Weak (non-existent..) code review 🙊 FTP to production 😳 Noticed internally (mid-victory lap) 🤬 Rapid change in dev processes Flawless Victory! (almost)
48. 🦾 Developer doing local testing 🙈 Weak (non-existent..) code review 🙊 FTP to production 😳 Noticed internally (mid-victory lap) 🤬 Rapid change in dev processes Flawless Victory! (almost)
51. https://blog.cloudflare.com/ai-labyrinth/
54. Bandwidth Hogs
55. • Property websites need to be attractive - strong imagery • Images make up about 60% of a site’s payload • Bandwidth is expensive!
57. 🔥🔥
58. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere
59. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere
60. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
61. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
62. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
63. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
64. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
65. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
66. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
67. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
68. • Watermarking each image • Aggressive - visible, central • Obvious on other sites that the original lives elsewhere • Estate agents hated it - lots of complaints! • Needed to be more subtle
69. Devolving into an arms race We need a more technical solution!
71. • Competitors quickly got frustrated • Poor user experience • Stopped hotlinking us
72. • Competitors quickly got frustrated • Poor user experience • Stopped hotlinking us • They targeted other competitors
73. • Competitors quickly got frustrated • Poor user experience • Stopped hotlinking us • They targeted other competitors 🏃🧍🦁
74. Signed URLs • More browser restrictions on sharing of referral URLs • Application generates URL with signature • Typically includes timestamp and a shared secret • Server validates it
75. Signed URLs • More browser restrictions on sharing of referral URLs • Application generates URL with signature • Typically includes timestamp and a shared secret • Server validates it
76. 🧑💻 Image is in cache already? Retrieve image from cache Nginx + LUA scripting Decode sig Extract timestamp Check vs shared secret Return image Yes No Request: /img/house.jpg?sig=abc1234 Shared secret Valid sig? Add cache headers to image Yes No Error image
77. IP Restrictions • Mobile network in Ireland grouping user IPs • Threat detection was counting IPs per account • Think again about rate limiting strategies • 1 IP != 1 Person
78. IP Restrictions • Mobile network in Ireland grouping user IPs • Threat detection was counting IPs per account • Think again about rate limiting strategies • 1 IP != 1 Person
79. Phishing Expedition
80. Real Estate scams, very popular Beautiful photos, low price Owner “not available right now” Wire the money now, it’s yours!
81. Real Estate scams, very popular Beautiful photos, low price Owner “not available right now” Wire the money now, it’s yours!
82. Real Estate scams, very popular Beautiful photos, low price Owner “not available right now” Wire the money now, it’s yours!
83. Real Estate scams, very popular Beautiful photos, low price Owner “not available right now” Wire the money now, it’s yours!
84. Real Estate scams, very popular Beautiful photos, low price Owner “not available right now” Wire the money now, it’s yours! Let your users help you out!
85. 🧔
86. 🧔 Username & password
87. 🧔 Username & password Platform access
88. 🧔 Username & password 🏴☠
89. 🧔 Username & password Error message 🏴☠
90. 🧔 Username & password Error message 🏴☠ 🎟
91. 🏴☠ 😇
92. 🏴☠ 😇
93. 🏴☠ 😇
94. 🏴☠ 😇
95. 🏴☠ 😇
96. 🏴☠ 😇
97. 🏴☠ 😇
98. 🏴☠ 😇
99. • Got smarter over time • Copy css and js files - no more hotlinking
100. • Got smarter over time • Copy css and js files - no more hotlinking • Didn’t get them all…
101. • Got smarter over time • Copy css and js files - no more hotlinking • Didn’t get them all…
103. Honeypot • Tempting to nuke the site immediately • Set up a honeypot • Seemingly-real credentials • Track when they are used to log in • Check for other logins from the same device
105. https://www.haveibeenpwned.com/ • User education - checking URLs • Two Factor Authentication • HaveIBeenPwned.com
108. • User education - checking URLs • Two Factor Authentication • HaveIBeenPwned.com • Ugly emails 😬 https://www.haveibeenpwned.com/
109. Black Sunday Hack
110. • 1990s US, Pay TV taking off - sports & movies • More expensive for bars - charge $20 entry to customers for events • Card encryption cracked - thriving black market in cloned decoders
111. 📺 👨💼
112. 📺 👨💼 🏴☠
113. 📺 📺 👨💼 🏴☠
114. 📺 📺 👨💼 🏴☠
115. • Critical bit of the process to copy • Odd updates - not breaking pirated cards • No obvious functionality • Pirates copied them faithfully 📺 👨💼
130. Pieces combined to make programme that could run on the card!
131. Pieces combined to make programme that could run on the card! Overwrote part of internal card memory, sending machines into boot loop
132. Pieces combined to make programme that could run on the card! 98% of pirate cards effectively bricked overnight Overwrote part of internal card memory, sending machines into boot loop
133. Pieces combined to make programme that could run on the card! 98% of pirate cards effectively bricked overnight Overwrote part of internal card memory, sending machines into boot loop
134. Pieces combined to make programme that could run on the card! 98% of pirate cards effectively bricked overnight Overwrote part of internal card memory, sending machines into boot loop Developers took a small victory lap…
135. Pieces combined to make programme that could run on the card! 98% of pirate cards effectively bricked overnight Overwrote part of internal card memory, sending machines into boot loop Developers took a small victory lap…
140. Strategies • Tarpit - slow your adversary down, slow-responding code. • Honeypot - give them credentials that seem legit, but can be used to trace them. • Fake data - share data that looks good to machines, but needs a lot of human effort to clean up. • Swiss cheese defence - one layer is rarely enough. • Patience can be a virtue when building a counter-attack.
141. Takeaways 🥷 Know your adversary - what are they trying to do? 🕵 Monitoring - how are they trying to do it? ⏰ Immediate block vs longer-term counter-attack 👮 Know the law! Counter-hacking can get you in trouble in some areas
142. Takeaways 🥷 Know your adversary - what are they trying to do? 🕵 Monitoring - how are they trying to do it? ⏰ Immediate block vs longer-term counter-attack 👮 Know the law! Counter-hacking can get you in trouble in some areas 🦁 Lace up your running shoes!
143. • Revisting the Black Sunday Hack https://blog.codinghorror.com/revisiting-the-black-sunday-hack/ • Dublin Coastal Development Launch Site (Irish state broadcaster news coverage) http://dublincoastaldevelopment.com/
 https://www.rte.ie/archives/2016/0925/818719-artificial-islands-for-dublin-bay/ • Cloudflare AI Labyrinth https://blog.cloudflare.com/ai-labyrinth/ • HaveIBeenPwned.com https://www.haveibeenpwned.com Further Reading
144. Thank you! Conroyp.com 
 @conroyp 
 paul@conroyp.com 🌍 🌐 📧