Suparn-SymfonyOnline-June-2025-Slides

Transcript

1. Efficient Web Scraping with Symfony & PHP Suparn Padma Patra Developer at Class Central
2. whoami 📍 Currently based in Jaipur, India ✏ Former teacher at DPS Gaya ❤ Passionate about MOOCs, learning & teaching Personal Education Work 📘 Pursuing PhD in Computer Science (Pre-submission seminar completed) 🎓 MSc in Artificial Intelligence (Gold Medalist) 📚 B.Sc. in Science & B.Ed. in Education 🚀 Joined Class Central in April 2019 (working remotely from India) 🧱 Full Stack Engineer working with Symfony & PHP 🕷 Maintains and writes scrapers that collect the course metadata from different providers.
4. SymfonyCasts Symfony UX: Turbo Course Page at Class Central https://www.classcentral.com/course/symfonycasts-screencast-turbo-291012
7. You Need Python for Web Scraping ❌ Wrong! Symfony + PHP = full scraping stack No need to switch languages Easier to integrate with existing Symfony apps
8. When to Choose PHP/Symfony for Web Scraping? Your app is already built with PHP/Symfony Comfortable in PHP? No need to switch just to scrape - Symfony has you covered. You want production-ready tools like HttpClient & Panther
9. Scraping Mistakes That Caused My Scrapers to Fail Too Often (In Early Days) 🏃 Started Coding Without Reverse Engineering 📍 Blindly Trusted sitemap.xml 🔥 Hammered Server Despite 403 Errors
10. ⏱ Rate limiting and IP blocking 🔄 Constantly changing site structures 🤖 CAPTCHA and anti-bot systems 🎭 JavaScript-heavy SPAs Common Scraping Challenges (The Reality of Modern Web Scraping)
11. The 70% (Language Independent): • 🕵 Understanding site structure • 🔐 Analyzing API calls in DevTools • 🎭 Detecting anti-bot patterns • 🗺 Mapping data relationships • 🔄 Handling dynamic content • 📊 Planning extraction strategy LLMs still can't do this Web Scraping = 70% Reverse Engineering + 30% Coding The 30% (Any Language Works): • ✍ Writing the actual scraper • 📦 Implementing what you discovered • 🔧 Basic HTTP requests + parsing LLMs can do this
12. 1⃣ One-Time vs Recurring? • Just once? → Use Chrome extensions like: 🔹 Link Grabber, Instant Data Scraper, etc. → Text Editor (like PhpStorm / VSCode) with extensions like Beautify JSON, etc. • Recurring? → Build a full scraper with PHP/Symfony 2⃣ How to Get All URLs? • Check for sitemap.xml • Try on-site search (note infinite scroll, pagination, search limits: 10K, etc.) • Use filters/tags/category pages to generate a complete URL list Reverse Engineering a Website for Scraping
13. 3⃣ Inspect Their Defenses • Do they use Cloudflare, Akamai, or anti-bot services? • Look for honeypot links, trap fields in forms • Do they block unknown user-agents or IPs? 4⃣ Check Rate Limits & Behaviors • How many pages before throttling or blocking? • Test with different delays, headers, and user-agents • Use browser dev tools to simulate real user interaction Reverse Engineering a Website for Scraping (Contd.)
14. Reverse Engineering a Website for Scraping (Contd.) 5⃣ Plan Your Strategy Accordingly • Headless browser needed (Panther)? • How to rotate IPs and user-agents? • Is it worth caching/storing intermediate HTML? • What retry and backoff strategy will you use? • How often will scraping run — once or recurring? • How will you detect and avoid bot traps or honeypots?
15. 🌐 HTTP Capabilities • ✅ file_get_contents() + stream_context_create() – Simple requests with headers • 🌀 cURL – Advanced requests, cookies, proxies, redirects, and POST handling • 🔄 curl_multi_* – Perform asynchronous/multi-threaded scraping • 🎭 Custom headers, user-agents, timeouts, and cookies supported The Core PHP Stack for Web Scraping
16. 🧰 Parsing Arsenal • 🧱 DOMDocument – Native, reliable HTML parsing • 🔎 DOMXPath – CSS-like, precise element querying • 📦 SimpleXML – Clean parsing for XML/RSS feeds • 🧵 preg_match() / preg_match_all() – When regex is the best tool for the job The Core PHP Stack for Web Scraping (Contd.)
17. The Core PHP Stack for Web Scraping (Contd.) ⚙ Data Processing Tools • 📥 json_decode() – Parse API or embedded JSON • 🔗 parse_url() – Break apart URLs cleanly • 🔧 http_build_query() – Reconstruct query strings dynamically • 🌐 mb_string – Safely handle multilingual or encoded content • 📄 fputcsv() / fopen() – Easily write structured data to CSV files
18. 🧠 HttpClient (symfony/http-client) – Fast, async HTTP requests with retries & headers 🧼 DomCrawler (symfony/dom-crawler) – Clean, CSS-style HTML parsing 🕸 Panther (symfony/panther) – Headless browser for JavaScript-heavy sites 📦 Messenger (symfony/messenger) – Queue-based architecture for scaling & retries 🛠 Doctrine (doctrine/orm) – Structured, normalized data storage ⏱ Scheduler (symfony/scheduler) / CRON – Automate recurring scraping jobs The Symfony Stack for Efficient Scraping
19. https://books.toscrape.com/ - Demo website for web scraping purposes
20. https://books.toscrape.com/catalogue/page-2.html - URL for page 2
21. https://books.toscrape.com/catalogue/page-51.html - When checked for page 51
24. Area 🚧 Raw PHP Script ✅ Symfony-Based Approach HTTP Handling file_get_contents() with no headers or retry HttpClient with headers, timeouts, retries HTML Parsing DOMDocument + DOMXPath (with suppressed errors) DomCrawler with clean, readable selectors Error Handling Silent failures, no logs Exceptions + Logging with Monolog Modularity Inline, procedural logic Structured services, reusable, testable Scalability & Resilience Single-threaded loop, no retries Messenger + Scheduler for parallel jobs and retries Key Improvements with Symfony for Web Scraping
25. ⚖ Legal Responsibility: Always refer Terms of Service, robots.txt, copyright laws 🛡 Ethical Guidelines: Always identify your bot and respect website resources ⚠ Technical Boundaries: Implement rate limiting and never overload servers 🚫 What NOT to Scrape: Personal data, copyrighted content, or login-protected areas ✅ Best Practice: When in doubt, seek permission or use official APIs Disclaimer
26. Thanks! Suparn Padma Patra 📧 suparn@classcentral.com 🔗 https://www.linkedin.com/in/suparnpatra