RealEstate Data Platform
End-to-end real estate scraping and admin platform: multi-strategy scrapers feeding a React dashboard
// At a glance
Problem
There was no unified, resilient way to collect and normalize property listings from multiple Italian real estate sites, each with different anti-bot defenses, or to browse and manage the resulting leads.
Solution
Built a multi-strategy scraper suite (Playwright with network-request monitoring, a fast direct-API scraper, and a Scrapy spider) that can run independently or together for resilience against site/anti-bot changes, normalizing listings into a relational schema, served through a feature-based React SPA with RBAC, lead management, and PDF export.
// Key highlights
- →Adversarial web scraping combining headless browser automation, API reverse-engineering, and anti-bot evasion
- →Relational database schema unifying data scraped from multiple heterogeneous sources
- →Orchestrated scraper runs with independently toggleable Playwright/API/Scrapy strategies for resilience
- →Production-style React frontend with React Query caching, RBAC-driven UI, i18n, and PDF report generation
- →Scalable, feature-based frontend architecture
An end-to-end real estate data platform that scrapes Italian property listing sites (Casa.it, Trovacasa) using multiple techniques—headless browser automation, direct API calls, and Scrapy spiders—normalizes and persists the listings to a database, and serves them through a React admin dashboard with role-based access, lead management, and property browsing.
Tech Stack
- Scraping/Backend (Python): Playwright, Scrapy + scrapy-playwright, Selenium/webdriver-manager, BeautifulSoup4, lxml, aiohttp, requests, pandas, numpy, SQLAlchemy, SQLite, loguru, fake-useragent, ratelimit, python-dotenv
- Frontend (client/): React 19, Vite 7, React Router 7, TanStack React Query 5, Tailwind CSS 3 (tailwind-merge, tailwind-variants), Axios, i18next/react-i18next, Lucide icons, Sonner, Tippy.js, jsPDF/jspdf-autotable, Luxon, DOMPurify
- Mock/Dev backend: json-server, Node server.cjs, run concurrently with Vite dev server
- Tooling: ESLint 9 (flat config), PostCSS, Autoprefixer
Key Features
- Multi-strategy scraper suite: Playwright with network-request monitoring/API discovery, a fast direct-API scraper (no browser), and a Scrapy spider
- Infinite-scroll handling, cross-source deduplication, rate limiting, and structured JSON/DB output
- Authentication with role-based access control (Roles, Users management)
- Lead management module
- Property/offer listings with filters and a details modal
- Multi-language support (i18next)
- PDF report export
- Orchestrated scraper runs via
run_all_scrapers.pywith CLI flags to toggle Playwright/API/Scrapy independently
Architecture
The project separates data acquisition from presentation: Scrappers/ (Python) collects and normalizes listing data from Casa.it and Trovacasa into a relational schema (designed in conception/), while client/ is a React SPA consuming that data via REST/JSON. The frontend follows a feature-based structure (src/features/{leads,offers,roles,users}, each with its own hooks and components) rather than flat pages, supporting scalability. The scraper orchestrator is built for resilience against anti-bot measures and site/API changes by allowing independent toggling of scraping methods.
My Role
- Built adversarial web scraping pipelines combining headless browser automation, API reverse-engineering, and anti-bot evasion (rate limiting, rotating user agents)
- Designed a relational database schema to unify data scraped from multiple heterogeneous sources
- Implemented a production-style React frontend using React Query for caching, RBAC-driven UI, i18n, and PDF generation
- Structured the frontend using a scalable, feature-based architecture