Features

Scraping

Listings are extracted from ss.lv with BeautifulSoup. The target city and listing type are set with FALTS_FOR_SALE_BASE_URL — see Configuration.

Database storage

Scraped listings are stored in PostgreSQL. Each row keeps the listing's price, size, price per square metre, floor, address, the date it was first listed and, once it disappears from the source, the date it was removed.

Email reports

A scheduled job sends a summary of new and removed listings. This requires a SendGrid API key; without one, the rest of the pipeline still runs.

CSV export

The database can be exported to CSV for analysis. Two converters in the repository handle this:

• pg_dump_to_csv.py — turns a PostgreSQL dump into one CSV per table, without needing a database connection
• csv_to_json.py — aggregates a per-listing CSV into the JSON files these charts read

Scheduling

Scraping and reporting run on a schedule, via cron or the built-in task scheduler.

Cloud integration

Dumps and exports are stored in AWS S3. These contain per-listing data and belong in private storage — never in the repository or in this site's docs/ directory.

Published statistics

The data charts show removals per year and per month, how long listings stayed up, and the median price per square metre over time.