Building a Web Scraper for Douban Group Posts

In this tutorial, we’ll build a Python web scraper that downloads featured posts from Douban.com’s minimalist lifestyle group. The scraper will save both text content and images from each post into separate folders.

Prerequisites

• Python 3.x
• Basic understanding of HTML and web scraping
• Required Python packages: requests, beautifulsoup4, hashlib

Project Structure

douban-scraper/
├── scraper.py     # Main scraper class
├── main.py        # Script to run the scraper
└── posts/         # Downloaded posts (auto-generated)

Step 1: Creating the Scraper Class

First, let’s create scraper.py with our main scraper class:

import requests
from bs4 import BeautifulSoup
import os
import time
import re
from urllib.parse import urljoin
import hashlib

class DoubanScraper:
    def __init__(self):
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
        }

    def sanitize_filename(self, filename):
        # Remove invalid characters and handle long filenames
        sanitized = re.sub(r'[<>:"/\\|?*]', '', filename)
        hash_suffix = hashlib.md5(filename.encode()).hexdigest()[:6]
        if len(sanitized) > 30:
            sanitized = sanitized[:30] + f"_{hash_suffix}"
        return sanitized

    def get_post_content(self, url):
        response = requests.get(url, headers=self.headers)
        soup = BeautifulSoup(response.text, 'html.parser')

        content = soup.find('div', class_='topic-content')
        if not content:
            return None, []

        images = []
        for img in content.find_all('img'):
            img_url = img.get('src')
            if img_url:
                images.append(urljoin(url, img_url))

        return content.text.strip(), images

Step 2: Adding Save Functionality

Add the save_post method to the DoubanScraper class:

def save_post(self, title, author, content, images, base_url):
    folder_name = self.sanitize_filename(title)

    try:
        os.makedirs(folder_name, exist_ok=True)

        image_paths = []
        for i, img_url in enumerate(images):
            img_path = f"{folder_name}/image_{i+1}.jpg"
            try:
                img_data = requests.get(img_url, headers=self.headers).content
                with open(img_path, 'wb') as f:
                    f.write(img_data)
                image_paths.append(img_path)
            except Exception as e:
                print(f"Failed to save image: {e}")
                continue

        md_content = f"""# {title}

Author: {author}
Source: {base_url}

## Content
{content}

## Images
"""

        for img_path in image_paths:
            md_content += f"![Image]({img_path})\n"

        with open(f"{folder_name}/post.md", 'w', encoding='utf-8') as f:
            f.write(md_content)

        print(f"Successfully saved post to folder: {folder_name}")

    except Exception as e:
        print(f"Failed to save post: {e}")

Step 3: Creating the Main Script

Create main.py to run the scraper:

from scraper import DoubanScraper
import requests
from bs4 import BeautifulSoup
import time

def main():
    # List of posts to skip (optional)
    skip_titles = ["Post Title 1", "Post Title 2"]

    scraper = DoubanScraper()
    base_url = "https://www.douban.com/group/minimalists/discussion?start=0&type=elite"

    response = requests.get(base_url, headers=scraper.headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    table = soup.find('table', class_='olt')
    if not table:
        print("Failed to get post list. Login might be required.")
        return

    rows = table.find_all('tr')[1:]  # Skip header row

    for row in rows:
        title_cell = row.find('td', class_='title')
        if not title_cell:
            continue

        link = title_cell.find('a')
        if not link:
            continue

        title = link.text.strip()

        if any(skip_title in title for skip_title in skip_titles):
            print(f"Skipping post: {title}")
            continue

        url = link['href']

        author_cell = row.find('td', attrs={'nowrap': 'nowrap'})
        author = author_cell.find('a').text if author_cell and author_cell.find('a') else "Unknown"

        print(f"Processing: {title} by {author}")

        content, images = scraper.get_post_content(url)
        if content:
            scraper.save_post(title, author, content, images, url)

        time.sleep(2)  # Avoid hitting rate limits

if __name__ == "__main__":
    main()

Key Features

1. Filename Sanitization: Handles long filenames and removes invalid characters
2. Image Download: Saves all images from posts
3. Markdown Generation: Creates formatted markdown files with post content
4. Rate Limiting: Includes delays to avoid being blocked
5. Skip Functionality: Allows skipping specific posts
6. Error Handling: Robust error handling for network issues

Challenges Addressed

1. Windows Path Length Limitation: Implemented filename truncation with hash suffixes
2. Anti-Scraping Measures: Added user-agent headers and delays
3. Character Encoding: Proper handling of Chinese characters
4. File Organization: Structured storage of posts and images

Usage

• Install required packages:

pip install requests beautifulsoup4

• Run the scraper:

python main.py

Output Structure

For each post, the scraper creates:

• A folder named after the post title
• A markdown file containing the post content
• Downloaded images from the post

Best Practices

1. Always respect the website’s robots.txt
2. Implement reasonable delays between requests
3. Handle errors gracefully
4. Use proper user-agent headers
5. Sanitize filenames for cross-platform compatibility

Conclusion

This scraper demonstrates how to effectively download and organize content from Douban groups while handling common challenges in web scraping. The code is extensible and can be modified for other similar websites.

Remember to check the website’s terms of service and robots.txt before scraping, and always scrape responsibly!

Note: This tutorial is for educational purposes only. Always ensure you have permission to scrape content from websites.