How to Extract Keywords from Any Website Using Python

Keyword tools show you what people search for. They don’t show you what your competitors actually write about. That’s a different signal, and you can extract it directly from their pages.

This guide covers how to scrape a competitor’s content and surface the most-used 2- and 3-word phrases using Python.

What You’re Actually Extracting

Keyword tools give you search volume data. Scraping competitor pages gives you semantic density — the terms they repeat most, which often reveals how they structure their topic coverage.

From any page you can pull:

• Bigrams (2-word phrases): “keyword research”, “meta description”, “content gap”
• Trigrams (3-word phrases): “technical seo audit”, “search engine results”, “internal linking structure”
• Frequency count: how often each phrase appears

The hypothesis is simple: if a competitor’s top-ranking page uses “schema markup” 18 times and you don’t mention it once, that’s a gap worth addressing.

Setup

pip install httpx parsel nltk pandas

You’ll also need NLTK’s stopwords:

import nltk
nltk.download('punkt_tab')
nltk.download('stopwords')

The Keyword Extractor

import httpx
from parsel import Selector
from collections import Counter
from nltk import ngrams
from nltk.corpus import stopwords
import re
import pandas as pd

STOPWORDS = set(stopwords.words('english'))

def scrape_keywords(url: str, n_gram_range=(2, 3), top_n=20):
    response = httpx.get(url, follow_redirects=True, timeout=15)
    selector = Selector(response.text)

    # Extract visible body text only (skip nav/footer)
    raw = " ".join(selector.xpath("//main//text() | //article//text() | //p//text()").getall())

    # Normalize and tokenize
    words = re.findall(r'\b[a-z]{3,}\b', raw.lower())
    words = [w for w in words if w not in STOPWORDS]

    results = {}
    for n in range(n_gram_range[0], n_gram_range[1] + 1):
        grams = [" ".join(g) for g in ngrams(words, n)]
        results[f"{n}-gram"] = Counter(grams).most_common(top_n)

    return results

# Run it
url = "https://competitor.com/your-target-topic"
keywords = scrape_keywords(url)

for gram_type, phrases in keywords.items():
    df = pd.DataFrame(phrases, columns=["phrase", "count"])
    print(f"\n--- {gram_type} ---")
    print(df.to_string(index=False))

What each part does:

• xpath("//main//text() | //article//text() | //p//text()") — targets content areas, ignoring nav, sidebars, and footers
• re.findall(r'\b[a-z]{3,}\b', ...) — strips punctuation, numbers, and single/double-character tokens
• STOPWORDS — filter removes "the", "and", "with" etc. so only meaningful terms surface
• Counter().most_common(top_n) — ranks by frequency, returns top N phrases

Comparing Against Your Own Page

Run the scraper against your page and theirs, then diff the results:

competitor_terms = {phrase for phrase, _ in scrape_keywords(competitor_url)["2-gram"]}
your_terms = {phrase for phrase, _ in scrape_keywords(your_url)["2-gram"]}

gaps = competitor_terms - your_terms
print("Missing from your content:", gaps)

Anything in gaps is a term they treat as core vocabulary that you don't. Worth reviewing whether it belongs in your content — not stuffing it in, but checking if it represents a concept you've underexplained.

When This Breaks

JavaScript-rendered content (React, Vue, Next.js): httpx fetches raw HTML, so if the page content is injected client-side, you'll get an empty or near-empty text body.

The cleaner fix is using a web scraping API that handles JS rendering, retries, and anti-bot layers for you, so your script stays focused on parsing, not infrastructure. HasData’s Web Scraping API, for example, can return fully-rendered plain text with a single request — no proxy setup or timeout handling on your end.

For JS-heavy pages, replace scrape_keywords() with this:

def scrape_keywords_js(url: str, api_key: str, n_gram_range=(2, 3), top_n=20):
    response = httpx.post(
        "https://api.hasdata.com/scrape/web",
        headers={"x-api-key": api_key, "Content-Type": "application/json"},
        json={"url": url, "jsRendering": True, "outputFormat": ["text"]},
    )
    response.raise_for_status()
    raw = response.text  # plain text, no HTML parsing needed

    words = re.findall(r'\b[a-z]{3,}\b', raw.lower())
    words = [w for w in words if w not in STOPWORDS]

    results = {}
    for n in range(n_gram_range[0], n_gram_range[1] + 1):
        grams = [" ".join(g) for g in ngrams(words, n)]
        results[f"{n}-gram"] = Counter(grams).most_common(top_n)

    return results

Since the API returns plain text directly, the Selector step is gone — response.text feeds straight into the tokenizer.

Practical Workflow

1. Pick your target keyword, pull the top 3–5 ranking competitor URLs from Google
2. Run scrape_keywords() on each, collect all bigrams/trigrams
3. Merge and sort by combined frequency across all competitors
4. Cross-reference against your own page
5. Export to CSV, review the gaps in a spreadsheet

all_phrases = Counter()
for url in competitor_urls:
    for phrase, count in scrape_keywords(url)["2-gram"]:
        all_phrases[phrase] += count

pd.DataFrame(all_phrases.most_common(30), columns=["phrase", "count"]).to_csv("gaps.csv", index=False)

This gives you a single ranked list of the terms competitors collectively treat as important — a practical starting point for content review, not a replacement for thinking.

This article covers one use case from a broader guide. For the full breakdown — metadata audits, schema analysis, rank tracking, and more scraping scripts — see Web Scraping for SEO: Technical Guide. If you want more advanced Python scripts for SEO automation, Python for SEO covers additional workflows.