Learning Web Scraping Changed How I See the Internet

It started as a way to make money on Fiverr. It ended up rewiring my brain.

I didn’t plan to learn web scraping. I was supposed to be finishing a Python course so I could start applying for data entry gigs. Something stable. Something that didn’t involve me staring at code I barely understood.

But late one night, while half-watching a YouTube video about “top Fiverr skills for beginners,” I saw someone offering $50 to scrape product listings from a few e‑commerce sites. Fifty bucks. For copying and pasting, basically. At least that’s what I thought.

I had no idea what I was signing up for.

I still remember the first time I got a requests.get() to work. I was in my room in Dhaka, the ceiling fan making that clicking noise it always does before load shedding hits. I wrote maybe four lines of Python, hit run, and suddenly there it was — an entire page’s HTML, spit out in the terminal. Ugly, unformatted, completely unreadable. But it was the internet, raw. I felt like I’d just picked a lock I didn’t know existed.

That feeling lasted about twelve minutes.

Then I realized the page didn’t actually contain the product prices I needed. They were loaded dynamically, probably with JavaScript. And every time I refreshed too fast, the site started throwing CAPTCHAs at me. I Googled “Python beautifulsoup example” at least fifty times that week. Copy-pasted code from Stack Overflow without understanding a single line. At one point I even restarted my laptop because the error message looked too complicated and I thought maybe Windows needed a moment to think.

It was humbling.

Somewhere in the middle of all that, I fell into the YouTube hole. I’d open my browser to look up “Selenium chromedriver mismatch error” and end up watching a documentary about abandoned shopping malls in America. This happened more times than I care to admit. I told myself it was “research fatigue.” It was really just my brain refusing to deal with another red terminal line.

And then there was the Fiverr client. A kind-sounding person from somewhere in Europe who needed a scraper for a site with infinite scroll and product filters. I said yes even though I’d never used Selenium properly. I spent three days trying to make it work. I fought with chromedriver versions. I learned what “headless” meant by crashing my browser repeatedly. I sent the client a half-working script, and they politely told me it didn’t meet the requirements. No drama, no bad review. Just a quiet “thanks for your effort.” I closed Fiverr and didn’t open it again for two weeks.

That kind of failure doesn’t feel dramatic. It just sits in your chest and makes you wonder if you’re too slow for this.

But I kept going. Not because I’m disciplined — I’m really not. I kept going because scraping had started to change how I looked at the internet. Every website I visited, I began seeing in two layers: the pretty one everyone else sees, and the raw one underneath. The one built out of <div> tags, JSON responses, tracking scripts, and ad networks that ping dozens of servers before the page even finishes loading.

I couldn’t unsee it.

I’d visit Daraz to check the price of a phone, and instead of scrolling through deals, I’d open Developer Tools just to see what API call was returning the product data. I noticed how much of my browsing time was spent waiting for third-party scripts I never asked for. It felt like walking into a store and realizing half the walls are just mirrors designed to keep you inside.

The internet had stopped being a place I simply consumed. It became a massive, messy database, and for the first time I could see the structure underneath. Not all of it, obviously. But enough to feel like the curtain had been pulled back a little.

I’m not a great web scraper even now. I still get blocked. I still forget to rotate user agents. I still haven’t figured out async scraping properly. My scripts are held together with comments like # this works, don’t touch. But I’ve built a few things that actually helped people — a price tracker for a small business owner, a notification system for a friend waiting for exam results to be published on a university site. Small tools. Nothing flashy.

And I’ve become more careful. When you scrape, you see how much data sites collect without you realizing. I started using ad blockers more. I stopped automatically clicking “accept all cookies” like it was a terms-of-service popup I could ignore. I’m not a privacy fanatic, but scraping made me aware that the web is a two-way mirror, and most of us are only looking in one direction.

If you’re thinking of learning web scraping, here’s what I’d tell my beginner self:

Start simple. Use requests and BeautifulSoup. Understand the DOM before jumping to Selenium. When a site blocks you, don’t take it personally — read the robots.txt file and slow down. You’re a guest, not a hacker.

Watch the network tab. I learned more about how websites work from that tab than from any tutorial. See what’s actually being sent and received. It’s the difference between copying code and understanding the conversation between your browser and the server.

Don’t believe every “easy $100” gig you see. Most of those clients need scrapers for heavily protected sites. You’ll spend days fighting Cloudflare and end up exhausted. Build a few projects for yourself first.

Give yourself permission to be bad at it for a while. I spent months feeling like an imposter. You probably will too. That’s just part of learning something where the rules are invisible and constantly changing.

I don’t think web scraping is some life-changing secret. But it did teach me that the internet isn’t magic. It’s just a conversation between computers, and once you learn a little of the language, everything looks different. Even YouTube recommendations. Even product listings. Even your own browsing habits.

The internet is still full of cat videos and distractions. But now I know it’s also a massive, layered machine, and I have the keys to a tiny corner of it. That’s enough for me.