Export WiFi points locations from Wigle gets only 100 points

I want to export all WiFi points in a city from Wigle. I found a [href=”https://medium.com/@neuralnets/building-a-wifi-spots-map-of-networks-around-you-with-wigle-and-python-5adf72a48140" rel=”nofollow noreferrer”>tutorial]() and managed to export only 100 points, but how to export all points? I tried to change the first parameter, but that didn't help.

Here’s my code:

import requests
from requests.auth import HTTPBasicAuth
import pandas as pd
from pandas import json_normalize
import folium
import geopandas

# SETTING WiGLE USERNAME & PASSWORD FOR API CALL:
wigle_username = 'i can share it if you need'
wigle_password = 'i can share it if you need'

city = 'Sao Paulo'
# SETTING PARAMETERS:
payload = {'first': '1', 'freenet': 'false', 'paynet': 'false', 'addresscode': city, 'api_key': (wigle_username + wigle_password).encode()}
# FETCHING JSON RESPONSE FROM WiGLE:
details = requests.get(url='https://api.wigle.net/api/v2/network/geocode', params=payload, auth=HTTPBasicAuth(wigle_username, wigle_password)).json()

# SETTING PARAMETERS:
payload = {'latrange1':'-23.7106507', 'latrange2':'-23.3906507', 'longrange1':'-46.7933824', 'longrange2':'-46.4733824', 'api_key': (wigle_username + wigle_password).encode()}
# FETCHING JSON RESPONSE FROM WiGLE:
results = requests.get(url='https://api.wigle.net/api/v2/network/search', params=payload, auth=HTTPBasicAuth(wigle_username, wigle_password)).json()

# EXTRACTING 'RESULTS' AS A PANDAS DATAFRAME TO WORK WITH:
df = json_normalize(results['results'])
# RENAMING COLUMNS FOR GEOPLOTLIB:
df = df.rename(columns={'trilat': 'lat', 'trilong': 'lon'})
cols = list(df.columns)
# PREVIEWING AVAILABLE INFORMATION:
print(f"Result obtained has {df.shape[0]} rows and {df.shape[1]} columns in it. \n\nThe list of columns include {cols}")

gdf = geopandas.GeoDataFrame(
    df, geometry=geopandas.points_from_xy(df.lon, df.lat), crs="EPSG:4326"
)

interactive_map1 = folium.Map(
    location=(-23.568399795078903, -46.6474148638533),
    zoom_start=10
)

addresses_layer = folium.features.GeoJson(gdf)
addresses_layer.add_to(interactive_map1)
interactive_map1

Solution

After receiving each response, store the searchAfter provided in the payload:

search_after = results['searchAfter']

and then use that value in the subsequent request:

payload = {'latrange1':'-23.7106507', 'latrange2':'-23.3906507', 'longrange1':'-46.7933824', 'longrange2':'-46.4733824', 'searchAfter': search_after} # note: you shouldn't put your credentials in the payload

the first response will also contain a value that will tell you the total number of results for the quad so you can determine how many iterations are necessary:

target_iterations = results['totalResults']/100

note that if the target results exceed your daily threshold, you’ll need to wait until the next calendar day to continue — be sure to test for response.status_code == 429 and break accordingly, preserving your searchAfter

Answered By — AndyC

Answer Checked By — Senaida (FixIt Volunteer)

This Answer collected from stackoverflow, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0