Gobuster: The basics

This room focuses on an introduction to Gobuster, an offensive security tool used for enumeration.

Task 1: Introduction

This room focuses on the offensive security tool Gobuster, often used for reconnaissance. We will explore how this tool can enumerate web directories, subdomains, and virtual hosts. This room will follow a hands-on approach where you can follow along with the commands explained and execute them yourself to see the results.

Learning Objectives

• Understanding the basics of enumeration
• How to use Gobuster to enumerate web directories and files
• How to use Gobuster to enumerate subdomains
• How to use Gobuster to enumerate virtual hosts
• How to use a wordlist

Task 2: Environment and Setup

For this room, we will use an Ubuntu 20.04 VM acting as a web server. This web server hosts multiple subdomains and vhosts. The web server also has two content management systems (CMS) installed. These are Wordpress and Joomla.

Throughout this room, we will be using the AttackBox, where Gobuster is already installed, to enumerate the web server directories and subdomains. However, if you prefer to use your own machine instead of the AttackBox, you must be connected to the TryHackMe VPN and have Gobuster installed. You can find installation instructions for Gobuster on your own machine in the official Gobuster GitHub repository(opens in new tab).

You can start the web server by clicking the Start Lab Machine button below. The VM will take approximately 2 minutes to boot up. Direct access to this web server is not required. To start the AttackBox, click the Start AttackBox button at the top of the page.

Important: We work in a local network with a DNS server on the web server. To ensure we can resolve the domains used throughout this room, you need to change the /etc/resolv-dnsmasq file:

• Open up a terminal on the the AttackBox and enter the command: sudo nano /etc/resolv-dnsmasq.
• Insert nameserver 10.64.162.222 as the first line.
• Save the file by pressing CTRL+O, followed by pressing ENTER, and then exit the editor by pressing CTRL+X.
• Enter the command /etc/init.d/dnsmasq restart to restart the Dnsmasq service.

The file should look something like this:

root@tryhackme:~# cat /etc/resolv-dnsmasq 
nameserver 10.64.162.222
nameserver 169.254.169.253

Task 3: Gobuster: Introduction

Gobuster is an open-source offensive tool written in Golang. It enumerates web directories, DNS subdomains, vhosts, Amazon S3 buckets, and Google Cloud Storage by brute force, using specific wordlists and handling the incoming responses. Many security professionals use this tool for penetration testing, bug bounty hunting, and cyber security assessments. Looking at the phases of ethical hacking, we can place Gobuster between the reconnaissance and scanning phases.

Before exploring Gobuster, let’s briefly discuss the concepts of enumeration and Brute Force.

Enumeration

Enumeration is the act of listing all the available resources, whether they are accessible or not. For example, Gobuster enumerates web directories.

Brute Force

Brute force is the act of trying every possibility until a match is found. It is like having ten keys and trying them all on a lock until one fits. Gobuster uses wordlists for this purpose.

Gobuster: Overview

Gobuster is included by default in distributions like Kali Linux. Let’s start by looking at Gobuster’s help page. This help page gives us a good overview of its functionalities and options.

Enter the following command: gobuster --help. You should get the help page for the Gobuster tool as shown below:

The help page contains multiple sections:

• Usage: Shows the syntax on how to use the command.
• Available Commands: Multiple commands are available to aid us in enumerating directories, files, DNS subdomains, Google Cloud Storage buckets, and Amazon AWS S3 buckets. Throughout this room, we will focus on the dir, dns, and vhost commands. We will cover each of them in the following tasks.
• Flags: These are specific options we can configure to customize our commands. Let’s look at the flags we will often use throughout this room:

Example

Let us look at an example of how we would use these commands and flags together to enumerate a web directory:

gobuster dir -u "http://www.example.thm/" -w /usr/share/wordlists/dirb/small.txt -t 64

• gobuster dir indicates that we will use the directory and file enumeration mode.
• -u "http://www.example.thm/" tells Gobuster that the target URL is http://example.thm/(opens in new tab).
• -w /usr/share/wordlists/dirb/small.txt directs Gobuster to use the small.txt wordlist to brute force the web directories. Gobuster will use each entry in the wordlist to form a new URL and send a GET request to that URL. If the first entry of the wordlist were images, Gobuster would send a GET request to http://example.thm/images/.(opens in new tab)
• -t 64 sets the number of threads Gobuster will use to 64. This improves the performance drastically.

Now that we have a quick overview of Gobuster, let’s explore the different modes and their use cases in the following tasks.

• gobuster dir indicates that we will use the directory and file enumeration mode.
• -u "http://www.example.thm/" tells Gobuster that the target URL is http://example.thm/(opens in new tab).
• -w /usr/share/wordlists/dirb/small.txt directs Gobuster to use the small.txt wordlist to brute force the web directories. Gobuster will use each entry in the wordlist to form a new URL and send a GET request to that URL. If the first entry of the wordlist were images, Gobuster would send a GET request to http://example.thm/images/.(opens in new tab)
• -t 64 sets the number of threads Gobuster will use to 64. This improves the performance drastically.

Now that we have a quick overview of Gobuster, let’s explore the different modes and their use cases in the following tasks.

Questions:

What flag do we use to specify the target URL?

ANS: -u

What command do we use for the subdomain enumeration mode?

ANS: dns

Task 4: Use Case: Directory and File Enumeration

Gobuster has a dir mode, allowing users to enumerate website directories and their files. This mode is useful when you are performing a penetration test and would like to see what the directory structure of a website is and what files it contains. Often, directory structures of websites and web apps follow a particular convention, making them susceptible to Brute Force using wordlists. For example, the directory structure on the web server hosting WordPress looks something like this:

root@tryhackme:~# tree -L 3 -d
.
└── html
    └── wordpress
        ├── wp-admin
        ├── wp-content
        └── wp-includes

Gobuster is powerful because it allows you to scan the website and return the status codes. These status codes immediately tell you if you, as an outside user, can request that directory or not.

Help

If you want a complete overview of what the Gobuster dir command can offer, you can look at the help page. Seeing the extensive help page for the dir command can somewhat be intimidating. So, we will focus on the most essential flags in this room. Type the following command to display the help: gobuster dir --help.

Many flags are used to fine-tune the gobuster dir command. It is out of scope to go over each one of them, but in the table below, we have listed the flags that cover most of the scenarios:

How To Use dir Mode

To run Gobuster in dir mode, use the following command format:

gobuster dir -u "http://www.example.thm" -w /path/to/wordlist

Notice that the command also includes the flags -u and -w, in addition to the dir keyword. These two flags are required for the Gobuster directory enumeration to work. Let us look at a practical example of how to enumerate directories and files with Gobuster dir mode:

gobuster dir -u "http://www.example.thm" -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt -r

This command scans all the directories located at *www.example.thm using the wordlist directory-list-2.3-medium.txt*. Let’s look a bit closer at each part of the command:

• gobuster dir: Configures Gobuster to use the directory and file enumeration mode.
• -u [http://www.example.thm:](http://www.example.thm:)
• The URL will be the base path where Gobuster starts looking. So, the URL above is using the root web directory. For example, in a typical Apache installation on Linux, this is /var/www/html. So if you have a “resources” directory and you want to enumerate that directory, you’d set the URL as http://www.example.thm/resources. You can also think of this like [http://www.example.thm/path/to/folder.](http://www.example.thm/path/to/folder.)
• The URL must contain the protocol used, in this case, HTTP. This is important and required. If you pass the wrong protocol, the scan will fail.
• In the host part of the URL, you can either fill in the IP or the HOSTNAME. However, it is important to mention that when using the IP, you may target a different website than intended. A web server can host multiple websites using one IP (this technique is also called virtual hosting). Use the HOSTNAME if you want to be sure.
• Gobuster does not enumerate recursively. So, if the results show a directory path you are interested in, you will have to enumerate that specific directory.
• -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt configures Gobuster to use the directory-list-2.3-medium.txt wordlist to enumerate. Each entry of the wordlist is appended to the configured URL.
• -r configures Gobuster to follow the redirect responses received from the sent requests. If a status code 301 was received, Gobuster will navigate to the redirect URL that is included in the response.

Let’s look at a second example where we use the -x flag to specify what type of files we want to enumerate:

gobuster dir -u "http://www.example.thm" -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt -x .php,.js

This command will look for directories located at http://example.thm(opens in new tab) using the wordlist directory-list-2.3-medium.txt. In addition to directory listing, this command also lists all the files that have a .php or .js extension.

Questions

Which flag do we have to add to our command to skip the TLS verification? Enter the long flag notation. ANS: — no-tls-validation

Enumerate the directories of www.offensivetools.thm. Which directory catches your attention? ANS: secret

Continue enumerating the directory found in question 2. You will find an interesting file there with a .js extension. What is the flag found in this file? HINT FOR FLAG: gobuster dir -u "http://www.offensivetools.thm/secret" -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt -x .js -t 10

HINT: curl http://www.offensivetools.thm/secret/flag.js

Task 5: Use case: Subdomain Enumeration

The next mode we’ll focus on is the dns mode. This mode allows Gobuster to brute force subdomains. During a penetration test, checking the subdomains of your target’s top domain is essential. Just because something is patched in the regular domain, it doesn't mean it is also patched in the subdomain. An opportunity to exploit a vulnerability in one of these subdomains may exist. For example, if TryHackMe owns tryhackme.thm and mobile.tryhackme.thm, there may be a vulnerability in mobile.tryhackme.thm that is not present in tryhackme.thm. That is why it is important to search for subdomains as well!

Help

If you want a complete overview of what the Gobuster dns command can offer, you can have a look at the help page. Seeing the extensive help page for the dns command can be intimidating. So, we will focus on the most important flags in this room. Type the following command to display the help: gobuster dns --help

The dns mode offers fewer flags than the dir mode. But these are more than enough to cover most DNS subdomain enumeration scenarios. Let us have a look at some of the commonly used flags:

How to Use DNS Mode

To run Gobuster in DNS mode, use the following command syntax: gobuster dns -d example.thm -w /path/to/wordlist

Notice that the command also includes the flags -d and -w, in addition to the dns keyword. These two flags are required for the Gobuster subdomain enumeration to work. Let us look at an example of how to enumerate subdomains with Gobuster dns mode:

gobuster dns -d example.thm -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt

• gobuster dns enumerates subdomains on the configured domain.
• -d example.thm sets the target to the example.thm domain.
• -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt sets the wordlist to subdomains-top1million-5000.txt. Gobuster uses each entry of this list to construct a new DNS query. If the first entry of this list is 'all', the query would be all.example.thm.

Go ahead and enter the command for yourself. You should get the following output:

root@tryhackme:~# gobuster dns -d example.thm -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt 
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Domain:     example.thm
[+] Threads:    10
[+] Timeout:    1s
[+] Wordlist:   /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt
===============================================================
Starting gobuster in DNS enumeration mode
===============================================================
Found: www.example.thm

Found: shop.example.thm

Found: academy.example.thm

Found: primary.example.thm

Progress: 4989 / 4990 (99.98%)
===============================================================
Finished
===============================================================

Questions:

Apart from the dns keyword and the -w flag, which shorthand flag is required for the command to work? ANS: -d

Use the commands learned in this task, how many subdomains are configured for the offensivetools.thm domain? ANS: 4

Task Use Case: Vhost Enumeration

The last and final mode we’ll focus on is the vhost mode. This mode allows Gobuster to brute force virtual hosts. Virtual hosts are different websites on the same machine. Sometimes, they look like subdomains, but don’t be deceived! Virtual hosts are IP-based and are running on the same server. Subdomains are set up in DNS. The difference between vhost and dns mode is in the way Gobuster scans:

• vhost mode will navigate to the URL created by combining the configured HOSTNAME (-u flag) with an entry of a wordlist.
• dns mode will do a DNS lookup to the FQDN created by combining the configured domain name (-d flag) with an entry of a wordlist.

Help

If you want a complete overview of what the Gobuster vhost command can offer, you can have a look at the help page. Seeing the extensive help page for the vhost command can be intimidating. So, we will focus on the most important flags in this room. Type the following command to display the help: gobuster vhost --help

The vhost mode offers flags similar to those of the dir mode. Let us have a look at some of the commonly used flags:

How To Use vhost Mode

To run Gobuster in vhost mode, type the following command:

gobuster vhost -u "http://example.thm" -w /path/to/wordlist

Notice that the command also includes the flags -u and -w, in addition to the vhost keyword. These two flags are required for the Gobuster vhost enumeration to work. Let us look at a practical example of how to enumerate virtual hosts with Gobuster vhost mode:

root@tryhackme:~# gobuster vhost -u "http://10.64.162.222" --domain example.thm -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt --append-domain --exclude-length 250-320 
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:              http://10.10.94.214
[+] Method:           GET
[+] Threads:          10
[+] Wordlist:         /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt
[+] User Agent:       gobuster/3.6
[+] Timeout:          10s
[+] Append Domain:    true
[+] Exclude Length:   250,254,263,274,283,293,294,299,253,261,269,277,285,290,300,257,258,270,278,282,291,252,260,264,268,271,279,280,289,251,256,262,265,272,297,287,292,295,255,266,276,284,286,296,267,273,275,281,288,259,298
===============================================================
Starting gobuster in VHOST enumeration mode
===============================================================
Found: blog.example.thm Status: 200 [Size: 1493]
Found: shop.example.thm Status: 200 [Size: 2983]
Found: www.example.thm Status: 200 [Size: 84352]
Found: chelyabinsk-rnoc-rr02.backbone.example.thm Status: 404 [Size: 304]
Found: academy.example.thm Status: 200 [Size: 434]
Progress: 4989 / 4990 (99.98%)
===============================================================
Finished
===============================================================

You will notice that this command is much more complex than the base command syntax. It contains many more configured flags. This will often be the case in realistic tests, depending on how the infrastructure of the domain to test has been set up. In our case, we don’t have a fully set up DNS infrastructure. This requires us to give in extra flags like --domain and --append-domain. We need to look at the web requests Gobuster sends to understand better how these flags work. Below, you can see a basic GET request to www.example.thm:

GET / HTTP/1.1
Host: www.example.thm
User-Agent: gobuster/3.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive

Gobuster will send multiple requests, each time changing the Host: part of the request. The value of Host: in this example is www.example.thm. We can break this down into three parts:

• www: This is the subdomain. This is the part that Gobuster will fill in with each entry of the configured wordlist.
• .example: This is the second-level domain. You can configure this with the --domain flag (this needs to be configured together with the top-level domain).
• .thm: This is the top-level domain. You can configure this with the --domain flag (this needs to be configured together with the second-level domain).

Now that we know how Gobuster sends its request, let’s break down the command and examine each flag more closely:

• gobuster vhost instructs Gobuster to enumerate virtual hosts.
• -u "http://10.64.162.222" sets the URL to browse to 10.64.162.222.
• -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt configures Gobuster to use the subdomains-top1million-5000.txt wordlist. Gobuster appends each entry in the wordlist to the configured domain. If no domain is explicitly configured with the --domain flag, Gobuster will extract it from the URL. E.g., test.example.thm, help.example.thm, etc. If any subdomains are found, Gobuster will report them to you in the terminal.
• --domain example.thm sets the top- and second-level domains in the Hostname: part of the request to example.thm.
• --append-domain appends the configured domain to each entry in the wordlist. If this flag is not configured, the set hostname would be www, blog, etc. This will cause the command to work incorrectly and display false positives.
• --exclude-length filters the responses we get from the sent web requests. With this flag, we can filter out the false positives. If you run the command without this flag, you will notice you will get a lot of false positives like "Found: Orion.example.thm Status: 404 [Size: 279]" or "Found: pm.example.thm Status: 404 [Size: 276]". These false positives typically have a similar response size, so we can use this to filter out most false positives. We expect to get a 200 OK response back to have a true positive. There are, however, exceptions, but it is not in the scope of this room to go deeper into these.

Questions:

Use the commands learned in this task to answer the following question: How many vhosts on the offensivetools.thm domain reply with a status code 200? ANS: 4

Task 7 : Conclusion

This room taught us about the offensive tool Gobuster. This tool enumerates directories, files, DNS subdomains, and virtual hosts.

We have covered three different modes of the Gobuster tool:

• dns mode: enumerates dns subdomains.
• dir mode: enumerates directories.
• vhost mode: enumerates virtual hosts.

For each mode, we covered the required flags to configure and additional optional flags that fine-tune the desired results.

We have highlighted the difference between virtual hosts and subdomains and the way Gobuster scans for these:

• dns mode uses the DNS services to scan for subdomains using the configured domain and wordlist.
• vhost mode sends web requests using the configured URL and wordlist.

At the end of each task, we directly applied the skills we had learned through hands-on exercises.