12 Bash Tricks for Working with Text Files (That Every DevOps Engineer Should Know)

learn these patterns, and you’ll spend less time manually inspecting files

1. Search Multiple Patterns with a Single grep

Most people know:

grep ERROR app.log

But production logs rarely contain only one keyword.

Instead, search multiple patterns.

grep -E "ERROR|WARN|CRITICAL" app.log

Output

WARN Disk usage above 80%
ERROR Database timeout
CRITICAL Kernel panic

Ignore case:

grep -Ei "error|warning|critical" app.log

Search recursively:

grep -R -E "password|secret|token" .

Perfect for scanning repositories.

Instead of running three greps

grep ERROR

grep WARN

grep CRITICAL

you scan the file once. It’s much faster.

2. Display Context Around Matching Lines

When debugging logs, the matching line alone is rarely useful.

Instead:

grep -A 5 ERROR app.log

Shows five lines after.

ERROR Connection refused
Retrying...
Retrying...
Connected

Show before:

grep -B 5 ERROR app.log

Both:

grep -C 5 ERROR app.log

Excellent for incident response.

3. Extract Columns from Messy Files Using awk

Imagine this CSV.

Alice,Finance,5500
Bob,Sales,6200
John,Engineering,9000

Need names only?

awk -F',' '{print $1}' employees.csv

Need names and salary?

awk -F',' '{print $1,$3}' employees.csv

Need only Engineering?

awk -F',' '$2=="Engineering"{print $1,$3}' employees.csv

Output

John 9000

Why awk beats cut cut only slices.

awk can:

• compare
• filter
• calculate
• format
• aggregate

Example

Average salary

awk -F',' '{sum+=$3} END {print sum/NR}' employees.csv

4. Replace Text Safely Across Hundreds of Files

Changing one file:

sed -i 's/http:/https:/g' config.conf

Entire project:

find . -name "*.conf" -exec sed -i 's/http:/https:/g' {} +

As a production tip, preview first:

grep -R "http:" .

Never blindly edit hundreds of files.

5. View Large Files Efficiently with less

Most engineers do this:

cat app.log

For large logs, that’s painful. Instead, do:

less app.log

Some useful shortcuts:

/ERROR     Search forward
n          Next match
N          Previous match
G          Jump to end
g          Jump to beginning

Show line numbers:

less -N app.log

Unlike cat, less lets you navigate huge files interactively without flooding your terminal.

When you’re troubleshooting production incidents, this is often faster than opening an editor.

6. Remove Duplicate Lines Without Sorting

Most tutorials suggest:

sort file | uniq

But this destroys ordering.

Instead:

awk '!seen[$0]++' file.txt

Example: Input

apple
banana
apple
orange
banana

Output

apple
banana
orange

Original order preserved.

Perfect for log processing.

7. Compare Two Configuration Files Like a Pro

Instead of

cat old.conf

cat new.conf

Use

diff -u old.conf new.conf

Output

-listen 80;
+listen 443;

Even better

sdiff old.conf new.conf

Side-by-side comparison.

Excellent before deployments.

8. Extract File Paths and Fields Efficiently

Suppose

john@example.com
alice@test.com

Extract usernames

cut -d@ -f1 users.txt

Output

john
alice

If you want to extract the domains:

cut -d@ -f2 users.txt

And you get:

example.com
test.com

Working with file paths? Instead of complicated pipelines:

basename /var/log/nginx/access.log

Output:

access.log

Need the directory:

dirname /var/log/nginx/access.log

Output:

/var/log/nginx

Shell utilities like basename and dirname are easier to read and maintain than multi-command pipelines.

9. Join Related Files Without Opening Excel

employees.csv

101,Alice
102,Bob
103,John

salary.csv

101,5000
102,8000
103,6200

Join them

join -t',' employees.csv salary.csv

Output

101,Alice,5000
102,Bob,8000

Great for inventory reports.

Important: join Requires Sorted Files

The following may produce incorrect results:

join -t',' employees.csv salary.csv

Sort first:

sort -t',' -k1,1 employees.csv > employees.sorted

sort -t',' -k1,1 salary.csv > salary.sorted

join -t',' employees.sorted salary.sorted

join matches records based on a common field.

If the files aren’t sorted, valid matches may be skipped.

This is one of the most common mistakes engineers make when learning join.

10. Process Massive Files Without Loading Them into Memory

Instead of

cat huge.log

Use streaming.

while read -r line
do
    echo "$line"
done < huge.log

Now you can process files over 100GB.

Example

while read -r line
do
    if [[ $line == *ERROR* ]]; then
        echo "$line"
    fi
done < app.log

Memory efficient.

Both cat and while read stream data.

The advantage of while read isn't lower memory usage.

The advantage is that you can inspect, transform, filter, or act on each line as it arrives.

For example:

while IFS= read -r line
do
    [[ $line == *ERROR* ]] && echo "$line"
done < app.log

This pattern is useful when processing very large files where opening the entire dataset in an editor isn’t practical.

11. Find the Most Common Entries in a Log

Suppose authentication failures.

user1
user2
user1
user3
user1

Count occurrences

sort auth.log | uniq -c | sort -nr

Output

3 user1
1 user2
1 user3

Real-world example

Most common HTTP status

awk '{print $9}' access.log | sort | uniq -c | sort -nr

Output

5421 200
324 404
22 500

Excellent for monitoring.

12. Build a Mini Text Processing Pipeline

Experienced Linux engineers chain simple tools.

Example

Count failed SSH logins.

grep "Failed password" auth.log \
| awk '{print $(NF-3)}' \
| sort \
| uniq -c \
| sort -nr

Output

15 192.168.1.25
9 10.0.0.4
4 172.16.0.8

Pipeline explanation

grep
 ↓
awk
 ↓
sort
 ↓
uniq
 ↓
sort

Each tool performs one job well.

Let’s see some extras

Process substitution

Compare command outputs directly.

diff <(sort old.txt) <(sort new.txt)

Read compressed logs without extracting

zgrep ERROR app.log.gz

View files while they’re changing

tail -F app.log

Unlike tail -f, -F follows files across log rotation.

Show invisible characters

cat -A file.txt

Useful for diagnosing CRLF vs. LF line-ending issues, trailing spaces, and hidden tabs.

Analyze an Application Log

Instead of manually inspecting logs, automate the analysis.

#!/usr/bin/env bash

set -euo pipefail

if [[ $# -ne 1 ]]; then
    echo "Usage: $0 <access-log>" >&2
    exit 1
fi

LOG_FILE="$1"

if [[ ! -f "$LOG_FILE" ]]; then
    echo "File not found: $LOG_FILE" >&2
    exit 1
fi

echo "=== Log Summary ==="

echo
echo "Top 10 IP Addresses:"
awk '{print $1}' "$LOG_FILE" \
| sort \
| uniq -c \
| sort -nr \
| head

echo
echo "HTTP Status Codes:"
awk '{print $9}' "$LOG_FILE" \
| sort \
| uniq -c \
| sort -nr

echo
echo "Top Requested URLs:"
awk '{print $7}' "$LOG_FILE" \
| sort \
| uniq -c \
| sort -nr \
| head

echo
echo "Recent 5xx Errors:"
grep -E ' 5[0-9]{2} ' "$LOG_FILE" || true

Run:

chmod +x analyze-log.sh
./analyze-log.sh access.log

[embed]Create an Automated Disk Cleanup System with Bash ensure disk space never becomes an emergencymedium.com

[embed]12 Bash One-Liners That Actually Solved Production Incidents solve problems across Linux servers, Docker hosts, Kubernetes clusters, and Git repositoriesmedium.com