For a senior backend engineer, Linux commands are more than tools for navigating directories.

They become a production debugging language.

When a Rails application is consuming too much memory, a log contains millions of lines, a deployment introduces unexpected configuration changes, or a background worker suddenly starts failing, knowing how to combine commands such as sed, awk, grep, find, xargs, sort, uniq, cut, tr, jq, and ps can save hours.

The real skill is not memorizing commands. It is understanding how to compose them into pipelines.

command1 | command2 | command3

This article focuses on commands and techniques that become particularly valuable at a senior engineering level.

1. grep – Search With Intent

Most developers know:

grep "ERROR" production.log

But grep becomes much more powerful with regular expressions and recursive searches.

Search recursively

grep -R "ActiveRecord::Deadlocked" log/

Useful when you don’t know which file contains the problem.

Ignore case

grep -Ri "timeout" .

Show line numbers

grep -n "connection refused" production.log

Search multiple patterns

grep -E "ERROR|FATAL|Exception" production.log

Show context around matches

grep -C 5 "NoMethodError" production.log

This is extremely useful for application logs because the surrounding lines often contain request IDs, parameters, stack traces, or timestamps.

When to use

Use grep when your primary operation is:

“Find lines matching this condition.”

2. sed – Stream Editing

sed is one of the most useful Linux commands for manipulating text without opening an editor.

The simplest example:

sed 's/foo/bar/g' file.txt

Replace every foo with bar.

Delete lines

Delete empty lines:

sed '/^$/d' file.txt

Delete lines containing DEBUG:

sed '/DEBUG/d' production.log

Print specific lines

sed -n '100,150p' production.log

This displays lines 100 through 150.

Very useful when investigating a specific portion of a huge log file.

Modify a configuration file

For example:

sed -i 's/RAILS_LOG_LEVEL=info/RAILS_LOG_LEVEL=debug/' .env

-i modifies the file in place.

Be careful with production configuration files. Prefer making a backup when appropriate:

sed -i.bak 's/old_value/new_value/g' config.yml

Advanced use: remove sensitive information

Suppose logs contain email addresses:

User login: john@example.com
User login: alice@example.com

We can mask them:

sed -E 's/[[:alnum:]._%+-]+@[[:alnum:].-]+\.[A-Za-z]{2,}/[REDACTED]/g' app.log

This is useful when sanitizing logs before sharing them.

When to use sed

Think:

“I want to transform or filter text while streaming it.”

3. awk – Lightweight Data Processing

awk is one of the most important commands for senior engineers.

It treats input as structured columns.

Suppose:

101 John 4500
102 Alice 6000
103 Bob 5000

Run:

awk '{print $1, $3}' users.txt

Output:

101 4500
102 6000
103 5000

Filter records

awk '$3 > 5000 {print $1, $2, $3}' users.txt

Now only users earning more than 5000 are printed.

Calculate values

awk '{sum += $3} END {print sum}' users.txt

Calculate the total salary.

Average:

awk '{sum += $3; count++} END {print sum/count}' users.txt

Processing logs

Imagine an Nginx log:

10.0.0.1 GET /users 200
10.0.0.2 GET /users 500
10.0.0.3 GET /products 200
10.0.0.4 GET /users 500

Extract HTTP status:

awk '{print $4}' access.log

Count status codes:

awk '{print $4}' access.log | sort | uniq -c

Result:

2 200
2 500

awk with conditions

awk '$4 >= 500 {print}' access.log

Find server errors.

When to use awk

Think:

“My input has columns/records and I need to filter, transform, aggregate, or calculate something.”

For quick operational data analysis, awk can often replace writing a small script.

4. cut – Extract Columns

For simple column extraction, cut is usually easier than awk.

Example:

cut -d',' -f1 users.csv

Extract the first CSV field.

Multiple fields:

cut -d',' -f1,3 users.csv

Character ranges:

cut -c1-10 file.txt

Use cut when the operation is straightforward.

Use awk when logic becomes conditional or computational.

5. sort + uniq – Finding Patterns

These commands become extremely powerful together.

Suppose you want to find the most common URLs:

awk '{print $7}' access.log |
sort |
uniq -c |
sort -nr

Example:

1500 /api/users
980  /api/orders
450  /health

This is a classic production-analysis pipeline.

Why sort before uniq?

uniq only detects adjacent duplicate lines.

Therefore:

sort file.txt | uniq

is usually required.

6. head and tail – Inspect Large Files Safely

Instead of opening a 10 GB log:

head -n 50 production.log

Last 100 lines:

tail -n 100 production.log

The real power is:

tail -f production.log

Follow new log entries in real time.

For Rails applications this is particularly useful during deployments:

tail -f log/production.log

You can combine it with grep:

tail -f production.log | grep --line-buffered "ERROR"

Now you’re effectively monitoring errors as they occur.

7. find – Locate Files Precisely

Find Ruby files:

find app/ -type f -name "*.rb"

Find files modified recently:

find log/ -type f -mtime -1

Find large files:

find /var/log -type f -size +500M

Find and execute a command:

find tmp/ -type f -name "*.tmp" -delete

Be careful with destructive commands.

A safer approach is:

find tmp/ -type f -name "*.tmp" -print

Inspect the result first.

8. xargs – Turn Output Into Arguments

Suppose:

find tmp/ -type f -name "*.tmp"

returns many files.

You can pass them to another command:

find tmp/ -type f -name "*.tmp" -print0 |
xargs -0 rm

-print0 and -0 are important because filenames can contain spaces or special characters.

Another example:

grep -Rl "TODO" app/ | xargs wc -l

This finds files containing TODO and counts their lines.

9. ps – Understand Running Processes

For a Rails server:

ps aux | grep puma

More useful:

ps aux --sort=-%mem | head

Find processes consuming the most memory.

CPU:

ps aux --sort=-%cpu | head

This can quickly identify runaway workers.

10. top and htop – Live System Diagnosis

top

For an easier interactive interface:

htop

Use these when diagnosing:

• High CPU
• Memory pressure
• Load
• Runaway processes
• Number of workers
• Process states

For application debugging, don’t look only at Rails logs. Always correlate application behavior with OS-level resource usage.

11. df vs du

These commands answer different questions.

Disk filesystem usage

df -h

Answers:

How full is the filesystem?

Directory usage

du -sh log/

Answers:

What is consuming the space?

Find the largest directories:

du -sh * | sort -hr | head

This is extremely useful when a server suddenly reports:

No space left on device

12. lsof – Discover Who Owns a Resource

Find which process is using port 3000:

lsof -i :3000

Find processes using a file:

lsof /var/log/production.log

Find deleted files still consuming disk:

lsof +L1

This last one is particularly valuable.

A process may keep a deleted log file open. du may not show the file anymore, while disk space remains consumed until the process releases it.

13. ss – Network Investigation

Modern Linux systems commonly use ss for socket inspection.

Check listening ports:

ss -lntp

Check established connections:

ss -nt

Find connections to port 5432:

ss -nt | grep ':5432'

This can help investigate:

• PostgreSQL connection exhaustion
• Unexpected network connections
• Services not listening
• Connection buildup

14. jq – JSON From the Command Line

Modern APIs produce JSON everywhere.

Suppose:

{
  "users": [
    {"id": 1, "name": "John"},
    {"id": 2, "name": "Alice"}
  ]
}

Extract names:

jq '.users[].name' response.json

Output:

"John"
"Alice"

Transform it:

jq -r '.users[] | "\(.id),\(.name)"' response.json

This becomes especially powerful when debugging APIs:

curl -s https://example.com/api/users |
jq '.users[] | select(.active == true)'

15. curl – API Debugging From the Shell

Instead of immediately reaching for Postman:

curl -i https://example.com/health

POST JSON:

curl -X POST https://example.com/api/users \
  -H "Content-Type: application/json" \
  -d '{"name":"John"}'

Measure request timing:

curl -o /dev/null -s \
  -w 'HTTP: %{http_code}\nTime: %{time_total}s\n' \
  https://example.com

This is extremely useful when debugging production APIs.

16. tee – See and Save Output Simultaneously

bundle exec rails db:migrate 2>&1 | tee migration.log

The output is displayed on the terminal while simultaneously being written to a file.

Useful during deployments and troubleshooting.

17. Powerful Pipelines

The real senior-level skill comes from combining commands.

For example, identify the most frequent 500 responses:

grep " 500 " access.log |
awk '{print $7}' |
sort |
uniq -c |
sort -nr |
head -20

Or find the largest log files:

find /var/log -type f -size +100M -print |
xargs -r ls -lh |
sort -k5 -hr

Or monitor Rails errors:

tail -f log/production.log |
grep --line-buffered -E "ERROR|FATAL|Exception"

18. A Practical Senior Engineer Mental Model

Instead of memorizing hundreds of commands, categorize them.

The most important progression is:

Basic Linux
    ↓
Individual commands
    ↓
Pipelines
    ↓
Conditional filtering
    ↓
Aggregation
    ↓
Production diagnosis

A senior engineer should be comfortable turning an unclear operational question into a shell pipeline.

For example:

“Which API endpoints are causing the most HTTP 500 errors right now?”

Instead of manually opening a log file, you should naturally arrive at something like:

grep " 500 " access.log |
awk '{print $7}' |
sort |
uniq -c |
sort -nr |
head -20

That is the real power of Linux:

small, composable tools solving complex operational problems.

For Rails engineers especially, mastering these commands means you can diagnose the application, process, filesystem, network, and logs from the same shell instead of relying entirely on application-level tooling.

Happy commanding!