AWK is one of the most powerful and lightweight text-processing tools available on Linux and Unix systems. If you work with log files, CSV data, system reports, or structured text, AWK can replace complex scripts with simple one‑liners.
This guide is written for beginners, but it scales all the way to real-world automation.
In this tutorial, you will learn:
- What AWK is and how it works internally
- AWK syntax, patterns, and actions
- Printing and reordering columns
- BEGIN and END blocks
- Filtering rows using conditions
- Writing and executing AWK scripts
- Practical Linux examples you can reuse daily
What is AWK in Linux or Unix?
AWK is an interpreted programming language designed for pattern scanning and text processing. It reads input line by line, splits each line into fields, and applies actions when patterns match.
Unlike traditional programming languages, AWK is data-driven — you describe what to look for and what to do when you find it.
AWK is commonly used for:
- Parsing log files
- Extracting columns from text
- Generating reports
- Filtering structured data
Types of AWK implementations
Most modern Linux distributions ship with GNU AWK (gawk), which is backward compatible with classic AWK.
- AWK – Original implementation
- NAWK – New AWK
- GAWK – GNU AWK (most common)
On Linux, running awk usually executes gawk.
Common AWK variables (quick reference)
These built-in variables are used frequently in real-world AWK scripts:
| Variable | Meaning |
|---|---|
$0 |
Entire input line |
$1, $2, … |
Field (column) values |
NF |
Number of fields in current line |
NR |
Current record (line) number |
FS |
Input field separator |
OFS |
Output field separator |
RS |
Record separator |
Bookmark this table — it answers most AWK questions instantly.
AWK vs sed vs cut (When to Use What)
Understanding when to use the right text-processing tool avoids overengineering simple tasks and keeps commands readable and efficient.
| Tool | Best used for |
|---|---|
awk |
Column-based processing, conditions, calculations, reporting |
sed |
Line-based text replacement and stream editing |
cut |
Fast extraction of fixed columns using delimiters |
grep |
Simple pattern matching and line filtering |
Rule of thumb:
- Need logic, math, or conditions → awk
- Need simple search or replace → sed
- Need fast column extraction → cut
- Need only pattern matching → grep
AWK vs grep (Pattern-Only Use Cases)
Use grep when you only need to match lines based on a pattern.
Example using grep:
grep "ERROR" app.log
Use AWK when you need pattern matching with field-level control.
Equivalent AWK command:
awk '/ERROR/ { print }' app.log
Match a pattern in a specific field (not possible with plain grep):
awk '$3 ~ /ERROR/ { print }' app.log
Combine pattern matching with conditions:
awk '/ERROR/ && $5 > 100 { print }' app.log
AWK Basics & Syntax
AWK command syntax (pattern and action)
An AWK program consists of patterns and actions:
awk 'pattern { action }' file
Both pattern and action are optional:
- Pattern only → print matching lines
- Action only → run action on all lines
AWK syntax: pattern-only example
The awk command syntax with a pattern only is as follows:
awk '/mail/' /etc/passwd
This prints all lines containing the word mail.
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
Sample input file used in all examples
All examples below use the same input file so you can focus on learning AWK, not guessing the data.
userdata.txt
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
Running and Writing AWK Programs
Running AWK from a source file
Instead of writing long one-liners, you can store AWK programs in a file
and execute them using the -f option. This improves readability and
maintainability.
Create an AWK source file (script.awk):
{
print $1, $2
}
Run the AWK program on an input file:
awk -f script.awk data.txt
Use multiple input files:
awk -f script.awk file1.txt file2.txt
Writing Executable AWK Scripts
AWK scripts can be made executable like shell scripts by adding a shebang line.
Create an executable AWK script:
#!/usr/bin/awk -f
{
print $1, $2
}
Make the script executable:
chmod +x script.awk
Run the script directly:
./script.awk data.txt
You can also pass command-line arguments:
./script.awk file1.txt file2.txt
Comments in AWK
Comments are used to document AWK scripts and improve readability. Any text after # is ignored during execution.
# This is a comment
BEGIN { print "Hello" }
Comments are especially helpful in long or shared AWK scripts.
Basic Printing & Line Processing
Print every line (basic AWK example)
This is the most basic use of AWK. It prints every input line exactly as it is, without modifying the content.
awk '{ print }' userdata.txt
Sample output:
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
Print a specific line number
You can selectively print a particular line by matching NR with the desired line number.
awk 'NR == 2' userdata.txt
Sample output:
1 Deepak 31 deepak
Print even-numbered lines
Using the modulo operator with NR, you can filter even-numbered lines from the file.
awk 'NR % 2 == 0' userdata.txt
Sample output:
1 Deepak 31 deepak
3 Amit 33 amit
Print odd-numbered lines
This command prints only odd-numbered lines using a modulo operation on NR.
awk 'NR % 2 == 1' userdata.txt
Sample output:
id Name Age username
2 Rahul 32 rahul
4 Sumit 33 sumit
Print a range of lines using NR
The range operator allows printing a continuous block of lines based on line numbers.
awk 'NR == 2, NR == 4' userdata.txt
Sample output:
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
Print a range of lines using patterns
Instead of line numbers, you can define a start and end pattern to print a block of lines.
awk '/Deepak/,/Amit/' userdata.txt
Sample output:
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
BEGIN and END Blocks
BEGIN and END blocks in AWK
The BEGIN block runs before any input is processed, and the END block runs after all input lines are processed. These blocks are commonly used for headers, footers, and initialization.
awk 'BEGIN { print "== Start ==" }
{ print }
END { print "== End ==" }' userdata.txt
Sample output:
== Start ==
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
== End ==
Print header using BEGIN block
The BEGIN block is executed before input processing starts, making it ideal for printing headers.
awk 'BEGIN { print "ID Name Age User" } { print }' userdata.txt
Sample output:
ID Name Age User
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
Print the last line using END block
Since variables retain their last assigned value, the END block can be used to print the final record.
awk '{ last = $0 } END { print last }' userdata.txt
Sample output:
4 Sumit 33 sumit
Field and Column Operations
Print selected columns
AWK treats each column as a field. Fields are accessed using $1, $2, $3, and so on.
awk '{ print $1, $2 }' userdata.txt
Sample output:
id Name
1 Deepak
2 Rahul
3 Amit
4 Sumit
This is commonly used to extract specific columns from structured data.
Print columns with conditions
Patterns can be combined with actions so that columns are printed only when a condition matches.
awk '/deepak/ { print $1, $2, $3 }' userdata.txt
Sample output:
1 Deepak 31
This helps filter and format output in a single command.
Reorder columns with custom text
This example reorders fields and formats the output using custom text. It prints the Name ($2) followed by the username ($4) inside parentheses.
awk '{ print $2 " (" $4 ")" }' userdata.txt
Sample output:
Deepak (deepak)
Rahul (rahul)
Amit (amit)
Sumit (sumit)
This is useful for generating readable reports.
Printing the number of columns in a line (NF)
NF is a built-in AWK variable that stores the number of fields in the current line. It is often used for validation and dynamic field access.
awk '{ print NF, $1, $NF }' userdata.txt
Sample output:
4 id username
4 1 deepak
4 2 rahul
4 3 amit
4 4 sumit
This example prints the field count, the first field, and the last field of each line.
Change the field separator (FS)
By default, AWK uses spaces as field separators. You can change this using -F or the FS variable. This example uses /etc/passwd, where fields are separated by colons.
awk -F: '{ print $1 }' /etc/passwd
Sample output:
root
bin
daemon
Pattern Matching & Regular Expressions
Match pattern and print lines
When only a pattern is specified, AWK automatically prints lines that match the pattern.
awk '/rahul/' userdata.txt
Sample output:
2 Rahul 32 rahul
This is useful for quickly filtering records based on text content.
Match a pattern in a specific field using (~)
The match operator (~) is used to check whether a specific field
matches a regular expression.
Command used:
awk '$2 ~ /a/' userdata.txt
Explanation:
$2refers to the second column, which is theNamefield.~means “matches the pattern”./a/is a regular expression that matches any value containing the lowercase letter a.- AWK prints the entire line when the condition is true.
Sample output:
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
Match beginning of a field using (^)
The caret (^) is a regex anchor that matches the start of a field.
Command used:
awk '$2 ~ /^A/' userdata.txt
Explanation:
$2refers to the Name column.^Ameans the field must start with the letterA.- Only names beginning with uppercase
Aare matched. - AWK prints the entire line when the condition is true.
Sample output:
3 Amit 33 amit
Why this line matched:
- Amit starts with the letter
A - Other names do not start with
A
Match character classes using []
Character classes ([]) allow matching one character from a
defined set.
Command used:
awk '$2 ~ /^[AD]/' userdata.txt
Explanation:
^[AD]means the field must start with eitherAorD.- Character classes match exactly one character at that position.
- Matching is case-sensitive by default.
Sample output:
1 Deepak 31 deepak
3 Amit 33 amit
Why these lines matched:
- Deepak starts with
D - Amit starts with
A
Match end of a field using ($)
The dollar sign ($) is a regex anchor that matches the end of a
field.
Command used:
awk '$2 ~ /k$/' userdata.txt
Explanation:
k$means the field must end with the letterk.$ensures the match occurs at the end, not anywhere in the field.
Sample output:
1 Deepak 31 deepak
Why this line matched:
- Deepak ends with the letter
k - Other names end with different characters
Conditional Logic & Comparisons
Numeric comparison using relational operators
AWK treats fields as numbers when used with relational operators
such as >, <, >=, <=, ==, and !=.
awk '$3 >= 32' userdata.txt
Explanation:
$3refers to the Age column.>= 32selects records where the numeric value of Age is greater than or equal to 32.- AWK automatically performs numeric comparison without explicit type conversion.
Sample output:
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
String comparison in a field
String comparisons in AWK must use double quotes around the string value.
awk '$4 == "deepak"' userdata.txt
Explanation:
$4refers to the username column.== "deepak"performs an exact string match.- String comparisons are case-sensitive by default.
Sample output:
1 Deepak 31 deepak
Why this line matched:
- The username value exactly matches
deepak
Combine multiple conditions using logical operators
AWK supports logical operators such as:
&&→ AND||→ OR!→ NOT
awk '$3 >= 30 && $3 <= 32' userdata.txt
Explanation:
- The first condition checks Age ≥ 30
- The second condition checks Age ≤ 32
- Both conditions must be true for the record to be printed.
Sample output:
1 Deepak 31 deepak
2 Rahul 32 rahul
Why these lines matched:
- Deepak → Age 31 (within range)
- Rahul → Age 32 (within range)
Counting, Statistics & Built-in Variables
Count total number of lines in a file
NR is a built-in AWK variable that stores the current record (line) number.
Since NR keeps increasing as lines are read, its final value represents
the total number of lines in the file.
awk 'END { print NR }' userdata.txt
Explanation:
- AWK processes the file line by line.
NRincrements automatically for each line.- The
ENDblock runs after the last line is processed. - Printing
NRin theENDblock outputs the total line count.
This is functionally similar to: wc -l userdata.txt
Sample output:
5
Count characters using length()
The length() function returns the number of characters in a string.
When applied to $0, it counts characters in the entire line.
awk '{ print length($0), $0 }' userdata.txt
Explanation:
$0represents the entire current line.length($0)counts all characters, including spaces.- The output shows the character count followed by the original line.
Sample output:
23 id Name Age username
22 1 Deepak 31 deepak
22 2 Rahul 32 rahul
22 3 Amit 33 amit
22 4 Sumit 33 sumit
Remove empty lines using NF
NF represents the number of fields in the current line.
For empty or blank lines, NF evaluates to 0.
awk 'NF > 0' userdata.txt
Explanation:
- Lines with at least one field have
NF > 0. - Empty or whitespace-only lines have
NF == 0. - AWK prints only lines that satisfy the condition.
This effectively removes:
- Blank lines
- Lines containing only spaces or tabs
Sample output:
id Name Age username
1 Deepak 31 deepak
2 Rahul 32 rahul
3 Amit 33 amit
4 Sumit 33 sumit
Print line numbers using NR
NR is a built-in variable that stores the current line number. Prefixing it prints line numbers alongside each record.
awk '{ print NR, $0 }' userdata.txt
Explanation:
NRprefixes each line with its line number.$0prints the full line content.- The output shows numbered lines, similar to
nlorcat -n.
Sample output:
1 id Name Age username
2 1 Deepak 31 deepak
3 2 Rahul 32 rahul
4 3 Amit 33 amit
5 4 Sumit 33 sumit
Control Structures and Programming in AWK
Control structures: AWK while loop
AWK supports control structures such as loops. This example uses a while loop to print numbers from 1 to 4.
awk 'BEGIN { n=1; while (n < 5) { print n; n++ } }'
Explanation:
- The
BEGINblock runs before any input is read. n=1initializes a variable.while (n < 5)keeps looping as long as the condition is true.print noutputs the current value ofn.n++incrementsnby 1 on each iteration.
Sample output:
1
2
3
4
Using Variables in AWK
Variables in AWK are created automatically when first used. They can store numbers, strings, or computed values.
Assign and use a variable:
awk '{ total = total + $3 } END { print total }' userdata.txt
Explanation:
totalis a user-defined variable.$3refers to the Age column.- The variable accumulates values for each line.
- The
ENDblock prints the final result.
Pass variables from the command line using -v:
awk -v min=32 '$3 >= min { print $1, $3 }' userdata.txt
Explanation:
-v min=32defines a variable before processing starts.- This avoids hardcoding values inside the AWK program.
- Useful for reusable and parameterized scripts.
Use variables inside BEGIN block:
awk -v header="Name Age" 'BEGIN { print header } { print $1, $2 }' userdata.txt
Mental model:
- Variables are untyped (AWK decides number vs string automatically)
- Variables persist across lines
-vis the preferred way to pass external values
Using if-else Conditions in AWK
AWK supports conditional logic using if, else if, and else.
Classify values using if-else:
awk '{
if ($2 >= 30)
print $1, "Senior"
else
print $1, "Junior"
}' data.txt
Multiple conditions using else if:
awk '{
if ($2 >= 40)
print $1, "Expert"
else if ($2 >= 30)
print $1, "Intermediate"
else
print $1, "Beginner"
}' data.txt
Use if-else inside BEGIN block:
awk 'BEGIN {
if (1)
print "AWK condition check passed"
else
print "This will never run"
}'
Using for Loops in AWK
AWK supports for loops for iterating over fields, records, or arrays.
Print all fields in each line using a loop:
awk '{
for (i = 1; i <= NF; i++)
print $i
}' file.txt
Print field number along with value:
awk '{
for (i = 1; i <= NF; i++)
print "Field", i ":", $i
}' file.txt
Associative Arrays in AWK
AWK arrays are associative (key-value based), not index-based.
Count occurrences of the first column:
awk '{ count[$1]++ } END {
for (item in count)
print item, count[item]
}' data.txt
Group and sum values by key:
awk '{ sum[$1] += $2 } END {
for (item in sum)
print item, sum[item]
}' data.txt
Track unique values:
awk '!seen[$1]++ { print $1 }' data.txt
Using Functions in AWK
You can define and reuse functions inside AWK programs.
Define and use a function:
awk '
function classify(age) {
if (age >= 40)
return "Expert"
else if (age >= 30)
return "Intermediate"
else
return "Beginner"
}
{
print $1, classify($2)
}' data.txt
Use functions with BEGIN block:
awk '
function banner() {
print "=== Processing File ==="
}
BEGIN {
banner()
}
{ print $0 }
' file.txt
Advanced Data Handling
AWK for CSV and Delimiter-Based Files
By default, AWK uses whitespace as the field separator. For CSV or other
delimiter-based files, specify the delimiter using -F.
Example CSV file (data.csv):
Name,Age,City
Alice,30,Bangalore
Bob,25,Chennai
Carol,28,Hyderabad
Print the first and third columns from a CSV file:
awk -F',' '{ print $1, $3 }' data.csv
Skip the header row:
awk -F',' 'NR > 1 { print $1, $3 }' data.csv
Using OFS to Control Output Formatting
OFS (Output Field Separator) controls how fields are separated in the output.
Print columns separated by |:
awk '{ OFS=" | "; print $1, $2, $3 }' file.txt
Set both input and output field separators:
awk -F',' '{ OFS=" | "; print $1, $2, $3 }' data.csv
Format output as CSV explicitly:
awk '{ OFS=","; print $1, $2, $3 }' file.txt
Practical AWK One-Liners for Logs
Extract IP addresses from access logs:
awk '{ print $1 }' access.log
Count requests per IP:
awk '{ count[$1]++ } END {
for (ip in count)
print ip, count[ip]
}' access.log
Filter logs by HTTP status code:
awk '$9 == 404 { print $0 }' access.log
Print timestamp and request URL:
awk '{ print $4, $7 }' access.log
Summary
AWK is a powerful yet lightweight text-processing tool that excels at working with structured data such as logs, command output, and delimited files. In this tutorial, you learned AWK progressively—from basic syntax to practical, real-world use cases—without unnecessary complexity.
By the end of this guide, you should be comfortable with:
- Understanding how AWK processes input line by line
- Using patterns and actions effectively
- Printing, filtering, and reordering columns
- Working with built-in variables like
NR,NF, andFS - Writing reusable AWK scripts with
BEGINandENDblocks - Applying conditions, comparisons, and regular expressions
- Using AWK loops and control structures
This tutorial is intentionally structured as a cheat sheet + hands-on guide, making it easy to revisit specific examples whenever you need them in day-to-day Linux or Unix work.
If you practice these examples regularly, AWK can replace many complex shell scripts with simple, fast, and readable one-liners.
References
Use the following references to deepen your understanding or explore advanced AWK features:
- GNU AWK Manual – Official and most comprehensive reference for AWK
- AWK man page – Quick command-line reference
- Learning AWK Programming (O’Reilly) – In-depth book for serious learners
- POSIX AWK Specification – Standard behavior across Unix systems
Next steps
To continue improving your AWK skills:
- Apply AWK to real log files and command output
- Combine AWK with shell pipelines such as
grep,
sort, anduniq - Write small AWK scripts instead of long shell loops
- Explore advanced topics such as associative arrays and functions
Mastering these concepts will make AWK one of the most productive tools in your Linux tool
