Bash Script Generator
Creates bash scripts with argument parsing, error handling, logging, and input validation following current standards. Use when creating new bash or shell scripts, .sh files, CLI tools, scripting automation, text processing workflows (grep/awk/sed pipelines), or building production-ready command-line utilities. Trigger phrases include 'write a bash script', 'create a shell script', 'generate a .sh file', 'bash command', 'scripting', or 'automate with bash'.
npx skills add github:pantheon-org/tekhne --skill development--scripting--bash-script--generatorOverview
This skill generates production-ready bash scripts with best practices built-in: strict mode, error handling, logging, argument parsing, input validation, and cleanup traps. Use for system administration, text processing, API clients, automation workflows, and scheduled tasks.
Mindset
Treat every generated script as if it will run unattended, at 3am, on a machine nobody is watching. Assume any command can fail and any variable can be empty; the script’s job is to fail loudly and safely rather than continue silently on bad data. Never trust set -e alone to make a script safe — always pair it with -u and -o pipefail, because a script that only sets -e still lets pipeline failures and unset-variable typos through undetected. Always register a trap before doing anything destructive, because a script that creates temp files or partial output without a cleanup trap leaves garbage (or worse, corrupt state) behind on every failure path, not just the happy path. Verify every external dependency (command -v) before using it rather than letting a missing binary produce a cryptic “command not found” three functions deep into execution. Write status output to stderr, not stdout, so a script’s logs never get accidentally piped into whatever consumes its actual output.
When to Use
Use this skill when writing a new bash or POSIX sh script, converting an ad-hoc one-liner or cron job into a maintainable script, adding argument parsing, logging, or error handling to an existing script, or when a script needs to run unattended (cron, CI, systemd timer) and must fail safely instead of silently.
When NOT to Use
Do not use this skill to write a script in a general-purpose language (Python, TypeScript, Go) — reach for bash only when the task is genuinely shell-shaped (orchestrating other commands, simple text pipelines, filesystem operations). Do not use it for a one-off interactive command typed directly at a prompt; the strict-mode and trap-handling ceremony this skill teaches is overhead that only pays off once a script is going to be saved, reused, or scheduled.
Pre-Generation Checklist (REQUIRED)
Before writing any script, complete these steps:
-
Clarify ambiguities — ask if any of the following are unclear:
- Input data format (nginx log, JSON, CSV, custom?)
- Large file handling (>100MB)?
- Error handling preference (fail fast / continue / retry)?
- Security context (sensitive data, elevated privileges)?
- Portability needs (bash-specific vs POSIX sh)?
- Output format (human-readable, JSON, CSV)?
-
Explain your approach — before writing code, briefly describe:
- Script architecture and main functions
- Tool selections (grep/awk/sed) with rationale from
references/text-processing-guide.md - Key design decisions and customization points
-
Use the template for standard scripts (CLI tools, automation scripts):
bash scripts/generate_script_template.sh standard output-script.shThen customize for your specific use case.
Generation Workflow
Stage 1 – Understand Requirements
Determine: purpose, input/output sources, bash vs POSIX sh, argument needs, error handling strategy, performance constraints, security requirements. Use AskUserQuestion for anything unclear.
Stage 2 – Architecture Planning
- Select functions, config management, logging strategy, error handling approach
- Tool selection: grep (pattern matching/filtering), awk (structured data, calculations), sed (substitutions, stream editing), find (filesystem), curl/wget (HTTP)
- Plan
set -euo pipefail, error functions, andtrapcleanup
Stage 3 – Script Structure
#!/usr/bin/env bash
#
# Script Name: script-name.sh
# Description: Brief description
# Created: YYYY-MM-DD
#
set -euo pipefail
IFS=$'\n\t'
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
cleanup() {
local exit_code=$?
# Remove temp files, release locks, etc.
exit "${exit_code}"
}
trap cleanup EXIT ERR INT TERM
Stage 4 – Core Functions
Full implementations are in assets/templates/standard-template.sh. Key signatures to include:
Logging (one line per level):
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') - $*" >&2; }
log_warn() { echo "[WARN] $(date '+%Y-%m-%d %H:%M:%S') - $*" >&2; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - $*" >&2; }
log_fatal() { echo "[FATAL] $(date '+%Y-%m-%d %H:%M:%S') - $*" >&2; exit 1; }
# Add log_debug with LOG_LEVEL guard from template when DEBUG support is needed
Error handling:
die() { log_error "$@"; exit 1; }
check_command() { command -v "$1" &>/dev/null || die "Required command '$1' not found."; }
validate_file() { [[ -f "$1" ]] || die "File not found: $1"; [[ -r "$1" ]] || die "File not readable: $1"; }
Argument parsing (getopts):
usage() {
cat << EOF
Usage: ${SCRIPT_NAME} [OPTIONS] [ARGUMENTS]
Options:
-h Show this help and exit
-v Enable verbose output
-f FILE Input file path
-o FILE Output file path
-d Enable debug logging
Examples:
${SCRIPT_NAME} -f input.txt -o output.txt
EOF
}
parse_args() {
while getopts ":hvf:o:d" opt; do
case ${opt} in
h) usage; exit 0 ;;
v) VERBOSE=true ;;
f) INPUT_FILE="${OPTARG}" ;;
o) OUTPUT_FILE="${OPTARG}" ;;
d) LOG_LEVEL="DEBUG" ;;
\?) echo "Invalid option: -${OPTARG}" >&2; usage; exit 1 ;;
:) echo "Option -${OPTARG} requires an argument" >&2; usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
}
Stage 5 – Business Logic
- Text processing: use
references/text-processing-guide.mdfor grep/awk/sed selection - System administration: include prerequisite validation, backup, rollback, progress indicators
- API clients: include HTTP error handling, retry logic, authentication, response parsing
Stage 6 – Main Function
main() {
parse_args "$@" # From Stage 4
check_command "grep"
check_command "awk"
[[ -n "${INPUT_FILE:-}" ]] || die "Input file not specified. Use -f option."
validate_file "${INPUT_FILE}"
log_info "Starting processing..." # From Stage 4
# Main logic here
log_info "Processing completed successfully"
}
main "$@"
Stage 7 – Documentation
#######################################
# Brief description of function
# Globals:
# VARIABLE_NAME
# Arguments:
# $1 - Description
# Outputs:
# Writes results to stdout
# Returns:
# 0 if successful, non-zero on error
#######################################
Stage 8 – Validate
After generating any script, invoke devops-skills:bash-script-validator:
- Generate the script
- Run validator; review syntax, ShellCheck, security, performance results
- Fix issues and re-validate until all checks pass
- Provide Post-Generation Summary (see below)
Key Best Practices
Security:
- Always quote variables:
"${var}"not$var - Validate inputs:
[[ "${val}" =~ ^[a-zA-Z0-9/_.-]+$ ]] || die "Invalid" - Never
evaluser input; usecasestatements instead
Performance:
- Avoid useless
cat:grep pattern filenotcat file | grep pattern - Single-pass awk:
awk '/ERROR/{e++} /WARN/{w++} END{print e,w}' log
Maintainability:
- Functions follow single responsibility
- Use
readonlyfor constants - Meaningful variable names; comments for complex logic
Portability (POSIX sh):
- Avoid bash arrays,
[[ ]],$BASH_SOURCE; test withsh -n script.sh
Common Script Patterns
See references/script-patterns.md for full templates including text processing and parallel batch processing. Quick reference for simple CLI tools:
Pattern 1 – Simple CLI tool:
#!/usr/bin/env bash
set -euo pipefail
# For production scripts, use the full logging and argument parsing
# functions from Stage 4 above. This minimal example demonstrates structure:
usage() { cat << EOF
Usage: ${0##*/} [OPTIONS] FILE
-h Show help
-v Verbose
-o Output file
EOF
}
main() {
local verbose=false output_file="" input_file=""
while getopts ":hvo:" opt; do
case ${opt} in
h) usage; exit 0 ;; v) verbose=true ;; o) output_file="${OPTARG}" ;;
*) echo "Invalid option: -${OPTARG}" >&2; usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
input_file="${1:-}"
[[ -n "${input_file}" ]] || { echo "Error: FILE required" >&2; usage; exit 1; }
[[ -f "${input_file}" ]] || { echo "Error: File not found: ${input_file}" >&2; exit 1; }
if [[ -n "${output_file}" ]]; then
process_file "${input_file}" > "${output_file}"
else
process_file "${input_file}"
fi
}
process_file() { local file="$1"; cat "${file}"; }
main "$@"
For text processing (grep/awk/sed pipelines) and parallel batch processing patterns, see references/script-patterns.md.
Post-Generation Summary (REQUIRED)
After every script, provide:
## Generated Script Summary
**File:** path/to/script.sh
**Architecture:** [main functions and purposes]
**Tool Selection:**
- grep: [why used]
- awk: [why used]
- sed: [why used / not needed]
**Key Features:** [list]
**Customization Points:** [variables/functions to modify]
**Usage:**
./script.sh --help
./script.sh -v input.log
**Validation Status:** ✅ Passed ShellCheck / ❌ Issues found (fixing...)
**Documentation References:**
- references/text-processing-guide.md (tool selection)
- references/script-patterns.md (argument parsing)
Anti-Patterns
NEVER start a bash script without set -euo pipefail
- WHY: Without these flags, errors are silently ignored:
-ecauses exit on error,-uerrors on unset variables,-o pipefailcatches pipeline failures. - BAD:
#!/usr/bin/env bashwith no error flags and silent failures. - GOOD:
#!/usr/bin/env bashfollowed immediately byset -euo pipefailas the first two lines of every script.
NEVER use unquoted variable expansions in command arguments
- WHY: Word splitting and glob expansion on unquoted
$VARcause subtle bugs when values contain spaces or special characters. - BAD:
cp $SOURCE $DEST - GOOD:
cp "$SOURCE" "$DEST"
NEVER use ls | grep to filter files
- WHY:
lsoutput is not reliably parseable; it is locale-dependent and breaks on filenames with special characters. - BAD:
ls *.log | grep error - GOOD:
for f in *.log; do grep error "$f"; doneorfind . -name "*.log" -exec grep error {} +
NEVER hardcode absolute paths for tools like python, node, or bash
- WHY: Paths differ across operating systems and distributions, making scripts non-portable.
- BAD:
#!/usr/bin/python3or/usr/local/bin/node script.js - GOOD: Use
#!/usr/bin/env python3, rely on PATH fornode script.js, or guard withcommand -v python3 || { echo "python3 required"; exit 1; }.
NEVER use eval to execute dynamically constructed commands
- WHY:
evalenables code injection when any part of the command comes from user input or external data. - BAD:
eval "rm -rf $USER_INPUT" - GOOD: Use arrays for command construction:
cmd=(rm -rf "$USER_INPUT"); "${cmd[@]}"
References
Internal references (offline, load as context):
references/bash-scripting-guide.md— strict mode, functions, arrays, parameter expansionreferences/script-patterns.md— argument parsing, logging, retry logic, lock files, parallel processingreferences/text-processing-guide.md— grep/awk/sed selection, pipelines, large-file optimizationreferences/generation-best-practices.md— naming, documentation, testing, security, portabilityassets/templates/standard-template.sh— production-ready template with all componentsexamples/log-analyzer.sh— grep/awk/sed usage demonstrationscripts/generate_script_template.sh— template generator tool
Official documentation:
All generated scripts are automatically validated using devops-skills:bash-script-validator to ensure correct syntax, ShellCheck compliance, security, and performance.
Explicit NEVER/WHY/consequence triplets with side-by-side bad/good examples. Generic warnings score low — concrete failure modes with real impact (security, correctness, performance) score high.
Runtime validation via eval scenarios proving the skill actually changes agent behaviour. Requires evals/ directory, instructions.json, ≥3 scenarios, and ≥80% instruction coverage in summary.json.
Appropriate rigidity for the skill type: strict rules for critical/security skills, balanced steps for process skills, flexible options for tool skills. Mismatched calibration (too rigid or too loose) reduces score.
Expert-only knowledge the AI couldn't derive on its own — domain-specific patterns, production gotchas, decision frameworks, and non-obvious tool behaviour. High scores mean <5% redundancy with what the model already knows.
A clear philosophy plus numbered step-by-step workflows with explicit entry/exit points. Should answer: what is the core principle, how to execute it, and when (and when not) to apply it.
Rich activation keywords and trigger scenarios in the description so the skill fires when needed. Best descriptions read like an exhaustive list of the tasks and concepts that should invoke this skill.
Copy-paste-ready code, complete runnable examples, real file paths and commands. Pseudocode and abstract descriptions score low — concrete, executable content scores high.
SKILL.md as a <100-line navigation hub pointing to focused references/ files. Content should be lazy-loaded on demand, not front-loaded. Includes actionable 'When to Use' conditions per reference.
Proper frontmatter, single-task focus, activation keywords, and cross-harness portability. Skills must work across 40+ agentic harnesses without hardcoded paths or agent-specific references.
Expert-only knowledge the AI couldn't derive on its own — domain-specific patterns, production gotchas, decision frameworks, and non-obvious tool behaviour. High scores mean <5% redundancy with what the model already knows.
A clear philosophy plus numbered step-by-step workflows with explicit entry/exit points. Should answer: what is the core principle, how to execute it, and when (and when not) to apply it.
Explicit NEVER/WHY/consequence triplets with side-by-side bad/good examples. Generic warnings score low — concrete failure modes with real impact (security, correctness, performance) score high.
Proper frontmatter, single-task focus, activation keywords, and cross-harness portability. Skills must work across 40+ agentic harnesses without hardcoded paths or agent-specific references.
SKILL.md as a <100-line navigation hub pointing to focused references/ files. Content should be lazy-loaded on demand, not front-loaded. Includes actionable 'When to Use' conditions per reference.
Appropriate rigidity for the skill type: strict rules for critical/security skills, balanced steps for process skills, flexible options for tool skills. Mismatched calibration (too rigid or too loose) reduces score.
Rich activation keywords and trigger scenarios in the description so the skill fires when needed. Best descriptions read like an exhaustive list of the tasks and concepts that should invoke this skill.
Copy-paste-ready code, complete runnable examples, real file paths and commands. Pseudocode and abstract descriptions score low — concrete, executable content scores high.
No eval scenarios available.
Bash Scripting Guide
Table of Contents
- Introduction
- Bash vs POSIX sh
- Strict Mode and Error Handling
- Variables and Parameter Expansion
- Functions and Scope
- Arrays and Associative Arrays
- Control Structures
- Process and Command Substitution
- Best Practices
- Common Pitfalls
Introduction
Bash (Bourne Again Shell) is a powerful Unix shell and command language. This guide covers modern bash scripting practices and patterns for creating robust, maintainable scripts.
Bash vs POSIX sh
Key Differences
Bash-specific features (not in POSIX sh):
- Arrays:
arr=(one two three) - Associative arrays:
declare -A map=([key]=value) [[conditional expressions$(( ))arithmetic expansion with more operators${var//pattern/replacement}parameter expansion- Process substitution:
<(command) selectkeyword for menus**recursive globbing withshopt -s globstar
POSIX sh compatible:
- Basic variable assignment and substitution
[test command (single brackets)casestatements- Basic parameter expansion
- Command substitution with
$() - Functions (with different syntax)
When to Choose
Use Bash when:
- Script runs on modern Linux/macOS systems
- Need arrays or associative arrays
- Want advanced string manipulation
- Targeting bash-specific environments
Use POSIX sh when:
- Maximum portability required
- Running on minimal systems (embedded, containers)
- Need to run on different Unix variants
- Following strict POSIX compliance requirements
Strict Mode and Error Handling
Essential: set -euo pipefail
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
Explanation:
set -e(errexit): Exit immediately if a command exits with non-zero statusset -u(nounset): Treat unset variables as an errorset -o pipefail: Return value of pipeline is status of last command to exit with non-zero statusIFS=$'\n\t': Set Internal Field Separator to newline and tab only (prevents word splitting issues)
When to Disable Strict Mode Temporarily
# Disable errexit for commands that are expected to fail
set +e
command_that_might_fail
exit_code=$?
set -e
# Or use || true for single commands
command_that_might_fail || true
# Or handle error explicitly
if ! command_that_might_fail; then
echo "Command failed, but continuing..."
fi
Signal Handling with trap
# Cleanup function
cleanup() {
local exit_code=$?
echo "Cleaning up..." >&2
rm -f "${temp_file}"
exit "${exit_code}"
}
# Set traps
trap cleanup EXIT # Always run cleanup on exit
trap cleanup ERR # Run cleanup on error
trap cleanup INT TERM # Run cleanup on interrupt or termination
# Create temp file
temp_file=$(mktemp)
# Rest of script...
Error Handling Patterns
# Pattern 1: Die function
die() {
echo "ERROR: $*" >&2
exit 1
}
[[ -f "${file}" ]] || die "File not found: ${file}"
# Pattern 2: Check function return values
if ! do_something; then
echo "do_something failed" >&2
return 1
fi
# Pattern 3: Command substitution with error handling
output=$(command 2>&1) || {
echo "Command failed: ${output}" >&2
exit 1
}
# Pattern 4: Validate prerequisites
check_command() {
command -v "$1" &> /dev/null || die "Required command not found: $1"
}
check_command "jq"
check_command "curl"
Variables and Parameter Expansion
Variable Naming Conventions
# Constants - uppercase with readonly
readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/myapp/config.conf"
# Environment variables - uppercase
export PATH="${HOME}/bin:${PATH}"
export LOG_LEVEL="INFO"
# Local variables - lowercase
local counter=0
local temp_file=""
# Function names - lowercase with underscores
process_data() {
local input="$1"
# ...
}
Always Quote Variables
# Good - properly quoted
rm "${file}"
cp "${source}" "${destination}"
echo "Value: ${variable}"
# Bad - unquoted (prone to word splitting and globbing)
rm $file
cp $source $destination
echo "Value: $variable"
Parameter Expansion
# Default values
${var:-default} # Use default if var is unset or empty
${var:=default} # Set var to default if unset or empty
${var:?error message} # Exit with error message if var is unset or empty
${var:+alternative} # Use alternative if var is set
# String manipulation
${var#pattern} # Remove shortest match from beginning
${var##pattern} # Remove longest match from beginning
${var%pattern} # Remove shortest match from end
${var%%pattern} # Remove longest match from end
${var/pattern/replacement} # Replace first match
${var//pattern/replacement} # Replace all matches
${var^} # Uppercase first character
${var^^} # Uppercase all characters
${var,} # Lowercase first character
${var,,} # Lowercase all characters
# Length and substring
${#var} # Length of var
${var:offset} # Substring from offset to end
${var:offset:length} # Substring from offset with length
# Examples
file="/path/to/file.txt"
${file##*/} # file.txt (basename)
${file%.*} # /path/to/file (remove extension)
${file##*.} # txt (extension only)
${file%/*} # /path/to (dirname)
Functions and Scope
Function Definition
# POSIX style (portable)
function_name() {
# function body
}
# Bash-specific (not portable to sh)
function function_name {
# function body
}
# Recommended: POSIX style with local variables
process_file() {
local input_file="$1"
local output_file="$2"
# Process file
grep "pattern" "${input_file}" > "${output_file}"
}
Variable Scope
# Global variable
GLOBAL_VAR="global"
my_function() {
# Local variable - only visible in function
local local_var="local"
# Modifying global variable
GLOBAL_VAR="modified"
# Function parameter access
local param1="$1"
local param2="$2"
echo "Params: ${param1} ${param2}"
}
my_function "arg1" "arg2"
Return Values
# Functions return exit status (0-255)
check_file() {
local file="$1"
[[ -f "${file}" ]] && return 0 || return 1
}
# Use function return status
if check_file "data.txt"; then
echo "File exists"
fi
# Return data via stdout
get_value() {
echo "computed value"
}
# Capture output
result=$(get_value)
# Return data via variable (using nameref in bash 4.3+)
get_data() {
local -n result_var=$1
result_var="computed value"
}
get_data my_result
echo "${my_result}"
Arrays and Associative Arrays
Indexed Arrays (Bash-specific)
# Array creation
arr=() # Empty array
arr=(one two three) # Initialize with values
arr[0]="first" # Assign to specific index
# Array operations
arr+=("four") # Append
${arr[0]} # Access element
${arr[@]} # All elements (as separate words)
${arr[*]} # All elements (as single word)
${#arr[@]} # Number of elements
${!arr[@]} # Indices
# Iterating over array
for item in "${arr[@]}"; do
echo "${item}"
done
# Iterating with indices
for i in "${!arr[@]}"; do
echo "Index $i: ${arr[i]}"
done
# Array slicing
${arr[@]:offset:length} # Slice array
# Remove element
unset 'arr[1]' # Remove specific element
Associative Arrays (Bash 4.0+)
# Declaration required
declare -A map
# Assignment
map[key1]="value1"
map[key2]="value2"
# Or initialize
declare -A map=([key1]="value1" [key2]="value2")
# Access
${map[key1]} # Get value
${map[@]} # All values
${!map[@]} # All keys
${#map[@]} # Number of elements
# Check if key exists
if [[ -v map[key1] ]]; then
echo "key1 exists"
fi
# Iterate over keys and values
for key in "${!map[@]}"; do
echo "${key}: ${map[${key}]}"
done
POSIX Alternative to Arrays
# Use positional parameters
set -- one two three
# Access
echo "$1" # one
echo "$2" # two
echo "$#" # count: 3
# Iterate
for item in "$@"; do
echo "${item}"
done
# Add item
set -- "$@" "four"
# Remove first item
shift
Control Structures
Conditional Expressions
# Bash [[ ... ]] (recommended for bash)
if [[ -f "${file}" ]]; then
echo "File exists"
fi
if [[ "${var}" == "value" ]]; then
echo "Match"
fi
if [[ "${var}" =~ ^[0-9]+$ ]]; then
echo "Numeric"
fi
# POSIX [ ... ] (portable)
if [ -f "${file}" ]; then
echo "File exists"
fi
# File tests
[[ -e file ]] # Exists
[[ -f file ]] # Regular file
[[ -d file ]] # Directory
[[ -L file ]] # Symbolic link
[[ -r file ]] # Readable
[[ -w file ]] # Writable
[[ -x file ]] # Executable
[[ -s file ]] # Not empty
# String tests
[[ -z "${var}" ]] # Empty string
[[ -n "${var}" ]] # Non-empty string
[[ "${a}" == "${b}" ]] # Equal
[[ "${a}" != "${b}" ]] # Not equal
[[ "${a}" < "${b}" ]] # Lexicographically less (bash only)
# Numeric tests
[[ "${a}" -eq "${b}" ]] # Equal
[[ "${a}" -ne "${b}" ]] # Not equal
[[ "${a}" -lt "${b}" ]] # Less than
[[ "${a}" -le "${b}" ]] # Less than or equal
[[ "${a}" -gt "${b}" ]] # Greater than
[[ "${a}" -ge "${b}" ]] # Greater than or equal
# Logical operators
[[ condition1 && condition2 ]] # AND
[[ condition1 || condition2 ]] # OR
[[ ! condition ]] # NOT
case Statements
case "${var}" in
pattern1)
# commands
;;
pattern2|pattern3)
# Multiple patterns
;;
*)
# Default case
;;
esac
# Example with patterns
case "${file}" in
*.txt)
echo "Text file"
;;
*.jpg|*.png)
echo "Image file"
;;
*)
echo "Unknown type"
;;
esac
Loops
# while loop
while condition; do
# commands
done
# until loop
until condition; do
# commands
done
# for loop (C-style, bash only)
for ((i=0; i<10; i++)); do
echo "${i}"
done
# for loop (iterating over values)
for item in one two three; do
echo "${item}"
done
# for loop (iterating over files)
for file in *.txt; do
echo "${file}"
done
# for loop (iterating over command output)
while IFS= read -r line; do
echo "${line}"
done < file.txt
# Or with command substitution (avoid for large output)
for file in $(find . -name "*.txt"); do
echo "${file}"
done
Process and Command Substitution
Command Substitution
# Recommended: $( ... )
result=$(command)
result=$(command arg1 arg2)
# Nested command substitution
outer=$(echo "Inner: $(echo "value")")
# Not recommended: backticks (legacy)
result=`command`
Process Substitution (Bash-specific)
# <( ... ) creates a named pipe/file descriptor
# Treat command output as a file
# Compare output of two commands
diff <(ls dir1) <(ls dir2)
# Use multiple inputs
paste <(cut -f1 file1) <(cut -f2 file2)
# Output redirection with process substitution
command > >(tee stdout.log) 2> >(tee stderr.log >&2)
Best Practices
Script Structure
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
# ============================================================================
# Script Name: example.sh
# Description: Brief description
# Author: Your Name
# Created: 2025-01-23
# ============================================================================
# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
# Global variables
VERBOSE=false
DRY_RUN=false
# Functions
usage() {
# ...
}
cleanup() {
# ...
}
main() {
# ...
}
# Signal handlers
trap cleanup EXIT ERR INT TERM
# Execute main
main "$@"
Always Use Quotes
# Good
echo "${variable}"
cp "${source}" "${dest}"
[[ -f "${file}" ]]
# Bad (unsafe)
echo $variable
cp $source $dest
[[ -f $file ]]
Use readonly for Constants
readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/config"
Prefer $() Over Backticks
# Good
output=$(command)
result=$(first $(second))
# Bad
output=`command`
result=`first \`second\`` # Hard to read
Check Command Existence
if ! command -v required_cmd &> /dev/null; then
echo "Error: required_cmd not found" >&2
exit 1
fi
Validate Inputs
# Check argument count
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <file>" >&2
exit 1
fi
# Validate file exists
[[ -f "${file}" ]] || { echo "File not found: ${file}" >&2; exit 1; }
# Validate numeric input
[[ "${count}" =~ ^[0-9]+$ ]] || { echo "Count must be numeric" >&2; exit 1; }
Common Pitfalls
Word Splitting
# Problem: Filename with spaces
file="my file.txt"
rm $file # Tries to remove "my" and "file.txt"
# Solution: Quote variables
rm "${file}" # Correctly removes "my file.txt"
Globbing
# Problem: Pattern in variable
pattern="*.txt"
echo $pattern # Expands to list of .txt files
# Solution: Quote to prevent globbing
echo "${pattern}" # Prints "*.txt"
Useless Use of Cat (UUOC)
# Bad: Unnecessary cat
cat file.txt | grep "pattern"
# Good: Direct input
grep "pattern" file.txt
# Bad: cat in loop
cat file.txt | while read line; do
echo "${line}"
done
# Good: redirect to while
while read -r line; do
echo "${line}"
done < file.txt
Not Handling Spaces in Filenames
# Bad: Will break on filenames with spaces
for file in $(find . -name "*.txt"); do
process "${file}"
done
# Good: Use while read
find . -name "*.txt" -print0 | while IFS= read -r -d '' file; do
process "${file}"
done
# Or use globbing
for file in ./**/*.txt; do
process "${file}"
done
Ignoring Command Exit Status
# Bad: Ignoring failure
command_that_might_fail
next_command
# Good: Check exit status
if command_that_might_fail; then
next_command
else
echo "Command failed" >&2
exit 1
fi
# Or with errexit
command_that_might_fail || { echo "Failed" >&2; exit 1; }
References
- GNU Bash Manual
- Google Shell Style Guide
- ShellCheck - Script analysis tool
- Bash Guide for Beginners
Script Generation Best Practices
Guidelines for generating high-quality, maintainable bash scripts.
Core Principles
- Security First - Validate inputs, quote variables, avoid injection
- Fail Fast - Use strict mode, check errors immediately
- Self-Documenting - Clear names, usage text, comments for complex logic
- Testable - Modular functions, predictable behavior
- Maintainable - Consistent style, organized structure
Script Structure Template
#!/usr/bin/env bash
#
# Script Name: descriptive-name.sh
# Description: What it does in one line
# Usage: script.sh [OPTIONS] ARGUMENTS
# Author: Name
# Created: Date
#
set -euo pipefail
IFS=$'\n\t'
# Constants (UPPERCASE, readonly)
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
# Global variables (lowercase or Mixed_Case)
verbose=false
dry_run=false
# Functions (lowercase_with_underscores)
usage() { }
cleanup() { }
main() { }
# Signal handlers
trap cleanup EXIT ERR INT TERM
# Execute
main "$@"
Naming Conventions
# Constants - UPPERCASE with readonly
readonly MAX_RETRIES=3
readonly CONFIG_FILE="/etc/app.conf"
# Environment variables - UPPERCASE
export PATH="${HOME}/bin:${PATH}"
export LOG_LEVEL="INFO"
# Global variables - lowercase or Mixed_Case
script_version="1.0.0"
temp_directory=""
# Functions - lowercase_with_underscores
process_file() { }
send_notification() { }
# Local variables - lowercase
local count=0
local file_path=""
Security Best Practices
# 1. Always quote variables
rm "${file}" # Good
rm $file # Bad
# 2. Validate all inputs
[[ "${input}" =~ ^[a-zA-Z0-9_-]+$ ]] || die "Invalid input"
# 3. Never use eval with user input
eval "${user_command}" # Dangerous!
# 4. Validate file paths
[[ "${file}" =~ /etc ]] && die "Cannot modify /etc"
[[ -f "${file}" ]] || die "File not found"
# 5. Use $() instead of backticks
output=$(command) # Good
output=`command` # Bad
# 6. Set safe IFS
IFS=$'\n\t'
Error Handling Patterns
# Pattern 1: Die function
die() {
echo "ERROR: $*" >&2
exit 1
}
# Pattern 2: Check prerequisites
check_command() {
command -v "$1" &> /dev/null || die "Required: $1"
}
# Pattern 3: Validate inputs
[[ $# -ge 1 ]] || die "Usage: $0 FILE"
[[ -f "$1" ]] || die "File not found: $1"
# Pattern 4: Cleanup on exit
cleanup() {
[[ -n "${temp_dir:-}" ]] && rm -rf "${temp_dir}"
}
trap cleanup EXIT
Function Design
# Good function design
#######################################
# Process a log file and extract errors
# Globals:
# LOG_LEVEL
# Arguments:
# $1 - Path to log file
# $2 - Output file (optional)
# Outputs:
# Writes errors to stdout or file
# Returns:
# 0 on success, 1 on error
#######################################
process_log_file() {
local log_file="$1"
local output_file="${2:-}"
# Validate
[[ -f "${log_file}" ]] || return 1
# Process
local errors
errors=$(grep "ERROR" "${log_file}")
# Output
if [[ -n "${output_file}" ]]; then
echo "${errors}" > "${output_file}"
else
echo "${errors}"
fi
return 0
}
Code Organization
# Recommended order:
1. Shebang and header comments
2. Strict mode settings
3. Constants
4. Global variables
5. Helper functions (general → specific)
6. Main logic functions
7. Main function
8. Signal handlers
9. Main execution
Generated Code Quality Checklist
- Proper shebang:
#!/usr/bin/env bash - Strict mode enabled:
set -euo pipefail - All variables quoted:
"${var}" - Constants marked readonly
- Functions documented
- Error handling implemented
- Usage/help function included
- Input validation present
- Cleanup on exit (trap)
- No ShellCheck warnings
- Comments for complex logic
- Consistent formatting
References
Bash Script Patterns
Common patterns and templates for bash script generation.
Table of Contents
- Argument Parsing Patterns
- Configuration File Handling
- Logging Frameworks
- Parallel Processing
- Lock Files
- Signal Handling
- Retry Logic
Argument Parsing Patterns
Simple getopts Pattern
#!/usr/bin/env bash
set -euo pipefail
usage() {
cat << EOF
Usage: ${0##*/} [OPTIONS] FILE
Options:
-h Show this help
-v Verbose output
-f FILE Input file
-o FILE Output file
EOF
}
main() {
local verbose=false
local input_file=""
local output_file=""
while getopts ":hvf:o:" opt; do
case ${opt} in
h) usage; exit 0 ;;
v) verbose=true ;;
f) input_file="${OPTARG}" ;;
o) output_file="${OPTARG}" ;;
:) echo "Option -${OPTARG} requires an argument" >&2; exit 1 ;;
\?) echo "Invalid option: -${OPTARG}" >&2; exit 1 ;;
esac
done
shift $((OPTIND - 1))
# Validation
[[ -n "${input_file}" ]] || { echo "Error: -f required" >&2; exit 1; }
# Process
echo "Processing ${input_file}..."
}
main "$@"
Long Options Pattern
# Parse both short and long options
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
usage
exit 0
;;
-v|--verbose)
VERBOSE=true
shift
;;
-f|--file)
INPUT_FILE="$2"
shift 2
;;
-o|--output)
OUTPUT_FILE="$2"
shift 2
;;
--)
shift
break
;;
-*)
echo "Unknown option: $1" >&2
exit 1
;;
*)
break
;;
esac
done
# Remaining arguments
REMAINING_ARGS=("$@")
}
Subcommand Pattern
#!/usr/bin/env bash
set -euo pipefail
cmd_start() {
echo "Starting service..."
}
cmd_stop() {
echo "Stopping service..."
}
cmd_status() {
echo "Checking status..."
}
usage() {
cat << EOF
Usage: ${0##*/} COMMAND [OPTIONS]
Commands:
start Start the service
stop Stop the service
status Check service status
Options:
-h, --help Show this help
EOF
}
main() {
[[ $# -lt 1 ]] && { usage; exit 1; }
local command="$1"
shift
case "${command}" in
start) cmd_start "$@" ;;
stop) cmd_stop "$@" ;;
status) cmd_status "$@" ;;
-h|--help) usage; exit 0 ;;
*) echo "Unknown command: ${command}" >&2; usage; exit 1 ;;
esac
}
main "$@"
Configuration File Handling
Source-based Configuration
# config.conf file
CONFIG_VALUE="something"
MAX_RETRIES=3
API_URL="https://api.example.com"
# In script
load_config() {
local config_file="${1:-config.conf}"
if [[ -f "${config_file}" ]]; then
# shellcheck source=/dev/null
source "${config_file}"
else
echo "Warning: Config file not found: ${config_file}" >&2
fi
}
load_config "/etc/myapp/config.conf"
Key-Value Configuration Parser
# config.conf format:
# key=value
# # comments
load_config() {
local config_file="$1"
while IFS='=' read -r key value; do
# Skip empty lines and comments
[[ -z "${key}" || "${key}" =~ ^[[:space:]]*# ]] && continue
# Trim whitespace
key=$(echo "${key}" | tr -d '[:space:]')
value=$(echo "${value}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
# Export as variable
declare -g "${key}=${value}"
done < "${config_file}"
}
INI-style Configuration Parser
# Parse INI format [section] key=value
parse_ini() {
local file="$1"
local section=""
while IFS= read -r line; do
# Skip empty lines and comments
[[ -z "${line}" || "${line}" =~ ^[[:space:]]*[#;] ]] && continue
# Section header
if [[ "${line}" =~ ^\[([^]]+)\] ]]; then
section="${BASH_REMATCH[1]}"
continue
fi
# Key=value
if [[ "${line}" =~ ^([^=]+)=(.*)$ ]]; then
local key="${BASH_REMATCH[1]}"
local value="${BASH_REMATCH[2]}"
key=$(echo "${key}" | tr -d '[:space:]')
value=$(echo "${value}" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
# Store in associative array
config["${section}.${key}"]="${value}"
fi
done < "${file}"
}
Logging Frameworks
Simple Logging with Levels
# Log level: DEBUG=0, INFO=1, WARN=2, ERROR=3
LOG_LEVEL=${LOG_LEVEL:-1}
log_debug() {
[[ ${LOG_LEVEL} -le 0 ]] && echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2
}
log_info() {
[[ ${LOG_LEVEL} -le 1 ]] && echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2
}
log_warn() {
[[ ${LOG_LEVEL} -le 2 ]] && echo "[WARN] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2
}
log_error() {
echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2
}
File-based Logging
readonly LOG_FILE="${LOG_FILE:-/var/log/myscript.log}"
log_to_file() {
local level="$1"
shift
echo "[${level}] $(date '+%Y-%m-%d %H:%M:%S') $*" >> "${LOG_FILE}"
}
log_info() {
local msg="$*"
echo "[INFO] ${msg}" >&2
log_to_file "INFO" "${msg}"
}
Structured JSON Logging
log_json() {
local level="$1"
local message="$2"
local timestamp=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
cat <<EOF
{"timestamp":"${timestamp}","level":"${level}","message":"${message}","script":"${SCRIPT_NAME}"}
EOF
}
log_info() {
log_json "INFO" "$*" >&2
}
Parallel Processing
Using xargs for Parallel Execution
# Process files in parallel
find . -name "*.txt" -print0 | xargs -0 -P 4 -I {} process_file {}
# With function export
process_file() {
echo "Processing $1..."
# ... processing logic
}
export -f process_file
find . -name "*.txt" | xargs -P 4 -I {} bash -c 'process_file "$@"' _ {}
Using GNU Parallel
# Requires: apt-get install parallel
# Simple parallel execution
parallel process_file ::: file1.txt file2.txt file3.txt
# From file list
cat files.txt | parallel process_file
# With progress bar
parallel --bar process_file ::: *.txt
# Control number of jobs
parallel -j 4 process_file ::: *.txt
Background Jobs Pattern
# Track background jobs
pids=()
# Start jobs
for file in *.txt; do
process_file "${file}" &
pids+=($!)
done
# Wait for all jobs
for pid in "${pids[@]}"; do
if wait "${pid}"; then
echo "Job ${pid} completed successfully"
else
echo "Job ${pid} failed" >&2
fi
done
Lock Files
Simple Lock File
readonly LOCK_FILE="/var/lock/myscript.lock"
acquire_lock() {
if [[ -f "${LOCK_FILE}" ]]; then
echo "Another instance is running (lock file exists)" >&2
exit 1
fi
echo $$ > "${LOCK_FILE}"
trap 'rm -f "${LOCK_FILE}"' EXIT
}
acquire_lock
PID-based Lock with Stale Lock Detection
acquire_lock() {
local lock_file="/var/lock/myscript.lock"
if [[ -f "${lock_file}" ]]; then
local old_pid=$(cat "${lock_file}")
# Check if process is still running
if kill -0 "${old_pid}" 2>/dev/null; then
echo "Another instance (PID ${old_pid}) is running" >&2
return 1
else
echo "Removing stale lock file" >&2
rm -f "${lock_file}"
fi
fi
echo $$ > "${lock_file}"
trap 'rm -f "${lock_file}"' EXIT
}
Using flock for Atomic Locking
# Requires flock command
exec 200>/var/lock/myscript.lock
flock -n 200 || { echo "Another instance is running" >&2; exit 1; }
# Script runs exclusively
# Lock is released when script exits
Signal Handling
Cleanup on Exit
cleanup() {
local exit_code=$?
echo "Cleaning up..." >&2
# Remove temp files
[[ -n "${temp_dir:-}" ]] && rm -rf "${temp_dir}"
# Release locks
[[ -f "${lock_file:-}" ]] && rm -f "${lock_file}"
exit "${exit_code}"
}
trap cleanup EXIT
Handling Multiple Signals
handle_sigint() {
echo "Received SIGINT, cleaning up..." >&2
cleanup
exit 130 # Standard exit code for SIGINT
}
handle_sigterm() {
echo "Received SIGTERM, cleaning up..." >&2
cleanup
exit 143 # Standard exit code for SIGTERM
}
trap handle_sigint INT
trap handle_sigterm TERM
trap cleanup EXIT ERR
Graceful Shutdown
SHUTDOWN=false
handle_signal() {
echo "Shutdown signal received, finishing current work..." >&2
SHUTDOWN=true
}
trap handle_signal INT TERM
# Main processing loop
while [[ "${SHUTDOWN}" == "false" ]]; do
process_next_item || break
done
echo "Graceful shutdown complete" >&2
Retry Logic
Simple Retry with Backoff
retry() {
local max_attempts=3
local delay=1
local attempt=1
while [[ ${attempt} -le ${max_attempts} ]]; do
if "$@"; then
return 0
else
echo "Attempt ${attempt} failed, retrying in ${delay}s..." >&2
sleep "${delay}"
((attempt++))
((delay*=2)) # Exponential backoff
fi
done
echo "All ${max_attempts} attempts failed" >&2
return 1
}
# Usage
retry curl -f https://api.example.com/data
Advanced Retry with Custom Parameters
retry_with_backoff() {
local max_attempts="${1}"
local delay="${2}"
local max_delay="${3:-60}"
shift 3
local attempt=1
while [[ ${attempt} -le ${max_attempts} ]]; do
if "$@"; then
return 0
fi
if [[ ${attempt} -lt ${max_attempts} ]]; then
echo "Attempt ${attempt}/${max_attempts} failed" >&2
echo "Retrying in ${delay}s..." >&2
sleep "${delay}"
# Exponential backoff with max cap
delay=$((delay * 2))
[[ ${delay} -gt ${max_delay} ]] && delay=${max_delay}
fi
((attempt++))
done
echo "All ${max_attempts} attempts failed" >&2
return 1
}
# Usage: retry_with_backoff MAX_ATTEMPTS INITIAL_DELAY MAX_DELAY command args...
retry_with_backoff 5 1 30 curl -f https://api.example.com/data
Retry with Jitter
retry_with_jitter() {
local max_attempts="$1"
local base_delay="$2"
shift 2
local attempt=1
while [[ ${attempt} -le ${max_attempts} ]]; do
if "$@"; then
return 0
fi
if [[ ${attempt} -lt ${max_attempts} ]]; then
# Add random jitter (0-100% of delay)
local jitter=$((RANDOM % base_delay))
local delay=$((base_delay + jitter))
echo "Attempt ${attempt} failed, retrying in ${delay}s..." >&2
sleep "${delay}"
# Exponential backoff
((base_delay*=2))
fi
((attempt++))
done
return 1
}
References
Text Processing Guide
Guide for choosing and using grep, awk, sed, and other text processing tools effectively in bash scripts.
Decision Tree: Which Tool to Use?
Is it a simple pattern match/filter?
├─ YES → Use grep
└─ NO
├─ Is it field/column-based data?
│ └─ YES → Use awk
└─ NO
├─ Is it find-and-replace or deletion?
│ └─ YES → Use sed
└─ NO
└─ Complex processing → Use awk or consider Python/Perl
grep: Pattern Matching and Filtering
When to Use grep
- Searching for patterns in files
- Filtering lines by regex
- Simple line extraction
- Counting matches
- Finding files containing patterns
Common grep Patterns
# Basic search
grep "pattern" file.txt
# Case-insensitive
grep -i "error" log.txt
# Invert match (lines NOT containing pattern)
grep -v "DEBUG" log.txt
# Count matches
grep -c "ERROR" log.txt
# Show line numbers
grep -n "TODO" *.sh
# Extended regex (ERE)
grep -E "(error|fail|critical)" log.txt
# Recursive directory search
grep -r "function_name" src/
# Show filename only
grep -l "pattern" *.txt
# Show context (lines before/after)
grep -A 2 -B 2 "ERROR" log.txt # 2 lines after and before
# Multiple patterns
grep -e "error" -e "fail" log.txt
# Read patterns from file
grep -f patterns.txt input.txt
# Whole word match
grep -w "test" file.txt # Matches "test" but not "testing"
# Fixed string (not regex)
grep -F "a.b" file.txt # Matches literal "a.b", not regex
# Binary file handling
grep -a "pattern" binary_file # Treat binary as text
grep for Log Analysis
# Extract error messages from last hour
find /var/log -name "*.log" -mmin -60 -exec grep "ERROR" {} +
# Count errors by type
grep "ERROR" app.log | cut -d':' -f3 | sort | uniq -c | sort -rn
# Find errors excluding known issues
grep "ERROR" app.log | grep -v -f known_errors.txt
awk: Field-Based Text Processing
When to Use awk
- Processing structured data (CSV, logs, tables)
- Extracting specific fields
- Performing calculations
- Generating reports
- Complex conditional logic on fields
awk Basics
# Print specific fields (space-delimited by default)
awk '{print $1, $3}' file.txt
# Custom delimiter
awk -F',' '{print $1, $3}' data.csv
awk -F':' '{print $1}' /etc/passwd
# Multiple delimiters
awk -F'[,:]' '{print $1}' file.txt
# Print last field
awk '{print $NF}' file.txt
# Print all but first field
awk '{$1=""; print $0}' file.txt
awk Conditionals
# Print lines where field 3 > 100
awk '$3 > 100' data.txt
# Print lines where field matches pattern
awk '$2 ~ /error/' log.txt
# Print lines where field does NOT match
awk '$2 !~ /DEBUG/' log.txt
# Multiple conditions
awk '$3 > 100 && $4 < 500' data.txt
# If-else logic
awk '{if ($3 > 100) print "High:", $0; else print "Low:", $0}' data.txt
awk Calculations
# Sum values in column 3
awk '{sum += $3} END {print sum}' numbers.txt
# Average
awk '{sum += $3; count++} END {print sum/count}' numbers.txt
# Find max value
awk 'BEGIN {max=0} {if ($1 > max) max=$1} END {print max}' numbers.txt
# Count lines matching condition
awk '$3 > 100 {count++} END {print count}' data.txt
awk Formatted Output
# Printf-style formatting
awk '{printf "Name: %-20s Age: %3d\n", $1, $2}' people.txt
# Tab-separated output
awk 'BEGIN {OFS="\t"} {print $1, $2, $3}' file.txt
# Custom output formatting
awk '{printf "%s: %10.2f\n", $1, $2}' data.txt
awk Built-in Variables
NF # Number of fields in current line
NR # Current line number
FNR # Line number in current file
FS # Input field separator
OFS # Output field separator
RS # Input record separator
ORS # Output record separator
FILENAME # Current filename
# Examples
awk '{print NR, NF, $0}' file.txt # Line number, field count, full line
awk 'NR==10' file.txt # Print line 10
awk 'NF > 5' file.txt # Lines with more than 5 fields
awk for Log Analysis
# Apache/Nginx access log analysis
# Extract status codes and count
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# Summarize traffic by IP
awk '{ip[$1]++} END {for (i in ip) print ip[i], i}' access.log | sort -rn
# Calculate average response time (field 11)
awk '{sum += $11; count++} END {print sum/count}' access.log
# Extract requests by hour
awk '{print substr($4, 2, 14)}' access.log | uniq -c
sed: Stream Editing
When to Use sed
- Find and replace operations
- Deleting specific lines
- In-place file editing
- Simple transformations
sed Substitution
# Basic substitution (first occurrence per line)
sed 's/old/new/' file.txt
# Global substitution (all occurrences)
sed 's/old/new/g' file.txt
# Case-insensitive substitution
sed 's/old/new/gi' file.txt
# In-place editing
sed -i 's/old/new/g' file.txt
# Backup before in-place edit
sed -i.bak 's/old/new/g' file.txt
# Replace only on specific line
sed '5s/old/new/' file.txt
# Replace on lines matching pattern
sed '/ERROR/s/old/new/g' file.txt
# Use different delimiter
sed 's|/usr/local|/opt|g' file.txt
# Backreferences
sed 's/\([0-9]*\)-\([0-9]*\)/\2-\1/' file.txt
# Multiple substitutions
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt
sed Deletion
# Delete specific line
sed '5d' file.txt
# Delete range of lines
sed '5,10d' file.txt
# Delete lines matching pattern
sed '/pattern/d' file.txt
# Delete empty lines
sed '/^$/d' file.txt
# Delete lines NOT matching pattern
sed '/pattern/!d' file.txt
sed Line Operations
# Print specific line
sed -n '10p' file.txt
# Print range
sed -n '10,20p' file.txt
# Print lines matching pattern
sed -n '/ERROR/p' file.txt
# Insert line before match
sed '/pattern/i\New line before' file.txt
# Append line after match
sed '/pattern/a\New line after' file.txt
# Change entire line
sed '/pattern/c\Replacement line' file.txt
sed Advanced Patterns
# Remove comments
sed 's/#.*//' file.txt
# Remove leading whitespace
sed 's/^[ \t]*//' file.txt
# Remove trailing whitespace
sed 's/[ \t]*$//' file.txt
# Remove HTML tags
sed 's/<[^>]*>//g' file.html
# Extract text between delimiters
sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' file.html
Combining Tools: Pipeline Patterns
grep + awk
# Filter then extract fields
grep "ERROR" log.txt | awk '{print $1, $5}'
# Filter multiple patterns, process
grep -E "ERROR|WARN" log.txt | awk '{count[$2]++} END {for (i in count) print i, count[i]}'
sed + awk
# Clean then process
sed 's/[^[:print:]]//g' data.txt | awk '{sum += $2} END {print sum}'
# Remove comments, extract fields
sed 's/#.*//' config.txt | awk -F'=' '{print $1}'
Complete Pipeline Example
# Analyze web server logs
cat access.log \
| grep "GET" \
| grep -v "robot" \
| sed 's/.*HTTP\/[0-9.]*" //' \
| awk '$1 >= 200 && $1 < 300 {success++} $1 >= 400 {fail++} END {print "Success:", success, "Fail:", fail}'
Performance Tips
grep Performance
# Use -F for fixed strings (faster than regex)
grep -F "literal.string" large_file.txt
# Use -m to stop after N matches
grep -m 10 "pattern" large_file.txt
# Parallel grep for large files
parallel -j4 grep "pattern" ::: chunk1 chunk2 chunk3 chunk4
awk Performance
# Exit early if possible
awk '{if (condition) {print; exit}}' large_file.txt
# Process only needed lines
awk 'NR > 1000 {exit} {process}' large_file.txt
# Use built-in functions efficiently
awk '{count[$1]++} END {for (i in count) print i, count[i]}' file.txt
sed Performance
# Minimize patterns
sed -e 's/a/b/g' -e 's/c/d/g' file.txt # Better than multiple sed calls
# Use in-place editing for large files
sed -i 's/old/new/g' large_file.txt # Avoids loading entire file
Avoid Useless Use of cat
# Bad
cat file.txt | grep "pattern"
cat file.txt | awk '{print $1}'
cat file.txt | sed 's/old/new/g'
# Good
grep "pattern" file.txt
awk '{print $1}' file.txt
sed 's/old/new/g' file.txt
Real-World Examples
Example 1: CSV Processing
# Extract specific columns from CSV
awk -F',' '{print $1, $3, $5}' data.csv
# Filter rows by value
awk -F',' '$3 > 1000 {print $0}' data.csv
# Calculate sum per category
awk -F',' '{sum[$1] += $3} END {for (cat in sum) print cat, sum[cat]}' sales.csv
Example 2: Log Analysis
# Error rate over time
grep "ERROR" app.log \
| awk '{print $1}' \
| uniq -c \
| awk '{print $2, $1}'
# Top 10 error messages
grep "ERROR" app.log \
| sed 's/.*ERROR: //' \
| sort \
| uniq -c \
| sort -rn \
| head -10
Example 3: Configuration File Processing
# Extract non-comment, non-empty lines
sed -e 's/#.*//' -e '/^$/d' config.txt
# Convert KEY=VALUE to JSON
awk -F'=' 'BEGIN {print "{"} {printf " \"%s\": \"%s\",\n", $1, $2} END {print "}"}' config.txt