Explaining Streams and Redirection in Bash

The aim of this page📝is to describe input and output streams in bash and the possibilities of redirection. A stream is a way of sequentially accessing/modifying information; regardless of the form (file, program, memory, terminal). The conceptualization of streams in SICP is opposed to the object orientation in a way of managing complexity via conventional interfaces. — stream processing one being a flow where only beginning and end matter — object orientation is a chain of states where every modification matters. These are notes on Pluralsight lectures from https://app.pluralsight.com/profile/author/reindertjan-ekker — I believe this particular course has been archived (there are more on bash).

1. There are three, numerically represended, standard streams — stdin (0), stdout (1), stderr (2) — accompanied by a null stream dedicated for discarding output

• whenever the shell script starts, it automatically connects with the 3 standard streams

• when a script is launched interactively, all standard streams connect to the terminal
• therefore, you cannot notice from which stream are messages originating
• e.g. you cannot notice that errors arrive from stderr stream

# EXAMPLE: CANNOT SAY THAT ERROR MESSAGES COMES FROM STDERR STREAM
#!/bin/bash
date=$(date)
topic="$1"
filename="./${topic}notes.txt"
read -r -p "Your note:" note
if [[ -n $note ]]; then
    echo "Note '$note' saved to $filename"
    echo "$date: $note" >>"$filename"
else
    echo "No input; note was not saved" >/dev/stderr
fi

# EXAMPLE EXECUTED
>>> $ ./b33_01.sh
Your note: #nothing here
No input; note was not saved

• but e.g. errors, the stream-separation makes it possible to treat it differently than normal output

2. Redirection

• Input redirection (<) allows us to take the contents of a file and use it as input for a command.
• Output redirection (>) allows us to take the output of a command and write it to a file.
• for example, the cat command usually reads the provided filename input and prints the output to the terminal which is its standard output stream
• in the example below, you can also print it to another file and thus use it as a copy command (cp)

# EXAMPLE: using `cat` as `cp` command
echo hello > hello.txt
cat hello.txt
cat < hello.txt
cat < hello.txt > hello2.txt
cat hello2.txt

• The command cat is executed.
• The contents of hello.txt are used as input for the cat command.
• The output of the cat command is written to the file hello2.txt.
• The contents of hello.txt are effectively copied into hello2.txt.
• What was intuitively confusing for me is the order in which I’ve read that command
• Seeing a filename surrounded by brackets, it seems “natural” to start reading from the middle
• That is of course incorrect

3. Explanation of the Example Code:

1. echo hello > hello.txt: This command writes the string "hello" to the file hello.txt.
2. cat hello.txt: This command reads and displays the contents of hello.txt.
3. cat < hello.txt: This command uses the contents of hello.txt as input for the cat command and display them.
4. cat < hello.txt > hello2.txt: This command uses the contents of hello.txt as input for the cat command and writes the output to hello2.txt.
5. cat hello2.txt: This command reads and displays the contents of hello2.txt, which should be the same as the contents of hello.txt.

By combining input and output redirection, we can manipulate the flow of data in shell commands and perform various operations efficiently.

3.1. Input redirection

• using <
• in simple use-cases, the input redirection to file in cat and grep is

grep milk < shoppingnotes.txt
# SYNONYMOUS
grep milk shoppingnotes.txt

#!/bin/bash
date=$(date)
topic="$1"
filename="./${topic}notes.txt"
read -r -p "Your note:" note
if [[ -n $note ]]; then
    echo "Note '$note' saved to $filename"
    echo "$date: $note" >>"$filename"
else
    echo "No input; note was not saved" >/dev/stderr
fi

• when however you have read within the script that expects input from a user. it is a different story

$ cat b33_01.txt
test note hello world
$ ./b33_01.sh < b33_01.txt
Note 'test note hello world' saved to ./notes.txt
$ cat notes.txt
Thu Jan 20 11:57:27 UTC 2022: test note hello world

3.2. Output redirection

• by default output redirection operators redirect only standard output and not standard error stream

ls > listing.txt
ls >> listing.txt

• > operator redirects the output of the ls and (re)writes listing.txt
• >> operator redirects the output of the ls and appends to listing.txt

#!/bin/bash
date=$(date)
topic="$1"
filename="./${topic}notes.txt"
read -r -p "Your note:" note
if [[ -n $note ]]; then
    echo "Note '$note' saved to $filename"
    echo "$date: $note" >>"$filename"
else
    echo "No input; note was not saved" >/dev/stderr
fi

# EXAMPLE: REDIRECTING ONLY TO STDOUTPUT -> ERRORS STILL GET PRINTED
$ ./b33_01.sh > b33_01.txt
Your note: 
No input; note was not saved

3.3. Pipe

• |
• connects the standard output from 1 command and passes that to standard input to another command

ls | grep x

3.4. Specific Stream Redirection

• prefix redirection command with a number representation to a given stream, e.g. 2>

cmd 2> /dev/null

• usecase: error redirection; e.g. discarding all of them
• for example the following discards all errors
• remember that the number representation of standard i/o streams defaults to one, no need to specify when redirecting

3.5. Redirection to a Specific Stream

• use &<stream_number>
• usecase: sending both error and output to a single file
• the following redirects all standard output to the standard error strream

>&2

• the following redirects all standard errors to the standard output

2>&1

• the following sends standard output to the log file and all errors to standard output and therewith to the log file

<cmd> > logfile 2>&1

• the above is idiomatic: don’t send errors to logfile; the two streams would overwrite each other in the file

<cmd> > logfile 2>logfile

• don’t use other options that historically offered sending both streams to a single file. but you may run into these in scripts written by others

&>
>&

4. order does matter

• these are the same

cmd < inputfile > outputfile
> outputfile cmd < inputfile

• these are different

cmd > logfile 2>&1  #sends errors to the logfile
2>&1 >logfile cmd   #sends errors to stdout and errors end up in the terminal; then stdout will end up in a logfile

5. links

java — What is InputStream & Output Stream?

Why and when do we use them? — Stack Overflow java — Byte Stream and Character stream — Stack Overflow