Bdtyktl

If the meaning of foo > bar would depend on whether there is a command named bar that would make using redirection a lot harder and more error prone: Every time I want to redirect to a file I first had to check whether there's a command named like my destination file.

There's a vital difference between the two operators:

1. ls > log.txt --> This command sends the output to the log.txt file.




2. ls | grep file.txt --> This command sends the output of the ls to grep command through the use of pipe (|), and the grep command searches for file.txt in the in the input provided to it by the previous command.

If you had to perform the same task using the first scenario, then it would be:

ls > log.txt; grep 'file.txt' log.txt

So a pipe (with |) is used to send the output to other command, whereas redirection (with >) is used to redirect the output to some file.

From the Unix and Linux System Administration Handbook:

Redirection

The shell interprets the symbols <,>, and >> as instructions to reroute a command's input or output to or from a file.

Pipes

To connect the STDOUT of one command to the STDIN of another use the | symbol, commonly known as a pipe.

So my interpretation is: If it's command to command, use a pipe. If you are outputting to or from a file use the redirect.

There's a big syntactic difference between the two:

1. A redirect is an argument to a program


2. A pipe separates two commands

You can think of redirects like this: cat [<infile] [>outfile]. This implies order doesn't matter: cat <infile >outfile is the same as cat >outfile <infile. You can even mix redirects up with other arguments: cat >outfile <infile -b and cat <infile -b >outfile are both perfectly fine. Also you can string together more than one input or output (inputs will be read sequentially and all output will be written to each output file): cat >outfile1 >outfile2 <infile1 <infile2. The target or source of a redirect can be either a filename or the name of a stream (like &1, at least in bash).

But pipes totally separate one command from another command, you can't mix them in with arguments:

[command1] | [command2]

The pipe takes everything written to standard output from command1 and sends it to the standard input of command2.

You can also combine piping and redirection. For example:

cat <infile >outfile | cat <infile2 >outfile2

The first cat will read lines from infile, then simultaneously write each line to outfile and send it to the second cat.

In the second cat, standard input first reads from the pipe (the contents of infile), then reads from infile2, writing each line to outfile2. After running this, outfile will be a copy of infile, and outfile2 will contain infile followed by infile2.

Finally, you actually do something really similar to your example using "here string" redirection (bash family only) and backticks:

grep blah <<<`ls`

will give the same result as

ls | grep blah

But I think the redirection version will first read all of the output of ls into a buffer (in memory), and then feed that buffer to grep one line at a time, whereas the piped version will take each line from ls as it emerges, and pass that line to grep.

_{Note:The answer reflects my own understanding of these mechanisms up to date, accumulated over research and reading of the answers by the peers on this site and unix.stackexchange.com, and will be updated as time goes on. Don't hesitate to ask questions or suggest improvements in the comments. I also suggest you try to see how syscalls work in shell with strace command. Also please don't be intimidated by the notion of internals or syscalls - you don't have to know or be able to use them in order to understand how shell does things, but they definitely help understanding.}

TL;DR

• 
  | pipes are not associated with an entry on disk, therefore do not have an inode number of disk filesystem (but do have inode in pipefs virtual filesystem in kernel-space), but redirections often involve files, which do have disk entries and therefore have corresponding inode.


• pipes are not lseek()'able so commands can't read some data and then rewind back, but when you redirect with > or < usually it's a file which is lseek() able object, so commands can navigate however they please.


• redirections are manipulations on file descriptors, which can be many; pipes have only two file descriptors - one for left command and one for right command


• redirection on standard streams and pipes are both buffered.


• pipes almost always involve forking and therefore pairs of processes are involved; redirections - not always, though in both cases resulting file descriptors are inherited by sub-processes.


• pipes always connect file descriptors (a pair), redirections - either use a pathname or file descriptors.


• pipes are Inter-Process Communication method, while redirections are just manipulations on open files or file-like objects


• both employ dup2() syscalls underneath the hood to provide copies of file descriptors, where actual flow of data occurs.


• redirections can be applied "globally" with exec built-in command ( see this and this ), so if you do exec > output.txt every command will write to output.txt from then on. | pipes are applied only for current command (which means either simple command or subshell like seq 5 | (head -n1; head -n2) or compound commands.



• When redirection is done on files, things like echo "TEST" > file and echo "TEST" >> file both use open() syscall on that file (see also) and get file descriptor from it to pass it to dup2(). Pipes | only use pipe() and dup2() syscall.




• As far as commands being executed, pipes and redirection are no more than file descriptors - file-like objects, to which they may write blindly, or manipulate them internally (which may produce unexpected behaviors; apt for instance, tends to not even write to stdout if it knows there's redirection).

Introduction

In order to understand how these two mechanisms differ, it's necessary to understand their essential properties, the history behind the two, and their roots in C programming language. In fact, knowing what file descriptors are, and how dup2() and pipe() system calls work is essential, as well as lseek(). Shell is meant as a way of making these mechanisms abstract to the user, but digging deeper than the abstraction helps understand the true nature of shell's behavior.

The Origins of Redirections and Pipes

According to Dennis Ritche's article Prophetic Petroglyphs, pipes originated from a 1964 internal memo by Malcolm Douglas McIlroy, at the time when they were working on Multics operating system. Quote:

To put my strongest concerns into a nutshell:

1. We should have some ways of connecting programs like garden hose--screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also.

What's apparent is that at the time programs were capable of writing to disk, however that was inefficient if output was large. To quote Brian Kernighan's explanation in Unix Pipeline video :

First, you don't have to write one big massive program - you've got existing smaller programs that may already do parts of the job...Another is that it's possible that the amount of data you're procesing would not fit if you stored it in a file...because remember, we're back in the days when disks on these things had, if you were lucky, a Megabyte or two of data...So the pipeline never had to instantiate the whole output.

Thus conceptual difference is apparent: pipes are a mechanism of making programs talk to one another. Redirections - are way of writing to file at basic level. In both cases, shell makes these two things easy, but underneath the hood, there's whole lot of going on.

Going deeper: syscalls and internal workings of the shell

We start with the notion of file descriptor. File descriptors describe basically an open file (whether that's a file on disk, or in memory, or anonymous file), which is represented by an integer number. The two standard data streams (stdin,stdout,stderr) are file descriptors 0,1, and 2 respectively. Where do they come from ? Well, in shell commands the file descriptors are inherited from their parent - shell. And it's true in general for all processes - child process inherits parent's file descriptors. For daemons it is common to close all inherited file descriptors and/or redirect to other places.

Back to redirection. What is it really ? It's a mechanism that tells the shell to prepare file descriptors for command (because redirections are done by shell before command runs), and point them where the user suggested. The standard definition of output redirection is

[n]>word

That [n] there is the file descriptor number. When you do echo "Something" > /dev/null the number 1 is implied there, and echo 2> /dev/null.

Underneath the hood this is done by duplicating file descriptor via dup2() system call. Let's take df > /dev/null. The shell will create a child process where df runs, but before that it will open /dev/null as file descriptor #3, and dup2(3,1) will be issued, which makes a copy of file descriptor 3 and the copy will be 1. You know how you have two files file1.txt and file2.txt, and when you do cp file1.txt file2.txt you'll have two same files, but you can manipulate them independently ? That's kinda the same thing happening here. Often you can see that before running, the bash will do dup(1,10) to make a copy file descriptor #1 which is stdout ( and that copy will be fd #10 ) in order to restore it later. Important is to note that when you consider built-in commands (which are part of shell itself, and have no file in /bin or elsewhere) or simple commands in non-interactive shell, the shell doesn't create a child process.

And then we have things like [n]>&[m] and [n]&<[m]. This is duplicating file descriptors, which the same mechanism as dup2() only now it's in the shell syntax, conveniently available for the user.

One of the important things to note about redirection is that their order is not fixed, but is significant to how shell interprets what user wants. Compare the following:

# Make copy of where fd 2 points , then redirect fd 2

$ ls -l /proc/self/fd/  3>&2  2> /dev/null

total 0

lrwx------ 1 user user 64 Sep 13 00:08 0 -> /dev/pts/0

lrwx------ 1 user user 64 Sep 13 00:08 1 -> /dev/pts/0

l-wx------ 1 user user 64 Sep 13 00:08 2 -> /dev/null

lrwx------ 1 runner user 64 Sep 13 00:08 3 -> /dev/pts/0

lr-x------ 1 user user 64 Sep 13 00:08 4 -> /proc/29/fd



# redirect fd #2 first, then clone it

$ ls -l /proc/self/fd/    2> /dev/null 3>&2

total 0

lrwx------ 1 user user 64 Sep 13 00:08 0 -> /dev/pts/0

lrwx------ 1 user user 64 Sep 13 00:08 1 -> /dev/pts/0

l-wx------ 1 user user 64 Sep 13 00:08 2 -> /dev/null

l-wx------ 1 user user 64 Sep 13 00:08 3 -> /dev/null

lr-x------ 1 user user 64 Sep 13 00:08 4 -> /proc/31/fd

The practical use of these in shell scripting can be versatile:

• saving output to variable of a program that only writes to stderr


• swapping stderr and stdout


• separating even input lines from odd input lines

and many other.

Plumbing with pipe() and dup2()

So how do pipes get created ? Via pipe() syscall, which will take as input an array (aka list) called pipefd of two items of type int (integer). Those two integers are file descriptors. The pipefd[0] will be the read end of the pipe and pipefd[1] will be the write end. So in df | grep 'foo', grep will get copy of pipefd[0] and df will get a copy of pipefd[1]. But how ? Of course, with the magic of dup2() syscall. For df in our example, let's say pipefd[1] has #4, so the shell will make a child, do dup2(4,1) (remember my cp example ?), and then do execve() to actually run df. Naturally, df will inherit file descriptor #1, but will be unaware that it's no longer pointing at terminal, but actually fd #4, which is actually the write end of the pipe. Naturally, same thing will occur with grep 'foo' except with different numbers of file descriptors.

Now, interesting question: could we make pipes that redirect fd #2 as well, not just fd #1 ? Yes, in fact that's what |& does in bash. The POSIX standard requires shell command language to support df 2>&1 | grep 'foo' syntax for that purpose, but bash does |& as well.

What's important to note is that pipes always deal with file descriptors. There exists FIFO or named pipe, which has a filename on disk and let's you use it as a file, but behaves like a pipe. But the | types of pipes are what's known as anonymous pipe - they have no filename, because they're really just two objects connected together. The fact that we're not dealing with files also makes an important implication: pipes aren't lseek()'able. Files, either in memory or on disk, are static - programs can use lseek() syscall to jump to byte 120, then back to byte 10, then forward all the way to the end. Pipes are not static - they're sequential, and therefore you cannot rewind data you get from them with lseek(). This is what makes some programs aware if they're reading from file or from pipe, and thus they can make necessary adjustments for efficient performance; in other words, a prog can detect if I do cat file.txt | prog or prog < input.txt. Real work example of that is tail.

The other two very interesting property of pipes is that they have a buffer, which on Linux is 4096 bytes, and they actually have a filesystem as defined in Linux source code ! They're not simply an object for passing data around, they are a datastructure themselves ! In fact, because there exists pipefs filesystem, which manages both pipes and FIFOs, pipes have an inode number on their respective filesystem:

# Stdout of ls is wired to pipe

$ ls -l /proc/self/fd/  | cat  

lrwx------ 1 user user 64 Sep 13 00:02 0 -> /dev/pts/0

l-wx------ 1 user user 64 Sep 13 00:02 1 -> pipe:[15655630]

lrwx------ 1 user user 64 Sep 13 00:02 2 -> /dev/pts/0

lr-x------ 1 user user 64 Sep 13 00:02 3 -> /proc/22/fd

# stdin of ls is wired to pipe

$ true | ls -l /proc/self/fd/0

lr-x------ 1 user user 64 Sep 13 03:58 /proc/self/fd/0 -> 'pipe:[54741]'

On Linux pipes are uni-directional, just like redirection. On some Unix-like implementations - there are bi-directional pipes. Although with magic of shell scripting, you can make bi-directional pipes on Linux as well.

See Also:

• How to make output of any shell command unbuffered?


• 
  Wikipedia example of how pipeline is created in C using pipe() syscall and dup2().


• why pipes are used instead of input redirection


• What is the differences between &> and 2>&1


• Redirections such as <<,<<< are implemented as anonymous (unlinked) temp files in bash and ksh, while < <() uses anonymous pipes ; /bin/dash uses pipes for <<. See What's the difference between <<, <<< and < < in bash?
  

To add to the other answers, there are subtle semantic difference too - e.g. pipes close more readily than redirects:

seq 5 | (head -n1; head -n1)                # just 1

seq 5 > tmp5; (head -n1; head -n1) < tmp5   # 1 and 2

seq 5 | (read LINE; echo $LINE; head -n1)   # 1 and 2

In the first example, when the first call to head finishes, it closes the pipe, and seq terminates, so there's no input available for the second head.

In the second example, head consumes the first line, but when it closes it's own stdin pipe, the file remains open for the next call to use.

The third example shows that if we use read to avoid closing the pipe it is still available within the subprocess.

So the "stream" is the thing that we shunt data through (stdin etc), and is the same in both cases, but the pipe connects streams from two processes, where a redirection connects a streams between a process and a file, so you can see source of both the similarities and differences.

P.S. If you're as curious about and/or surprised by those examples as I was, you can get dig in further using trap to see how the processes resolve, E.g:

(trap 'echo seq EXITed >&2' EXIT; seq 5) | (trap 'echo all done' EXIT; (trap 'echo first head exited' EXIT; head -n1)

echo '.'

(trap 'echo second head exited' EXIT; head -n1))

Sometimes the first process closes before 1 is printed, sometimes afterwards.

I also found it interesting to use exec <&- to close the stream from the redirection to approximate the behaviour of the pipe (albeit with an error):

seq 5 > tmp5

(trap 'echo all done' EXIT

(trap 'echo first head exited' EXIT; head -n1)

echo '.'

exec <&-

(trap 'echo second head exited' EXIT; head -n1)) < tmp5`

I've hit a problem with this in C today. Essentially Pipe's have different semantics to redirects as well, even when sent to stdin. Really I think given the differences, pipes should go somewhere other than stdin, so that stdin and lets call it stdpipe (to make an arbitrary differential) can be handled in different ways.

Consider this. When piping one program output to another fstat seems to return zero as the st_size despite ls -lha /proc/{PID}/fd showing that there is a file. When redirecting a file this is not the case (at least on debian wheezy, stretch and jessie vanilla and ubuntu 14.04, 16.04 vanilla.

If you cat /proc/{PID}/fd/0 with a redirection you'll be able to repeat to read as many times as you like. If you do this with a pipe you'll notice that the second time you run the task consecutively, you don't get the same output.