Bash: create anonymous fifo
We all know mkfifo
and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp
and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes)
; in Bash there is exec 5<>file
so one would expect something like "exec 5<> -"
or "pipe <5 >6"
-is there something like that in Bash?
bash script pipe
add a comment |
We all know mkfifo
and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp
and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes)
; in Bash there is exec 5<>file
so one would expect something like "exec 5<> -"
or "pipe <5 >6"
-is there something like that in Bash?
bash script pipe
add a comment |
We all know mkfifo
and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp
and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes)
; in Bash there is exec 5<>file
so one would expect something like "exec 5<> -"
or "pipe <5 >6"
-is there something like that in Bash?
bash script pipe
We all know mkfifo
and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp
and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes)
; in Bash there is exec 5<>file
so one would expect something like "exec 5<> -"
or "pipe <5 >6"
-is there something like that in Bash?
bash script pipe
bash script pipe
asked Sep 3 '10 at 15:18
Adrian PanasiukAdrian Panasiuk
5301714
5301714
add a comment |
add a comment |
7 Answers
7
active
oldest
votes
You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:
# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-
If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.
# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
add a comment |
While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.
In bash, ksh and zsh, assuming your system supports /dev/fd
(most do nowadays), you can tie the input or the output of a command to a file name: <(command)
expands to a file name that designates a pipe connected to the output from command
, and >(command)
expands to a file name that designates a pipe connected to the input of command
. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,
diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)
This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1)
is equivalent to command1 | command2
, except that its status is that of command2
. Another use case is exec > >(postprocessing)
, which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing
.
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something likepipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).
– Gilles
Aug 17 '15 at 10:14
add a comment |
Bash 4 has coprocesses.
A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.
The format for a coprocess is:
coproc [NAME] command [redirections]
add a comment |
As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:coproc THING { dothing; }
Now your FDs are in${THING[*]}
and you can runcoproc OTHERTHING { dothing; }
and send and receive things to and from both.
– clacke
Mar 9 at 13:57
add a comment |
While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:)
, as long as you stay on Linux.
So the shortest and simplest answer to your question is:
exec 5<> <(:)
On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
Ifexec
is passed and anonymous fifo that is opened only for reading, thenexec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a-bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation sayingexec 5<> <(:)
is explicated permitted.
– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doingopen(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash<>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765
– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed toexec 5<>
, then enterfun() { ls -l $1; ls -lH $1; }; fun <(:)
.
– David Anderson
Mar 10 at 16:46
|
show 1 more comment
The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)
. The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_upper=$(ulimit -n)"
if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
|| 10#$1 -ge MakeFIFO_upper ]] || ! eval exec "$1<> " <(:) 2>"/dev/null"; then
echo "$FUNCNAME: $1: Could not create FIFO" >&2
return "1"
fi
}
The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
. The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
if ! eval exec "$1<> " <(:) 2>"/dev/null"; then
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
fi
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.
fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"
Below is an example of reading the entire contents of the anonymous FIFO.
echo "EOF" >&"$fd"
while read -u "$fd" message; do
[[ $message != *EOF ]] || break
echo "$message"
done
This produces the following output.
Now is the
time for all
good men
The command below closes the anonymous FIFO.
eval exec "$fd>&-"
References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security
add a comment |
Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:
# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f184307%2fbash-create-anonymous-fifo%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
7 Answers
7
active
oldest
votes
7 Answers
7
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:
# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-
If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.
# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
add a comment |
You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:
# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-
If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.
# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
add a comment |
You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:
# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-
If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.
# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:
# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-
If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.
# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
edited Aug 17 '13 at 23:56
answered Aug 17 '13 at 23:22
htamashtamas
48144
48144
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
add a comment |
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
You can combine this with automatic finding of unused file descriptors: stackoverflow.com/questions/8297415/…
– CMCDragonkai
May 31 '18 at 4:22
add a comment |
While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.
In bash, ksh and zsh, assuming your system supports /dev/fd
(most do nowadays), you can tie the input or the output of a command to a file name: <(command)
expands to a file name that designates a pipe connected to the output from command
, and >(command)
expands to a file name that designates a pipe connected to the input of command
. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,
diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)
This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1)
is equivalent to command1 | command2
, except that its status is that of command2
. Another use case is exec > >(postprocessing)
, which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing
.
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something likepipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).
– Gilles
Aug 17 '15 at 10:14
add a comment |
While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.
In bash, ksh and zsh, assuming your system supports /dev/fd
(most do nowadays), you can tie the input or the output of a command to a file name: <(command)
expands to a file name that designates a pipe connected to the output from command
, and >(command)
expands to a file name that designates a pipe connected to the input of command
. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,
diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)
This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1)
is equivalent to command1 | command2
, except that its status is that of command2
. Another use case is exec > >(postprocessing)
, which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing
.
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something likepipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).
– Gilles
Aug 17 '15 at 10:14
add a comment |
While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.
In bash, ksh and zsh, assuming your system supports /dev/fd
(most do nowadays), you can tie the input or the output of a command to a file name: <(command)
expands to a file name that designates a pipe connected to the output from command
, and >(command)
expands to a file name that designates a pipe connected to the input of command
. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,
diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)
This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1)
is equivalent to command1 | command2
, except that its status is that of command2
. Another use case is exec > >(postprocessing)
, which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing
.
While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.
In bash, ksh and zsh, assuming your system supports /dev/fd
(most do nowadays), you can tie the input or the output of a command to a file name: <(command)
expands to a file name that designates a pipe connected to the output from command
, and >(command)
expands to a file name that designates a pipe connected to the input of command
. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,
diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)
This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1)
is equivalent to command1 | command2
, except that its status is that of command2
. Another use case is exec > >(postprocessing)
, which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing
.
answered Sep 3 '10 at 19:01
GillesGilles
53.2k15115162
53.2k15115162
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something likepipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).
– Gilles
Aug 17 '15 at 10:14
add a comment |
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something likepipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).
– Gilles
Aug 17 '15 at 10:14
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once?
– Eyal
Aug 17 '15 at 9:15
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something like
pipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).– Gilles
Aug 17 '15 at 10:14
@Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something like
pipe:[123456]
). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway).– Gilles
Aug 17 '15 at 10:14
add a comment |
Bash 4 has coprocesses.
A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.
The format for a coprocess is:
coproc [NAME] command [redirections]
add a comment |
Bash 4 has coprocesses.
A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.
The format for a coprocess is:
coproc [NAME] command [redirections]
add a comment |
Bash 4 has coprocesses.
A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.
The format for a coprocess is:
coproc [NAME] command [redirections]
Bash 4 has coprocesses.
A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.
The format for a coprocess is:
coproc [NAME] command [redirections]
edited Jul 26 '17 at 20:00
answered Sep 3 '10 at 16:00
Dennis WilliamsonDennis Williamson
78.1k14130167
78.1k14130167
add a comment |
add a comment |
As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:coproc THING { dothing; }
Now your FDs are in${THING[*]}
and you can runcoproc OTHERTHING { dothing; }
and send and receive things to and from both.
– clacke
Mar 9 at 13:57
add a comment |
As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:coproc THING { dothing; }
Now your FDs are in${THING[*]}
and you can runcoproc OTHERTHING { dothing; }
and send and receive things to and from both.
– clacke
Mar 9 at 13:57
add a comment |
As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.
As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.
answered Oct 8 '12 at 1:49
Radu CRadu C
1313
1313
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:coproc THING { dothing; }
Now your FDs are in${THING[*]}
and you can runcoproc OTHERTHING { dothing; }
and send and receive things to and from both.
– clacke
Mar 9 at 13:57
add a comment |
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:coproc THING { dothing; }
Now your FDs are in${THING[*]}
and you can runcoproc OTHERTHING { dothing; }
and send and receive things to and from both.
– clacke
Mar 9 at 13:57
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:
coproc THING { dothing; }
Now your FDs are in ${THING[*]}
and you can run coproc OTHERTHING { dothing; }
and send and receive things to and from both.– clacke
Mar 9 at 13:57
Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead:
coproc THING { dothing; }
Now your FDs are in ${THING[*]}
and you can run coproc OTHERTHING { dothing; }
and send and receive things to and from both.– clacke
Mar 9 at 13:57
add a comment |
While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:)
, as long as you stay on Linux.
So the shortest and simplest answer to your question is:
exec 5<> <(:)
On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
Ifexec
is passed and anonymous fifo that is opened only for reading, thenexec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a-bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation sayingexec 5<> <(:)
is explicated permitted.
– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doingopen(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash<>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765
– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed toexec 5<>
, then enterfun() { ls -l $1; ls -lH $1; }; fun <(:)
.
– David Anderson
Mar 10 at 16:46
|
show 1 more comment
While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:)
, as long as you stay on Linux.
So the shortest and simplest answer to your question is:
exec 5<> <(:)
On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
Ifexec
is passed and anonymous fifo that is opened only for reading, thenexec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a-bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation sayingexec 5<> <(:)
is explicated permitted.
– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doingopen(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash<>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765
– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed toexec 5<>
, then enterfun() { ls -l $1; ls -lH $1; }; fun <(:)
.
– David Anderson
Mar 10 at 16:46
|
show 1 more comment
While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:)
, as long as you stay on Linux.
So the shortest and simplest answer to your question is:
exec 5<> <(:)
On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.
While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:)
, as long as you stay on Linux.
So the shortest and simplest answer to your question is:
exec 5<> <(:)
On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.
answered Mar 9 at 15:22
clackeclacke
163111
163111
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
Ifexec
is passed and anonymous fifo that is opened only for reading, thenexec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a-bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation sayingexec 5<> <(:)
is explicated permitted.
– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doingopen(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash<>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765
– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed toexec 5<>
, then enterfun() { ls -l $1; ls -lH $1; }; fun <(:)
.
– David Anderson
Mar 10 at 16:46
|
show 1 more comment
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
Ifexec
is passed and anonymous fifo that is opened only for reading, thenexec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a-bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation sayingexec 5<> <(:)
is explicated permitted.
– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doingopen(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash<>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765
– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed toexec 5<>
, then enterfun() { ls -l $1; ls -lH $1; }; fun <(:)
.
– David Anderson
Mar 10 at 16:46
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed.
– David Anderson
Mar 9 at 16:19
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
@DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug?
– clacke
Mar 9 at 19:17
1
1
If
exec
is passed and anonymous fifo that is opened only for reading, then exec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a -bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation saying exec 5<> <(:)
is explicated permitted.– David Anderson
Mar 9 at 21:36
If
exec
is passed and anonymous fifo that is opened only for reading, then exec
should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a -bash: /dev/fd/5: Permission denied
message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation saying exec 5<> <(:)
is explicated permitted.– David Anderson
Mar 9 at 21:36
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doing
open(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash <>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765– clacke
Mar 10 at 6:33
@DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doing
open(..., O_RDWR)
on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash <>
: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765– clacke
Mar 10 at 6:33
Not that it matters, but if you want to see under Ubuntu what is passed to
exec 5<>
, then enter fun() { ls -l $1; ls -lH $1; }; fun <(:)
.– David Anderson
Mar 10 at 16:46
Not that it matters, but if you want to see under Ubuntu what is passed to
exec 5<>
, then enter fun() { ls -l $1; ls -lH $1; }; fun <(:)
.– David Anderson
Mar 10 at 16:46
|
show 1 more comment
The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)
. The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_upper=$(ulimit -n)"
if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
|| 10#$1 -ge MakeFIFO_upper ]] || ! eval exec "$1<> " <(:) 2>"/dev/null"; then
echo "$FUNCNAME: $1: Could not create FIFO" >&2
return "1"
fi
}
The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
. The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
if ! eval exec "$1<> " <(:) 2>"/dev/null"; then
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
fi
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.
fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"
Below is an example of reading the entire contents of the anonymous FIFO.
echo "EOF" >&"$fd"
while read -u "$fd" message; do
[[ $message != *EOF ]] || break
echo "$message"
done
This produces the following output.
Now is the
time for all
good men
The command below closes the anonymous FIFO.
eval exec "$fd>&-"
References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security
add a comment |
The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)
. The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_upper=$(ulimit -n)"
if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
|| 10#$1 -ge MakeFIFO_upper ]] || ! eval exec "$1<> " <(:) 2>"/dev/null"; then
echo "$FUNCNAME: $1: Could not create FIFO" >&2
return "1"
fi
}
The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
. The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
if ! eval exec "$1<> " <(:) 2>"/dev/null"; then
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
fi
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.
fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"
Below is an example of reading the entire contents of the anonymous FIFO.
echo "EOF" >&"$fd"
while read -u "$fd" message; do
[[ $message != *EOF ]] || break
echo "$message"
done
This produces the following output.
Now is the
time for all
good men
The command below closes the anonymous FIFO.
eval exec "$fd>&-"
References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security
add a comment |
The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)
. The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_upper=$(ulimit -n)"
if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
|| 10#$1 -ge MakeFIFO_upper ]] || ! eval exec "$1<> " <(:) 2>"/dev/null"; then
echo "$FUNCNAME: $1: Could not create FIFO" >&2
return "1"
fi
}
The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
. The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
if ! eval exec "$1<> " <(:) 2>"/dev/null"; then
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
fi
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.
fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"
Below is an example of reading the entire contents of the anonymous FIFO.
echo "EOF" >&"$fd"
while read -u "$fd" message; do
[[ $message != *EOF ]] || break
echo "$message"
done
This produces the following output.
Now is the
time for all
good men
The command below closes the anonymous FIFO.
eval exec "$fd>&-"
References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security
The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu)
. The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_upper=$(ulimit -n)"
if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
|| 10#$1 -ge MakeFIFO_upper ]] || ! eval exec "$1<> " <(:) 2>"/dev/null"; then
echo "$FUNCNAME: $1: Could not create FIFO" >&2
return "1"
fi
}
The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)
. The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.
MakeFIFO() {
local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)"
local "MakeFIFO_directory=" "MakeFIFO_file" "MakeFIFO_trap=$(trap -p EXIT)"
local "MakeFIFO_handler=${MakeFIFO_trap#*'}"
MakeFIFO_CleanUp() {
rm -rf "$MakeFIFO_directory"
"$MakeFIFO_handler"
}
MakeFIFO_handler="${MakeFIFO_handler%'*}"
[[ -n $MakeFIFO_trap ]] || MakeFIFO_trap="trap EXIT"
trap "MakeFIFO_CleanUp" EXIT
until "$MakeFIFO_success"; do
[[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
&& 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
if ! eval exec "$1<> " <(:) 2>"/dev/null"; then
MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
MakeFIFO_file="$MakeFIFO_directory/pipe"
mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
fi
MakeFIFO_success="true"
done
rm -rf "$MakeFIFO_directory"
eval "$MakeFIFO_trap"
"$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}
Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.
fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"
Below is an example of reading the entire contents of the anonymous FIFO.
echo "EOF" >&"$fd"
while read -u "$fd" message; do
[[ $message != *EOF ]] || break
echo "$message"
done
This produces the following output.
Now is the
time for all
good men
The command below closes the anonymous FIFO.
eval exec "$fd>&-"
References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security
edited Mar 9 at 16:05
answered Feb 14 at 13:31
David AndersonDavid Anderson
556411
556411
add a comment |
add a comment |
Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:
# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
add a comment |
Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:
# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
add a comment |
Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:
# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:
# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
answered Jul 5 '16 at 22:00
Luiz Felipe SilvaLuiz Felipe Silva
11
11
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
add a comment |
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
6
6
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
I can't help noticing that your one-liner has more than one line.
– Dmitry Grigoryev
Jul 6 '16 at 11:56
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f184307%2fbash-create-anonymous-fifo%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown