Use disk-backed buffer between pipes
I'd like to pipe a command into a slower one, with a rather big (~200GB) buffer in between. Here is an example use-case:
command1 | buffer -f file.buffer | command2
Is there a command to do that? The ones I saw only buffered to memory.
Now that I think about it, would this achieve the desired result?
#!/bin/bash
# buffer.sh
FILE="$1"
tail -F "$FILE" 2> /dev/null &
cat > "$FILE"
Though I'm not sure how to stop the tail once it has read everything...
bash pipe buffer
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
add a comment |
I'd like to pipe a command into a slower one, with a rather big (~200GB) buffer in between. Here is an example use-case:
command1 | buffer -f file.buffer | command2
Is there a command to do that? The ones I saw only buffered to memory.
Now that I think about it, would this achieve the desired result?
#!/bin/bash
# buffer.sh
FILE="$1"
tail -F "$FILE" 2> /dev/null &
cat > "$FILE"
Though I'm not sure how to stop the tail once it has read everything...
bash pipe buffer
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
add a comment |
I'd like to pipe a command into a slower one, with a rather big (~200GB) buffer in between. Here is an example use-case:
command1 | buffer -f file.buffer | command2
Is there a command to do that? The ones I saw only buffered to memory.
Now that I think about it, would this achieve the desired result?
#!/bin/bash
# buffer.sh
FILE="$1"
tail -F "$FILE" 2> /dev/null &
cat > "$FILE"
Though I'm not sure how to stop the tail once it has read everything...
bash pipe buffer
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
I'd like to pipe a command into a slower one, with a rather big (~200GB) buffer in between. Here is an example use-case:
command1 | buffer -f file.buffer | command2
Is there a command to do that? The ones I saw only buffered to memory.
Now that I think about it, would this achieve the desired result?
#!/bin/bash
# buffer.sh
FILE="$1"
tail -F "$FILE" 2> /dev/null &
cat > "$FILE"
Though I'm not sure how to stop the tail once it has read everything...
bash pipe buffer
bash pipe buffer
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
edited Aug 26 '15 at 17:35
Gyscos
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
asked Aug 26 '15 at 6:43
GyscosGyscos
1335
1335
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I just found mbuffer.
Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f962902%2fuse-disk-backed-buffer-between-pipes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I just found mbuffer.
Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
add a comment |
I just found mbuffer.
Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
add a comment |
I just found mbuffer.
Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
I just found mbuffer.
Apparently the option -t uses a temporary file for huge buffer, which is exactly what I was looking for. Alternatively with -T /path/to/file I can choose which of my mounted filesystems will hold the file.
I also note the file gets deleted just after it's opened, so ls doesn't show it. Still ls -l /proc/<PID of mbuffer>/fd confirms it's there.
So my example becomes:
command1 | mbuffer -T file.buffer -m 200G | command2
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
edited Jan 6 at 19:05
Kamil Maciorowski
26.1k155679
26.1k155679
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
answered Aug 27 '15 at 0:01
GyscosGyscos
1335
1335
add a comment |
add a comment |
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f962902%2fuse-disk-backed-buffer-between-pipes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
