How does `mbuffer -T` work?
I'm currently trying to make a tar backup to a tape with mbuffer acting as a buffer between e.g:
tar --acls -c /var/test | mbuffer -m 8G -P 100% -R 100M | dd of=/dev/nst0 bs=256k
But the buffer is still empty about 10-20 times. So I wanted to use a file based buffer with the following command:
tar --acls -c /var/test | mbuffer -m 60G -T /srv/testbuffer -P 100% -R 100M | dd of=/dev/nst0 bs=256k
My understanding was, that the option -T would create a file at the given location and use it as a read/write buffer.
But no file is created, and on one system mbuffer just hangs.
- Am I missing something?
- Does
mbuffercreate the file in a special way? - Does
mbuffercreate the file in another directory? - Does
mbufferextend/use the swap in some way?
As far as I read in the source of mbuffer (settings.c) mbuffer allocs space of the size of the tmpfile via malloc().
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the -T parameter?
ubuntu command-line
edited 2 days ago
Kamil Maciorowski
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm currently trying to make a tar backup to a tape with mbuffer acting as a buffer between e.g:
tar --acls -c /var/test | mbuffer -m 8G -P 100% -R 100M | dd of=/dev/nst0 bs=256k
But the buffer is still empty about 10-20 times. So I wanted to use a file based buffer with the following command:
tar --acls -c /var/test | mbuffer -m 60G -T /srv/testbuffer -P 100% -R 100M | dd of=/dev/nst0 bs=256k
My understanding was, that the option -T would create a file at the given location and use it as a read/write buffer.
But no file is created, and on one system mbuffer just hangs.
- Am I missing something?
- Does
mbuffercreate the file in a special way? - Does
mbuffercreate the file in another directory? - Does
mbufferextend/use the swap in some way?
As far as I read in the source of mbuffer (settings.c) mbuffer allocs space of the size of the tmpfile via malloc().
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the -T parameter?
ubuntu command-line
edited 2 days ago
Kamil Maciorowski
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
I'm currently trying to make a tar backup to a tape with mbuffer acting as a buffer between e.g:
tar --acls -c /var/test | mbuffer -m 8G -P 100% -R 100M | dd of=/dev/nst0 bs=256k
But the buffer is still empty about 10-20 times. So I wanted to use a file based buffer with the following command:
tar --acls -c /var/test | mbuffer -m 60G -T /srv/testbuffer -P 100% -R 100M | dd of=/dev/nst0 bs=256k
My understanding was, that the option -T would create a file at the given location and use it as a read/write buffer.
But no file is created, and on one system mbuffer just hangs.
- Am I missing something?
- Does
mbuffercreate the file in a special way? - Does
mbuffercreate the file in another directory? - Does
mbufferextend/use the swap in some way?
As far as I read in the source of mbuffer (settings.c) mbuffer allocs space of the size of the tmpfile via malloc().
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the -T parameter?
ubuntu command-line
edited 2 days ago
Kamil Maciorowski
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I'm currently trying to make a tar backup to a tape with mbuffer acting as a buffer between e.g:
tar --acls -c /var/test | mbuffer -m 8G -P 100% -R 100M | dd of=/dev/nst0 bs=256k
But the buffer is still empty about 10-20 times. So I wanted to use a file based buffer with the following command:
tar --acls -c /var/test | mbuffer -m 60G -T /srv/testbuffer -P 100% -R 100M | dd of=/dev/nst0 bs=256k
My understanding was, that the option -T would create a file at the given location and use it as a read/write buffer.
But no file is created, and on one system mbuffer just hangs.
- Am I missing something?
- Does
mbuffercreate the file in a special way? - Does
mbuffercreate the file in another directory? - Does
mbufferextend/use the swap in some way?
As far as I read in the source of mbuffer (settings.c) mbuffer allocs space of the size of the tmpfile via malloc().
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the -T parameter?
ubuntu command-line
ubuntu command-line
edited 2 days ago
Kamil Maciorowski
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 2 days ago
Kamil Maciorowski
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 2 days ago
Kamil Maciorowski
1,5391827
edited 2 days ago
Kamil Maciorowski
1,5391827
edited 2 days ago
Kamil Maciorowski
1,5391827
1,5391827
asked Jan 14 at 23:09
bpfrhbpfrh
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Jan 14 at 23:09
bpfrhbpfrh
133
asked Jan 14 at 23:09
bpfrhbpfrh
133
133
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
bpfrh is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
no file is created
mbuffer -m 60G -T /srv/testbuffer creates a file, opens it and unlinks it right after. If you check the file descriptors in /proc/<PID of mbuffer>/fd you will find one that points to
/srv/testbuffer (deleted)
This is quite a standard trick when it comes to temporary files. If mbuffer terminates for whatever reason (even if brutally killed), the filesystem will eventually free the space (in the worst case of power failure or kernel panic: after fsck in the future). There's no risk a totally unused, abandoned file will occupy your diskspace forever.
on one system
mbufferjust hangs
The tool actually reserves diskspace like fallocate (not just creates a sparse file like truncate) before it starts its main job of buffering data. Allocating a large file may take time. This delay depends on the filesystem type, possible fragmentation etc.
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the
-Tparameter?
No. The file exists in the filesystem appropriate for the chosen path (/srv/testbuffer in your case), but the path no longer exists. (Note: a new file with the same path will be a separate file with different inode number).
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
bpfrh is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f494511%2fhow-does-mbuffer-t-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
no file is created
mbuffer -m 60G -T /srv/testbuffer creates a file, opens it and unlinks it right after. If you check the file descriptors in /proc/<PID of mbuffer>/fd you will find one that points to
/srv/testbuffer (deleted)
This is quite a standard trick when it comes to temporary files. If mbuffer terminates for whatever reason (even if brutally killed), the filesystem will eventually free the space (in the worst case of power failure or kernel panic: after fsck in the future). There's no risk a totally unused, abandoned file will occupy your diskspace forever.
on one system
mbufferjust hangs
The tool actually reserves diskspace like fallocate (not just creates a sparse file like truncate) before it starts its main job of buffering data. Allocating a large file may take time. This delay depends on the filesystem type, possible fragmentation etc.
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the
-Tparameter?
No. The file exists in the filesystem appropriate for the chosen path (/srv/testbuffer in your case), but the path no longer exists. (Note: a new file with the same path will be a separate file with different inode number).
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
add a comment |
no file is created
mbuffer -m 60G -T /srv/testbuffer creates a file, opens it and unlinks it right after. If you check the file descriptors in /proc/<PID of mbuffer>/fd you will find one that points to
/srv/testbuffer (deleted)
This is quite a standard trick when it comes to temporary files. If mbuffer terminates for whatever reason (even if brutally killed), the filesystem will eventually free the space (in the worst case of power failure or kernel panic: after fsck in the future). There's no risk a totally unused, abandoned file will occupy your diskspace forever.
on one system
mbufferjust hangs
The tool actually reserves diskspace like fallocate (not just creates a sparse file like truncate) before it starts its main job of buffering data. Allocating a large file may take time. This delay depends on the filesystem type, possible fragmentation etc.
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the
-Tparameter?
No. The file exists in the filesystem appropriate for the chosen path (/srv/testbuffer in your case), but the path no longer exists. (Note: a new file with the same path will be a separate file with different inode number).
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
add a comment |
no file is created
mbuffer -m 60G -T /srv/testbuffer creates a file, opens it and unlinks it right after. If you check the file descriptors in /proc/<PID of mbuffer>/fd you will find one that points to
/srv/testbuffer (deleted)
This is quite a standard trick when it comes to temporary files. If mbuffer terminates for whatever reason (even if brutally killed), the filesystem will eventually free the space (in the worst case of power failure or kernel panic: after fsck in the future). There's no risk a totally unused, abandoned file will occupy your diskspace forever.
on one system
mbufferjust hangs
The tool actually reserves diskspace like fallocate (not just creates a sparse file like truncate) before it starts its main job of buffering data. Allocating a large file may take time. This delay depends on the filesystem type, possible fragmentation etc.
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the
-Tparameter?
No. The file exists in the filesystem appropriate for the chosen path (/srv/testbuffer in your case), but the path no longer exists. (Note: a new file with the same path will be a separate file with different inode number).
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
no file is created
mbuffer -m 60G -T /srv/testbuffer creates a file, opens it and unlinks it right after. If you check the file descriptors in /proc/<PID of mbuffer>/fd you will find one that points to
/srv/testbuffer (deleted)
This is quite a standard trick when it comes to temporary files. If mbuffer terminates for whatever reason (even if brutally killed), the filesystem will eventually free the space (in the worst case of power failure or kernel panic: after fsck in the future). There's no risk a totally unused, abandoned file will occupy your diskspace forever.
on one system
mbufferjust hangs
The tool actually reserves diskspace like fallocate (not just creates a sparse file like truncate) before it starts its main job of buffering data. Allocating a large file may take time. This delay depends on the filesystem type, possible fragmentation etc.
Does that mean that I still need the same amount of RAM, or at least swap, as if I would use it without the
-Tparameter?
No. The file exists in the filesystem appropriate for the chosen path (/srv/testbuffer in your case), but the path no longer exists. (Note: a new file with the same path will be a separate file with different inode number).
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
edited 2 days ago
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
answered 2 days ago
Kamil MaciorowskiKamil Maciorowski
1,5391827
1,5391827
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
add a comment |
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
Thanks, I also wrote to the maintainer of mbuffer and he gave me the same explanation. I was just about to Answer the question myself :)
– bpfrh
2 days ago
add a comment |
bpfrh is a new contributor. Be nice, and check out our Code of Conduct.
bpfrh is a new contributor. Be nice, and check out our Code of Conduct.
bpfrh is a new contributor. Be nice, and check out our Code of Conduct.
bpfrh is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f494511%2fhow-does-mbuffer-t-work%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
