remove all lines with nullbytes/corrupt data
So I recovered a text file from an old hdd, but I failed to completely recover all of the data. The data that wasn't correctly recovered has returned as null bytes. How can I remove every line from the file that contains these bytes?
Example of corrupt data
xE3
xAF
xE2
xBF
NUL
xBD
and a ton more...
I know NULL is equal to x00.
How can I remove every line containing corrupt data with sed rather than removing the bytes individually?
There are so many variations of bytes/corrupt data that I doubt I would be able to discover all of them with regex..
text-processing awk sed regular-expression null
edited Feb 28 at 23:13
katosh
1619
asked Feb 28 at 18:48
user339364user339364
61
add a comment |
So I recovered a text file from an old hdd, but I failed to completely recover all of the data. The data that wasn't correctly recovered has returned as null bytes. How can I remove every line from the file that contains these bytes?
Example of corrupt data
xE3
xAF
xE2
xBF
NUL
xBD
and a ton more...
I know NULL is equal to x00.
How can I remove every line containing corrupt data with sed rather than removing the bytes individually?
There are so many variations of bytes/corrupt data that I doubt I would be able to discover all of them with regex..
text-processing awk sed regular-expression null
edited Feb 28 at 23:13
katosh
1619
asked Feb 28 at 18:48
user339364user339364
61
add a comment |
So I recovered a text file from an old hdd, but I failed to completely recover all of the data. The data that wasn't correctly recovered has returned as null bytes. How can I remove every line from the file that contains these bytes?
Example of corrupt data
xE3
xAF
xE2
xBF
NUL
xBD
and a ton more...
I know NULL is equal to x00.
How can I remove every line containing corrupt data with sed rather than removing the bytes individually?
There are so many variations of bytes/corrupt data that I doubt I would be able to discover all of them with regex..
text-processing awk sed regular-expression null
edited Feb 28 at 23:13
katosh
1619
asked Feb 28 at 18:48
user339364user339364
61
So I recovered a text file from an old hdd, but I failed to completely recover all of the data. The data that wasn't correctly recovered has returned as null bytes. How can I remove every line from the file that contains these bytes?
Example of corrupt data
xE3
xAF
xE2
xBF
NUL
xBD
and a ton more...
I know NULL is equal to x00.
How can I remove every line containing corrupt data with sed rather than removing the bytes individually?
There are so many variations of bytes/corrupt data that I doubt I would be able to discover all of them with regex..
text-processing awk sed regular-expression null
text-processing awk sed regular-expression null
edited Feb 28 at 23:13
katosh
1619
asked Feb 28 at 18:48
user339364user339364
61
edited Feb 28 at 23:13
katosh
1619
asked Feb 28 at 18:48
user339364user339364
61
edited Feb 28 at 23:13
katosh
1619
edited Feb 28 at 23:13
katosh
1619
edited Feb 28 at 23:13
katosh
1619
1619
asked Feb 28 at 18:48
user339364user339364
61
asked Feb 28 at 18:48
user339364user339364
61
asked Feb 28 at 18:48
user339364user339364
61
61
add a comment |
add a comment |
5 Answers
5
active
oldest
votes
To remove lines that contain byte 0 or bytes 128 to 255, you can use:
perl -ne 'print unless /[200-377]/'
Or with GNU grep built with PCRE support:
LC_ALL=C grep -vaP '[200-377]'
See also the strings command to extract what looks like printable text from data.
To just remove those bytes:
tr -d '200-377'
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
add a comment |
You can try with this Perl command:
<in_file perl -lne's///g; print if $_'
in_file is the input. Pipe redirection can be used.
It removes NULs globally. You can adjust the regular expression to suit your needs.
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
add a comment |
You can remove all lines of yourfile that contain a null byte with gnu sed by
sed '/x0/d' yourfile
This also works in a pipe:
cat yourfile | sed '/x0/d'
answered Feb 28 at 21:36
katoshkatosh
1619
add a comment |
You might be able to use strings with a minimum length of, say, five characters
strings -w -n5 corrupted.txt
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
add a comment |
Yes. You can do it like this: sed -e '/x00/d' [filename] > [new_filename]
If you want to edit the file in-place: sed -i '/x00/d' [filename]
You can also, combine the two, change the original file and keep a backup copy: sed -i~ '/x00/d' [filename]
That will delete any line of the file that contains at least 1 NULL.
answered Feb 28 at 23:09
Scottie HScottie H
676
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f503617%2fremove-all-lines-with-nullbytes-corrupt-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
5 Answers
5
active
oldest
votes
5 Answers
5
active
oldest
votes
active
oldest
votes
active
oldest
votes
To remove lines that contain byte 0 or bytes 128 to 255, you can use:
perl -ne 'print unless /[200-377]/'
Or with GNU grep built with PCRE support:
LC_ALL=C grep -vaP '[200-377]'
See also the strings command to extract what looks like printable text from data.
To just remove those bytes:
tr -d '200-377'
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
add a comment |
To remove lines that contain byte 0 or bytes 128 to 255, you can use:
perl -ne 'print unless /[200-377]/'
Or with GNU grep built with PCRE support:
LC_ALL=C grep -vaP '[200-377]'
See also the strings command to extract what looks like printable text from data.
To just remove those bytes:
tr -d '200-377'
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
add a comment |
To remove lines that contain byte 0 or bytes 128 to 255, you can use:
perl -ne 'print unless /[200-377]/'
Or with GNU grep built with PCRE support:
LC_ALL=C grep -vaP '[200-377]'
See also the strings command to extract what looks like printable text from data.
To just remove those bytes:
tr -d '200-377'
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
To remove lines that contain byte 0 or bytes 128 to 255, you can use:
perl -ne 'print unless /[200-377]/'
Or with GNU grep built with PCRE support:
LC_ALL=C grep -vaP '[200-377]'
See also the strings command to extract what looks like printable text from data.
To just remove those bytes:
tr -d '200-377'
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
edited Mar 1 at 7:00
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
answered Feb 28 at 21:44
Stéphane ChazelasStéphane Chazelas
311k57586945
311k57586945
add a comment |
add a comment |
You can try with this Perl command:
<in_file perl -lne's///g; print if $_'
in_file is the input. Pipe redirection can be used.
It removes NULs globally. You can adjust the regular expression to suit your needs.
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
add a comment |
You can try with this Perl command:
<in_file perl -lne's///g; print if $_'
in_file is the input. Pipe redirection can be used.
It removes NULs globally. You can adjust the regular expression to suit your needs.
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
add a comment |
You can try with this Perl command:
<in_file perl -lne's///g; print if $_'
in_file is the input. Pipe redirection can be used.
It removes NULs globally. You can adjust the regular expression to suit your needs.
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
You can try with this Perl command:
<in_file perl -lne's///g; print if $_'
in_file is the input. Pipe redirection can be used.
It removes NULs globally. You can adjust the regular expression to suit your needs.
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
answered Feb 28 at 18:57
TomaszTomasz
10.2k53168
10.2k53168
add a comment |
add a comment |
You can remove all lines of yourfile that contain a null byte with gnu sed by
sed '/x0/d' yourfile
This also works in a pipe:
cat yourfile | sed '/x0/d'
answered Feb 28 at 21:36
katoshkatosh
1619
add a comment |
You can remove all lines of yourfile that contain a null byte with gnu sed by
sed '/x0/d' yourfile
This also works in a pipe:
cat yourfile | sed '/x0/d'
answered Feb 28 at 21:36
katoshkatosh
1619
add a comment |
You can remove all lines of yourfile that contain a null byte with gnu sed by
sed '/x0/d' yourfile
This also works in a pipe:
cat yourfile | sed '/x0/d'
answered Feb 28 at 21:36
katoshkatosh
1619
You can remove all lines of yourfile that contain a null byte with gnu sed by
sed '/x0/d' yourfile
This also works in a pipe:
cat yourfile | sed '/x0/d'
answered Feb 28 at 21:36
katoshkatosh
1619
answered Feb 28 at 21:36
katoshkatosh
1619
answered Feb 28 at 21:36
katoshkatosh
1619
answered Feb 28 at 21:36
katoshkatosh
1619
1619
add a comment |
add a comment |
You might be able to use strings with a minimum length of, say, five characters
strings -w -n5 corrupted.txt
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
add a comment |
You might be able to use strings with a minimum length of, say, five characters
strings -w -n5 corrupted.txt
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
add a comment |
You might be able to use strings with a minimum length of, say, five characters
strings -w -n5 corrupted.txt
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
You might be able to use strings with a minimum length of, say, five characters
strings -w -n5 corrupted.txt
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
answered Feb 28 at 21:54
roaimaroaima
45.8k758124
45.8k758124
add a comment |
add a comment |
Yes. You can do it like this: sed -e '/x00/d' [filename] > [new_filename]
If you want to edit the file in-place: sed -i '/x00/d' [filename]
You can also, combine the two, change the original file and keep a backup copy: sed -i~ '/x00/d' [filename]
That will delete any line of the file that contains at least 1 NULL.
answered Feb 28 at 23:09
Scottie HScottie H
676
add a comment |
Yes. You can do it like this: sed -e '/x00/d' [filename] > [new_filename]
If you want to edit the file in-place: sed -i '/x00/d' [filename]
You can also, combine the two, change the original file and keep a backup copy: sed -i~ '/x00/d' [filename]
That will delete any line of the file that contains at least 1 NULL.
answered Feb 28 at 23:09
Scottie HScottie H
676
add a comment |
Yes. You can do it like this: sed -e '/x00/d' [filename] > [new_filename]
If you want to edit the file in-place: sed -i '/x00/d' [filename]
You can also, combine the two, change the original file and keep a backup copy: sed -i~ '/x00/d' [filename]
That will delete any line of the file that contains at least 1 NULL.
answered Feb 28 at 23:09
Scottie HScottie H
676
Yes. You can do it like this: sed -e '/x00/d' [filename] > [new_filename]
If you want to edit the file in-place: sed -i '/x00/d' [filename]
You can also, combine the two, change the original file and keep a backup copy: sed -i~ '/x00/d' [filename]
That will delete any line of the file that contains at least 1 NULL.
answered Feb 28 at 23:09
Scottie HScottie H
676
answered Feb 28 at 23:09
Scottie HScottie H
676
answered Feb 28 at 23:09
Scottie HScottie H
676
answered Feb 28 at 23:09
Scottie HScottie H
676
676
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f503617%2fremove-all-lines-with-nullbytes-corrupt-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
