Shell - Read binary file character by character as hex
I'm trying to read serial input(/dev/tty*) character by character. I need this to parse the data following the corresponding protocol. My current method is as follows:
while read -rs -n 1 c; do
echo Read a char:
echo ${c}
done < "${SERIAL_PORT}"
This works, except for the fact that the data is binary and some bytes are x00, which is not allowed as a character in a variable. My aim is to first convert the byte into a hex string(two characters) which I will then store in the variable. This is where I get stuck. What is a good way to do this? I would prefer to stay posix compliant.
shell-script binary
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
add a comment |
I'm trying to read serial input(/dev/tty*) character by character. I need this to parse the data following the corresponding protocol. My current method is as follows:
while read -rs -n 1 c; do
echo Read a char:
echo ${c}
done < "${SERIAL_PORT}"
This works, except for the fact that the data is binary and some bytes are x00, which is not allowed as a character in a variable. My aim is to first convert the byte into a hex string(two characters) which I will then store in the variable. This is where I get stuck. What is a good way to do this? I would prefer to stay posix compliant.
shell-script binary
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
2
oddoes this. Note that you're already not POSIX compliant since you're using other options than just-rtoread.
– Kusalananda♦
Mar 7 at 8:54
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggestpyserialvia Python script. Or at a very leastscreenapplication. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that
– Sergiy Kolodyazhnyy
Mar 7 at 9:12
How fast is the serial data coming? You could do an infinite loop ofc=$(dd bs=1 count=1 < ... | od ... | awk ...)but it might take so long that some data would be missed..
– Mark Plotnick
Mar 7 at 21:42
add a comment |
I'm trying to read serial input(/dev/tty*) character by character. I need this to parse the data following the corresponding protocol. My current method is as follows:
while read -rs -n 1 c; do
echo Read a char:
echo ${c}
done < "${SERIAL_PORT}"
This works, except for the fact that the data is binary and some bytes are x00, which is not allowed as a character in a variable. My aim is to first convert the byte into a hex string(two characters) which I will then store in the variable. This is where I get stuck. What is a good way to do this? I would prefer to stay posix compliant.
shell-script binary
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
I'm trying to read serial input(/dev/tty*) character by character. I need this to parse the data following the corresponding protocol. My current method is as follows:
while read -rs -n 1 c; do
echo Read a char:
echo ${c}
done < "${SERIAL_PORT}"
This works, except for the fact that the data is binary and some bytes are x00, which is not allowed as a character in a variable. My aim is to first convert the byte into a hex string(two characters) which I will then store in the variable. This is where I get stuck. What is a good way to do this? I would prefer to stay posix compliant.
shell-script binary
shell-script binary
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
asked Mar 7 at 8:50
Aart StuurmanAart Stuurman
1011
1011
2
oddoes this. Note that you're already not POSIX compliant since you're using other options than just-rtoread.
– Kusalananda♦
Mar 7 at 8:54
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggestpyserialvia Python script. Or at a very leastscreenapplication. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that
– Sergiy Kolodyazhnyy
Mar 7 at 9:12
How fast is the serial data coming? You could do an infinite loop ofc=$(dd bs=1 count=1 < ... | od ... | awk ...)but it might take so long that some data would be missed..
– Mark Plotnick
Mar 7 at 21:42
add a comment |
2
oddoes this. Note that you're already not POSIX compliant since you're using other options than just-rtoread.
– Kusalananda♦
Mar 7 at 8:54
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggestpyserialvia Python script. Or at a very leastscreenapplication. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that
– Sergiy Kolodyazhnyy
Mar 7 at 9:12
How fast is the serial data coming? You could do an infinite loop ofc=$(dd bs=1 count=1 < ... | od ... | awk ...)but it might take so long that some data would be missed..
– Mark Plotnick
Mar 7 at 21:42
2
2
od does this. Note that you're already not POSIX compliant since you're using other options than just -r to read.– Kusalananda♦
Mar 7 at 8:54
od does this. Note that you're already not POSIX compliant since you're using other options than just -r to read.– Kusalananda♦
Mar 7 at 8:54
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggest
pyserial via Python script. Or at a very least screen application. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that– Sergiy Kolodyazhnyy
Mar 7 at 9:12
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggest
pyserial via Python script. Or at a very least screen application. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that– Sergiy Kolodyazhnyy
Mar 7 at 9:12
How fast is the serial data coming? You could do an infinite loop of
c=$(dd bs=1 count=1 < ... | od ... | awk ...) but it might take so long that some data would be missed..– Mark Plotnick
Mar 7 at 21:42
How fast is the serial data coming? You could do an infinite loop of
c=$(dd bs=1 count=1 < ... | od ... | awk ...) but it might take so long that some data would be missed..– Mark Plotnick
Mar 7 at 21:42
add a comment |
1 Answer
1
active
oldest
votes
Using od to create hexadecimal ASCII representation for each byte/character of input, and outputting these one at a time:
od -v -An -txC <input |
awk '{ for (i = 1; i <= NF; ++i) print $i }'
This will output one hexadecimal number per line.
You may then read this with read if you wish, but it would be better if you could do your processing inside of the already existing awk code.
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
add a comment |
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504865%2fshell-read-binary-file-character-by-character-as-hex%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Using od to create hexadecimal ASCII representation for each byte/character of input, and outputting these one at a time:
od -v -An -txC <input |
awk '{ for (i = 1; i <= NF; ++i) print $i }'
This will output one hexadecimal number per line.
You may then read this with read if you wish, but it would be better if you could do your processing inside of the already existing awk code.
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
add a comment |
Using od to create hexadecimal ASCII representation for each byte/character of input, and outputting these one at a time:
od -v -An -txC <input |
awk '{ for (i = 1; i <= NF; ++i) print $i }'
This will output one hexadecimal number per line.
You may then read this with read if you wish, but it would be better if you could do your processing inside of the already existing awk code.
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
add a comment |
Using od to create hexadecimal ASCII representation for each byte/character of input, and outputting these one at a time:
od -v -An -txC <input |
awk '{ for (i = 1; i <= NF; ++i) print $i }'
This will output one hexadecimal number per line.
You may then read this with read if you wish, but it would be better if you could do your processing inside of the already existing awk code.
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
Using od to create hexadecimal ASCII representation for each byte/character of input, and outputting these one at a time:
od -v -An -txC <input |
awk '{ for (i = 1; i <= NF; ++i) print $i }'
This will output one hexadecimal number per line.
You may then read this with read if you wish, but it would be better if you could do your processing inside of the already existing awk code.
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
edited Mar 7 at 9:42
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
answered Mar 7 at 9:02
Kusalananda♦Kusalananda
139k17259429
139k17259429
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
add a comment |
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
I am not sure if this is the issue, but I think od buffers until 16 bytes? Because if I remove the awk it displays as 16 bytes batches
– Aart Stuurman
Mar 7 at 9:22
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
For this system I need the characters as soon as they're available
– Aart Stuurman
Mar 7 at 9:23
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
@AartStuurman Then you need to use a language that can read the raw bytes from the input stream.
– Kusalananda♦
Mar 7 at 9:45
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f504865%2fshell-read-binary-file-character-by-character-as-hex%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
oddoes this. Note that you're already not POSIX compliant since you're using other options than just-rtoread.– Kusalananda♦
Mar 7 at 8:54
od can convert it's input to hex, but how to get in in a variable afterwards?
– Aart Stuurman
Mar 7 at 8:56
While it's definitely possible, probably you might want to read serial input via proper libraries instead of via shell script. I'd suggest
pyserialvia Python script. Or at a very leastscreenapplication. Those tools are designed exactly for the job. Shell scripts - not so suitable for low level stuff like that– Sergiy Kolodyazhnyy
Mar 7 at 9:12
How fast is the serial data coming? You could do an infinite loop of
c=$(dd bs=1 count=1 < ... | od ... | awk ...)but it might take so long that some data would be missed..– Mark Plotnick
Mar 7 at 21:42