How can I monitor the length of the accept queue?
I have a hypothesis: sometimes TCP connections arrive faster than my server can accept() them. They queue up until the queue overflows and then there are problems.
How can I confirm this is happening?
Can I monitor the length of the accept queue or the number of overflows? Is there a counter exposed somewhere?
linux networking monitoring
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
|
show 1 more comment
I have a hypothesis: sometimes TCP connections arrive faster than my server can accept() them. They queue up until the queue overflows and then there are problems.
How can I confirm this is happening?
Can I monitor the length of the accept queue or the number of overflows? Is there a counter exposed somewhere?
linux networking monitoring
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
You're looking fornetstat.
– Satō Katsura
Dec 7 '16 at 18:26
As far as I can tell,netstatonly shows the send and receive queue lengths, which is not the same as the accept queue.
– Phil Frost
Dec 7 '16 at 19:08
Yeah, it isn't shown by default.man netstat | less +/Flags
– Satō Katsura
Dec 7 '16 at 19:33
I'm not sure how those flags tell me the accept queue length -- in factnetstatdoesn't seem to showFlagsat all for TCP connections. From a little testing, it looks like the connections are shown asESTABLISHEDinnetstat, even if I try opening connections to a process that doeslisten()but neveraccept().
– Phil Frost
Dec 7 '16 at 20:33
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just countSYN_RECVthough. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting foraccept()to do its job, you're waiting forACKs to arrive from the connecting hosts to complete the connections.
– Satō Katsura
Dec 7 '16 at 20:47
|
show 1 more comment
I have a hypothesis: sometimes TCP connections arrive faster than my server can accept() them. They queue up until the queue overflows and then there are problems.
How can I confirm this is happening?
Can I monitor the length of the accept queue or the number of overflows? Is there a counter exposed somewhere?
linux networking monitoring
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
I have a hypothesis: sometimes TCP connections arrive faster than my server can accept() them. They queue up until the queue overflows and then there are problems.
How can I confirm this is happening?
Can I monitor the length of the accept queue or the number of overflows? Is there a counter exposed somewhere?
linux networking monitoring
linux networking monitoring
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
asked Dec 7 '16 at 18:17
Phil FrostPhil Frost
1995
1995
You're looking fornetstat.
– Satō Katsura
Dec 7 '16 at 18:26
As far as I can tell,netstatonly shows the send and receive queue lengths, which is not the same as the accept queue.
– Phil Frost
Dec 7 '16 at 19:08
Yeah, it isn't shown by default.man netstat | less +/Flags
– Satō Katsura
Dec 7 '16 at 19:33
I'm not sure how those flags tell me the accept queue length -- in factnetstatdoesn't seem to showFlagsat all for TCP connections. From a little testing, it looks like the connections are shown asESTABLISHEDinnetstat, even if I try opening connections to a process that doeslisten()but neveraccept().
– Phil Frost
Dec 7 '16 at 20:33
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just countSYN_RECVthough. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting foraccept()to do its job, you're waiting forACKs to arrive from the connecting hosts to complete the connections.
– Satō Katsura
Dec 7 '16 at 20:47
|
show 1 more comment
You're looking fornetstat.
– Satō Katsura
Dec 7 '16 at 18:26
As far as I can tell,netstatonly shows the send and receive queue lengths, which is not the same as the accept queue.
– Phil Frost
Dec 7 '16 at 19:08
Yeah, it isn't shown by default.man netstat | less +/Flags
– Satō Katsura
Dec 7 '16 at 19:33
I'm not sure how those flags tell me the accept queue length -- in factnetstatdoesn't seem to showFlagsat all for TCP connections. From a little testing, it looks like the connections are shown asESTABLISHEDinnetstat, even if I try opening connections to a process that doeslisten()but neveraccept().
– Phil Frost
Dec 7 '16 at 20:33
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just countSYN_RECVthough. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting foraccept()to do its job, you're waiting forACKs to arrive from the connecting hosts to complete the connections.
– Satō Katsura
Dec 7 '16 at 20:47
You're looking for
netstat.– Satō Katsura
Dec 7 '16 at 18:26
You're looking for
netstat.– Satō Katsura
Dec 7 '16 at 18:26
As far as I can tell,
netstat only shows the send and receive queue lengths, which is not the same as the accept queue.– Phil Frost
Dec 7 '16 at 19:08
As far as I can tell,
netstat only shows the send and receive queue lengths, which is not the same as the accept queue.– Phil Frost
Dec 7 '16 at 19:08
Yeah, it isn't shown by default.
man netstat | less +/Flags– Satō Katsura
Dec 7 '16 at 19:33
Yeah, it isn't shown by default.
man netstat | less +/Flags– Satō Katsura
Dec 7 '16 at 19:33
I'm not sure how those flags tell me the accept queue length -- in fact
netstat doesn't seem to show Flags at all for TCP connections. From a little testing, it looks like the connections are shown as ESTABLISHED in netstat, even if I try opening connections to a process that does listen() but never accept().– Phil Frost
Dec 7 '16 at 20:33
I'm not sure how those flags tell me the accept queue length -- in fact
netstat doesn't seem to show Flags at all for TCP connections. From a little testing, it looks like the connections are shown as ESTABLISHED in netstat, even if I try opening connections to a process that does listen() but never accept().– Phil Frost
Dec 7 '16 at 20:33
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just count
SYN_RECV though. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting for accept() to do its job, you're waiting for ACKs to arrive from the connecting hosts to complete the connections.– Satō Katsura
Dec 7 '16 at 20:47
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just count
SYN_RECV though. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting for accept() to do its job, you're waiting for ACKs to arrive from the connecting hosts to complete the connections.– Satō Katsura
Dec 7 '16 at 20:47
|
show 1 more comment
1 Answer
1
active
oldest
votes
Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.
7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10
As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f328746%2fhow-can-i-monitor-the-length-of-the-accept-queue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.
7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10
As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
add a comment |
Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.
7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10
As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
add a comment |
Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.
7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10
As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
Sysdig will provide some of this information at the end of each accept syscall, as the queuelen argument. It also shows the length of the queue as queuemax.
7598971 21:05:30.322229280 1 gunicorn (6451) < accept fd=13(<4t>127.0.0.1:45882->127.0.0.1:8003) tuple=127.0.0.1:45882->127.0.0.1:8003 queuepct=0 queuelen=0 queuemax=10
As far as I'm aware, it provides no mechanism to know exactly when or how many times the queue has overflowed. And it would be cumbersome to integrate this with periodic monitoring by collectd or similar.
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
answered Dec 8 '16 at 13:15
Phil FrostPhil Frost
1995
1995
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f328746%2fhow-can-i-monitor-the-length-of-the-accept-queue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You're looking for
netstat.– Satō Katsura
Dec 7 '16 at 18:26
As far as I can tell,
netstatonly shows the send and receive queue lengths, which is not the same as the accept queue.– Phil Frost
Dec 7 '16 at 19:08
Yeah, it isn't shown by default.
man netstat | less +/Flags– Satō Katsura
Dec 7 '16 at 19:33
I'm not sure how those flags tell me the accept queue length -- in fact
netstatdoesn't seem to showFlagsat all for TCP connections. From a little testing, it looks like the connections are shown asESTABLISHEDinnetstat, even if I try opening connections to a process that doeslisten()but neveraccept().– Phil Frost
Dec 7 '16 at 20:33
Right, looking at the sources it seems those flags are for UNIX sockets. For TCP you could just count
SYN_RECVthough. There is no other queue beyond that. I suppose the kernel can be told somehow to log dropped packets because of too many half-open connections, but there have been some 10+ years since I looked at networking with Linux, so I have no idea how to do that. On a side note: you aren't waiting foraccept()to do its job, you're waiting forACKs to arrive from the connecting hosts to complete the connections.– Satō Katsura
Dec 7 '16 at 20:47