bash fork: retry: Resource temporarily unavailable

I am trying to run a shell script which will create process using a shell script. I get Resource temporarily unavailable error. how to identify which limit (memory/process/filecount) is creating this problem. Below is my ulimit -a results.

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 563959

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) unlimited

cpu time               (seconds, -t) unlimited

max user processes              (-u) 10000000

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

You will have to give us more information. Perhaps post your bash script. i've never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.

– Doug Smythies
Nov 4 '16 at 22:01

It's a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done <$1 With few more iterations of run I understood that its not the process count which is limiting the execution but it is memory. When I run another simple script I am able to reach to count of 1900 processes but after which I again see this issue. Which memory exactly would play role here, it is stack memory in ulimits or any specific memory for bash process itself is limiting the execution

– Viswanath
Nov 8 '16 at 3:38

3

Possible duplicate of How can I monitor the memory usage?

– Elder Geek
Nov 9 '16 at 3:51

watch cat /proc/meminfo

– Elder Geek
Nov 9 '16 at 3:58

meminfo snapshot

– Viswanath
Nov 9 '16 at 7:36

|
show 10 more comments

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 563959

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) unlimited

cpu time               (seconds, -t) unlimited

max user processes              (-u) 10000000

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

You will have to give us more information. Perhaps post your bash script. i've never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.

– Doug Smythies
Nov 4 '16 at 22:01

It's a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done <$1 With few more iterations of run I understood that its not the process count which is limiting the execution but it is memory. When I run another simple script I am able to reach to count of 1900 processes but after which I again see this issue. Which memory exactly would play role here, it is stack memory in ulimits or any specific memory for bash process itself is limiting the execution

– Viswanath
Nov 8 '16 at 3:38

3

Possible duplicate of How can I monitor the memory usage?

– Elder Geek
Nov 9 '16 at 3:51

watch cat /proc/meminfo

– Elder Geek
Nov 9 '16 at 3:58

meminfo snapshot

– Viswanath
Nov 9 '16 at 7:36

|
show 10 more comments

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 563959

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) unlimited

cpu time               (seconds, -t) unlimited

max user processes              (-u) 10000000

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 563959

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) unlimited

cpu time               (seconds, -t) unlimited

max user processes              (-u) 10000000

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

bash memory-usage ulimit

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

edited Jan 9 at 23:55

Pablo Bianchi

2,4451530

asked Nov 4 '16 at 7:49

Viswanath

2114

asked Nov 4 '16 at 7:49

Viswanath

2114

asked Nov 4 '16 at 7:49

Viswanath

2114

You will have to give us more information. Perhaps post your bash script. i've never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.

– Doug Smythies
Nov 4 '16 at 22:01

It's a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done <$1 With few more iterations of run I understood that its not the process count which is limiting the execution but it is memory. When I run another simple script I am able to reach to count of 1900 processes but after which I again see this issue. Which memory exactly would play role here, it is stack memory in ulimits or any specific memory for bash process itself is limiting the execution

– Viswanath
Nov 8 '16 at 3:38

3

Possible duplicate of How can I monitor the memory usage?

– Elder Geek
Nov 9 '16 at 3:51

watch cat /proc/meminfo

– Elder Geek
Nov 9 '16 at 3:58

meminfo snapshot

– Viswanath
Nov 9 '16 at 7:36

|
show 10 more comments

You will have to give us more information. Perhaps post your bash script. i've never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.

– Doug Smythies
Nov 4 '16 at 22:01

It's a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done <$1 With few more iterations of run I understood that its not the process count which is limiting the execution but it is memory. When I run another simple script I am able to reach to count of 1900 processes but after which I again see this issue. Which memory exactly would play role here, it is stack memory in ulimits or any specific memory for bash process itself is limiting the execution

– Viswanath
Nov 8 '16 at 3:38

3

Possible duplicate of How can I monitor the memory usage?

– Elder Geek
Nov 9 '16 at 3:51

watch cat /proc/meminfo

– Elder Geek
Nov 9 '16 at 3:58

meminfo snapshot

– Viswanath
Nov 9 '16 at 7:36

You will have to give us more information. Perhaps post your bash script. i've never had trouble with this. Example: Before: ps aux | wc -l gave 183 and after running a script to create 4000 processes it gave 4183.

– Doug Smythies
Nov 4 '16 at 22:01

It's a simple script which will invoke background process for each invocation. #!/bin/sh while read zi; do (source example.sh &) done <$1 With few more iterations of run I understood that its not the process count which is limiting the execution but it is memory. When I run another simple script I am able to reach to count of 1900 processes but after which I again see this issue. Which memory exactly would play role here, it is stack memory in ulimits or any specific memory for bash process itself is limiting the execution

– Viswanath
Nov 8 '16 at 3:38

Possible duplicate of How can I monitor the memory usage?

– Elder Geek
Nov 9 '16 at 3:51

watch cat /proc/meminfo

– Elder Geek
Nov 9 '16 at 3:58

meminfo snapshot

– Viswanath
Nov 9 '16 at 7:36

|
show 10 more comments

1 Answer
1

active

oldest

votes

For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

12288

$ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

And if I use my "what is the thread limit" program (found here) to check, before:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

...

12100 threads so far ...

12200 threads so far ...

Failed with return code 11 creating thread 12281 (Resource temporarily unavailable).

Malloc worked, hmmm

and after:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

300 threads so far ...

...

14700 threads so far ...

14800 threads so far ...

14900 threads so far ...

Failed with return code 11 creating thread 14993 (Resource temporarily unavailable).

Malloc worked, hmmm

Of course, the numbers above are not exact because the "doug" user has a few other threads running, such as my SSH sessions to my sever. Check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

8

Program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */

/* originally from: http://www.volano.com/linuxnotes.html */



#include <stdlib.h>

#include <stdio.h>

#include <unistd.h>

#include <pthread.h>

#include <string.h>



#define MAX_THREADS 100000

#define PTHREAD_STACK_MIN 1*1024*1024*1024

int i;



void run(void) {

  sleep(60 * 60);

}



int main(int argc, char *argv) {

  int rc = 0;

  pthread_t thread[MAX_THREADS];

  pthread_attr_t thread_attr;



  pthread_attr_init(&thread_attr);

  pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN);



  printf("Creating threads ...n");

  for (i = 0; i < MAX_THREADS && rc == 0; i++) {

    rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL);

    if (rc == 0) {

      pthread_detach(thread[i]);

      if ((i + 1) % 100 == 0)

    printf("%i threads so far ...n", i + 1);

    }

    else

    {

      printf("Failed with return code %i creating thread %i (%s).n",

         rc, i + 1, strerror(rc));



      // can we allocate memory?

      char *block = NULL;

      block = malloc(65545);

      if(block == NULL)

        printf("Malloc failed too :( n");

      else

        printf("Malloc worked, hmmmn");

    }

  }

sleep(60*60); // ctrl+c to exit; makes it easier to see mem use

  exit(0);

}

See also here

Now, if you have enough memory, the next limit will be defined by the default maximum PID number, which is 32768, but is also writable. Obvioulsy in order to have more than 32768 simultaneous processes or tasks or threads their PID will have to be allowed to be higher:

$ cat /proc/sys/kernel/pid_max

32768

$ echo 80000 | sudo tee /proc/sys/kernel/pid_max

80000

$ cat /proc/sys/kernel/pid_max

80000

Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

And at this point, realize that the above listed program seems to have a limit of about 32768 threads, even if resources are still available, and so use another method. My test server with 16 gigabytes of memory seems to exhaust some other resource at about 62344 tasks, even though there does seem to still be memory available.

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

62344

top:

top - 13:48:26 up 21:08,  4 users,  load average: 281.52, 134.90, 70.93

Tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie

%Cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache

KiB Swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

37884 doug      20   0  108052  68920   3104 R   5.7  0.4   1:23.08 top

24075 doug      20   0    4360    652    576 S   0.4  0.0   0:00.31 consume

26006 doug      20   0    4360    796    720 S   0.4  0.0   0:00.09 consume

30062 doug      20   0    4360    732    656 S   0.4  0.0   0:00.17 consume

21009 doug      20   0    4360    748    672 S   0.3  0.0   0:00.26 consume

Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):

$ ulimit -i

62340

doug@s15:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62340

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32768

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62340

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

If I raise those limits, I am able to go to 69000 threads, which is all I asked for and as far as I'm going to take this answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

69011

top:

top - 16:39:43 up 33 min,  3 users,  load average: 314.59, 181.48, 105.27

Tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie

%Cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache

KiB Swap: 16472060 total, 16472060 free,        0 used.  2866316 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 4166 doug      20   0  115408  75956   3252 R   5.1  0.5   1:30.52 top

62667 doug      20   0   28916   1516   1320 R   3.7  0.0   0:01.14 ps

73184 doug      20   0    7196   4396   1600 S   0.9  0.0   0:09.95 try_stuff5

 2038 doug      20   0    4360    652    576 S   0.4  0.0   0:00.34 consume

 4594 doug      20   0    4360    652    580 S   0.4  0.0   0:00.14 consume

 5435 doug      20   0    4360    652    576 S   0.4  0.0   0:00.24 consume

 8891 doug      20   0    4360    688    612 S   0.4  0.0   0:00.14 consume

At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. I tried 118000 threads and the system totally bogged down and I had many of these errors:

Feb 17 16:13:02 s15 kernel: [  967.907305] INFO: task waiter:119371 blocked for more than 120 seconds.

Feb 17 16:13:02 s15 kernel: [  967.907335]       Not tainted 4.10.0-rc8-stock #194

Feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

And my load average ballooned to ~29000. But I just left the computer for an hour and it sorted itself out. I staggered the spin out of the threads by 200 microseconds per spin out, and then was able to run 118000 threads just fine (admittedly at very little resource use per thread).

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

1

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

add a comment |

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f845380%2fbash-fork-retry-resource-temporarily-unavailable%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

12288

$ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

And if I use my "what is the thread limit" program (found here) to check, before:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

...

12100 threads so far ...

12200 threads so far ...

Failed with return code 11 creating thread 12281 (Resource temporarily unavailable).

Malloc worked, hmmm

and after:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

300 threads so far ...

...

14700 threads so far ...

14800 threads so far ...

14900 threads so far ...

Failed with return code 11 creating thread 14993 (Resource temporarily unavailable).

Malloc worked, hmmm

Of course, the numbers above are not exact because the "doug" user has a few other threads running, such as my SSH sessions to my sever. Check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

8

Program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */

/* originally from: http://www.volano.com/linuxnotes.html */



#include <stdlib.h>

#include <stdio.h>

#include <unistd.h>

#include <pthread.h>

#include <string.h>



#define MAX_THREADS 100000

#define PTHREAD_STACK_MIN 1*1024*1024*1024

int i;



void run(void) {

  sleep(60 * 60);

}



int main(int argc, char *argv) {

  int rc = 0;

  pthread_t thread[MAX_THREADS];

  pthread_attr_t thread_attr;



  pthread_attr_init(&thread_attr);

  pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN);



  printf("Creating threads ...n");

  for (i = 0; i < MAX_THREADS && rc == 0; i++) {

    rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL);

    if (rc == 0) {

      pthread_detach(thread[i]);

      if ((i + 1) % 100 == 0)

    printf("%i threads so far ...n", i + 1);

    }

    else

    {

      printf("Failed with return code %i creating thread %i (%s).n",

         rc, i + 1, strerror(rc));



      // can we allocate memory?

      char *block = NULL;

      block = malloc(65545);

      if(block == NULL)

        printf("Malloc failed too :( n");

      else

        printf("Malloc worked, hmmmn");

    }

  }

sleep(60*60); // ctrl+c to exit; makes it easier to see mem use

  exit(0);

}

See also here

$ cat /proc/sys/kernel/pid_max

32768

$ echo 80000 | sudo tee /proc/sys/kernel/pid_max

80000

$ cat /proc/sys/kernel/pid_max

80000

Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

62344

top:

top - 13:48:26 up 21:08,  4 users,  load average: 281.52, 134.90, 70.93

Tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie

%Cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache

KiB Swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

37884 doug      20   0  108052  68920   3104 R   5.7  0.4   1:23.08 top

24075 doug      20   0    4360    652    576 S   0.4  0.0   0:00.31 consume

26006 doug      20   0    4360    796    720 S   0.4  0.0   0:00.09 consume

30062 doug      20   0    4360    732    656 S   0.4  0.0   0:00.17 consume

21009 doug      20   0    4360    748    672 S   0.3  0.0   0:00.26 consume

Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):

$ ulimit -i

62340

doug@s15:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62340

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32768

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62340

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

If I raise those limits, I am able to go to 69000 threads, which is all I asked for and as far as I'm going to take this answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

69011

top:

top - 16:39:43 up 33 min,  3 users,  load average: 314.59, 181.48, 105.27

Tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie

%Cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache

KiB Swap: 16472060 total, 16472060 free,        0 used.  2866316 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 4166 doug      20   0  115408  75956   3252 R   5.1  0.5   1:30.52 top

62667 doug      20   0   28916   1516   1320 R   3.7  0.0   0:01.14 ps

73184 doug      20   0    7196   4396   1600 S   0.9  0.0   0:09.95 try_stuff5

 2038 doug      20   0    4360    652    576 S   0.4  0.0   0:00.34 consume

 4594 doug      20   0    4360    652    580 S   0.4  0.0   0:00.14 consume

 5435 doug      20   0    4360    652    576 S   0.4  0.0   0:00.24 consume

 8891 doug      20   0    4360    688    612 S   0.4  0.0   0:00.14 consume

At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. I tried 118000 threads and the system totally bogged down and I had many of these errors:

Feb 17 16:13:02 s15 kernel: [  967.907305] INFO: task waiter:119371 blocked for more than 120 seconds.

Feb 17 16:13:02 s15 kernel: [  967.907335]       Not tainted 4.10.0-rc8-stock #194

Feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

1

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

add a comment |

For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

12288

$ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

And if I use my "what is the thread limit" program (found here) to check, before:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

...

12100 threads so far ...

12200 threads so far ...

Failed with return code 11 creating thread 12281 (Resource temporarily unavailable).

Malloc worked, hmmm

and after:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

300 threads so far ...

...

14700 threads so far ...

14800 threads so far ...

14900 threads so far ...

Failed with return code 11 creating thread 14993 (Resource temporarily unavailable).

Malloc worked, hmmm

Of course, the numbers above are not exact because the "doug" user has a few other threads running, such as my SSH sessions to my sever. Check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

8

Program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */

/* originally from: http://www.volano.com/linuxnotes.html */



#include <stdlib.h>

#include <stdio.h>

#include <unistd.h>

#include <pthread.h>

#include <string.h>



#define MAX_THREADS 100000

#define PTHREAD_STACK_MIN 1*1024*1024*1024

int i;



void run(void) {

  sleep(60 * 60);

}



int main(int argc, char *argv) {

  int rc = 0;

  pthread_t thread[MAX_THREADS];

  pthread_attr_t thread_attr;



  pthread_attr_init(&thread_attr);

  pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN);



  printf("Creating threads ...n");

  for (i = 0; i < MAX_THREADS && rc == 0; i++) {

    rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL);

    if (rc == 0) {

      pthread_detach(thread[i]);

      if ((i + 1) % 100 == 0)

    printf("%i threads so far ...n", i + 1);

    }

    else

    {

      printf("Failed with return code %i creating thread %i (%s).n",

         rc, i + 1, strerror(rc));



      // can we allocate memory?

      char *block = NULL;

      block = malloc(65545);

      if(block == NULL)

        printf("Malloc failed too :( n");

      else

        printf("Malloc worked, hmmmn");

    }

  }

sleep(60*60); // ctrl+c to exit; makes it easier to see mem use

  exit(0);

}

See also here

$ cat /proc/sys/kernel/pid_max

32768

$ echo 80000 | sudo tee /proc/sys/kernel/pid_max

80000

$ cat /proc/sys/kernel/pid_max

80000

Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

62344

top:

top - 13:48:26 up 21:08,  4 users,  load average: 281.52, 134.90, 70.93

Tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie

%Cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache

KiB Swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

37884 doug      20   0  108052  68920   3104 R   5.7  0.4   1:23.08 top

24075 doug      20   0    4360    652    576 S   0.4  0.0   0:00.31 consume

26006 doug      20   0    4360    796    720 S   0.4  0.0   0:00.09 consume

30062 doug      20   0    4360    732    656 S   0.4  0.0   0:00.17 consume

21009 doug      20   0    4360    748    672 S   0.3  0.0   0:00.26 consume

Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):

$ ulimit -i

62340

doug@s15:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62340

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32768

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62340

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

If I raise those limits, I am able to go to 69000 threads, which is all I asked for and as far as I'm going to take this answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

69011

top:

top - 16:39:43 up 33 min,  3 users,  load average: 314.59, 181.48, 105.27

Tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie

%Cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache

KiB Swap: 16472060 total, 16472060 free,        0 used.  2866316 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 4166 doug      20   0  115408  75956   3252 R   5.1  0.5   1:30.52 top

62667 doug      20   0   28916   1516   1320 R   3.7  0.0   0:01.14 ps

73184 doug      20   0    7196   4396   1600 S   0.9  0.0   0:09.95 try_stuff5

 2038 doug      20   0    4360    652    576 S   0.4  0.0   0:00.34 consume

 4594 doug      20   0    4360    652    580 S   0.4  0.0   0:00.14 consume

 5435 doug      20   0    4360    652    576 S   0.4  0.0   0:00.24 consume

 8891 doug      20   0    4360    688    612 S   0.4  0.0   0:00.14 consume

At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. I tried 118000 threads and the system totally bogged down and I had many of these errors:

Feb 17 16:13:02 s15 kernel: [  967.907305] INFO: task waiter:119371 blocked for more than 120 seconds.

Feb 17 16:13:02 s15 kernel: [  967.907335]       Not tainted 4.10.0-rc8-stock #194

Feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

1

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

add a comment |

For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

12288

$ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

And if I use my "what is the thread limit" program (found here) to check, before:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

...

12100 threads so far ...

12200 threads so far ...

Failed with return code 11 creating thread 12281 (Resource temporarily unavailable).

Malloc worked, hmmm

and after:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

300 threads so far ...

...

14700 threads so far ...

14800 threads so far ...

14900 threads so far ...

Failed with return code 11 creating thread 14993 (Resource temporarily unavailable).

Malloc worked, hmmm

Of course, the numbers above are not exact because the "doug" user has a few other threads running, such as my SSH sessions to my sever. Check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

8

Program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */

/* originally from: http://www.volano.com/linuxnotes.html */



#include <stdlib.h>

#include <stdio.h>

#include <unistd.h>

#include <pthread.h>

#include <string.h>



#define MAX_THREADS 100000

#define PTHREAD_STACK_MIN 1*1024*1024*1024

int i;



void run(void) {

  sleep(60 * 60);

}



int main(int argc, char *argv) {

  int rc = 0;

  pthread_t thread[MAX_THREADS];

  pthread_attr_t thread_attr;



  pthread_attr_init(&thread_attr);

  pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN);



  printf("Creating threads ...n");

  for (i = 0; i < MAX_THREADS && rc == 0; i++) {

    rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL);

    if (rc == 0) {

      pthread_detach(thread[i]);

      if ((i + 1) % 100 == 0)

    printf("%i threads so far ...n", i + 1);

    }

    else

    {

      printf("Failed with return code %i creating thread %i (%s).n",

         rc, i + 1, strerror(rc));



      // can we allocate memory?

      char *block = NULL;

      block = malloc(65545);

      if(block == NULL)

        printf("Malloc failed too :( n");

      else

        printf("Malloc worked, hmmmn");

    }

  }

sleep(60*60); // ctrl+c to exit; makes it easier to see mem use

  exit(0);

}

See also here

$ cat /proc/sys/kernel/pid_max

32768

$ echo 80000 | sudo tee /proc/sys/kernel/pid_max

80000

$ cat /proc/sys/kernel/pid_max

80000

Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

62344

top:

top - 13:48:26 up 21:08,  4 users,  load average: 281.52, 134.90, 70.93

Tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie

%Cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache

KiB Swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

37884 doug      20   0  108052  68920   3104 R   5.7  0.4   1:23.08 top

24075 doug      20   0    4360    652    576 S   0.4  0.0   0:00.31 consume

26006 doug      20   0    4360    796    720 S   0.4  0.0   0:00.09 consume

30062 doug      20   0    4360    732    656 S   0.4  0.0   0:00.17 consume

21009 doug      20   0    4360    748    672 S   0.3  0.0   0:00.26 consume

Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):

$ ulimit -i

62340

doug@s15:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62340

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32768

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62340

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

If I raise those limits, I am able to go to 69000 threads, which is all I asked for and as far as I'm going to take this answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

69011

top:

top - 16:39:43 up 33 min,  3 users,  load average: 314.59, 181.48, 105.27

Tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie

%Cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache

KiB Swap: 16472060 total, 16472060 free,        0 used.  2866316 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 4166 doug      20   0  115408  75956   3252 R   5.1  0.5   1:30.52 top

62667 doug      20   0   28916   1516   1320 R   3.7  0.0   0:01.14 ps

73184 doug      20   0    7196   4396   1600 S   0.9  0.0   0:09.95 try_stuff5

 2038 doug      20   0    4360    652    576 S   0.4  0.0   0:00.34 consume

 4594 doug      20   0    4360    652    580 S   0.4  0.0   0:00.14 consume

 5435 doug      20   0    4360    652    576 S   0.4  0.0   0:00.24 consume

 8891 doug      20   0    4360    688    612 S   0.4  0.0   0:00.14 consume

At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. I tried 118000 threads and the system totally bogged down and I had many of these errors:

Feb 17 16:13:02 s15 kernel: [  967.907305] INFO: task waiter:119371 blocked for more than 120 seconds.

Feb 17 16:13:02 s15 kernel: [  967.907335]       Not tainted 4.10.0-rc8-stock #194

Feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

For the case in the comments, where you were not using much memory per thread, you were hitting the cgroup limits. You will find the default to be around 12288, but the value is writable:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

12288

$ echo 15000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

15000

And if I use my "what is the thread limit" program (found here) to check, before:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

...

12100 threads so far ...

12200 threads so far ...

Failed with return code 11 creating thread 12281 (Resource temporarily unavailable).

Malloc worked, hmmm

and after:

$ ./thread-limit

Creating threads ...

100 threads so far ...

200 threads so far ...

300 threads so far ...

...

14700 threads so far ...

14800 threads so far ...

14900 threads so far ...

Failed with return code 11 creating thread 14993 (Resource temporarily unavailable).

Malloc worked, hmmm

Of course, the numbers above are not exact because the "doug" user has a few other threads running, such as my SSH sessions to my sever. Check with:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

8

Program used:

/* compile with:   gcc -pthread -o thread-limit thread-limit.c */

/* originally from: http://www.volano.com/linuxnotes.html */



#include <stdlib.h>

#include <stdio.h>

#include <unistd.h>

#include <pthread.h>

#include <string.h>



#define MAX_THREADS 100000

#define PTHREAD_STACK_MIN 1*1024*1024*1024

int i;



void run(void) {

  sleep(60 * 60);

}



int main(int argc, char *argv) {

  int rc = 0;

  pthread_t thread[MAX_THREADS];

  pthread_attr_t thread_attr;



  pthread_attr_init(&thread_attr);

  pthread_attr_setstacksize(&thread_attr, PTHREAD_STACK_MIN);



  printf("Creating threads ...n");

  for (i = 0; i < MAX_THREADS && rc == 0; i++) {

    rc = pthread_create(&(thread[i]), &thread_attr, (void *) &run, NULL);

    if (rc == 0) {

      pthread_detach(thread[i]);

      if ((i + 1) % 100 == 0)

    printf("%i threads so far ...n", i + 1);

    }

    else

    {

      printf("Failed with return code %i creating thread %i (%s).n",

         rc, i + 1, strerror(rc));



      // can we allocate memory?

      char *block = NULL;

      block = malloc(65545);

      if(block == NULL)

        printf("Malloc failed too :( n");

      else

        printf("Malloc worked, hmmmn");

    }

  }

sleep(60*60); // ctrl+c to exit; makes it easier to see mem use

  exit(0);

}

See also here

$ cat /proc/sys/kernel/pid_max

32768

$ echo 80000 | sudo tee /proc/sys/kernel/pid_max

80000

$ cat /proc/sys/kernel/pid_max

80000

Note that is quite on purpose that a number bigger than 2**16 was chosen, to see if it was actually allowed. And so now, set the cgroup max to, say 70000:

$ echo 70000 | sudo tee /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max

70000

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

62344

top:

top - 13:48:26 up 21:08,  4 users,  load average: 281.52, 134.90, 70.93

Tasks: 62535 total, 201 running, 62334 sleeping,   0 stopped,   0 zombie

%Cpu0  : 96.6 us,  2.4 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 95.7 us,  2.4 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu2  : 95.1 us,  3.1 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu3  : 93.5 us,  4.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu4  : 94.8 us,  3.4 sy,  0.0 ni,  1.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu5  : 95.5 us,  2.6 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu6  : 94.7 us,  3.5 sy,  0.0 ni,  1.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu7  : 93.8 us,  4.5 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999116 total,   758684 free, 10344908 used,  4895524 buff/cache

KiB Swap: 16472060 total, 16470396 free,     1664 used.  4031160 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

37884 doug      20   0  108052  68920   3104 R   5.7  0.4   1:23.08 top

24075 doug      20   0    4360    652    576 S   0.4  0.0   0:00.31 consume

26006 doug      20   0    4360    796    720 S   0.4  0.0   0:00.09 consume

30062 doug      20   0    4360    732    656 S   0.4  0.0   0:00.17 consume

21009 doug      20   0    4360    748    672 S   0.3  0.0   0:00.26 consume

Seems I finally hit my default ulimit settings for both user processes and number of timers (signals):

$ ulimit -i

62340

doug@s15:~$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 62340

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 32768

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 62340

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

If I raise those limits, I am able to go to 69000 threads, which is all I asked for and as far as I'm going to take this answer:

$ cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current

69011

top:

top - 16:39:43 up 33 min,  3 users,  load average: 314.59, 181.48, 105.27

Tasks: 69205 total, 234 running, 68971 sleeping,   0 stopped,   0 zombie

%Cpu(s): 93.7 us,  5.5 sy,  0.0 ni,  0.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem : 15999004 total,  2659452 free, 11393940 used,  1945612 buff/cache

KiB Swap: 16472060 total, 16472060 free,        0 used.  2866316 avail Mem



  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

 4166 doug      20   0  115408  75956   3252 R   5.1  0.5   1:30.52 top

62667 doug      20   0   28916   1516   1320 R   3.7  0.0   0:01.14 ps

73184 doug      20   0    7196   4396   1600 S   0.9  0.0   0:09.95 try_stuff5

 2038 doug      20   0    4360    652    576 S   0.4  0.0   0:00.34 consume

 4594 doug      20   0    4360    652    580 S   0.4  0.0   0:00.14 consume

 5435 doug      20   0    4360    652    576 S   0.4  0.0   0:00.24 consume

 8891 doug      20   0    4360    688    612 S   0.4  0.0   0:00.14 consume

At some point you will get into trouble, but it is absolutely amazing how gracefully the system bogs down. I tried 118000 threads and the system totally bogged down and I had many of these errors:

Feb 17 16:13:02 s15 kernel: [  967.907305] INFO: task waiter:119371 blocked for more than 120 seconds.

Feb 17 16:13:02 s15 kernel: [  967.907335]       Not tainted 4.10.0-rc8-stock #194

Feb 17 16:13:02 s15 kernel: [  967.907357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

edited Jun 27 '17 at 16:26

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

answered Feb 15 '17 at 16:49

Doug Smythies

7,17631428

1

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

add a comment |

1

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

On Ubuntu with systemd, the resource limits can be configured in /etc/systemd/system.conf . freedesktop.org/software/systemd/man/… explains the relevant configuration options. Setting "DefaultTasksAccounting=no" in /etc/systemd/system.conf removes the default of setting limits for pids.max.

– Lari Hotari
Nov 9 '17 at 13:14

Seems that adding "UserTasksMax=infinity" to /etc/systemd/logind.conf would remove the limit.

– Lari Hotari
Nov 9 '17 at 13:29

@LariHotari : Thanks for your information. Indeed your method works fine. I still have to raise the pid_max value and the ulimits. I might edit my answer to include your method.

– Doug Smythies
Nov 9 '17 at 17:03

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Ask Ubuntu!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Ytdyklly