Ytdyklly

I am running the following command:

nohup ./run_hyper_param_tuning.sh> ../../logs/hyper_model_15_b_Jan.txt 2>&1  & echo $! >save_pid.txt

It has been always working but suddenly it stopped working now. The script is the following:

#!/bin/bash



export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}



#added by Anaconda3 installer

export PATH="/home/javier/anaconda3/envs/AIenv/bin:$PATH"

#export PATH="/home/javier/anaconda3/bin:$PATH"



#source /home/javier/anaconda3/bin/activate AIenv



echo "STARTING HYPER-PARAMETER TUNING..."

backbones=("resnet101")

gradient_clip_norms=(10)

learning_rates=(0.001)

weight_decays=(0.0005)

layers=('heads')



for i in "${backbones[@]}"

do

  for j in "${gradient_clip_norms[@]}"

  do

    for k in "${learning_rates[@]}"

    do

      for l in "${weight_decays[@]}"

      do

        for m in "${layers[@]}"

        do

         echo "-----------------------------------------------------------------------------------"

         echo  "backbone:" $i " gradient_clip_norms:" $j " learning_rate:" $k " weight_decays:" $l "layers:" $m

         echo "-----------------------------------------------------------------------------------"

            python3 custom.py train --dataset=customImages/ --weights=coco --backbone=$i --grad_clip_norms=$j --learn_rate=$k --weight_decays=$l --layers=$m --epochs=100

         echo "successfully done"

        done

      done

    done

  done

done

The file where I redirect the output has the following message:

[1]+  Exit 125                nohup ./run_hyper_param_tuning.sh > ../../logs/hyper_model_15_b_Jan.txt 2>&1

However when I show again the content with cat, the file is empty.

How can I solve this?

This, according to the (GNU) nohup documentation, is nohup itself failing.

-bash-4.2$ man nohup | grep 125

-bash-4.2$ info nohup | grep 125

info: Writing node (coreutils.info.gz)nohup invocation...

info: Done.

     125 if 'nohup' itself fails, and 'POSIXLY_CORRECT' is not set

instead of 125.

Perhaps try a strace of nohup to see what it is failing over?