Ytdyklly

I built TensorFlow 1.12 from source using Bazel 15.0, Python 3.6.5, and GCC 7.3.0. It seems to work properly, but that when I import pandas before importing tensorflow it uses /lib64/libstdc++.so.6, rather than the one associated with GCC 7.3.0:

$ python3 -c "import tensorflow, pandas" # no error

$ python3 -c "import pandas, tensorflow"

Traceback (most recent call last):

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>

    from tensorflow.python.pywrap_tensorflow_internal import *

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>

    _pywrap_tensorflow_internal = swig_import_helper()

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper

    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)

  File "/apps/python/3.6.5/gcc-7.3.0/lib/python3.6/imp.py", line 243, in load_module

    return load_dynamic(name, filename, file)

  File "/apps/python/3.6.5/gcc-7.3.0/lib/python3.6/imp.py", line 343, in load_dynamic

    return _load(spec)

ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module>

    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module>

    from tensorflow.python import pywrap_tensorflow

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>

    raise ImportError(msg)

ImportError: Traceback (most recent call last):

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>

    from tensorflow.python.pywrap_tensorflow_internal import *

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>

    _pywrap_tensorflow_internal = swig_import_helper()

  File "/zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper

    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)

  File "/apps/python/3.6.5/gcc-7.3.0/lib/python3.6/imp.py", line 243, in load_module

    return load_dynamic(name, filename, file)

  File "/apps/python/3.6.5/gcc-7.3.0/lib/python3.6/imp.py", line 343, in load_dynamic

    return _load(spec)

ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /zapps7/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)





Failed to load the native TensorFlow runtime.



See https://www.tensorflow.org/install/errors



for some common reasons and solutions.  Include the entire stack trace

above this error message when asking for help.

I made sure when building to use -Wl,--enable-new-dtags,-rpath,/apps/gcc/7.3.0/lib64, and have confirmed that the RUNPATH is actually set:

$ readelf -d /apps/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so | grep RUNPATH

 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../_solib_k8/_U_S_Stensorflow_Spython_C_Upywrap_Utensorflow_Uinternal.so___Utensorflow:$ORIGIN/../../_solib_k8/_U@mkl_Ulinux_S_S_Cmkl_Ulibs_Ulinux___Uexternal_Smkl_Ulinux_Slib:$ORIGIN/:$ORIGIN/..:/apps/gcc/7.3.0/lib64]

$ ldd /apps/python-3.6.5-tensorflow/1.12.0/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so | grep libstdc++

    libstdc++.so.6 => /apps/gcc/7.3.0/lib64/libstdc++.so.6 (0x00007f82c9711000)

It seems odd that despite the fact that ldd finds the right libstdc++.so.6 (so probably not a duplicate of this issue), the wrong one is used under this circumstance. LD_LIBRARY_PATH and LD_PRELOAD aren't set, and the ld.so man page shows that RUNPATH should thus have precedence. It could be that I'm misunderstanding what should be happening--it's my impression that, absent LD_LIBRARY_PATH and LD_PRELOAD, an ELF's RUNPATH is the supreme authority on where libraries should be searched, and that whatever ldd sees should be exactly what is used at runtime. Am I misunderstanding? Is there a real way (i.e. NOT just sloppily using LD_LIBRARY_PATH) to fix this?

There's a little more background at this GitHub issue, although I don't know how much of it will be helpful.