Issue with keras installation in AWS R server instance

kashif · June 11, 2019, 6:17am

I have installed keras on my r server instance but whenever I try to run it I am facing this issue -

 Error: Python module keras was not found.

Detected Python configuration:

python:         /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
libpython:      /usr/lib64/python2.7/config/libpython2.7.so
pythonhome:     /usr:/usr
virtualenv:     /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/activate_this.py
version:        2.7.15 (default, Nov 28 2018, 22:38:08)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
numpy:          /home/mohammad.kashif/.virtualenvs/r-tensorflow/local/lib64/python2.7/site-packages/numpy
numpy_version:  1.14.5
keras:          /home/mohammad.kashif/.virtualenvs/r-tensorflow/local/lib64/python2.7/site-packages/keras

python versions found: 
 /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
 /usr/bin/python
 /usr/bin/python3

I have been trying to debug it but to no avail. Have gone through similar discourses in the community but that hasn't helped. Any advice on how to make it work would be very helpful.

I am working on a AWS AMI

Thanks

andrie · June 11, 2019, 6:21am

The easiest way to set up keras in an R session is to use:

keras::install_keras()

This will install keras, tensorflow and all other python dependencies you need, in a virtualenv where the R keras package knows where to find it.

kashif · June 11, 2019, 6:24am

Dear Andrie,
I have used keras::install_keras() command and the installation I a success and as you have said it is installed in a virtual environment, but when I am trying to run a code it is throwing up the aforementioned error

andrie · June 11, 2019, 6:26am

What is the value of

Sys.getenv("RETICULATE_PYTHON")

?

kashif · June 11, 2019, 6:27am

This is giving me an empty string, the value is -
""

andrie · June 11, 2019, 6:40am

I have just noticed that you're using Red Hat. This complicates things very substantially. If you have the option, I suggest you use a machine running Ubuntu instead.

In fact, TensorFlow is not officially supported on Red Hat (https://www.tensorflow.org/install)

I have heard that it is possible, but you will probably have to install TensorFlow from source.

kashif · June 11, 2019, 8:52am

Thanks a lot, my organization is using this so I will have to use it. I will try to install tensorflow from source and see if it works for me. Would mark it complete if it works

zkajdan · June 12, 2019, 6:29am

Can you show what happens if in the r-tensorflow virtualenv, you call python and execute:

>>> import tensorflow as tf
>>> tf.__version__
>>> sess = tf.Session()
>>> hello = tf.constant("hello")
>>> sess.run(hello)

kashif · June 12, 2019, 12:04pm

Hi zkajdan,

Please find the screen shot of the commands
rserver_python

zkajdan · June 12, 2019, 1:10pm

Hi,
thanks.
this says the TF version is 1.10.0, which can't be after a successful run of tensorflow::install_tensorflow() - this should install version 1.13.1 as of today...

Perhaps your R tensorflow version is older?
Can you try installing either the current tensorflow from CRAN, or the current version from github, and then try running

tensorflow::install_tensorflow()

again and check the output?

kashif · June 13, 2019, 4:56am

Dear zkajdan,

Please find screen shot of updated package, it is still not working for me.

The error that is throwing up is
Note: This error is after I have alreay run install_tensorflow() and install_keras() command

zkajdan · June 14, 2019, 7:18am

Can you try install_tensorflow() again and post the complete output?
(You don't need install_keras() after that, it will already install keras.)

Then, can you start a new R session and do, in this order

reticulate::py_discover_config()
reticulate::py_config()
tensorflow::tf_config()

and post the output?

After that, if there is no error so far, can you do
library(keras) and
m <- keras_model_sequential()

and if that throws an error like in your screenshot above, click "show traceback"?

kashif · June 14, 2019, 9:19am

Output of install_tensorflow()

> library(tensorflow)
> install_tensorflow()
Creating virtualenv for TensorFlow at  ~/.virtualenvs/r-tensorflow 
New python executable in /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
Installing setuptools, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python
Installing TensorFlow ...
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting tensorflow==1.13.1
  Using cached https://files.pythonhosted.org/packages/d2/ea/ab2c8c0e81bd051cc1180b104c75a865ab0fc66c89be992c4b20bbf6d624/tensorflow-1.13.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting h5py
  Using cached https://files.pythonhosted.org/packages/53/08/27e4e9a369321862ffdce80ff1770553e9daec65d98befb2e14e7478b698/h5py-2.9.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml
Collecting requests
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting Pillow
  Using cached https://files.pythonhosted.org/packages/b6/4b/5adc1109908266554fb978154c797c7d71aba43dd15508d8c1565648f6bc/Pillow-6.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting keras
  Using cached https://files.pythonhosted.org/packages/5e/10/aa32dad071ce52b5502266b5c659451cfd6ffcbf14e6c8c4f16c0ff5aaab/Keras-2.2.4-py2.py3-none-any.whl
Collecting tensorflow-hub
  Using cached https://files.pythonhosted.org/packages/10/5c/6f3698513cf1cd730a5ea66aec665d213adf9de59b34f362f270e0bd126f/tensorflow_hub-0.4.0-py2.py3-none-any.whl
Collecting tensorflow-probability
  Using cached https://files.pythonhosted.org/packages/4e/93/191e42ca27786d6875f74e7b756e34ef42f385f6d250bfc28aa48a1d1072/tensorflow_probability-0.6.0-py2.py3-none-any.whl
Collecting scipy
  Using cached https://files.pythonhosted.org/packages/1d/f6/7c16d60aeb3694e5611976cb4f1eaf1c6b7f1e7c55771d691013405a02ea/scipy-1.2.2-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.13.3 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/1f/c7/198496417c9c2f6226616cff7dedf2115a4f4d0276613bab842ec8ac1e23/numpy-1.16.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting mock>=2.0.0 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/05/d2/f94e68be6b17f46d2c353564da56e6fb89ef09faeeff3313a046cb810ca9/mock-3.0.5-py2.py3-none-any.whl
Collecting tensorflow-estimator<1.14.0rc0,>=1.13.0 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/bb/48/13f49fc3fa0fdf916aa1419013bb8f2ad09674c275b4046d5ee669a46873/tensorflow_estimator-1.13.0-py2.py3-none-any.whl
Collecting astor>=0.6.0 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/d1/4f/950dfae467b384fc96bc6469de25d832534f6b4441033c39f914efd13418/astor-0.8.0-py2.py3-none-any.whl
Collecting backports.weakref>=1.0rc1 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/88/ec/f598b633c3d5ffe267aaada57d961c94fdfa183c5c3ebda2b6d151943db6/backports.weakref-1.0.post1-py2.py3-none-any.whl
Collecting wheel (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/bb/10/44230dd6bf3563b8f227dbf344c908d412ad2ff48066476672f3a72e174e/wheel-0.33.4-py2.py3-none-any.whl
Collecting gast>=0.2.0 (from tensorflow==1.13.1)
Collecting termcolor>=1.1.0 (from tensorflow==1.13.1)
Collecting protobuf>=3.6.1 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/b2/a8/ad407cd2a56a052d92f602e164a9e16bede22079252af0db3838f375b6a8/protobuf-3.8.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting six>=1.10.0 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting keras-applications>=1.0.6 (from tensorflow==1.13.1)
Collecting grpcio>=1.8.6 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/8e/62/448199c50e05526f7fd336d8a87bd83a0c7d92188ae51ebea23aa5f3c37e/grpcio-1.21.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting absl-py>=0.1.6 (from tensorflow==1.13.1)
Collecting tensorboard<1.14.0,>=1.13.0 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/89/ac/48dd71c2bdc8d31e367f9b72f25ccb3b89bc6b9d664fee21f9a8efa5714d/tensorboard-1.13.1-py2-none-any.whl
Collecting keras-preprocessing>=1.0.5 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl
Collecting enum34>=1.1.6 (from tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from mock>=2.0.0->tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl
Collecting setuptools (from protobuf>=3.6.1->tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/ec/51/f45cea425fd5cb0b0380f5b0f048ebc1da5b417e48d304838c02d6288a1e/setuptools-41.0.1-py2.py3-none-any.whl
Collecting futures>=2.2.0; python_version < "3.2" (from grpcio>=1.8.6->tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting werkzeug>=0.11.15 (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/9f/57/92a497e38161ce40606c27a86759c6b92dd34fcdb33f64171ec559257c02/Werkzeug-0.15.4-py2.py3-none-any.whl
Collecting markdown>=2.6.8 (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1)
  Using cached https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl
ERROR: awscli 1.15.83 has requirement PyYAML<=3.13,>=3.10, but you'll have pyyaml 5.1.1 which is incompatible.
Installing collected packages: numpy, funcsigs, six, mock, enum34, absl-py, tensorflow-estimator, astor, backports.weakref, wheel, gast, termcolor, setuptools, protobuf, h5py, keras-applications, futures, grpcio, werkzeug, markdown, tensorboard, keras-preprocessing, tensorflow, pyyaml, chardet, idna, urllib3, certifi, requests, Pillow, scipy, keras, tensorflow-hub, tensorflow-probability
Successfully installed Pillow-6.0.0 absl-py-0.7.1 astor-0.8.0 backports.weakref-1.0.post1 certifi-2019.3.9 chardet-3.0.4 enum34-1.1.6 funcsigs-1.0.2 futures-3.2.0 gast-0.2.2 grpcio-1.21.1 h5py-2.9.0 idna-2.8 keras-2.2.4 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 mock-3.0.5 numpy-1.16.4 protobuf-3.8.0 pyyaml-5.1.1 requests-2.22.0 scipy-1.2.2 setuptools-41.0.1 six-1.12.0 tensorboard-1.13.1 tensorflow-1.13.1 tensorflow-estimator-1.13.0 tensorflow-hub-0.4.0 tensorflow-probability-0.6.0 termcolor-1.1.0 urllib3-1.25.3 werkzeug-0.15.4 wheel-0.33.4

Installation complete.


Restarting R session...

Following other commands as stated by you

> reticulate::py_discover_config()
python:         /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
libpython:      /usr/lib64/python2.7/config/libpython2.7.so
pythonhome:     /usr:/usr
virtualenv:     /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/activate_this.py
version:        2.7.15 (default, Nov 28 2018, 22:38:08)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
numpy:          /home/mohammad.kashif/.virtualenvs/r-tensorflow/local/lib64/python2.7/site-packages/numpy
numpy_version:  1.16.4

python versions found: 
 /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
 /usr/bin/python
 /usr/bin/python3
> reticulate::py_config()
python:         /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
libpython:      /usr/lib64/python2.7/config/libpython2.7.so
pythonhome:     /usr:/usr
virtualenv:     /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/activate_this.py
version:        2.7.15 (default, Nov 28 2018, 22:38:08)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
numpy:          /home/mohammad.kashif/.virtualenvs/r-tensorflow/local/lib64/python2.7/site-packages/numpy
numpy_version:  1.16.4
tensorflow:     /home/mohammad.kashif/.virtualenvs/r-tensorflow/local/lib/python2.7/site-packages/tensorflow

python versions found: 
 /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
 /usr/bin/python
 /usr/bin/python3
> tensorflow::tf_config()
Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
 /usr/bin/python2.7
 /usr/bin/python3.5

You can install TensorFlow using the install_tensorflow() function.

Although tensorflow is installed it is looking in a separate folder (I think so). Running the pending commands, we have -

> library(keras)
> m <- keras_model_sequential()
Error: Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /home/mohammad.kashif/.virtualenvs/r-tensorflow/bin/python
 /usr/bin/python2.7
 /usr/bin/python3.5

You can install TensorFlow using the install_tensorflow() function.

zkajdan · June 14, 2019, 4:46pm

Hm, I'd say the output looks pretty good - until it doesn't ...

Can you try this...

installing reticulate from github (and please print its version to be sure it's getting used)
restart R
reticulate::use_virtualenv("r-tensorflow")
reticulate::py_module_available("tensorflow")

thanks!

system · July 5, 2019, 4:46pm

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.