Struggling to install shiny server on Raspberry pi 4

cpsyctc · March 26, 2021, 4:22pm

I have a new Raspberry Pi 4 running Debian GNU/Linux 10 (buster) which I want to use as a shiny server. I have successfully installed and compiled R 4.0.4 and installed the shiny package and many other packages. However, I am completely failing to install the shiny server. I have seen 2018 posts here about doing this and one 2020 one about doing this using a docker container. I don't want the docker option and the other posts don't seem to give me the clues I need. I assumed I could follow Download Shiny Server - RStudio and the instructions here: Building Shiny Server from Source · rstudio/shiny-server Wiki · GitHub

The catch is that when I get to:

make
mkdir ../build
(cd .. && ./bin/npm install)
(cd .. && ./bin/node ./ext/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js rebuild)

I get an exec format error and I think the problem is that the version of node that is being used (# Install our private copy of Node.js) is not usable on the Raspberry ARM architecture.

I feel sure there is a way around this and that someone else has found it but I'm failing to see the answer.

TIA,

Chris

andresrcs · March 26, 2021, 5:47pm

If you are interested I have a blog post using a different approach

Also, if you share the error message you get, maybe I could help you with what you are currently doing.

cpsyctc · March 26, 2021, 6:15pm

Thanks Andrés,

That link started off giving me this error

Error 526

Ray ID: 63624dc2c80d40d2 • 2021-03-26 18:04:51 UTC

Invalid SSL certificate

but now it's working. Thought I should say.

I am a bit reluctant to go to ansible, which I've never used and don't know at all when it seems to me that it really is something that should be fairly easy to do using a tweak to get the ARM node. I also see that your ansible playbooks are aimed at a 32 bit Raspberry OS and I'm using the 64 bit. I'll hang on for now and see if anyone else can come up with the solution to fix that "exec format" error from the node bit of the compliation.

Very best,

Chris

andresrcs · March 26, 2021, 6:21pm

I suppose you are cloning the GitHub repository at its latest commit so you are installing the development version of shiny-server which uses Node.js 12.20, you need to modify the installation script to use the ARM version and to download from the official site instead of Joe Cheng's GitHub repository, these commands should do the trick.

sed -i '8s/.*/NODE_SHA256=abef7d431d6d0e067fe5797d4fe44039a5577f01ed9e40d7a3496cbb22502f55/' ../external/node/install-node.sh # node-v12.20.0-linux-armv7l.tar.xz
sed -i 's/linux-x64.tar.xz/linux-armv7l.tar.xz/' ../external/node/install-node.sh
sed -i 's/https:\/\/github.com\/jcheng5\/node-centos6\/releases\/download\//https:\/\/nodejs.org\/dist\//' ../external/node/install-node.sh

cpsyctc · March 26, 2021, 7:00pm

Thanks. I am sorry: I am working rather beyond the edge of my competences here. I am sure your interpretation of the problem is correct and exactly what I was looking for. Sadly, after running those lines I don't seem to have solved the problem and I wonder if you and I are talking about the same installation instructions. I was working from Building Shiny Server from Source · rstudio/shiny-server Wiki · GitHub and I don't see anything about Joe Cheng's repository there. I think you are referring to your instructions at Setting up your own shiny-server / rstudio-server on a Raspberry Pi 3B+

So I copied your "install shiny-server" command block there and tweaked it to this (in a file tmp which I made executable!)

root@raspberrypi:/home/chris# cat tmp
git clone https://github.com/rstudio/shiny-server.git
cd shiny-server
DIR=`pwd`
PATH=$DIR/bin:$PATH
mkdir tmp
cd tmp
PYTHON=`which python`
sudo cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DPYTHON="$PYTHON" ../
sudo make
mkdir ../build
### commented out 26.iii.21
# sed -i '8s/.*/NODE_SHA256=a865e69914c568fcb28be7a1bf970236725a06a8fc66530799300181d2584a49/' ../external/node/install-node.sh # node-v12.15.0-linux-armv7l.tar.xz
# sed -i 's/linux-x64.tar.xz/linux-armv7l.tar.xz/' ../external/node/install-node.sh
# sed -i 's/https:\/\/github.com\/jcheng5\/node-centos6\/releases\/download\//https:\/\/nodejs.org\/dist\//' ../external/node/install-node.sh
### replaced with these lines from https://forum.posit.co/t/struggling-to-install-shiny-server-on-raspberry-pi-4/100152/4
sed -i '8s/.*/NODE_SHA256=abef7d431d6d0e067fe5797d4fe44039a5577f01ed9e40d7a3496cbb22502f55/' ../external/node/install-node.sh # node-v12.20.0-linux-armv7l.tar.xz
sed -i 's/linux-x64.tar.xz/linux-armv7l.tar.xz/' ../external/node/install-node.sh
sed -i 's/https:\/\/github.com\/jcheng5\/node-centos6\/releases\/download\//https:\/\/nodejs.org\/dist\//' ../external/node/install-node.sh
### end of tweaks
(cd .. && sudo ./external/node/install-node.sh)
(cd .. && ./bin/npm --python="${PYTHON}" install --no-optional)
(cd .. && ./bin/npm --python="${PYTHON}" rebuild)
sudo make install

That runs up to this:

CMake Error at cmake_install.cmake:41 (file):
  file INSTALL cannot find "/home/chris/shiny-server/node_modules": No such
  file or directory.


make: *** [Makefile:108: install] Error 1

I feel I'm nearly there ... but not quite. I hugely appreciate your taking the time to help! Can you see the next (last?!) step I need to fix things? TIA, Chris

andresrcs · March 26, 2021, 7:18pm

You have to rebuild the node modules

(cd .. && ./bin/node ./ext/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js --python="${PYTHON}" rebuild)

Muy post here is very outdated, better read my blog post for reference.

Although, I can't guarantee it's going to work since I haven't tested these instructions with the latest development version of shiny server.

cpsyctc · March 26, 2021, 7:48pm

OK. I don't know where that would belong in the script I put up but straight at the bash prompt that is getting this:

./bin/node: 4: exec: ./bin/../ext/node/bin/node: not found

I don't think I understand your line about your blog. Sure the 2018 one is 2018 and explicitly for the Pi 3B+ and for 32 bit (I think). Are you saying there is a more recent one? Or are you saying I should only see that earlier version as a historical reference: yes, I get that. That was really why I went to the Rstudio official document about compiling the open source server from source.

Anyway: thanks and if that error message is transparent to you, I'd love to hear more. Throughout this I've had the sense the directory mappings I have in that script are part of the problem. I just feel increasingly out of my depth trying to debug this! Chris

andresrcs · March 26, 2021, 8:10pm

I update my blogpost from time to time so chances are that instructions there are more updated than the ones you can find on the post I posted on this forum.
I suspect you are executing the command from the wrong path, It is hard to tell what is the state your particular server is currently in so I would say that you should try to start over from the beginning.

cpsyctc · March 31, 2021, 3:20pm

Thanks. OK. I cleaned out everything and went back to your blog instructions. The cmake update ran OK and these lines ran fine:

git clone https://github.com/rstudio/shiny-server.git
cd shiny-server
DIR=`pwd`
PATH=$DIR/bin:$PATH
mkdir tmp
cd tmp
PYTHON=`which python`
sudo cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DPYTHON="$PYTHON" ../
sudo make
mkdir ../build
sed -i '8s/.*/NODE_SHA256=abef7d431d6d0e067fe5797d4fe44039a5577f01ed9e40d7a3496cbb22502f55/' ../external/node/install-node.sh # node-v12.20.0-linux-armv7l.tar.xz
sed -i 's/linux-x64.tar.xz/linux-armv7l.tar.xz/' ../external/node/install-node.sh
sed -i 's/https:\/\/github.com\/jcheng5\/node-centos6\/releases\/download\//https:\/\/nodejs.org\/dist\//' ../external/node/install-node.sh

However, the next line fails:

chris@raspberrypi:~/shiny-server/tmp$ (cd .. && ./bin/npm --python="${PYTHON}" install --no-optional)
./bin/npm: 4: exec: ./bin/../ext/node/bin/node: not found

I think that tells me that the mapping of the command to the directory structure is broken but I don't want to go further and try to fix anything myself as I'm sure it's safer and more efficient if you can look at this and tell me if you agree and, if so, what would fix it.

TIA,

Chris

andresrcs · April 1, 2021, 1:30am

It seems to me Node.js has not been successfully installed, you haven't mentioned if this command runs without problems (it comes before the line you are showing). It doesn't give you any error messages?

(cd .. && sudo ./external/node/install-node.sh)

Also, I think that since you are using a 64 bit OS you should be using the arm64 installer instead of the armv7l I use but I can't test this right now.

cpsyctc · April 1, 2021, 6:19am

Thanks. Sorry about clipping that last line, yes, it ran fine. What you say about the node version makes perfect sense (of course). I have used nvm to get node 15.12.0 installed. Would it be possible for me to use a tweaked version of your install script using that?

Very much appreciate your continuing help. Chris

andresrcs · April 2, 2021, 6:09pm

Shiny uses specific node versions, currently 12.20.0

You can try installing the arm64 version by modifying these two commands.

sed -i '8s/.*/NODE_SHA256=976285886f734ac4e13be8b34586136499b088aa32c6430ca8eee166d167dca5/' ../external/node/install-node.sh # node-v12.20.0-linux-arm64.tar.xz
sed -i 's/linux-x64.tar.xz/linux-arm64.tar.xz/' ../external/node/install-node.sh

PD. You can't apply this change over the already modified file that got downloaded the first time you cloned the GitHub repository so you should clone it again before applying these changes.

cpsyctc · April 5, 2021, 3:35am

Thanks again. Got back to this.

One thing I should have said earlier: to get reliable downloading of node things I had to add

2606:4700::6810:1323				registry.npmjs.org

To /etc/hosts as the machine is sitting in an ipv6 network at my ISP and they advised me that this fixes some intermittent failures pulling things from npmjs.org site.

OK. To the meat of the matter: I think you have helped me a bit closer but I am still hitting a problem. Here's the detail.

I started by deleting the old installation directories (after fresh login to the machine but I suspect that's irrelevant). Ran apt-get update & apt-get upgrade: some packages upgraded. Launched R and ran update.packages(): nothing to update. I already have cmake at 3.19.1 so skipped remaking that. So today's sequence was:

git clone https://github.com/rstudio/shiny-server.git
cd shiny-server
DIR=`pwd`
PATH=$DIR/bin:$PATH
mkdir tmp
cd tmp
PYTHON=`which python`
sudo cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DPYTHON="$PYTHON" ../
sudo make
mkdir ../build

Which produced a cmake warning:

chris@raspberrypi:~/shiny-server/tmp$ sudo cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DPYTHON="$PYTHON" ../
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

But I am guessing that's not fatal, just a warning about something that will need changing in the script at some point soon as cmake continues to update. It also ends with a warning which may be more important:

-- gunzip /home/chris/shiny-server/tmp/external/pandoc/pandoc-citeproc.gz
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON


-- Build files have been written to: /home/chris/shiny-server/tmp

However, again, to my inexpert eye, that is warning, not an error. My python at which python , i.e. /usr/bin/python, is 2.7.16.

Then I insert your new sed lines, replacing the ones in the blog so:

sed -i '8s/.*/NODE_SHA256=976285886f734ac4e13be8b34586136499b088aa32c6430ca8eee166d167dca5/' ../external/node/install-node.sh # node-v12.20.0-linux-arm64.tar.xz
sed -i 's/linux-x64.tar.xz/linux-arm64.tar.xz/' ../external/node/install-node.sh

No messages: good!
Then I continued with:

sed -i 's/https:\/\/github.com\/jcheng5\/node-centos6\/releases\/download\//https:\/\/nodejs.org\/dist\//' ../external/node/install-node.sh
(cd .. && sudo ./external/node/install-node.sh)

Seems fine:

Downloading Node v12.20.0 from https://nodejs.org/dist/v12.20.0/node-v12.20.0-linux-arm64.tar.xz
--2021-04-03 14:25:12--  https://nodejs.org/dist/v12.20.0/node-v12.20.0-linux-arm64.tar.xz
Resolving nodejs.org (nodejs.org)... 2606:4700:10::6814:172e, 2606:4700:10::6814:162e, 104.20.22.46, ...
Connecting to nodejs.org (nodejs.org)|2606:4700:10::6814:172e|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14038888 (13M) [application/x-xz]
Saving to: ‘/tmp/node-v12.20.0-linux-arm64.tar.xz’

/tmp/node-v12.20.0-linux-arm64.tar.xz                                  100%[==========================================================================================================================================================================>]  13.39M  27.9MB/s    in 0.5s    

2021-04-03 14:25:13 (27.9 MB/s) - ‘/tmp/node-v12.20.0-linux-arm64.tar.xz’ saved [14038888/14038888]

Extracting node-v12.20.0-linux-arm64.tar.xz

Sadly, that fails:

Downloading Node v12.20.0 from https://nodejs.org/dist/v12.20.0/node-v12.20.0-linux-arm64.tar.xz
--2021-04-03 14:25:12--  https://nodejs.org/dist/v12.20.0/node-v12.20.0-linux-arm64.tar.xz
Resolving nodejs.org (nodejs.org)... 2606:4700:10::6814:172e, 2606:4700:10::6814:162e, 104.20.22.46, ...
Connecting to nodejs.org (nodejs.org)|2606:4700:10::6814:172e|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14038888 (13M) [application/x-xz]
Saving to: ‘/tmp/node-v12.20.0-linux-arm64.tar.xz’

/tmp/node-v12.20.0-linux-arm64.tar.xz                                  100%[==========================================================================================================================================================================>]  13.39M  27.9MB/s    in 0.5s    

2021-04-03 14:25:13 (27.9 MB/s) - ‘/tmp/node-v12.20.0-linux-arm64.tar.xz’ saved [14038888/14038888]

Extracting node-v12.20.0-linux-arm64.tar.xz
chris@raspberrypi:~/shiny-server/tmp$ (cd .. && ./bin/npm --python="${PYTHON}" install --no-optional)

> shiny-server@1.5.17 install /home/chris/shiny-server
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: connect ECONNREFUSED 104.20.22.46:443
gyp ERR! stack     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
gyp ERR! System Linux 5.10.17-v8+
gyp ERR! command "/home/chris/shiny-server/ext/node/bin/node" "/home/chris/shiny-server/ext/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/chris/shiny-server
gyp ERR! node -v v12.20.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! shiny-server@1.5.17 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the shiny-server@1.5.17 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/chris/.npm/_logs/2021-04-03T14_26_32_871Z-debug.log

That's replicable (three tries, starting over from having deleted the directories each time). If I go to that log file it doesn't throw much more light on things for me but it may for you. Here are the last lines:

62 verbose unlock done using /home/chris/.npm/_locks/staging-43571a4f0bb3a8f4.lock for /home/chris/shiny-server/node_modules/.staging
63 timing stage:executeActions Completed in 43ms
64 timing stage:rollbackFailedOptional Completed in 0ms
65 silly install runPostinstallTopLevelLifecycles
66 silly build shiny-server@1.5.17
67 info linkStuff shiny-server@1.5.17
68 silly linkStuff shiny-server@1.5.17 has /home/chris as its parent node_modules
69 silly install shiny-server@1.5.17
70 info lifecycle shiny-server@1.5.17~install: shiny-server@1.5.17
71 verbose lifecycle shiny-server@1.5.17~install: unsafe-perm in lifecycle true
72 verbose lifecycle shiny-server@1.5.17~install: PATH: /home/chris/shiny-server/ext/node/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/chris/shiny-server/node_modules/.bin:/home/chris/shiny-server/ext/node/bin:/home/chris/shiny-server/bin:/home/chris/shiny-server/bin:/home/chris/shiny-server/shiny-server/bin:/home/chris/shiny-server/bin:/root/.nvm/versions/node/v15.12.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
73 verbose lifecycle shiny-server@1.5.17~install: CWD: /home/chris/shiny-server
74 silly lifecycle shiny-server@1.5.17~install: Args: [ '-c', 'node-gyp rebuild' ]
75 timing audit submit Completed in 332ms
76 http fetch POST 200 http://registry.npmjs.org/-/npm/v1/security/audits/quick 333ms
77 timing audit body Completed in 2ms
78 silly lifecycle shiny-server@1.5.17~install: Returned: code: 1  signal: null
79 info lifecycle shiny-server@1.5.17~install: Failed to exec install script
80 verbose stack Error: shiny-server@1.5.17 install: `node-gyp rebuild`
80 verbose stack Exit status 1
80 verbose stack     at EventEmitter.<anonymous> (/home/chris/shiny-server/ext/node/lib/node_modules/npm/node_modules/npm-lifecycle/index.js:332:16)
80 verbose stack     at EventEmitter.emit (events.js:314:20)
80 verbose stack     at ChildProcess.<anonymous> (/home/chris/shiny-server/ext/node/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
80 verbose stack     at ChildProcess.emit (events.js:314:20)
80 verbose stack     at maybeClose (internal/child_process.js:1022:16)
80 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)
81 verbose pkgid shiny-server@1.5.17
82 verbose cwd /home/chris/shiny-server
83 verbose Linux 5.10.17-v8+
84 verbose argv "/home/chris/shiny-server/ext/node/bin/node" "/home/chris/shiny-server/ext/node/lib/node_modules/npm/bin/npm-cli.js" "--python=/usr/bin/python" "install" "--no-optional"
85 verbose node v12.20.0
86 verbose npm  v6.14.8
87 error code ELIFECYCLE
88 error errno 1
89 error shiny-server@1.5.17 install: `node-gyp rebuild`
89 error Exit status 1
90 error Failed at the shiny-server@1.5.17 install script.
90 error This is probably not a problem with npm. There is likely additional logging output above.
91 verbose exit [ 1, true ]

Sorry I'm not able to help more but perhaps this information is more helpful to you than to me! I am very keen to get the server running on the machine and on that OS if it's humanly possible so I am very happy to run any further suggestions you have. Many thanks again, Chris

andresrcs · April 5, 2021, 2:06pm

This is starting to go beyond my understanding but I think this line is referring to a connectivity issue, sadly I can't offer any practical advice since it would be very hard to help you with system specific issues.

cpsyctc · April 5, 2021, 5:46pm

Totally understood. You have been extremely helpful as I blundered my way forward and I agree with you: on the face of it, this appears to be a connectivity issue but I am puzzled by that as it's entirely replicable and as other things seem to download fine and as I know the ISP have most connectivity issues sorted. A few thoughts, a mix of thinking aloud for me, some you may, if you still have time, be able to help with and some, as this is a public forum and not at all your personal responsibility, for anyone else who have wisdom and experience to share:

I am a bit surprised that there seem to be very few of us actively and publicly working on deploying the open source shiny server on Raspberry Pis. It seems so obviously to play to the strengths of both shiny and the Pi to be doing this that I'd have expected more to be doing this. Anyone else trying this or willing to?
I get the sense that Rstudio.com aren't very interested in the open source shiny server. That may be unfair but reflects my experience over several attempts to get to use it, actually spread over several years. Anyone up to correct me about that and point to evidence of active involvement in updating and maintaining it and perhaps increasing the number of platforms on which it might run?
I really don't understand node and it is clearly vital to this and I am not finding it easy to find information about it. Any suggestions anyone?
I am specifically puzzled that the open shiny server is tied to a particular, and old, version of node. That is probably just a reflection of #2 so the same question applies?

So, anyone got any thoughts, places I should look etc.?

Meanwhile:

I have ordered my own 8Gb Pi 4 for home use which may help clarify connectivity issues and make it feel slightly easier for me to play around with it more.
It's way beyond what my ISP owe but they have a history of support for Raspberrys and of being brilliant, so I will ask them if they have any thoughts.
I guess I do have options to fall back to the 32bit OS on either machine.

Thanks again Andres ... and here's hoping there are others out there with interest or thoughts! Very best, Chris

andresrcs · April 5, 2021, 6:13pm

Some follow ups on your comments.

Actually, there are quite a few people using shiny server on Raspberry Pi SBCs (My blog articles on the topic receive visits regularly). I think the problem here is that not many people has move to 64-bit OS' since official support for them is rather recent, so not many people has tested this niche topic on 64-bit systems.

For example, I can't test it my self since I only have one Raspberry Pi and it is currently being used in production-ish but if I had a spare Pi I would definitely give it a try.

I think this is partially true, I asked for official Raspberry Pi support for RStudio Server and they told me they don't have the resources to invest their time on such a niche topic, they prefer to support education and open source in other ways,and I can understand why. Resources are not limitless after all and you have to make the best you can with what you have.

But please make your request on the GitHub repository as a feature request, maybe if they see enough interest in the topic, they might reconsider priorities.

system · May 29, 2021, 10:13pm

This topic was automatically closed 54 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.