Running huggingfaces integration shiny demo on posit connect.

[leet]

I'm trying to publish the shiny demo found at dfalbel/gptneox-chat at main (huggingface.co) to a Posit connect server. This app will run in RStudio and push-button publishing from Rstudio successfully deploys on my target Posit connect instance.

However the app will not run, in fact it crashes abruptly. I suspect that there are deployment dialogue windows from tok or some other library loading the model and its crashing the app deployment. The logs from Posit Connect for the app console don't tell me what's crashing the app, is there a way to get more detailed logs from the app console on Posit connect?

2023/08/15 20:35:55.193482552 [rsc-session] Content GUID: ff36b760-d5d0-4629-a66e-501ca1012ea0
2023/08/15 20:35:55.193504152 [rsc-session] Content ID: 44
2023/08/15 20:35:55.193512710 [rsc-session] Bundle ID: 108
2023/08/15 20:35:55.394602685 Running on host: ccb-rsc-rstudio-connect-dd7b87dc4-x7hdj
2023/08/15 20:35:55.401376957 Linux distribution: Ubuntu 20.04.6 LTS (focal)
2023/08/15 20:35:55.403311437 Running as user: uid=999(rstudio-connect) gid=999(rstudio-connect) groups=999(rstudio-connect)
2023/08/15 20:35:55.403328509 Connect version: 2023.06.0
2023/08/15 20:35:55.403357076 LANG: en_US.UTF-8
2023/08/15 20:35:55.403403690 Working directory: /opt/rstudio-connect/mnt/app
2023/08/15 20:35:55.403632325 Using R 4.3.1
2023/08/15 20:35:55.403639521 R.home(): /opt/R/4.3.1/lib/R
2023/08/15 20:35:55.404054358 Content will use associated Packrat library
2023/08/15 20:35:55.440783857 Adding Packrat library to R_LIBS and .libPaths: /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1
2023/08/15 20:35:55.440925394 R_LIBS: /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1
2023/08/15 20:35:55.440932940 .libPaths(): /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1, /opt/R/4.3.1/lib/R/library
2023/08/15 20:35:55.462255956 shiny version: 1.7.5
2023/08/15 20:35:55.462266686 httpuv version: 1.6.11
2023/08/15 20:35:55.462290192 rmarkdown version: (none)
2023/08/15 20:35:55.462295406 knitr version: (none)
2023/08/15 20:35:55.462324338 jsonlite version: 1.8.7
2023/08/15 20:35:55.462333336 RJSONIO version: (none)
2023/08/15 20:35:55.462382596 htmltools version: 0.5.5
2023/08/15 20:35:55.462392272 reticulate version: (none)
2023/08/15 20:35:55.462549415 Using pandoc: /opt/rstudio-connect/ext/pandoc/2.16
2023/08/15 20:35:56.008746758 Using Shiny bookmarking base directory /opt/rstudio-connect/mnt/bookmarks
2023/08/15 20:35:56.008996017 
2023/08/15 20:35:56.009007817 Starting R with process ID: '8073'
2023/08/15 20:35:56.009516696 Shiny application starting ...
2023/08/15 20:35:56.235343823 
2023/08/15 20:35:56.235356258 Attaching package: ‘bslib’
2023/08/15 20:35:56.235394308 
2023/08/15 20:35:56.236063186 The following object is masked from ‘package:utils’:
2023/08/15 20:35:56.236070342 
2023/08/15 20:35:56.236090694     page
2023/08/15 20:35:56.236097824 
2023/08/15 20:35:57.130592015 
2023/08/15 20:35:57.130603687 Listening on http://127.0.0.1:42175
2023/08/15 20:35:59.117764980 Started loading model .... 
2023/08/15 20:36:01.187517840 Warning: Error in if: argument is of length zero
2023/08/15 20:36:01.193783368   [No stack trace available]
2023/08/15 20:36:01.194296699 From earlier call:
2023/08/15 20:36:01.216363865   15: <Anonymous>
2023/08/15 20:36:01.216373116   13: fn
2023/08/15 20:36:01.216400199    8: retry
2023/08/15 20:36:01.216401224    7: connect$retryingStartServer
2023/08/15 20:36:01.216409912    6: eval
2023/08/15 20:36:01.216410667    5: eval
2023/08/15 20:36:01.216416951    4: eval
2023/08/15 20:36:01.216417401    3: eval
2023/08/15 20:36:01.216428022    2: eval.parent
2023/08/15 20:36:01.216433555    1: local
2023/08/15 20:36:06.217826900 [rsc-session] Received signal: interrupt
2023/08/15 20:36:06.217894130 [rsc-session] Terminating subprocess with interrupt ...
2023/08/15 20:36:06.235821571 
2023/08/15 20:36:06.235884294 
2023/08/15 20:36:06.236754192 Shiny application exiting ...
2023/08/15 20:36:06.236771788 Execution halted
2023/08/15 20:36:06.281193007 [rsc-session] Terminated subprocess with signal: interrupt

[dfalbel]

Hi @leet ,

I believe this is caused by a failure in the CRAN builds of tok. I also get the same behavior in outside of posit connect. I just submited a fixed tok version to CRAN, should be fixed automatically in a few days.

A workaround could be pinning tok to the current dev version.

[leet]

Did not expect to get your response here! Thanks for looking into this.

Also while I have your attention: My goal is to have a shiny chat interface with Llama2. The offical Llama transformer models are currently gated on HF which probably means I need an auth token. The function from_pretrained gives me an error:

> tok::tokenizer$from_pretrained("meta-llama/Llama-2-7b-hf", auth_token = "hf_XXXXXXXXXXXX")
Error in `tok::tokenizer$from_pretrained()`:
! `auth_token` is currently unsupported.

Will this function be added in the upcoming CRAN release of Tok?

[dfalbel]

Wow, that's very nice plan!

It's not yet supported unfortunately, but I can work on it in the next few days and add support for the auth_token parameter. I'll reach out once it's done.

I opened Add support for `auth_token` · Issue #4 · mlverse/hfhub · GitHub to track.

[dfalbel]

tok v0.1.1 is now on CRAN and should fix the installation issues.

[dfalbel]

I've added support for authentication in:

Auth

and soon in tok with

Add support for authorization.

You will need hfhub v0.1.1 to authenticate, however you don't need the dev version of tok for it work, if you set the HUGGING_FACE_HUB_TOKEN env var, you can avoid setting the auth_token parameter and it will use the env var instead.

[leet]

@dfalbel

tok v0.1.1 is now on CRAN and should fix the installation issues.

Thanks Daniel! Unfortunately that was not the solution. I have updated tok to ver 0.1.1 from CRAN and published the app again. The log was virtually identical, and I still have the question of how to get more detailed reports.

I will however try out the dev versions of hfhub, might save me the trouble of finding a way to upload the model files to Posit Connect.

2023/08/19 01:14:44.268959565 [rsc-session] Content GUID: ff36b760-d5d0-4629-a66e-501ca1012ea0
2023/08/19 01:14:44.268992009 [rsc-session] Content ID: 44
2023/08/19 01:14:44.268996688 [rsc-session] Bundle ID: 111
2023/08/19 01:14:44.466159731 Running on host: ccb-rsc-rstudio-connect-55c55dd4bc-2wjl8
2023/08/19 01:14:44.472852807 Linux distribution: Ubuntu 20.04.6 LTS (focal)
2023/08/19 01:14:44.474767191 Running as user: uid=999(rstudio-connect) gid=999(rstudio-connect) groups=999(rstudio-connect)
2023/08/19 01:14:44.474777652 Connect version: 2023.07.0
2023/08/19 01:14:44.474815684 LANG: en_US.UTF-8
2023/08/19 01:14:44.474828888 Working directory: /opt/rstudio-connect/mnt/app
2023/08/19 01:14:44.475057138 Using R 4.3.1
2023/08/19 01:14:44.475064235 R.home(): /opt/R/4.3.1/lib/R
2023/08/19 01:14:44.475489017 Content will use associated Packrat library
2023/08/19 01:14:44.495240711 Adding Packrat library to R_LIBS and .libPaths: /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1
2023/08/19 01:14:44.495399174 R_LIBS: /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1
2023/08/19 01:14:44.495408887 .libPaths(): /opt/rstudio-connect/mnt/app/packrat/lib/x86_64-pc-linux-gnu/4.3.1, /opt/R/4.3.1/lib/R/library
2023/08/19 01:14:44.513440045 shiny version: 1.7.5
2023/08/19 01:14:44.513458476 httpuv version: 1.6.11
2023/08/19 01:14:44.513484725 rmarkdown version: (none)
2023/08/19 01:14:44.513486922 knitr version: (none)
2023/08/19 01:14:44.513511096 jsonlite version: 1.8.7
2023/08/19 01:14:44.513522550 RJSONIO version: (none)
2023/08/19 01:14:44.513539046 htmltools version: 0.5.5
2023/08/19 01:14:44.513539838 reticulate version: (none)
2023/08/19 01:14:44.513736593 Using pandoc: /opt/rstudio-connect/ext/pandoc/2.16
2023/08/19 01:14:45.040944678 Using Shiny bookmarking base directory /opt/rstudio-connect/mnt/bookmarks
2023/08/19 01:14:45.041200693 
2023/08/19 01:14:45.041211931 Starting R with process ID: '1078'
2023/08/19 01:14:45.041755080 Shiny application starting ...
2023/08/19 01:14:45.265216843 
2023/08/19 01:14:45.265229680 Attaching package: ‘bslib’
2023/08/19 01:14:45.265263665 
2023/08/19 01:14:45.265990545 The following object is masked from ‘package:utils’:
2023/08/19 01:14:45.266001511 
2023/08/19 01:14:45.266036870     page
2023/08/19 01:14:45.266038495 
2023/08/19 01:14:46.151036871 
2023/08/19 01:14:46.151050279 Listening on http://127.0.0.1:37367
2023/08/19 01:14:48.189081374 Started loading model .... 
2023/08/19 01:14:50.222572658 Warning: Error in if: argument is of length zero
2023/08/19 01:14:50.228848138   [No stack trace available]
2023/08/19 01:14:50.229308376 From earlier call:
2023/08/19 01:14:50.249078291   15: <Anonymous>
2023/08/19 01:14:50.249089082   13: fn
2023/08/19 01:14:50.249119007    8: retry
2023/08/19 01:14:50.249119797    7: connect$retryingStartServer
2023/08/19 01:14:50.249129006    6: eval
2023/08/19 01:14:50.249129726    5: eval
2023/08/19 01:14:50.249136595    4: eval
2023/08/19 01:14:50.249137108    3: eval
2023/08/19 01:14:50.249142843    2: eval.parent
2023/08/19 01:14:50.249143404    1: local
2023/08/19 01:14:55.250880393 [rsc-session] Received signal: interrupt
2023/08/19 01:14:55.250970293 [rsc-session] Terminating subprocess with interrupt ...
2023/08/19 01:14:55.268389734 
2023/08/19 01:14:55.268446307 
2023/08/19 01:14:55.269277484 Shiny application exiting ...
2023/08/19 01:14:55.269300383 Execution halted
2023/08/19 01:14:55.313205484 [rsc-session] Terminated subprocess with signal: interrupt

[dfalbel]

You can try adding

Sys.setenv(TORCH_INSTALL=1, TORCH_HOME="libtorch/")

For faster iterations while debugging, you can also do:

repo <- "EleutherAI/pythia-70m"

which downloads a much smaller model.

[leet]

Adding the Sys.setenv line seems to have fixed the crash on posit connect with tok 0.1.1 so I'm marking this as the solution. I cannot find any documentation for the pythia-70m system prompt so the smaller model generates nonsense, but it works.

Thanks!