Announcing a way to forward logs from RSConnect to Elasticsearch, Splunk, and many others

One of the things we've struggled with in using RSConnect is making Shiny app logs more useful. You can view logs for individual sessions in the browser (if you know what you're looking for), but there's no searching or aggregation of any kind. This is frustrating, because log aggregation is largely a solved problem -- there are many commercial and open-source options available, none of which RSConnect currently supports.

To solve this for our own uses, I wrote an RSConnect-specific plugin for Fluent Bit. It works by watching for new logs written to the filesystem and querying the Connect API for additional metadata to link these logs back to specific applications. Fluent Bit can then be used to forward the results to any of its supported backends, including Elasticsearch, Splunk, Stackdriver, Cloudwatch Logs, Datadog, Loki, and many others.

This should allow R users to easily export logs to their organisation's existing logging provider without needing to modify their reports or Shiny apps to do so. For example, log entries look like this in our Splunk instance:

The plugin is open-source and available on GitHub. The README contains information on building the plugin (I don't have prebuilt binaries available yet, sorry) and sample configuration files. If this looks interesting to you but you can't get it working, feel free to reach out or file an issue.

(Shoutout to @cole for some private correspondence on using Connect's APIs that I needed to make it work.)

7 Likes

Woohoo!! Well done! I can't wait to give this a shot - thanks for sharing and posting about it!

This is super interesting and useful @atheriel! Exactly what I was looking for! :slight_smile:

Are there any things planned on the RStudio side, e.g. official support?