Using graylog for centralising logs within the R ecosystem

apooRv · May 3, 2018, 9:56am

Hey folks,

We are trying to centralise the logging infrastructure inside the organisation. The tech stack is mostly NodeJS + Python. We use R for all data processing, reporting and analytics.

The engineering team is planning to use Graylog for some internal tools. Just wanted to understand if people here have some experience in connecting Graylog with the R ecosystem (Shiny applications, scripts, etc)

Python has a grapy package for doing the same. I think this can be used via reticulate. Any other approaches/help/learnings will be appreciated

Thanks

knilda · May 27, 2018, 11:36pm

Hello!

I had the same task, I used the REST API of Graylog and then read out the data with elasticsearchr. To display the data in a Shiny app, I made a script which is based on the movie explorer example in R Shiny. Does this help you? I can help out with script fragments if needed, and I am happy to exchange experience with you.

apooRv · May 29, 2018, 12:12pm

Thanks for the reply @knilda .. Please share your repo, will definitely have a look.

knilda · May 29, 2018, 4:59pm

You are very welcome, @apooRv. As I unterstand, graypy package is reporting logs from python to graylog. My script is getting data from graylog to analyse them in R. Is this what you want?

apooRv · June 11, 2018, 7:08am

@knilda was more interested in the ingestion of logs in Graylog, any inputs on that front ?

knilda · June 11, 2018, 8:28am

Sorry, I have no experience on that. I think its also possible via REST API.

cole · June 15, 2018, 7:08pm

You might have a look at the logging package, which has some examples of logging to a remote server via REST (I think this is it? Not sure if this is the best example):

github.com

mfrasca/r-logging/blob/master/handlers/pkg/R/sentry.R

##***********************************************************************
## this program is free software: you can redistribute it and/or
## modify it under the terms of the GNU General Public License as
## published by the Free Software Foundation, either version 3 of the
## License, or (at your option) any later version.
##
## this program is distributed in the hope that it will be useful, but
## WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
## General Public License for more details.
##
## You should have received a copy of the GNU General Public License
## along with the nens libraray.  If not, see
## <http://www.gnu.org/licenses/>.
##
## Copyright © 2011, 2012 by Mario Frasca
##
## Library    : logging
##
## Purpose    : implement a sentry logging handler

This file has been truncated. show original

knilda · July 6, 2018, 9:55pm

@apooRv this is my repo, in case you want to get your logging data from graylog back to R, this might help you https://github.com/knilda/shRlog.git

shapenaji · July 13, 2018, 3:28pm

I've been testing using @atheriel's excellent rsyslog package
(https://cran.r-project.org/web/packages/rsyslog/index.html) to forward via rsyslog. I can confirm that it successfully logs to the main syslog process through rstudio-connect, and then I forward that to a logging box.