Sparklyr - change log level during spark-submit

[Adam3S]

Hi!

I wonder if you can help me with the following problem.

What I need to achieve is to silence all the information logged to the console during ${SPARK_HOME}/bin/spark-submit execution.

I have already changed log level for spark itself by changing the appropriate line in ${SPARK_HOME}/conf/log4j.properties but still when I submit my R-sparklyr-script by providing sparklyr.jar I get the following info:

[user@host] ${SPARK_HOME}/bin/spark-submit --class sparklyr.Shell 'sparklyr.jar' script.R
21/07/01 16:12:25 INFO sparklyr: Session (999991) is starting under 127.0.0.1 port 4440
21/07/01 16:12:25 INFO sparklyr: Session (999991) found port 4440 is available
21/07/01 16:12:26 INFO sparklyr: Gateway (999991) is starting batch rscript
....

I deducted that sparklyr itself uses log4j as a logger but I don't know where to find the appropriate log4j.properites file.

Do you know any method to cope with it?

With regards,
Adam.

[yitaoli]

Sparklyr actually does not use log4j (or other similar framework), and there is no way to change the log level at the moment unfortunately. I have created an issue for this at Use log4j (or similar) for logging · Issue #3145 · sparklyr/sparklyr · GitHub

[Adam3S]

Thanks a lot for the response. It seems that my deduction was false then - it was based on the observation that log produced by sparklyr is similar to the one produced by log4j. Anyway I believe that this change would be appreciated by many

Once again thanks and looking forward to the changes