I want to know how I can manipulate dates and strings in sparklyr. It seems lubridate does not work. Also, functions like subtr and paste do not work either.
1 Like
sparklyr helps work with spark from R. It works with dplyr but use SPARK SQL on the back end. Know that you can use directly spark sql functions to work with data. There are some function for time and date. See this book chapter:
https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-functions-datetime.html
You can also use HIVE function in sparklyr
https://spark.rstudio.com/dplyr/#hive-functions
and hive as data time function :
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
For character, it is the same.
With all this I think you can work with date and character with sparklyr. The hability to use spark and hive functions is a very powerful feature !
3 Likes