@chuck_m
here is the example, play rmarkdown, sparklyr and airflow:
rmd_exe_base = """/bin/Rscript -e 'rmarkdown::render("/data/share/airflow/dags/project/Rmd/{job_name}.Rmd",params=list(exe_date="{exe_date}",master="yarn-client"),run_pandoc=F)' """
your_ops = BashOperator(
task_id='your_db.your_tbl',
depends_on_past=False,
retries = 3,
retry_delay=timedelta(0, 300),
bash_command=rmd_exe_base.format(exe_date = exe_date,job_name = "your_job"),
dag=dag)
and you can checkout the rmd_exe_base rendered command in airflow ui at task view.
And the advantage of Rmarkdown is the chunk can log the process bar automatically, and organize code and parameters very well.