Creating an azure-pipelines.yml for Azure DevOps Testing (Packages, Shiny, Scripts)

[KoderKow]

Hello everyone!

I have a few packages on GitHub and been through setting up Travis-CI, AppVeyor, and Coverage for them. My work environment started using Azure DevOps. Since these repos are private I have been seeing how to set up tests myself for different situations; packages, Shiny, and general R scripts.

I have an azure-pipelines.yml to test my package as shown:

pool:
  vmImage: 'ubuntu-16.04'

container:
  image: 'rocker/tidyverse:latest'


variables:
  _R_CHECK_FORCE_SUGGESTS_: false
  MAKEFLAGS: "-j 2"

steps:
- bash: R -q -e 'writeLines(".libPaths(\"~/R-private\")", ".Rprofile"); dir.create("~/R-private", recursive = TRUE)'
  displayName: "preliminaries"

- bash: R -q -e 'install.packages(c("covr", "roxygen2", "testthat", "remotes")); remotes::install_deps(dependencies = TRUE);'
  displayName: 'before_install'

- bash: R -q -e 'devtools::load_all(); options("testthat.output_file" = "test-results.xml"); devtools::test(reporter = JunitReporter$new());'
  displayName: 'test()'

- bash: R -q -e 'cov <- covr::package_coverage(); covr::to_cobertura(cov, "coverage.xml")'

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-*.xml'

I took a lot of this code from Support for azure pipelines · Issue #169 · ropensci/tic · GitHub. Everything works fine, I get the test() results within Azure DevOps. My issue is with my coverage results. The pipeline overview looks like this:

Click to Show Image

The warning text is below:

##[section]Starting: PublishCodeCoverageResults
==============================================================================
Task         : Publish code coverage results
Description  : Publish Cobertura or JaCoCo code coverage results from a build
Version      : 1.152.1
Author       : Microsoft Corporation
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=626485)
==============================================================================
##[warning]Please install dotnet core to enable automatic generation of Html report.
Reading code coverage summary from '/home/vsts/work/1/s/coverage.xml'
##[warning]No coverage data found. Check the build errors/warnings for more details.
##[section]Async Command Start: Publish code coverage
Modifying Cobertura Index file
Publishing code coverage files to TFS server.
Uploading 1 files
File upload succeed.
Published '/tmp/Code Coverage Report_68' as artifact 'Code Coverage Report_68'
##[section]Async Command End: Publish code coverage
##[section]Finishing: PublishCodeCoverageResults

So it says it published the report as an artifact, which I can see here by clicking Artifacts in the top right:

Click to Show Image

Click to Show Test Results (No Coverage Results)

Does anyone have a good next step or resources for me to learn how to approach this? I am new to Azure DevOps, Docker, and writing .yml files for testing like this. I have tried multiple different things but keep on running into issues with coverage results. I believe my lack in knowledge of how the docker image is structured along with the appropriate .yml steps is holding my progress.

I will eventually want to set up pipelines for building the R package, test R scripts, and test if a Shiny app runs without error.

Thank you for your time!

[KoderKow]

Found the solution! The current covr on CRAN does not spit out a .xml structure that is compatible with the Azure DevOps environment. This is fixed in the development version on GitHub, so switching install.packaes() to install_github() returns the expected test results. I also added a .net core installation to create the .html output of the coverage report to display within the testing section, this addresses the other warning the pipeline gave. Solution below!

pool:
  vmImage: 'ubuntu-16.04'

container:
  image: 'rocker/tidyverse:latest'


variables:
  _R_CHECK_FORCE_SUGGESTS_: false
  MAKEFLAGS: "-j 2"

steps:
- bash: R -q -e 'writeLines(".libPaths(\"~/R-private\")", ".Rprofile"); dir.create("~/R-private", recursive = TRUE)'
  displayName: "preliminaries"

- bash: R -q -e 'install.packages(c("roxygen2", "testthat", "remotes")); devtools::install_github("r-lib/covr"); remotes::install_deps(dependencies = TRUE);'
  displayName: 'install packages'

- bash: R -q -e 'devtools::load_all(); options("testthat.output_file" = "test-results.xml"); devtools::test(reporter = JunitReporter$new());'
  displayName: 'test()'

- bash: R -q -e 'cov <- covr::package_coverage(); covr::to_cobertura(cov, "coverage.xml")'
  displayName: 'take covr'

- task: DotNetCoreInstaller@0
  displayName: 'install dotnet core'
  inputs:
    packageType: sdk
    version: 2.2.203
    installationPath: $(Agent.ToolsDirectory)/dotnet

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '**/test-*.xml'

[jimhester]

You have made great progress, glad you resolved the issue! I have also been prototyping Azure Pipelines and have an example yaml file at https://github.com/jimhester/azuretest/blob/master/azure-pipelines.yml.

It runs R CMD check (via the rcmdcheck package) on Windows, macOS and Linux, along with a number of R versions, maybe it would serve as a useful example.

[KoderKow]

Wow! This is great!

I have been trying to learn to set up these documents so this will definitely help with future progress! Thanks Jim!