Multiple package dev with devtools/usethis

Recent changes in devtools and usethis have inconvenienced my workflow.

I often work developing more than one package - for example a statistical methods package that is reusable, and a project-specific package that handles loading data for the project I'm working on.

With old devtools, I could do: create("methods") ; create("project") and then load_all("methods") and load_all("project").

I could also do use_package("sp", pkg="methods") to add sp to my methods package.

But now every time I want to use something in one of my packages I have to call proj_set() because the usethis::use_x() functions don't take a pkg= option:

proj_set("methods"); use_package("p1")
proj_set("project") ; use_package("p2")

That's a whole lot more typing and doesn't make semantic sense because I'm not working on a "project" at that level - these are "packages".

Furthermore, having a default project set with proj_set means that if I forget which "project" I've currently got set as default and type use_package("fnord") I've added it to the wrong one. With the old pkg= argument the package being modified was explicit, not based on some invisible global context.

I dont see why the pkg argument was removed, or why its default (of ".") wasn't changed to something like pkg=proj_get() - that would give the same behaviour as now for those that work with proj_set(), but also allowed the flexibility to easily work with multiple packages anywhere on a file system.

So I think I'm going to have to write some wrapper functions that call proj_set, do something, then set the project to NULL, which you can only do with more hacks because proj_set(NULL) doesn't work and the value is in a locked binding so its unexported function time: usethis:::proj_set_(NULL)

There's a lot more friction in package development due to these changes. The reason I used devtools was because it was a lot less friction than existing methods at the time.

Comment and ideas welcome please....

Are you are trying to work on multiple packages within a single session?

Yes. There's a package of code that is specific to the data analysis task at hand, and one package that is developing statistical methodology. These two lots of code must be kept separate.

Development happens in a single R session because the working process is something like:

data = project::read_data_and_fix("./Data/foo.csv")
result = model::fitmodel(data)

Then I might edit fitmodel, reload, re-run the fit. Or edit read_data_and_fix, reload, rerun.

Or I might even have three packages on the go, for more than one statistical method development, which may belong in different packages.

That unfortunately does not fit well with the model envisaged by usethis, which is one session per package. We switched from the devtools model because as usethis became more complicated it become increasing crucial to pass pkg = pkg to every single function call. This was error prone and dangerous (because you don't want to affect the current package) so we rewrote to use active projects, which considerably simplifies the internal API. As far as I know, you are the first person to complain about the change.

What's the recommended workflow for developing more than one package adopting the "one session per package" methodology? Something like this for two R sessions R1 and R2:

R1> create_package("foo")
R1> load_all()

R2> create_package("bar")
R2> load_all()

R1> # edit stuff in R1 package
R1> load_all()
R1> # do stuff. Now I need it in R2:
R1> build()

R2> library(foo) #  assume libpath set to find the build from R1
R2> # do stuff with current R1 and R2
R2> # edit R2 stuff
R2> build()

R1> library(bar)

There isn't a recommendation because this isn't something I tend to do very often. But when I have to coordinate multiple packages I tend to install R1, then in R2 restart R and reload the package. (Or sometimes I'll call load_all() with the path to R1)

That isn't very useful to an interactive development process, and again its adding friction that wasn't present in the old devtools way of doing things.

As long as load_all never has its path argument removed I can carry on working with devtools but doing anything with usethis will remain awkward.

This can be done with some help from the withr package.

To create a package in the current directory without the unwanted side effect of changing the working directory:

with_dir(".",create_package("bar"))

this works in any directory:

with_dir("/tmp",create_package("foo"))

To use use_ functions in packages in any location, first define a with_pkg function a bit like this:

with_pkg = function(pkg, code){
     old = getwd()
     proj_set(pkg)
    on.exit(setwd(old));force(code)
}

Note that if you are using the concept of a current project then that function will stomp on it, version 2 will get the current project and save that and reset it as well as restore the working directory. This is more proof of concept

Then: with_pkg("/tmp/foo",use_package("sp")) will run use_package in the context of that folder as a package. The use of on.exit ensures errors don't leave you in the wrong place:

with_pkg("/tmp/foo",use_package("fnord"))

will not change your working directory if fnord doesn't exist.

See also usethis::with_project() to temporarily use a different project.

1 Like

That does the job nicely, but I still don't understand the use of "project" which is a non-standard concept in R against "package" which is.

I work with multiple concurrent packages alot myself (though in different R sessions). Because I still need to reload several packages regularily in my "main" session, I have my own helper function that wraps devtools::load_all() with the path where all my in-development packages lie hardcoded, so that i just have to do myload("mypkg"). Works pretty well and is pretty easy to implement.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.