Build fails on windows not able to find the package itself

Hi,
we're migrating to github actions and have a problem with windows builds.
When we use the default check_pak.yml, we always get errors that somewhere in the vignette the package is not installed. We then tried to build with no vignettes (both build-args and args). Then the package is not installed when running the examples.
I found some threads that address the problem that another package is missing, which was caused by building from source. However, the package I am checking itself is not found

Run rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check")
  rcmdcheck::rcmdcheck(args = c("--no-manual", "--as-cran"), error_on = "warning", check_dir = "check")
  shell: C:\R\bin\Rscript.EXE {0}
  env:
    R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
    RSPM: 
    GITHUB_PAT: ***
    R_LIBS_USER: D:\a\_temp\Library
    TZ: UTC
    _R_CHECK_SYSTEM_CLOCK_: FALSE
    NOT_CRAN: true
    _R_CHECK_CRAN_INCOMING_REMOTE_: false
-- R CMD build -----------------------------------------------------------------
* checking for file 'D:\a\seminr\seminr/DESCRIPTION' ... OK

* preparing 'seminr':

* checking DESCRIPTION meta-information ... OK

* installing the package to build vignettes

* creating vignettes ... ERROR

--- re-building 'SEMinR.Rmd' using rmarkdown

Quitting from lines 429-433 (SEMinR.Rmd) 

Error: Error: processing vignette 'SEMinR.Rmd' failed with diagnostics:

2 nodes produced errors; first error: there is no package called 'seminr'

--- failed re-building 'SEMinR.Rmd'



SUMMARY: processing the following file failed:

  'SEMinR.Rmd'



Error: Error: Vignette re-building failed.

Execution halted

Error: Error in proc$get_built_file() : Build process failed
Calls: <Anonymous> ... build_package -> with_envvar -> force -> <Anonymous>
Execution halted
Error: Process completed with exit code 1.

The lines (429-433) check is referring to are not even the first lines that use the package.

This is the branch this error occurs on: https://github.com/digitalemuendigkeit/seminr/tree/model-viz

Anyone have any idea what could go wrong herE?