How can speed up the running of pgmm models in R?

Hello,

I'm using R to run (pgmm models) from package "plm", I have data with 3.5 million obs. It takes a long time (4 hours) to run the model with 4 thousand obs. The RAM of the device is 16 GB. Is there any way to speed up the process?

Thank you.

Try running this example with 1K observations (just cut and paste; install tictoc if needed) and see how you compare. Then we can look at the specific model with example data and your specific model. See the FAQ: How to do a minimal reproducible example reprex for beginners.

suppressPackageStartupMessages({
  library(plm)
  library(tictoc)
})



# from documentation for pgmm
data("EmplUK", package = "plm")

tic()
## Arellano and Bond (1991), table 4 col. b 
z1 <- pgmm(log(emp) ~ lag(log(emp), 1:2) + lag(log(wage), 0:1)
           + log(capital) + lag(log(output), 0:1) | lag(log(emp), 2:99),
           data = EmplUK, effect = "twoways", model = "twosteps")
summary(z1, robust = FALSE)
#> Twoways effects Two steps model
#> 
#> Call:
#> pgmm(formula = log(emp) ~ lag(log(emp), 1:2) + lag(log(wage), 
#>     0:1) + log(capital) + lag(log(output), 0:1) | lag(log(emp), 
#>     2:99), data = EmplUK, effect = "twoways", model = "twosteps")
#> 
#> Unbalanced Panel: n = 140, T = 7-9, N = 1031
#> 
#> Number of Observations Used: 611
#> 
#> Residuals:
#>       Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
#> -0.6190677 -0.0255683  0.0000000 -0.0001339  0.0332013  0.6410272 
#> 
#> Coefficients:
#>                         Estimate Std. Error  z-value  Pr(>|z|)    
#> lag(log(emp), 1:2)1     0.474151   0.085303   5.5584 2.722e-08 ***
#> lag(log(emp), 1:2)2    -0.052967   0.027284  -1.9413 0.0522200 .  
#> lag(log(wage), 0:1)0   -0.513205   0.049345 -10.4003 < 2.2e-16 ***
#> lag(log(wage), 0:1)1    0.224640   0.080063   2.8058 0.0050192 ** 
#> log(capital)            0.292723   0.039463   7.4177 1.191e-13 ***
#> lag(log(output), 0:1)0  0.609775   0.108524   5.6188 1.923e-08 ***
#> lag(log(output), 0:1)1 -0.446373   0.124815  -3.5763 0.0003485 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Sargan test: chisq(25) = 30.11247 (p-value = 0.22011)
#> Autocorrelation test (1): normal = -2.427829 (p-value = 0.01519)
#> Autocorrelation test (2): normal = -0.3325401 (p-value = 0.73948)
#> Wald test for coefficients: chisq(7) = 371.9877 (p-value = < 2.22e-16)
#> Wald test for time dummies: chisq(6) = 26.9045 (p-value = 0.0001509)
toc()
#> 0.129 sec elapsed
sessionInfo()
#> R version 4.0.4 (2021-02-15)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Pop!_OS 20.10
#> 
#> Matrix products: default
#> BLAS:   /usr/local/lib/R/lib/libRblas.so
#> LAPACK: /usr/local/lib/R/lib/libRlapack.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] tictoc_1.0 plm_2.4-1 
#> 
#> loaded via a namespace (and not attached):
#>  [1] pillar_1.5.1      compiler_4.0.4    highr_0.8         miscTools_0.6-26 
#>  [5] tools_4.0.4       digest_0.6.27     nlme_3.1-152      evaluate_0.14    
#>  [9] lifecycle_1.0.0   tibble_3.1.0      debugme_1.1.0     lattice_0.20-41  
#> [13] pkgconfig_2.0.3   rlang_0.4.10      reprex_1.0.0      maxLik_1.4-6     
#> [17] yaml_2.2.1        xfun_0.22         styler_1.3.2      stringr_1.4.0    
#> [21] knitr_1.31        fs_1.5.0          vctrs_0.3.6       lmtest_0.9-38    
#> [25] grid_4.0.4        glue_1.4.2        bdsmatrix_1.3-4   fansi_0.4.2      
#> [29] Rdpack_2.1.1      rmarkdown_2.7     Formula_1.2-4     purrr_0.3.4      
#> [33] magrittr_2.0.1    backports_1.2.1   ellipsis_0.3.1    htmltools_0.5.1.1
#> [37] rbibutils_2.0     MASS_7.3-53.1     sandwich_3.0-0    utf8_1.2.1       
#> [41] stringi_1.5.3     crayon_1.4.1      zoo_1.8-9

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.