How to "connect" altogether results of iterations of for loops, lm, and dplyr, broom ?

[Andrzej]

Hi,
I would like to have all the results in dataframe or table:

List = list()
for(i in 1:100)
    {
       LM = lm(rnorm(10)~rnorm(10))
       List[[length(List)+1]] = LM
     }

I would like to have all iterations results, eg. first column will be number of iteration, next column will be formula, next column will be coefficients, etc.

Please help,
I tried to use magicfor package but was not able to make it work for me.
It has got a nice function:
magic_result_as_dataframe() , but it doesn't work with above code.
https://cran.r-project.org/web/packages/magicfor/vignettes/magicfor.html

[nirgrahamuk]

General advice. Before attempting to iterate, I recommend you fully produce one of the results manually yourself.
That means if you want certain things in a dataframe, that you construct such a dataframe.
Afterwords generalising and producing similar in a loop becomes a strait forward exercise. And often you can use purrr map rather than for loop

[nirgrahamuk]

You are printing... I thought you'd be dataframe making. But baby steps is fine!
One of the things you wanted were the coefficients. You could use https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/coef

But then you also said you are happy to use broom. Have you read about broom and tried to use its functions on your lm object?

[Andrzej]

I do baby steps because I do not know the other way.
Can you show me what you would do in order to fully produce one of the results manually, please ? That you suggested in your reply.
I have read about broom and it's functions. For example this works:

mtcars %>% group_by(am) %>% do(tidy(lm(mpg ~ wt, .)))

But I do not know ho to adapt this code to my case with with for loops or purrr package.

[nirgrahamuk]

Ok, next step for you is to try

tidy(LM)

[Andrzej]

OK, I tried this and it got me results of my LM object. This is good as well and it would be next four columns in my final dataframe:

So my final dataframe would look like:

How do I do that, please advise for a first element,
Thanks.

[Andrzej]

Hi,
I tried this and I am closer to a solution:

List = list()

for (i in 1:5)
{
  LM <- lm(rnorm(10) ~ rnorm(10))
  List[[length(List) + 1]] <- LM
}

My_dataframe_1 <- do.call(cbind, List) %>%  dplyr::as_data_frame(., rownames = "My_elements") %>% tibble::rownames_to_column()

I reduced the number of iterations to 5 to make it simpler.

How do I transpose this dataframe so My_element will go to columns and variables from V1 to V5 to rows ?

[siddharthprabhu]

I think you're going about this the wrong way. In my opinion, you should use all 3 of broom's tidying functions to generate the summary statistics that you need. Read more about them here.

library(tidyverse)
library(broom)

set.seed(42)
df <- tibble(id = rep(1:5, each = 10), x = rnorm(50), y = rnorm(50))

print(df, n = 15)
#> # A tibble: 50 x 3
#>       id       x       y
#>    <int>   <dbl>   <dbl>
#>  1     1  1.37    0.322 
#>  2     1 -0.565  -0.784 
#>  3     1  0.363   1.58  
#>  4     1  0.633   0.643 
#>  5     1  0.404   0.0898
#>  6     1 -0.106   0.277 
#>  7     1  1.51    0.679 
#>  8     1 -0.0947  0.0898
#>  9     1  2.02   -2.99  
#> 10     1 -0.0627  0.285 
#> 11     2  1.30   -0.367 
#> 12     2  2.29    0.185 
#> 13     2 -1.39    0.582 
#> 14     2 -0.279   1.40  
#> 15     2 -0.133  -0.727 
#> # ... with 35 more rows

regressions <- df %>%
  nest(data = -id) %>%
  mutate(
    fit = map(data, ~ lm(y ~ x, data = .x)),
    tidied = map(fit, tidy),
    glanced = map(fit, glance),
    augmented = map(fit, augment)
  )

unnest(regressions, tidied)
#> # A tibble: 10 x 10
#>       id data  fit   term  estimate std.error statistic p.value glanced
#>    <int> <lis> <lis> <chr>    <dbl>     <dbl>     <dbl>   <dbl> <list> 
#>  1     1 <tib~ <lm>  (Int~   0.303      0.461    0.659   0.529  <tibbl~
#>  2     1 <tib~ <lm>  x      -0.521      0.478   -1.09    0.308  <tibbl~
#>  3     2 <tib~ <lm>  (Int~   0.515      0.219    2.35    0.0470 <tibbl~
#>  4     2 <tib~ <lm>  x      -0.150      0.141   -1.07    0.317  <tibbl~
#>  5     3 <tib~ <lm>  (Int~  -0.265      0.241   -1.10    0.303  <tibbl~
#>  6     3 <tib~ <lm>  x      -0.264      0.217   -1.22    0.258  <tibbl~
#>  7     4 <tib~ <lm>  (Int~   0.199      0.263    0.757   0.471  <tibbl~
#>  8     4 <tib~ <lm>  x      -0.143      0.235   -0.609   0.559  <tibbl~
#>  9     5 <tib~ <lm>  (Int~  -0.0864     0.358   -0.241   0.815  <tibbl~
#> 10     5 <tib~ <lm>  x       0.0411     0.438    0.0940  0.927  <tibbl~
#> # ... with 1 more variable: augmented <list>
unnest(regressions, glanced)
#> # A tibble: 5 x 17
#>      id data  fit   tidied r.squared adj.r.squared sigma statistic p.value    df
#>   <int> <lis> <lis> <list>     <dbl>         <dbl> <dbl>     <dbl>   <dbl> <dbl>
#> 1     1 <tib~ <lm>  <tibb~   0.129          0.0202 1.20    1.19      0.308     1
#> 2     2 <tib~ <lm>  <tibb~   0.124          0.0150 0.690   1.14      0.317     1
#> 3     3 <tib~ <lm>  <tibb~   0.156          0.0507 0.751   1.48      0.258     1
#> 4     4 <tib~ <lm>  <tibb~   0.0443        -0.0752 0.786   0.371     0.559     1
#> 5     5 <tib~ <lm>  <tibb~   0.00110       -0.124  1.13    0.00884   0.927     1
#> # ... with 7 more variables: logLik <dbl>, AIC <dbl>, BIC <dbl>,
#> #   deviance <dbl>, df.residual <int>, nobs <int>, augmented <list>
unnest(regressions, augmented)
#> # A tibble: 50 x 13
#>       id data  fit   tidied glanced       y       x .fitted   .resid .std.resid
#>    <int> <lis> <lis> <list> <list>    <dbl>   <dbl>   <dbl>    <dbl>      <dbl>
#>  1     1 <tib~ <lm>  <tibb~ <tibbl~  0.322   1.37   -0.411   0.733      0.686  
#>  2     1 <tib~ <lm>  <tibb~ <tibbl~ -0.784  -0.565   0.598  -1.38      -1.37   
#>  3     1 <tib~ <lm>  <tibb~ <tibbl~  1.58    0.363   0.114   1.46       1.29   
#>  4     1 <tib~ <lm>  <tibb~ <tibbl~  0.643   0.633  -0.0262  0.669      0.589  
#>  5     1 <tib~ <lm>  <tibb~ <tibbl~  0.0898  0.404   0.0929 -0.00312   -0.00275
#>  6     1 <tib~ <lm>  <tibb~ <tibbl~  0.277  -0.106   0.359  -0.0822    -0.0751 
#>  7     1 <tib~ <lm>  <tibb~ <tibbl~  0.679   1.51   -0.484   1.16       1.12   
#>  8     1 <tib~ <lm>  <tibb~ <tibbl~  0.0898 -0.0947  0.353  -0.263     -0.240  
#>  9     1 <tib~ <lm>  <tibb~ <tibbl~ -2.99    2.02   -0.748  -2.25      -2.51   
#> 10     1 <tib~ <lm>  <tibb~ <tibbl~  0.285  -0.0627  0.336  -0.0512    -0.0466 
#> # ... with 40 more rows, and 3 more variables: .hat <dbl>, .sigma <dbl>,
#> #   .cooksd <dbl>

^{Created on 2020-12-14 by the reprex package (v0.3.0)}

[Andrzej]

Thank you very much, that is the way it should be,
anyway I still try to somehow to change table below so names/values in rows in column "My_elements" will go to columns and V1:V5 will go to row names. I tried pivot longer and pivot_wider but no success. If anybody has got an idea how to do it, I would be very grateful.

[Andrzej]

Hi again, I know that this post has got a solution but for learning/execising purposes only I tried to reshape this above dataframe and come up with this:

My_dataframe_2 <-   pivot_longer(My_dataframe_1, cols = V1:V5)

which gives me this:

and then I tried:

My_dataframe_3 <- My_dataframe_2 %>% pivot_wider(names_from = My_elements, values_from = Values)

which gives me this:

So this is almost what I wanted (that My_elements variable would go to columns) but my question is how to place everything within 1-5 rows only in order to have a dataframe without NULL values in many places ? Is it doable ?