Unnest breaking on lubridate columns

I am getting errors when trying to unnest a tibble column that has lubridate objects in it. I came across this issue after unnesting some columns that I had just nested. Here is a toy example that shows the error:

library(tidyverse)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
test <- tibble(A=1:5, B=hours(1:5))
test
#> # A tibble: 5 x 2
#>       A B       
#>   <int> <Period>
#> 1     1 1H 0M 0S
#> 2     2 2H 0M 0S
#> 3     3 3H 0M 0S
#> 4     4 4H 0M 0S
#> 5     5 5H 0M 0S
nested <- test %>% nest(B2=B)
nested
#> # A tibble: 5 x 2
#>       A B2              
#>   <int> <list>          
#> 1     1 <tibble [1 Γ— 1]>
#> 2     2 <tibble [1 Γ— 1]>
#> 3     3 <tibble [1 Γ— 1]>
#> 4     4 <tibble [1 Γ— 1]>
#> 5     5 <tibble [1 Γ— 1]>
unnested <- nested %>% unnest(cols=B2)
#> Error: No common type for `..1$B2$B` <Period> and `..2$B2$B` <Period>.

Created on 2020-03-17 by the reprex package (v0.3.0)

This seems like a bug, no? It seems like nesting should be reversible by unnesting if no other manipulation has been done to the data.

Here's my session:

sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.3

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] reprex_0.3.0      lubridate_1.7.4   magrittr_1.5      broom_0.5.2       forcats_0.5.0     stringr_1.4.0     dplyr_0.8.5       purrr_0.3.3       readr_1.3.1       tidyr_1.0.2       tibble_2.1.3      ggplot2_3.3.0    
[13] tidyverse_1.3.0   readxl_1.3.1      data.table_1.12.2

loaded via a namespace (and not attached):
 [1] tidyselect_1.0.0 xfun_0.9         haven_2.2.0      lattice_0.20-38  colorspace_1.4-1 vctrs_0.2.4      generics_0.0.2   htmltools_0.4.0  utf8_1.1.4       rlang_0.4.5      pillar_1.4.2     glue_1.3.1       withr_2.1.2     
[14] DBI_1.0.0        dbplyr_1.4.2     modelr_0.1.5     lifecycle_0.2.0  munsell_0.5.0    gtable_0.3.0     cellranger_1.1.0 rvest_0.3.5      evaluate_0.14    knitr_1.28       callr_3.3.1      ps_1.3.0         fansi_0.4.0     
[27] Rcpp_1.0.2       clipr_0.7.0      backports_1.1.4  scales_1.1.0     jsonlite_1.6     fs_1.3.1         digest_0.6.20    hms_0.5.3        stringi_1.4.6    processx_3.4.1   grid_3.6.1       cli_1.1.0        tools_3.6.1     
[40] whisker_0.3-2    crayon_1.3.4     pkgconfig_2.0.2  xml2_1.2.2       assertthat_0.2.1 rmarkdown_2.1    httr_1.4.1       rstudioapi_0.10  R6_2.4.0         nlme_3.1-140     compiler_3.6.1  

I wonder whether this is because Period objects have multiple slots -- unnesting seems to work fine with Duration objects (as well as Date and datetime objects).

Interval class objects have two slots: .Data, a numeric object equal to the number of seconds in the interval; and start, a POSIXct object that specifies the time when the interval starts.

So Interval objects have multiple slots, like Period objects. I tried to chase down the issue with debug(), and got as far as identifying vctrs::vector_rbind() as the function the raises the error, but I can't tell why. (vector_rbind() seems to invoke a hidden -- at least to me -- vctrs_rbind method, and I could get any further.)

1 Like

i was wondering if the ptype parameter to pass to unchop could be used as part of a solution, but I couldn't figure it....

Part of the issue is that unnest() and unchop() eventually call vctrs:::has_same_type() to compare list elements, and will lead to a raised error if it returns false:

library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following object is masked from 'package:base':
#> 
#>     date
vctrs:::has_same_type(hours(0), hours(1))
#> [1] FALSE

Created on 2020-03-18 by the reprex package (v0.3.0)

The only solution I can come up with is converting Period and Interval objects to duration objects before unnest() (which calls unchop()) is called, and then convert back afterwards. The downside is that conversion is not straightforward for Period objects, which are ambiguous lengths of time: for example, months(1) can represent 28, 29, 30, or 31 days, depending on the date it's applied to.

However, from the discussion on github, it sounds like the issue should be resolved soon.

1 Like

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

No solution yet, but adding some more context from the documentation for the Period class:

Period class objects have six slots.

  1. Data, a numeric object. The apparent amount of seconds to add to the period.
  2. minute, a numeric object. The apparent amount of minutes to add to the period.
  3. hour, a numeric object. The apparent amount of hours to add to the period.
  4. day, a numeric object. The apparent amount of days to add to the period.
  5. month, a numeric object. The apparent amount of months to add to the period.
  6. year, a numeric object. The apparent amount of years to add to the period.

I guess the next question would be if it works with Interval class objects (since @dromano has tested it out with Durations)

The answer seems to be a resounding no there:

suppressPackageStartupMessages(library(tidyverse))
library(lubridate)

date1 <- ymd_hms("2009-03-08 01:59:59")
date2 <- ymd_hms("2000-02-29 12:00:00")
intervals <- c(
  interval(date2, date1),
  interval((date2 - 1), (date1 - 1)),
  interval((date2 - 2), (date1 - 2)),
  interval((date2 - 3), (date1 - 3)),
  interval((date2 - 4), (date1 - 4))
)

int_tib <- tibble::tibble(a = 1:5, b = intervals)

int_tib
#> # A tibble: 5 x 2
#>       a b                                               
#>   <int> <Interval>                                      
#> 1     1 2000-02-29 12:00:00 UTC--2009-03-08 01:59:59 UTC
#> 2     2 2000-02-29 11:59:59 UTC--2009-03-08 01:59:58 UTC
#> 3     3 2000-02-29 11:59:58 UTC--2009-03-08 01:59:57 UTC
#> 4     4 2000-02-29 11:59:57 UTC--2009-03-08 01:59:56 UTC
#> 5     5 2000-02-29 11:59:56 UTC--2009-03-08 01:59:55 UTC

nested <- int_tib %>% nest(b2 = b)

nested
#> # A tibble: 5 x 2
#>       a b2              
#>   <int> <list>          
#> 1     1 <tibble [1 Γ— 1]>
#> 2     2 <tibble [1 Γ— 1]>
#> 3     3 <tibble [1 Γ— 1]>
#> 4     4 <tibble [1 Γ— 1]>
#> 5     5 <tibble [1 Γ— 1]>

unnested <- nested %>% unnest(cols = b2)
#> Error: No common type for `..1$b2$b` <Interval> and `..2$b2$b` <Interval>.
#> Backtrace:
#>      β–ˆ
#>   1. β”œβ”€nested %>% unnest(cols = b2)
#>   2. β”‚ β”œβ”€base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
#>   3. β”‚ └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
#>   4. β”‚   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
#>   5. β”‚     └─`_fseq`(`_lhs`)
#>   6. β”‚       └─magrittr::freduce(value, `_function_list`)
#>   7. β”‚         β”œβ”€base::withVisible(function_list[[k]](value))
#>   8. β”‚         └─function_list[[k]](value)
#>   9. β”‚           β”œβ”€tidyr::unnest(., cols = b2)
#>  10. β”‚           └─tidyr:::unnest.data.frame(., cols = b2)
#>  11. β”‚             └─tidyr::unchop(data, !!cols, keep_empty = keep_empty, ptype = ptype)
#>  12. β”‚               └─vctrs::vec_rbind(!!!x, .ptype = ptype)
#>  13. β”œβ”€vctrs:::vec_ptype2_dispatch_s3(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
#>  14. └─vctrs:::vec_ptype2.default(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
#>  15.   └─vctrs::stop_incompatible_type(x, y, x_arg = x_arg, y_arg = y_arg)
#>  16.     └─vctrs:::stop_incompatible(...)
#>  17.       └─vctrs:::stop_vctrs(...)

Created on 2020-03-18 by the reprex package (v0.3.0.9001)

Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.2 (2019-12-12)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2020-03-18                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version      date       lib source                              
#>  assertthat    0.2.1        2019-03-21 [1] CRAN (R 3.6.0)                      
#>  backports     1.1.5        2019-10-02 [1] CRAN (R 3.6.0)                      
#>  broom         0.5.5        2020-02-29 [1] CRAN (R 3.6.0)                      
#>  cellranger    1.1.0        2016-07-27 [1] CRAN (R 3.6.0)                      
#>  cli           2.0.2.9000   2020-03-02 [1] Github (r-lib/cli@a8b708e)          
#>  colorspace    1.4-1        2019-03-18 [1] CRAN (R 3.6.0)                      
#>  crayon        1.3.4        2020-02-02 [1] Github (r-lib/crayon@f4bc7b8)       
#>  DBI           1.1.0        2019-12-15 [1] CRAN (R 3.6.1)                      
#>  dbplyr        1.4.2        2019-06-17 [1] CRAN (R 3.6.0)                      
#>  digest        0.6.25       2020-02-23 [1] CRAN (R 3.6.0)                      
#>  dplyr       * 0.8.99.9001  2020-03-18 [1] Github (tidyverse/dplyr@fbefa13)    
#>  evaluate      0.14         2019-05-28 [1] CRAN (R 3.6.0)                      
#>  fansi         0.4.1        2020-01-09 [1] Github (brodieG/fansi@7ccb892)      
#>  forcats     * 0.5.0.9000   2020-03-04 [1] Github (tidyverse/forcats@1c93496)  
#>  fs            1.3.2.9000   2020-03-18 [1] Github (r-lib/fs@1744061)           
#>  generics      0.0.2        2018-11-29 [1] CRAN (R 3.6.0)                      
#>  ggplot2     * 3.3.0.9000   2020-03-16 [1] Github (tidyverse/ggplot2@3ddfc3f)  
#>  glue          1.3.2        2020-03-16 [1] Github (tidyverse/glue@0cbbb17)     
#>  gtable        0.3.0        2019-03-25 [1] CRAN (R 3.6.0)                      
#>  haven         2.2.0        2019-11-08 [1] CRAN (R 3.6.0)                      
#>  highr         0.8          2019-03-20 [1] CRAN (R 3.6.0)                      
#>  hms           0.5.3        2020-01-08 [1] CRAN (R 3.6.1)                      
#>  htmltools     0.4.0        2019-10-04 [1] CRAN (R 3.6.0)                      
#>  httr          1.4.1        2019-08-05 [1] CRAN (R 3.6.1)                      
#>  jsonlite      1.6.1        2020-02-02 [1] CRAN (R 3.6.1)                      
#>  knitr         1.28         2020-02-06 [1] CRAN (R 3.6.0)                      
#>  lattice       0.20-40      2020-02-19 [1] CRAN (R 3.6.0)                      
#>  lifecycle     0.2.0.9000   2020-03-16 [1] Github (r-lib/lifecycle@355dcba)    
#>  lubridate   * 1.7.4.9000   2020-03-18 [1] Github (tidyverse/lubridate@b922845)
#>  magrittr      1.5          2014-11-22 [1] CRAN (R 3.6.0)                      
#>  modelr        0.1.6        2020-02-22 [1] CRAN (R 3.6.0)                      
#>  munsell       0.5.0        2018-06-12 [1] CRAN (R 3.6.0)                      
#>  nlme          3.1-145      2020-03-04 [1] CRAN (R 3.6.0)                      
#>  pillar        1.4.3.9000   2020-01-06 [1] Github (r-lib/pillar@8f5918c)       
#>  pkgconfig     2.0.3        2019-09-22 [1] CRAN (R 3.6.1)                      
#>  purrr       * 0.3.3.9000   2020-03-10 [1] Github (tidyverse/purrr@aa7bc7f)    
#>  R6            2.4.1        2019-11-12 [1] CRAN (R 3.6.0)                      
#>  Rcpp          1.0.4.2      2020-03-18 [1] Github (RcppCore/Rcpp@20e462f)      
#>  readr       * 1.3.1        2018-12-21 [1] CRAN (R 3.6.0)                      
#>  readxl        1.3.1        2019-03-13 [1] CRAN (R 3.6.0)                      
#>  reprex        0.3.0.9001   2020-02-02 [1] Github (tidyverse/reprex@a019cc4)   
#>  rlang         0.4.5.9000   2020-03-18 [1] Github (r-lib/rlang@a90b04b)        
#>  rmarkdown     2.1          2020-01-20 [1] CRAN (R 3.6.1)                      
#>  rstudioapi    0.11         2020-02-07 [1] CRAN (R 3.6.0)                      
#>  rvest         0.3.5        2019-11-08 [1] CRAN (R 3.6.0)                      
#>  scales        1.1.0.9000   2020-03-16 [1] Github (r-lib/scales@1e2c918)       
#>  sessioninfo   1.1.1        2018-11-05 [1] CRAN (R 3.6.0)                      
#>  stringi       1.4.6        2020-02-17 [1] CRAN (R 3.6.0)                      
#>  stringr     * 1.4.0        2019-02-10 [1] CRAN (R 3.6.0)                      
#>  styler        1.3.2        2020-02-23 [1] CRAN (R 3.6.1)                      
#>  tibble      * 2.99.99.9014 2020-03-18 [1] Github (tidyverse/tibble@7673f9e)   
#>  tidyr       * 1.0.2.9000   2020-03-10 [1] Github (tidyverse/tidyr@bcfc5f0)    
#>  tidyselect    1.0.0.9000   2020-02-02 [1] Github (r-lib/tidyselect@bb145af)   
#>  tidyverse   * 1.3.0        2019-11-21 [1] CRAN (R 3.6.1)                      
#>  utf8          1.1.4        2018-05-24 [1] CRAN (R 3.6.0)                      
#>  vctrs         0.2.99.9010  2020-03-18 [1] Github (r-lib/vctrs@27b4b96)        
#>  withr         2.1.2.9000   2020-02-02 [1] Github (r-lib/withr@16d47fd)        
#>  xfun          0.12         2020-01-13 [1] CRAN (R 3.6.0)                      
#>  xml2          1.2.5        2020-03-11 [1] CRAN (R 3.6.2)                      
#>  yaml          2.2.1        2020-02-01 [1] CRAN (R 3.6.1)                      
#>  ymlthis       0.1.2.9000   2020-03-16 [1] Github (r-lib/ymlthis@d4f983a)      
#> 
#> [1] /Library/Frameworks/R.framework/Versions/3.6/Resources/library

Thanks for the links! I didn't know about the last_error() and last_trace() functions. Here's the trace I get:

> rlang::last_trace()
<error/vctrs_error_incompatible_type>
No common type for `..1$data$B` <Period> and `..2$data$B` <Period>.
Backtrace:
     β–ˆ
  1. β”œβ”€nested %>% unnest(data)
  2. β”‚ β”œβ”€base::withVisible(eval(quote(`_fseq`(`_lhs`)), env, env))
  3. β”‚ └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  4. β”‚   └─base::eval(quote(`_fseq`(`_lhs`)), env, env)
  5. β”‚     └─`_fseq`(`_lhs`)
  6. β”‚       └─magrittr::freduce(value, `_function_list`)
  7. β”‚         β”œβ”€base::withVisible(function_list[[k]](value))
  8. β”‚         └─function_list[[k]](value)
  9. β”‚           β”œβ”€tidyr::unnest(., data)
 10. β”‚           └─tidyr:::unnest.data.frame(., data)
 11. β”‚             └─tidyr::unchop(data, !!cols, keep_empty = keep_empty, ptype = ptype)
 12. β”‚               └─vctrs::vec_rbind(!!!x, .ptype = ptype)
 13. β”œβ”€vctrs:::vec_ptype2_dispatch_s3(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 14. └─vctrs:::vec_ptype2.default(x = x, y = y, x_arg = x_arg, y_arg = y_arg)
 15.   └─vctrs::stop_incompatible_type(x, y, x_arg = x_arg, y_arg = y_arg)
 16.     └─vctrs:::stop_incompatible(...)
 17.       └─vctrs:::stop_vctrs(...)

and when I inspected the x and y values passed in line 13, their slots -- although named identically -- are not interpreted as identical attributes so failed the compatibility test. (Note the .Data slot did not show as an attribute, so didn't seem involved -- I think this is the only slot in the other data types.)

Not sure where this goes, but thought I'd share it in case someone else can pick up the trail.

Sounds like it's related to


I initially thought it was related to:

I'd follow the GitHub issue at this point, since Davis seemed to be confident that the fix would be there. :slightly_smiling_face:

1 Like