Optimisation: Efficient way to reshape a list of matrices to a 3D array table object

[Leon]

Given this list of matrices:

my_list <- list(
  D = matrix(
    data = 1:12,
    nrow = 3,
    ncol = 4,
    dimnames = list(c("A", "B", "C"),
                    c("F", "G", "H", "I"))),
  E = matrix(
    data = 13:24,
    nrow = 3,
    ncol = 4,
    dimnames = list(c("A", "B", "C"),
                    c("F", "G", "H", "I")))
)

Yielding:

$D
  F G H  I
A 1 4 7 10
B 2 5 8 11
C 3 6 9 12

$E
   F  G  H  I
A 13 16 19 22
B 14 17 20 23
C 15 18 21 24

I want to convert it to a 3D array table object, like so:

my_tbl <- as.table(
  x = array(
    data = unlist(my_list),
    dim = c(3, 4, 2),
    dimnames = list(
      X = c("A", "B", "C"),
      Y = c("F", "G", "H", "I"),
      Z = c("D", "E"))))

Yielding:

, , Z = D

   Y
X    F  G  H  I
  A  1  4  7 10
  B  2  5  8 11
  C  3  6  9 12

, , Z = E

   Y
X    F  G  H  I
  A 13 16 19 22
  B 14 17 20 23
  C 15 18 21 24

However, due to the call to unlist(), this turns out to either be rather slow with big matrices or result in a vector memory exhausted (limit reached?)-error, simply because the big matrices are coerced to one really big vector.

So, any ideas on how to do this conversion efficiently?

[nirgrahamuk]

This method might be approx 50x faster ...

x <- 400
y <- 500

my_list <- list(
  D = matrix(
    data = 1:(x*y),
    nrow = x,
    ncol = y),
  E = matrix(
    data = ((x*y)+1):(2*x*y),
    nrow = x,
    ncol = y)
)

f1 <- function(l){
  as.table(
    x = array(
      data = unlist(l),
      dim = c(x,y,2)))
}

f2 <- function(l){
  as.table(
    x=array(
      data=reduce(.x=my_list,c),
      dim =c(x,y,2)
    )
  )
}

bench::mark(f1(my_list),
            f2(my_list))

[Leon]

Thanks @nirgrahamuk - As I see it, we still have that r-e-a-l-l-y big vector as a result of the call to reduce()?

[nirgrahamuk]

In general, performance is a trade off , cpu/storage/ram etc.
It might be that your requirements are such that you need a lower level programming language to implement closer to what you want. Perhaps look into using C++ via Rcpp, or Julia language ?

You didn't really clue me in to how big your big matrixes are ? are they much bigger than 400x500 for example ?