# Optimisation: Efficient way to reshape a list of matrices to a 3D array table object

Given this list of matrices:

``````my_list <- list(
D = matrix(
data = 1:12,
nrow = 3,
ncol = 4,
dimnames = list(c("A", "B", "C"),
c("F", "G", "H", "I"))),
E = matrix(
data = 13:24,
nrow = 3,
ncol = 4,
dimnames = list(c("A", "B", "C"),
c("F", "G", "H", "I")))
)
``````

Yielding:

``````\$D
F G H  I
A 1 4 7 10
B 2 5 8 11
C 3 6 9 12

\$E
F  G  H  I
A 13 16 19 22
B 14 17 20 23
C 15 18 21 24
``````

I want to convert it to a 3D array table object, like so:

``````my_tbl <- as.table(
x = array(
data = unlist(my_list),
dim = c(3, 4, 2),
dimnames = list(
X = c("A", "B", "C"),
Y = c("F", "G", "H", "I"),
Z = c("D", "E"))))
``````

Yielding:

``````, , Z = D

Y
X    F  G  H  I
A  1  4  7 10
B  2  5  8 11
C  3  6  9 12

, , Z = E

Y
X    F  G  H  I
A 13 16 19 22
B 14 17 20 23
C 15 18 21 24
``````

However, due to the call to `unlist()`, this turns out to either be rather slow with big matrices or result in a `vector memory exhausted (limit reached?)`-error, simply because the big matrices are coerced to one really big vector.

So, any ideas on how to do this conversion efficiently?

This method might be approx 50x faster ...

``````x <- 400
y <- 500

my_list <- list(
D = matrix(
data = 1:(x*y),
nrow = x,
ncol = y),
E = matrix(
data = ((x*y)+1):(2*x*y),
nrow = x,
ncol = y)
)

f1 <- function(l){
as.table(
x = array(
data = unlist(l),
dim = c(x,y,2)))
}

f2 <- function(l){
as.table(
x=array(
data=reduce(.x=my_list,c),
dim =c(x,y,2)
)
)
}

bench::mark(f1(my_list),
f2(my_list))``````

Thanks @nirgrahamuk - As I see it, we still have that r-e-a-l-l-y big vector as a result of the call to `reduce()`?

In general, performance is a trade off , cpu/storage/ram etc.
It might be that your requirements are such that you need a lower level programming language to implement closer to what you want. Perhaps look into using C++ via Rcpp, or Julia language ?

You didn't really clue me in to how big your big matrixes are ? are they much bigger than 400x500 for example ?

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.