3D convolution using keras?

[donbibi129]

Hey,

Does anyone have any samples R scripts for creating a keras model with 3D conv layers?

I have MRI data (176 x 256 x 256), and I want to create a basic model to process my list of 3D Arrays representing the MRI data.

model <- keras_model_sequential()

model %>% layer_conv_3d(1, kernel_size = c(3,3,3), input_shape=c(176, 256, 256,3))

summary(model)

# Compile
model %>% 
  compile(loss = 'binary_crossentropy',
          optimizer = 'adam',
          metrics = 'accuracy')
 # Fit model
history <- model %>%
  fit(train_data,
      train_labels,
      epoch = 200,
      batch_size = 32,
      validation_split = 0.2)

That just outputs this error:

 Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Error when checking input: expected conv3d_2_input to have 5 dimensions, but got array with shape (176, 256, 256) 

Found very little information online, I'm not even sure, if I should be using 3D conv, although the data is 3D.

Kind regards.

[mattwarkentin]

Hi @donbibi129,

If you have 3D data then it is definitely possible to use 3D convolutional kernels as the hidden layers in your model. Is your MRI data colour?

Based on your input shape of c(176, 256, 256, 3) it seems like you are specifying the input as 176 slices of dimension 256x256 each with 3 colour channels, is this correct? But since you are using this layer as the first layer in your model, you need to specify the data_format. Note this part of the help document (?layer_conv_3d):

When using this layer as the first layer in a model, provide the keyword argument input_shape (list of integers, does not include the sample axis), e.g. input_shape=c(128L, 128L, 128L, 3L) for 128x128x128 volumes with a single channel, in data_format="channels_last" .

Try this:

model <- keras_model_sequential()

model %>% 
  layer_conv_3d(filters = 1, 
                kernel_size = c(3, 3, 3),
                data_format = 'channels_last'
                input_shape=c(176, 256, 256, 3))

[donbibi129]

Thank you for the reply,

I've made a few mistakes in regards to the colour channel, the MRI data is monochromatic, so it's a single colour channel, I've amended my parameter to reflect this.

The train_data is an R List with 70 elements, and each element is a 3D array with the following dimensions num [1:176, 1:256, 1:256].

I've ammended my code per your corrections, and provided the correct colour channel parameter for monochrome:

model <- keras_model_sequential()

model %>% layer_conv_3d(filters = 1, 
                        kernel_size = c(3,3,3),
                        data_format = 'channels_last',
                        input_shape=c(176, 256, 256, 1))

Unfortunately the same error persists, maybe I'm missing something

Regards.

[mattwarkentin]

I don't have your data so I made some synthetic data to replicate your issue. So basically your input data should be an array with dimensions number of subjects * x * y * z * channels, where x*y*z is the dimensions of the MRI cube of image data.

Okay, so here is my simulated data:

# Image data, 10 subjects with images as 100x100x100 pixels, 1 channel (monochromatic)
# Data are between 0,1 (corresponding to 0 to 255 rescaled)
train_x <- runif(10*100*100*100*1, 0, 1)
train_x <- array_reshape(train_x, dim = c(10, 100, 100, 100, 1))

# Outcome data, binary
train_y <- to_categorical(sample(c(0,1), size = 10, replace = TRUE))

The dimensions of the train_x is 10 100 100 100 1. When we specify the input_shape to a keras model, we leave off the first dimension, which is assumed to be the samples dimension (number of subjects/samples).

So next we can build a model with only a single convolution layer (10 kernels), flatten it, and have it lead to the final layer which predicts a binary outcome (0 or 1).

model <- keras_model_sequential()

model %>% 
  layer_conv_3d(filters = 10,
                kernel_size = c(3,3,3),
                input_shape = c(100, 100, 100, 1),
                data_format = 'channels_last') %>% 
  layer_flatten() %>% 
  layer_dense(units = 2, activation = 'softmax')

model %>% 
  compile(loss = 'binary_crossentropy',
          optimizer = 'adam',
          metrics = 'accuracy')

# Fit model
history <- 
  model %>%
  fit(train_x,
      train_y,
      epoch = 200,
      batch_size = 32,
      validation_split = 0.2)

This runs for me. So I think you need to reshape your data. Take it from a list of arrays to an array with 5 dimensions.

[donbibi129]

Hey @mattwarkentin,

Thank you for that :), I've made changes as per your suggestion and created a 5D array, and can confirm the following code works:

 Turn train_data & test_data into a 5D array from a list of 4D arrays.
train_data <- abind(train_data, along=0)
gc() # Perform GC.
test_data <- abind(test_data, along=0)
gc() # Perform GC.

train_labels <- to_categorical(train_labels)
test_labels <- to_categorical(test_labels)

gc() # Perform GC.

# Create sequential model
model <- keras_model_sequential()

model %>% layer_conv_3d(filters = 70, 
                        kernel_size = c(3,3,3),
                        input_shape=c(176, 256, 256, 1),
                        data_format = 'channels_last') %>%
  layer_flatten() %>% 
  layer_dense(units = 2, activation = 'softmax')

# Compile
model %>% 
  compile(loss = 'binary_crossentropy',
          optimizer = 'adam',
          metrics = 'accuracy')
 # Fit model
history <- model %>%
  fit(train_data,
      train_labels,
      epoch = 200,
      batch_size = 10,
      validation_split = 0.2)

Regards.