Read Feather from s3

As a bit of a follow-up from my previous question - I have a file that is in the arrow/feather format (saved from python) on my S3, and I want to read it into R on my EC2.

Based on the comments here, I tried to use the aws.s3::get_object() function, which gets a huge Raw object...but then I can't turn that object into the dataframe

library(arrow)
library(aws.s3)

bucket <- 'my_bucket'
filename  <- "my_filename.f"

obj1 <- aws.s3::get_object(object = filename, bucket = bucket)
obj2 <- rawToChar(obj1)
# Error in rawToChar(obj1) : long vectors not supported yet: raw.c:68
df1 <- arrow::read_feather(obj2)
# Error in ipc___feather___Reader__Read(self, columns) : 
#  NotImplemented: LZ4 codec support not built

I even tried to do it all together

df1 <- aws.s3::s3read_using(FUN = arrow::read_feather, object = filename, bucket = bucket)
# Error in ipc___feather___Reader__Read(self, columns) : 
#  NotImplemented: LZ4 codec support not built

Any ideas?

Never mind - I found the problem. Based on this post, when I saved the file in python, I should have included an argument compression='uncompressed' into the write_feather() function

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.