automatic skip "download failed" case

[yangkai9999]

Hello Team,

I use quantmod and TTR package for some analysis. it need to download available stock data. But it is not for all of them. As you can see in the code below, I have to spent many time to manually remove some of them. But it is too many to remove.

it there a way to automatic skip "download failed" case, so I don't need to manually remove it?

Thanks,
Kai

library(quantmod)
options(getSymbols.auto.assign = FALSE)
require(TTR)
tickersList <- stockSymbols() #get all stock symbols list
#remove error symbols

# ----- use loop
# drop problem stocks
tickersList <- tickersList[!tickersList$Symbol %in% 
                             c("ADRA-UN","ADRA-WT","ADRT-UN","AP-WT","ATEST","ATEST-A","ATEST-B",
                               "ATEST-C","BMTX-WT","BRMK-WT","DC-WT","DMYY","DMYY-UN","DMYY-WT",
                               "ECF-PA","FLAG-WT","FOXO-WT","GLTA-UN","GLTA-WT","GLU-PA","GROY-WT",
                               "HNRA-WT","HPX-WT","IBO","KLR-WT","LGL-WT","MIMO-WT","MIMO-WTA","MIMO-WTB",
                               "MIMO-WTC","NILE-PD","PCG-PG","PHGE-UN","PHGE-WT","POL-WT","RCOR-WT","SACH-PA",
                               "SATX-WTA","SBEV-WT","SKYH-WT","STR-WT","TDW-WT","TWND-WT","VHAQ-R","VHAQ-UN",
                               "VHAQ-WT","IGZ","ZVV","ZBZX","ZTEST","ZEXIT","ZIEXT","ZXIET","AACIW",
                               "ACABW","ACACW","ACAHW","ACAXR","ACAXW","ACBAW","TYLG","FYLG","ACDCW","HYLG",
                               "OARK","BKGI","SNPD","SNPG","SNPV","ACONW","ACQRW","AAC-WT","ACAQ-WT",
                               "ACDI-WT","ACHR-WT","ACP-PA","ACRO-WT","ADC-PA","ADEX-WT","AEVA-WT"), ]
# use for loop to do plot
for (i in tickersList$Symbol) {
  myname <- getSymbols(i)
  head(myname)
  # chartSeries(myname, TA=NULL)
}

error message below

> # use for loop to do plot
> for (i in tickersList$Symbol) {
+   myname <- getSymbols(i)
+   head(myname)
+   # chartSeries(myname, TA=NULL)
+ }
Warning: AFTR-WT download failed; trying again.
Error in getSymbols.yahoo(Symbols = "AFTR-WT", env = <environment>, verbose = FALSE,  : 
  Unable to import “AFTR-WT”.
AFTR-WT download failed after two attempts. Error message:
HTTP error 404.
In addition: Warning message:
ABR-PF contains missing values. Some functions will not work if objects contain missing values in the middle of the series. Consider using na.omit(), na.approx(), na.fill(), etc to remove or replace them.

[yangkai9999]

Note: if you want to try the code above, it may table long time. it depend on your internet speed

[HanOostdijk]

Maybe the code below helps you.
I selected three symbols (two that gave you an error and one that did not) in mytickerslist.
I use the getSymbols function but 'guarded' by the tryCatch function that in case of an error
only writes the symbol to the variable error_ticker.
After exiting the tryCatch I check this variable: if set I add it to the list error_tickers otherwise I assign the values of the symbol (ticker) to a variable with the same name in the global environment.
In my case I now have the variable (data.frame) ABR-PF in the workspace and the other two symbols (ADRA-UN and AFTR-WT ) in the list error_tickers.
Try if this is what you need. If not let us know.

library(quantmod)
options(getSymbols.auto.assign = FALSE)
require(TTR)
tickersList <- stockSymbols()

mytickersList <- tickersList |>
  dplyr::filter (Symbol %in% c("ADRA-UN","AFTR-WT", "ABR-PF"))

error_tickers <- list()

for (i in mytickersList$Symbol) {
  myname <- 0
  error_ticker <- ""
  tryCatch(
    myname <- getSymbols(i) ,
    error = function(e) {
      assign("error_ticker", i,envir = .GlobalEnv)
    },
    finally = print("done with this ticker")
  )
  if (error_ticker != "") {
    error_tickers <- append(error_tickers, error_ticker)
  } else {
    assign(i, myname)
  }
}
print(error_tickers)

[yangkai9999]

Hi HanOostdijk,
Thank you for your solution. I was trying last night. It works on first ~3000 loops, the rest of 9000+ loops doesn't work (see the message below). I checked the symbol this morning.
for example, in the list below ZYNE, ZYXI can be loaded, the rest of them can not. I guess ~3000 loops meet the max loop iterations in R?

Thanks,
Kai

Warning: ZWZZT download failed; trying again.
[1] "done with this ticker"
Warning: ZXYZ-A download failed; trying again.
[1] "done with this ticker"
Warning: ZXZZT download failed; trying again.
[1] "done with this ticker"
Warning: ZYNE download failed; trying again.
[1] "done with this ticker"
Warning: ZYXI download failed; trying again.
[1] "done with this ticker"
Warning: NA download failed; trying again.
[1] "done with this ticker"
Error in if (error_ticker != "") { : 
  missing value where TRUE/FALSE needed
In addition: There were 50 or more warnings (use warnings() to see the first 50)

[HanOostdijk]

Hello @yangkai9999,
sorry to hear that this did not work out for you as I had hoped it would.
I don't think there is a limit in R for iterations (at least not one in the order of 12000)
but it could very well be a limit in the number of getSymbols requests that a server allows you in a certain timeframe.
I had a quick search in the quantmod documentation but I could not find such a restriction documented.
No further suggestions from me other than trying if inserting a Sys.sleep in the loop helps and trying to analyze the reason that some symbols are in error and others are not.

Maybe one of the other forum-members is able to help.

[yangkai9999]

Hi HanOostdijk,
Many Thanks for your help!
Kai