extracting the weight that maximize my function- R code

safara · February 17, 2023, 5:12pm

Hi,
My calculation contains 3 steps . The first step after generating 1000 random weigths I will calculate my variable y, which has the definition ; y_it = w^T_i * r_t. where i=1,...,1000
No y_it become a matrix of 1000 observations.

my second step is to fit a quantile regression for each y_i , so in total for a specific tau I will have 1000 quantile values.
I want to do this for 99 sequence of taus ; tau= .01, .02 ,....., .99
So as a result I will have a matrix of 1000x99
each column correspond to a specific values of tau.
For instance for tau=.01 I will have 1000 rows values ,each quantile lies in these rows correspond to different y

my code for step 1 and 2 after I generated the random weights all_wts[i,] as follows:

#this is to calculate y observations 
all_yts1 <- matrix(nrow = nrow(return) ,
                   ncol = 1000 )
for( i in 1:1000){
  all_yts1[,i]<-rowSums(all_wts1[i,] * return) } 

###calculate quantile matrix 

tau=seq(0.01,.99,by=.01)
quant.matrix.reg3<- matrix(nrow = np1 ,ncol = length(tau) )
for( i in 1:np1){
  quant.matrix.reg3[i,]<- rq(all_yts1[,i][-1]~ x.reg[,3], tau)$rho
  colnames(quant.matrix.reg3)<- tau
}

I am using rho values in the code because my third step is to allocate the maximum value of each column . The code for step 3 as follows:

opt.w3.alltaus=matrix(ncol = length(tau) )
for(j in 1: length(tau)){
  opt.w3.alltaus[,j]=which( quant.matrix.reg3[,j]== max(quant.matrix.reg3[,j]),arr.ind = T) 
}

so in total I am expecting 99 maximum values , each value represent specific tau .
for example, the maximum value in the column contains tau=.01 , is the row number 577. ( this what the codes tells me )

As a final step, since y_it= w^T_i * r_t. I want to extract the w values that lying in the row 577 , is there anyway to do that rather than do it manually , because extracting 99 w values will be a time consuming especially if you are doing this process for several regressors.

Any advice to improve my code will be appreciated

Thanks

nirgrahamuk · February 17, 2023, 3:17pm

I think you are saying; you determined the row numbers of interest and stored them here; it should be straightforward to use those numbers to look up actual values they correspond to.
Consider this example



(my_df <- data.frame(vec1=sample.int(10,size=10,replace=FALSE),
            vec2=sample.int(10,size=10,replace=FALSE)))

(colnum <- ncol(my_df))

what_big_mat <- which_big_mat <- matrix(ncol = colnum,nrow = 1)

for(i in seq_len(colnum)){
  which_big_mat[,i] <- which.max(my_df[,i])
  what_big_mat[,i] <- my_df[which_big_mat[,i],i]
}
which_big_mat
what_big_mat

safara · February 18, 2023, 12:36pm

thank you for your response. I actually don't want to find the maximum value. I want the weight ( w) that makes these values. I hope this makes sense

FactOREO · February 18, 2023, 1:36pm

I would like to know if you could you provide some sample data, e.g. what is your return variable? Just the general structure with some dummy data and your expected result, so that we talk about the same thing and your requirements are clear.

safara · February 18, 2023, 3:45pm

ok I simplified my data to make it easy to understand. we start with w , I included the grid of w , w is a matrix of 10 x 4

y = w^T * return, I included y , the return is just a vector of 250 observation. y will be a matrix of 250 x 10

I calculated the quantile regression for two values of tau e.g tau=.01 and tau=.06 . So I will have a matrix of 10 x 2 . My final step is to allocate what row has the maximum value from each column in the previous step.

I am allocating the max row because this will tell which w I used. e.g. row 9 has the maximum value in tau=.01, this tells me that w_9 =[ 0.226853071236893, 0.0855937166659081, 0.380296797258491 0.307256414838707] is the one that maximize the quantile regression. is there any way to extract w_9 vector without doing it manually ? because in my data the data is much bigger than this example..

Update::: I tried to include the data but I could not , it says new users can not upload attachments.

I hope my explanation helped or if there is any way to upload my data please help me

jrkrideau · February 18, 2023, 4:38pm

A handy way to supply some sample data is the dput() function. In the case of a large dataset something like dput(head(mydata, 100)) should supply the data we need. Just do dput(mydata) where mydata is your data. Copy the output and paste it here.

safara · February 18, 2023, 5:16pm

Thank you.

this is my w :
dput(w)

structure(c(0.125317152319512, 0.0745439292667052, 0.586077416953187,
0.293766825283739, 0.250302495915074, 0.485794185165549, 0.254711032662707,
0.379498294249792, 0.226853071236893, 0.389874337579722, 0.175638364717147,
0.33205501745088, 0.0575595762802929, 0.164240433092228, 0.345972920695575,
0.110256795228951, 0.368038594116812, 0.148514686946719, 0.0855937166659081,
0.0529870782366488, 0.270380451403551, 0.349162694152531, 0.191884202849581,
0.329179547703677, 0.132554766707465, 0.338694291691177, 0.012763479721988,
0.210360477419355, 0.380296797258491, 0.355253292662155, 0.42866403155979,
0.244238359129883, 0.164478803916939, 0.212813193920356, 0.271169816681886,
0.065254727914323, 0.364486893498494, 0.261626541384135, 0.307256414838707,
0.201885291521473), .Dim = c(10L, 4L))

my return :
structure(c(-0.998211538740179, -0.43432868782487, -0.525312554636326,
1.71122629775979, 1.79203296404209, -0.415539432255763, 0.985070931254484,
0.202075147679626, 0.788809256395595, 0.560738859870469, 0.740264359665588,
-0.755588398187079, 0.732776338490826, -0.0913109703556181, 2.80715374522682,
-3.36438278656583, 1.37597520604356, -0.583050929843676, 1.16272207218291,
-2.54679928694541, 0.866341157919148, 0.646763510440085, 1.31853477071724,
-0.262324296677008, -1.07896543186659, 0.318108259552208, 0.294484288231178,
0.458871664662386, 0.911475824923524, 1.0431700335368, -0.198908968226785,
0.976037546904873, -0.255925704675519, -1.23768194400569, -5.1020164810434,
-5.85497797625401, -2.00508135881581, -4.92079906872824, -2.5861170297131,
3.52136735775739, -5.28336392211601, 6.87767743233172, -4.20110756156529,
-2.46385672697755, -9.64286626325626, 4.9715102414706, -7.97685237502757,
-13.2663044620875, 17.5956307714163, -14.8327584889306, 0.823430549293214,
-16.0387869893928, 4.4033104569972, -3.90286578978376, -7.2158830037786,
19.7885283693386, 7.02848597838246, 3.40161763606437, -5.01148470344503,
1.72070614187483, -5.30030356411579, -9.51467239449855, -1.52752743594293,
-4.07351135699408, 13.0623018205046, 4.32846674435527, 5.01051019243546,
2.9322571340102, -4.85107041711821, -2.17108235434313, -5.34451491995656,
-2.9552871479873, 7.16200391165751, -3.94449850130503, -3.00875110697616,
1.24346843260899, -0.0969721288111458, 0.857336668180153, 2.2470184479209,
3.61367342657246, 8.61038415008561, -5.19944496166529, -3.26364099338159,
-2.22104497264919, 0.41589825475139, -3.20983165064934, 3.19830804431573,
3.14387459943388, -4.7712509293901, -2.60001941903223, -6.34325664834998,
7.14589529342273, -1.91537146150909, 7.258653702924, -1.30929400459907,
3.89959056312028, -0.996907874128272, -0.558161740274965, 5.4355856126473,
7.07019290170274, -3.18156694187319, -3.09660534564724, 1.75168695389525,
2.36963390750473, 6.1935984097655, 1.07578557924057, 2.95023194675421,
3.52766657612857, -2.86460406838369, -3.9706027717064, -7.38761863783122,
3.07593855565846, 2.20793585627943, 1.59377902385049, -1.58415755282926,
-1.74659492756541, -1.20139501526841, -1.49718334107805, -0.0201186765236727,
-4.04364476590287, 2.45188405118153, -4.63302397373662, 1.17059869141221,
0.716848970532169, -1.21533226397483, 0.297270621895862, 2.35724072896923,
-3.76646653961767, -0.118336840912203, -2.54006349614606, 2.89465581002277,
0.35333298522362, 1.35882986815856, 2.62242399295234, -1.07394773899196,
-1.19061458675462, -1.24750534229605, 2.44850352614918, 0.217762305925184,
0.13456551968245, -1.39585183023305, 0.481372834906235, -1.44054118981165,
2.37571112439312, -2.13241063943066, -1.41514516325829, 0.235471583512492,
-0.374872172925933, 2.33332994147428, 0.554074919471237, 3.32202576372858,
2.45056481814974, 1.60097031327542, -1.54194719388885, -1.24692561909905,
-0.00995269621943962, -2.88967459054339, -0.906200203500784,
0.227320828835786, -0.175608743026956, -0.591072381003954, 3.2538167526547,
-0.282285692987472, -0.658397904888197, 3.1313487080415, 0.989866464402045,
-0.930791037834755, 0.862501684108086, 1.86596120350462, 0.057451882948456,
1.15167980909812, -1.91082534830551, 0.106050626078869, -0.977981185824195,
0.572459726363039, 3.71309429333113, -0.551529332871148, -0.658327394197045,
-1.25333862087871, -1.1725285364574, -5.22911044668746, 0.315286300762807,
-3.0099210810886, -0.115178278666139, 0.845036591033654, 2.62471287655792,
-1.0886814552129, 2.52538673212035, 1.07154710110384, 0.275942814162367,
2.21906721175342, -2.07155182109746, 1.57943715028397, 2.5593462102993,
0.263651760926198, 0.253585488137187, -1.51217225786597, -0.190643466376983,
-0.363217760847956, 0.458588737985544, -2.35327953007332, 0.651719635667369,
-1.67183735393488, 3.2589451641841, -3.70358498901231, -4.14494388609299,
-2.03332903849853, -3.57085280476843, -0.580280826629576, 0.186493391624953,
2.33979911520956, 3.04731149894142, 0.114174072590068, 1.75830860045529,
-1.45793031878219, 19.3821358492108, -1.2172325843907, -4.25483123843913,
-0.0180046800143607, 3.44144720239719, 3.15013641558322, -1.86269046197181,
-1.94181291195994, -0.0875724736879135, -1.37616585463665, 3.07016200928931,
3.63707467466785, 0.157698500716119, 0.00828792974960635), index = structure(c(1578009600,
1578268800, 1578355200, 1578441600, 1578528000, 1578614400, 1578873600,
1578960000, 1579046400, 1579132800, 1579219200, 1579564800, 1579651200,
1579737600, 1579824000, 1580083200, 1580169600, 1580256000, 1580342400,
1580428800, 1580688000, 1580774400, 1580860800, 1580947200, 1581033600,
1581292800, 1581379200, 1581465600, 1581552000, 1581638400, 1581984000,
1582070400, 1582156800, 1582243200, 1582502400, 1582588800, 1582675200,
1582761600, 1582848000, 1583107200, 1583193600, 1583280000, 1583366400,
1583452800, 1583712000, 1583798400, 1583884800, 1583971200, 1584057600,
1584316800, 1584403200, 1584489600, 1584576000, 1584662400, 1584921600,
1585008000, 1585094400, 1585180800, 1585267200, 1585526400, 1585612800,
1585699200, 1585785600, 1585872000, 1586131200, 1586217600, 1586304000,
1586390400, 1586736000, 1586822400, 1586908800, 1586995200, 1587081600,
1587340800, 1587427200, 1587513600, 1587600000, 1587686400, 1587945600,
1588032000, 1588118400, 1588204800, 1588291200, 1588550400, 1588636800,
1588723200, 1588809600, 1588896000, 1589155200, 1589241600, 1589328000,
1589414400, 1589500800, 1589760000, 1589846400, 1589932800, 1590019200,
1590105600, 1590451200, 1590537600, 1590624000, 1590710400, 1590969600,
1591056000, 1591142400, 1591228800, 1591315200, 1591574400, 1591660800,
1591747200, 1591833600, 1591920000, 1592179200, 1592265600, 1592352000,
1592438400, 1592524800, 1592784000, 1592870400, 1592956800, 1593043200,
1593129600, 1593388800, 1593475200, 1593561600, 1593648000, 1593993600,
1594080000, 1594166400, 1594252800, 1594339200, 1594598400, 1594684800,
1594771200, 1594857600, 1594944000, 1595203200, 1595289600, 1595376000,
1595462400, 1595548800, 1595808000, 1595894400, 1595980800, 1596067200,
1596153600, 1596412800, 1596499200, 1596585600, 1596672000, 1596758400,
1597017600, 1597104000, 1597190400, 1597276800, 1597363200, 1597622400,
1597708800, 1597795200, 1597881600, 1597968000, 1598227200, 1598313600,
1598400000, 1598486400, 1598572800, 1598832000, 1598918400, 1599004800,
1599091200, 1599177600, 1599523200, 1599609600, 1599696000, 1599782400,
1600041600, 1600128000, 1600214400, 1600300800, 1600387200, 1600646400,
1600732800, 1600819200, 1600905600, 1600992000, 1601251200, 1601337600,
1601424000, 1601510400, 1601596800, 1601856000, 1601942400, 1602028800,
1602115200, 1602201600, 1602460800, 1602547200, 1602633600, 1602720000,
1602806400, 1603065600, 1603152000, 1603238400, 1603324800, 1603411200,
1603670400, 1603756800, 1603843200, 1603929600, 1604016000, 1604275200,
1604361600, 1604448000, 1604534400, 1604620800, 1604880000, 1604966400,
1605052800, 1605139200, 1605225600, 1605484800, 1605571200, 1605657600,
1605744000, 1605830400, 1606089600, 1606176000, 1606262400, 1606435200
), tzone = "UTC", tclass = "Date"), class = c("xts", "zoo"), src = "yahoo", updated = structure(1676732920.66705, class = c("POSIXct",
"POSIXt")), na.action = structure(1L, class = "omit", index = 1578009600), .Dim = c(229L,
1L), .Dimnames = list(NULL, "AXP"))

y= w^T * return , y data will be

dput(head(all_yts1, 20))
structure(c(-0.580967834494634, 0.507479398520152, -0.420854718217509,
0.545327344777418, 1.48328874190858, -0.293625991202183, 0.305454078766549,
0.18354409382499, -0.223466797315721, 0.580862249627697, 0.093945885725705,
-1.56131016252159, -0.476762273181072, 0.345414257000697, -0.74792389468483,
-2.65922432356954, 1.06955037166253, 1.27629873597617, 0.176749981189388,
-2.8848014502591, -0.682793111538542, 0.338465981597116, -0.0989938196758222,
0.33623890210593, 1.32988386108682, -0.324540690803226, 0.393960533719863,
-0.0813038876872903, -0.0448979164648601, 0.366017111430232,
-0.353681936021797, -1.71232564549603, -0.282959462709465, 0.0430018031915345,
0.501480023298234, -2.60470957665064, 1.35383309408865, 0.796740064136971,
0.419135983401322, -3.01604293461369, -0.839358917255938, 0.424576533042379,
-0.471775206085039, 0.0784079358584315, 1.71556810085991, -1.17016536203262,
1.34041486536764, 0.835357120754332, 0.293835725559358, 0.531185080394375,
0.717161308572643, -1.76416374713785, 0.246223902871955, 1.57858711950979,
-0.210974309719014, -1.80206389546457, 1.38822974695243, 1.50555474943637,
0.0883825237289841, -3.65852857957864, -0.762501702062449, 0.43560154873925,
-0.339210969899278, 0.0665577348751397, 1.5840134787506, -0.678784914637079,
0.840170320767625, 0.410803305390874, 0.042364436845636, 0.388495187440142,
0.261540773631844, -1.86288752808977, -0.0251643691441288, 0.672623501131463,
0.235582504999306, -2.21587957726957, 1.45846861608843, 1.18394682263418,
0.311403180187427, -3.1452044499659, -0.660629140842264, 0.239891717259511,
-0.0788256129512871, 0.800738235482466, 1.23380931108273, -0.602144117612077,
0.538204211000472, -0.00703576126436964, 0.175849678321056, 0.632012070740937,
-0.342756274452651, -1.29495521287396, -0.274157465133583, 0.704684467493195,
-0.244967521772638, -2.50690114286587, 1.00281879316964, 0.923572090435656,
0.135432428989633, -3.6111139648543, -0.901895938918957, 0.405678639654747,
-0.352258106261181, -0.27440102663697, 1.71239589366546, -1.06242638278374,
1.33379252735141, 0.708525448709335, 0.265301105906324, 0.297469228860143,
0.548567796145825, -2.05870816717739, 0.345506992090962, 1.13839181811808,
0.72377745057785, -1.80355278021766, 1.7055308672342, 1.2478443676009,
0.338308147727992, -3.41684303506543, -0.578720508011014, 0.214090985182973,
-0.0828735776157977, 1.20773675844956, 1.13541549603013, -0.56509847148058,
0.378895565533564, -0.0895037224493505, 0.174473624420912, 0.823105984953412,
-0.441045114539761, -0.982473036479242, -0.452566757896381, 0.84670503438,
-0.929229569105583, -2.65030889664673, 0.692155470765946, 0.989118506173803,
-0.036909730391039, -3.78114756601329, -0.729493658881572, 0.415551249969171,
-0.379927361119737, 0.341520433633738, 1.55589724374715, -0.793086106402882,
0.875396015970581, 0.476385785841237, 0.112322605302255, 0.556581158110974,
0.325656995743469, -1.62643313650011, -0.0675937718548307, 1.02411620050389,
-0.348832964870822, -2.19445786723307, 1.23107656645279, 1.31939520677256,
0.133041931770422, -3.40358277379452, -0.707953338153062, 0.564116630333372,
-0.506745100070673, 0.00724222289076494, 1.6887343181863, -0.510173347007619,
0.688659520799145, 0.483247935407562, -0.160625604584246, 0.392033994419187,
0.452695883937555, -1.94499132467416, -0.16042673294228, 0.500433069605947,
-0.118281704700903, -2.32321599974747, 1.44141344908114, 1.36327453699102,
0.307673197323693, -2.8034910726218, -0.806286999114405, 0.523443501176467,
-0.505273882737212, -0.168511417045818, 1.76166702140827, -0.827565510262399,
1.06507174525865, 0.700747644664492, 0.0368674051311197, 0.363427920963221,
0.648382837513699, -2.02323806584575, 0.109769719122347, 0.928208207631413,
0.143245414559511, -2.01278416340327, 1.56967057126975, 1.41717789382817,
0.288469687900237, -3.09280662997926), .Dim = c(20L, 10L))

quantile regression for two values of tau e.g tau=.01 and tau=.06 as follow :

dput(quant.matrix.reg1)
structure(c(31.0132004757793, 29.731942064456, 30.8044482731088,
29.3487557479765, 28.368598706984, 29.8900075294188, 28.1435428938265,
29.5148039317729, 31.7521366904934, 30.0404922387529, 105.683565308846,
107.373263664155, 98.988538212407, 102.537110474698, 104.955906616041,
100.54527789393, 105.045438323192, 101.383868555427, 103.556929792691,
100.909647795285), .Dim = c(10L, 2L), .Dimnames = list(NULL,
c("0.01", "0.06")))

dput(max_values_row)
structure(c(9L, 2L), .Dim = 1:2, .Dimnames = list(NULL, c("0.01",
"0.06")))

so row 9 and row 2 has the. maximum values. which means w_9 = [ 0.22685307 0.08559372 0.38029680 0.30725641] is the one maximize my quantile function at tau = .01

How can I store those w that maximize immediately with out do it manually ? I need it because I have a big matrix for w and 99 taus in my real data

FactOREO · February 18, 2023, 7:16pm

You did not specifiy what kind of return you expect. Every weights row is a vector, should the output be a matrix with n rows, each row corresponding to a vector of weights which maximizes your function? Or should it be a long (or wide) data.frame? Or even a list of vectors?

Since a list would be safest, consider this function to use:

get_max_weights <- function(weights_mat, quant_reg_mat) {
    ### function to go through every column, find the maximum value index
    ### and return the row holding the corresponding weights

    # get the index of every maximizing row
    col_idx <- apply(X = quant_reg_mat, MARGIN =  2, FUN = which.max)

    # return a list of vectors
    lapply(col_idx, \(x) weights_mat[x,])
}

This will give you a list with a vector of weights maximizing your objective per element.

safara · February 18, 2023, 8:06pm

I am sorry I did not understand your question. But my main goal is to find which w vector maximize my quantile function. my quantile function depends on y and y depends on w.
w is a matrix of 10 x 4

y = w^T * r , r is 250 observation. y will be a matrix of 250 x 10

if you are familiar with quantile regression I need to specify tau value , assume I fitted the regression for
two values of tau. e.g tau=.01 and tau=.06 . as a final output I will have a matrix of 10 x 2 , each column for specific value of tau.

I am afraid that the way I code my problem is making it complicated to me to find the w vector directly ! I applied your function but I did not understand how to use it ? I just run it and it did not give me any result

FactOREO · February 18, 2023, 9:39pm

Okay, so my function uses your weights matrix (the one you provided with dput()) as well as your quantile regression results (your quant.matrix.reg1 object). It then gets the index with the maximum value from the latter and looks for the row from your weights matrix, where the weights for that value come from. The result is a list, with your given matrices I get the following output when running my function:

> get_max_weights(weights_mat = weights, quant_reg_mat = quantile_reg)
$`0.01`
[1] 0.22685307 0.08559372 0.38029680 0.30725641

$`0.06`
[1] 0.07454393 0.33205502 0.34916269 0.24423836

Each list element has the tau value as name and the weights vector as entry. If you wish to have a matrix with your values, you can use the following:

get_max_weights_as_matrix <- function(weights_mat, quant_reg_mat) {
    ### function to go through every column, find the maximum value index
    ### and return the row holding the corrsponsing weights

    # get the index of every maximizing row
    col_idx <- apply(quant_reg_mat, 2, which.max)

    # return a matrix
    rbind(sapply(col_idx, \(x) weights_mat[x,]))
}
get_max_weights_as_matrix(weights_mat = weights, quant_reg_mat = quantile_reg)

           0.01       0.06
[1,] 0.22685307 0.07454393
[2,] 0.08559372 0.33205502
[3,] 0.38029680 0.34916269
[4,] 0.30725641 0.24423836

Each column has your tau value as header and the weights as row entries. Since I've used the exact same data as you have provided (just renamed quant.matrix.reg1 as quantile_reg and w as weights), you should get the expected output with the full dataset. Note that I do not use everything beforehand, since you have said you only want to find the weights which correspond to the maximum value from your quantile regression result. Everything else doesn't matter for the concret question you've asked.

If you have to store the values inside your for-loops, it will be quite a mess (for-loops are a mess as well, better try to use the apply() family, but that's not the main point atm). You should split the tasks and calculate the weights and regression results beforehand and go through the results afterwards, e.g. with the function(s) I've provided above. As I understood your request, they do what you demanded. Just feed them with your matrices like you did in one of your posts above.

safara · February 19, 2023, 7:50am

Thank you very much for your help ,you saved my day!!!!

safara · February 19, 2023, 9:56am

Can I ask you another question please, if I want to calculate the quantile regression for different x values , is there any way to consider a function to apply it easily for each x rather than repeat the code over and over . e.g

to calculate the quantile of x1 we use

quant.matrix.reg1<- matrix(nrow = np1 ,ncol = length(tau) )
for( i in 1:np1){
  quant.matrix.reg1[i,]<- rq(all_yts1[,i][-1]~ x.reg[,1], tau)$rho
  colnames(quant.matrix.reg1)<- tau
}

and for x2 I use

quant.matrix.reg2<- matrix(nrow = np1 ,ncol = length(tau) )
for( i in 1:np1){
  quant.matrix.reg2[i,]<- rq(all_yts1[,i][-1]~ x.reg[,2], tau)$rho
  colnames(quant.matrix.reg2)<- tau
}

and so on. is there any way to do the same what you have done , by defining a function . 
At the end of the day, for each tau  I will use the weight matrix for x1,x1,... etc 

Thank you very much

FactOREO · February 19, 2023, 2:06pm

If I understand your code for the first value correctly (I don't use quantreg, so correct me if Iam wrong), you are calculation a result vector rho with

and assign it the the ith row in your quant.matrix.reg1 object to obtain a \text{np1} \times \tau matrix.
Therefore it should be enough to wrap your for-loop inside a function, which takes the correct column from your x.reg object as well as tau as an input and return this matrix, like this:

make_quant_reg_matrix <- function(idx = 1, x_vec, tau) {

    # I don't know what np1 is, but since you don't change it I assume it is a constant
    # and can be left as is, if you have assigned it in your globalEnv
    out <- matrix(nrow = np1, ncol = length(tau) )
    # do your calculations
    for ( i in seq_along(np1)) { #seq_along is safer than 1:np1
    # substituted x_vec and idx to make the function usable for apply family
    # all_yts has to be defined in your globalEnv
    out[i,] <- rq(all_yts1[,i][-1] ~ x_vec[,idx], tau)$rho
    colnames(out) <- tau
    }
    return(out) # return is optional, but makes it clear that the function is done at this step
}

This would give you the same result as your manual calculations, but with arguments to run through. You could then go ahead and e.g. use lapply() to get a list of result matrices.

safara · February 20, 2023, 10:11am

Thank you very much!

Yes this is correct.

so to run the function I should define the following in my golbalEnv : all_yts , tau , np1 which is just a constant number and x.reg which is a matrix of all my x.reg . then I write down

make_quant_reg_matrix(x_vec = x.reg,tau=tau )

but it gave me

structure(c(31.0132004757793, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 54.0491925630904,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 71.3050660449517, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 85.0811563210061,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 96.1868951458365, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 105.683565308846,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 114.502013778766, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 122.710941679345,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 130.465363007909, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 137.813266241493,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, 144.645465484747, NA, NA, NA, NA, NA, NA, NA, NA,
NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 151.133578923139,

this is just the first row of my matrix quantile_reg1 with NA for the rest of the matrix. Any advice is appreciated

FactOREO · February 22, 2023, 6:51pm

If only your x vector is varying, then this will work (hopefully with the correct results, had just a few minutes to think about your problem).:

### data
weights_mat <- structure(c(0.125317152319512, 0.0745439292667052, 0.586077416953187,
                           0.293766825283739, 0.250302495915074, 0.485794185165549, 0.254711032662707,
                           0.379498294249792, 0.226853071236893, 0.389874337579722, 0.175638364717147,
                           0.33205501745088, 0.0575595762802929, 0.164240433092228, 0.345972920695575,
                           0.110256795228951, 0.368038594116812, 0.148514686946719, 0.0855937166659081,
                           0.0529870782366488, 0.270380451403551, 0.349162694152531, 0.191884202849581,
                           0.329179547703677, 0.132554766707465, 0.338694291691177, 0.012763479721988,
                           0.210360477419355, 0.380296797258491, 0.355253292662155, 0.42866403155979,
                           0.244238359129883, 0.164478803916939, 0.212813193920356, 0.271169816681886,
                           0.065254727914323, 0.364486893498494, 0.261626541384135, 0.307256414838707,
                           0.201885291521473), .Dim = c(10L, 4L))
quant_reg_mat <- structure(c(31.0132004757793, 29.731942064456, 30.8044482731088,
                             29.3487557479765, 28.368598706984, 29.8900075294188, 28.1435428938265,
                             29.5148039317729, 31.7521366904934, 30.0404922387529, 105.683565308846,
                             107.373263664155, 98.988538212407, 102.537110474698, 104.955906616041,
                             100.54527789393, 105.045438323192, 101.383868555427, 103.556929792691,
                             100.909647795285), .Dim = c(10L, 2L), .Dimnames = list(NULL,
                                                                                    c("0.01", "0.06")))
### weight extracting function
get_max_weights_as_matrix <- function(weights_mat, quant_reg_mat) {
  ### function to go through every column, find the maximum value index
  ### and return the row holding the corrsponsing weights
  
  # get the index of every maximizing row
  col_idx <- apply(quant_reg_mat, 2, which.max)
  
  # return a matrix
  rbind(sapply(col_idx, \(x) weights_mat[x,]))
}

get_max_weights_as_matrix(weights_mat, quant_reg_mat)
#>            0.01       0.06
#> [1,] 0.22685307 0.07454393
#> [2,] 0.08559372 0.33205502
#> [3,] 0.38029680 0.34916269
#> [4,] 0.30725641 0.24423836

# define np1
np1 <- 10
tau <- seq.default(0.01,0.99,0.01)
y_mat <- matrix(sample(1:10,30,TRUE), nrow = 3)
x_vec1 <- c(5,4,3)
x_vec2 <- c(5,4,3)

huge_function <- function(x_vec,np1,tau,y_mat, weights) {
  # initialize result matrix
  quant_reg_mat <- matrix(nrow = np1, ncol = length(tau))
  for( i in seq.default(1,np1,1)){
    # quant_reg_mat[i,] <- rq(y_mat[,i][-1]~ x.reg[,1], tau)$rho
    # just as placeholder
    quant_reg_mat[i,] <- rnorm(length(tau))
  }
  # set colnames outside the loop
  colnames(quant_reg_mat) <- tau
  # get the optimal weights vector
  get_max_weights_as_matrix(weights_mat = weights, quant_reg_mat = quant_reg_mat)
}

lapply(list(x_vec1, x_vec2),huge_function,  np1, tau, y_mat, weights_mat)
#> [[1]]
#>           0.01       0.02       0.03      0.04       0.05       0.06       0.07
#> [1,] 0.2503025 0.38987434 0.22685307 0.2937668 0.48579419 0.58607742 0.48579419
#> [2,] 0.3459729 0.05298708 0.08559372 0.1642404 0.11025680 0.05755958 0.11025680
#> [3,] 0.1325548 0.35525329 0.38029680 0.3291795 0.33869429 0.19188420 0.33869429
#> [4,] 0.2711698 0.20188529 0.30725641 0.2128132 0.06525473 0.16447880 0.06525473
#>            0.08      0.09       0.1       0.11       0.12       0.13      0.14
#> [1,] 0.07454393 0.1253172 0.1253172 0.22685307 0.58607742 0.48579419 0.3794983
#> [2,] 0.33205502 0.1756384 0.1756384 0.08559372 0.05755958 0.11025680 0.1485147
#> [3,] 0.34916269 0.2703805 0.2703805 0.38029680 0.19188420 0.33869429 0.2103605
#> [4,] 0.24423836 0.4286640 0.4286640 0.30725641 0.16447880 0.06525473 0.2616265
#>            0.15       0.16      0.17       0.18       0.19        0.2      0.21
#> [1,] 0.22685307 0.58607742 0.3794983 0.48579419 0.48579419 0.48579419 0.2937668
#> [2,] 0.08559372 0.05755958 0.1485147 0.11025680 0.11025680 0.11025680 0.1642404
#> [3,] 0.38029680 0.19188420 0.2103605 0.33869429 0.33869429 0.33869429 0.3291795
#> [4,] 0.30725641 0.16447880 0.2616265 0.06525473 0.06525473 0.06525473 0.2128132
#>            0.22       0.23       0.24       0.25       0.26       0.27
#> [1,] 0.58607742 0.22685307 0.58607742 0.22685307 0.25471103 0.07454393
#> [2,] 0.05755958 0.08559372 0.05755958 0.08559372 0.36803859 0.33205502
#> [3,] 0.19188420 0.38029680 0.19188420 0.38029680 0.01276348 0.34916269
#> [4,] 0.16447880 0.30725641 0.16447880 0.30725641 0.36448689 0.24423836
#>           0.28       0.29       0.3       0.31      0.32       0.33       0.34
#> [1,] 0.2503025 0.38987434 0.2503025 0.07454393 0.1253172 0.22685307 0.38987434
#> [2,] 0.3459729 0.05298708 0.3459729 0.33205502 0.1756384 0.08559372 0.05298708
#> [3,] 0.1325548 0.35525329 0.1325548 0.34916269 0.2703805 0.38029680 0.35525329
#> [4,] 0.2711698 0.20188529 0.2711698 0.24423836 0.4286640 0.30725641 0.20188529
#>            0.35      0.36      0.37       0.38      0.39       0.4       0.41
#> [1,] 0.58607742 0.1253172 0.3794983 0.25471103 0.1253172 0.2937668 0.58607742
#> [2,] 0.05755958 0.1756384 0.1485147 0.36803859 0.1756384 0.1642404 0.05755958
#> [3,] 0.19188420 0.2703805 0.2103605 0.01276348 0.2703805 0.3291795 0.19188420
#> [4,] 0.16447880 0.4286640 0.2616265 0.36448689 0.4286640 0.2128132 0.16447880
#>            0.42      0.43      0.44      0.45       0.46      0.47       0.48
#> [1,] 0.48579419 0.2937668 0.2503025 0.2937668 0.38987434 0.2937668 0.38987434
#> [2,] 0.11025680 0.1642404 0.3459729 0.1642404 0.05298708 0.1642404 0.05298708
#> [3,] 0.33869429 0.3291795 0.1325548 0.3291795 0.35525329 0.3291795 0.35525329
#> [4,] 0.06525473 0.2128132 0.2711698 0.2128132 0.20188529 0.2128132 0.20188529
#>           0.49        0.5      0.51       0.52      0.53      0.54       0.55
#> [1,] 0.1253172 0.58607742 0.1253172 0.07454393 0.1253172 0.3794983 0.58607742
#> [2,] 0.1756384 0.05755958 0.1756384 0.33205502 0.1756384 0.1485147 0.05755958
#> [3,] 0.2703805 0.19188420 0.2703805 0.34916269 0.2703805 0.2103605 0.19188420
#> [4,] 0.4286640 0.16447880 0.4286640 0.24423836 0.4286640 0.2616265 0.16447880
#>            0.56       0.57       0.58      0.59       0.6      0.61       0.62
#> [1,] 0.38987434 0.07454393 0.25471103 0.2503025 0.2503025 0.1253172 0.48579419
#> [2,] 0.05298708 0.33205502 0.36803859 0.3459729 0.3459729 0.1756384 0.11025680
#> [3,] 0.35525329 0.34916269 0.01276348 0.1325548 0.1325548 0.2703805 0.33869429
#> [4,] 0.20188529 0.24423836 0.36448689 0.2711698 0.2711698 0.4286640 0.06525473
#>           0.63       0.64       0.65      0.66       0.67       0.68       0.69
#> [1,] 0.1253172 0.48579419 0.58607742 0.3794983 0.07454393 0.07454393 0.22685307
#> [2,] 0.1756384 0.11025680 0.05755958 0.1485147 0.33205502 0.33205502 0.08559372
#> [3,] 0.2703805 0.33869429 0.19188420 0.2103605 0.34916269 0.34916269 0.38029680
#> [4,] 0.4286640 0.06525473 0.16447880 0.2616265 0.24423836 0.24423836 0.30725641
#>             0.7      0.71      0.72       0.73      0.74       0.75       0.76
#> [1,] 0.22685307 0.2937668 0.1253172 0.38987434 0.1253172 0.22685307 0.58607742
#> [2,] 0.08559372 0.1642404 0.1756384 0.05298708 0.1756384 0.08559372 0.05755958
#> [3,] 0.38029680 0.3291795 0.2703805 0.35525329 0.2703805 0.38029680 0.19188420
#> [4,] 0.30725641 0.2128132 0.4286640 0.20188529 0.4286640 0.30725641 0.16447880
#>            0.77       0.78       0.79        0.8      0.81      0.82      0.83
#> [1,] 0.58607742 0.48579419 0.48579419 0.25471103 0.2937668 0.1253172 0.2937668
#> [2,] 0.05755958 0.11025680 0.11025680 0.36803859 0.1642404 0.1756384 0.1642404
#> [3,] 0.19188420 0.33869429 0.33869429 0.01276348 0.3291795 0.2703805 0.3291795
#> [4,] 0.16447880 0.06525473 0.06525473 0.36448689 0.2128132 0.4286640 0.2128132
#>            0.84       0.85      0.86       0.87      0.88      0.89       0.9
#> [1,] 0.48579419 0.58607742 0.1253172 0.22685307 0.2503025 0.3794983 0.2937668
#> [2,] 0.11025680 0.05755958 0.1756384 0.08559372 0.3459729 0.1485147 0.1642404
#> [3,] 0.33869429 0.19188420 0.2703805 0.38029680 0.1325548 0.2103605 0.3291795
#> [4,] 0.06525473 0.16447880 0.4286640 0.30725641 0.2711698 0.2616265 0.2128132
#>            0.91      0.92       0.93       0.94       0.95      0.96      0.97
#> [1,] 0.58607742 0.2937668 0.58607742 0.22685307 0.25471103 0.2937668 0.1253172
#> [2,] 0.05755958 0.1642404 0.05755958 0.08559372 0.36803859 0.1642404 0.1756384
#> [3,] 0.19188420 0.3291795 0.19188420 0.38029680 0.01276348 0.3291795 0.2703805
#> [4,] 0.16447880 0.2128132 0.16447880 0.30725641 0.36448689 0.2128132 0.4286640
#>            0.98      0.99
#> [1,] 0.48579419 0.2937668
#> [2,] 0.11025680 0.1642404
#> [3,] 0.33869429 0.3291795
#> [4,] 0.06525473 0.2128132
#> 
#> [[2]]
#>            0.01       0.02      0.03       0.04       0.05      0.06       0.07
#> [1,] 0.22685307 0.25471103 0.2503025 0.22685307 0.22685307 0.2503025 0.22685307
#> [2,] 0.08559372 0.36803859 0.3459729 0.08559372 0.08559372 0.3459729 0.08559372
#> [3,] 0.38029680 0.01276348 0.1325548 0.38029680 0.38029680 0.1325548 0.38029680
#> [4,] 0.30725641 0.36448689 0.2711698 0.30725641 0.30725641 0.2711698 0.30725641
#>            0.08       0.09        0.1       0.11       0.12       0.13
#> [1,] 0.48579419 0.38987434 0.58607742 0.38987434 0.48579419 0.38987434
#> [2,] 0.11025680 0.05298708 0.05755958 0.05298708 0.11025680 0.05298708
#> [3,] 0.33869429 0.35525329 0.19188420 0.35525329 0.33869429 0.35525329
#> [4,] 0.06525473 0.20188529 0.16447880 0.20188529 0.06525473 0.20188529
#>            0.14       0.15      0.16       0.17       0.18      0.19        0.2
#> [1,] 0.25471103 0.25471103 0.2937668 0.38987434 0.38987434 0.1253172 0.22685307
#> [2,] 0.36803859 0.36803859 0.1642404 0.05298708 0.05298708 0.1756384 0.08559372
#> [3,] 0.01276348 0.01276348 0.3291795 0.35525329 0.35525329 0.2703805 0.38029680
#> [4,] 0.36448689 0.36448689 0.2128132 0.20188529 0.20188529 0.4286640 0.30725641
#>           0.21      0.22       0.23      0.24       0.25       0.26      0.27
#> [1,] 0.3794983 0.1253172 0.48579419 0.2503025 0.25471103 0.38987434 0.2503025
#> [2,] 0.1485147 0.1756384 0.11025680 0.3459729 0.36803859 0.05298708 0.3459729
#> [3,] 0.2103605 0.2703805 0.33869429 0.1325548 0.01276348 0.35525329 0.1325548
#> [4,] 0.2616265 0.4286640 0.06525473 0.2711698 0.36448689 0.20188529 0.2711698
#>           0.28       0.29       0.3       0.31      0.32      0.33       0.34
#> [1,] 0.3794983 0.38987434 0.3794983 0.22685307 0.2937668 0.1253172 0.58607742
#> [2,] 0.1485147 0.05298708 0.1485147 0.08559372 0.1642404 0.1756384 0.05755958
#> [3,] 0.2103605 0.35525329 0.2103605 0.38029680 0.3291795 0.2703805 0.19188420
#> [4,] 0.2616265 0.20188529 0.2616265 0.30725641 0.2128132 0.4286640 0.16447880
#>           0.35       0.36       0.37       0.38      0.39       0.4       0.41
#> [1,] 0.3794983 0.48579419 0.22685307 0.25471103 0.2503025 0.2937668 0.58607742
#> [2,] 0.1485147 0.11025680 0.08559372 0.36803859 0.3459729 0.1642404 0.05755958
#> [3,] 0.2103605 0.33869429 0.38029680 0.01276348 0.1325548 0.3291795 0.19188420
#> [4,] 0.2616265 0.06525473 0.30725641 0.36448689 0.2711698 0.2128132 0.16447880
#>            0.42       0.43       0.44      0.45      0.46       0.47       0.48
#> [1,] 0.48579419 0.07454393 0.48579419 0.2503025 0.3794983 0.58607742 0.58607742
#> [2,] 0.11025680 0.33205502 0.11025680 0.3459729 0.1485147 0.05755958 0.05755958
#> [3,] 0.33869429 0.34916269 0.33869429 0.1325548 0.2103605 0.19188420 0.19188420
#> [4,] 0.06525473 0.24423836 0.06525473 0.2711698 0.2616265 0.16447880 0.16447880
#>           0.49        0.5       0.51      0.52       0.53       0.54       0.55
#> [1,] 0.2503025 0.07454393 0.07454393 0.1253172 0.58607742 0.58607742 0.22685307
#> [2,] 0.3459729 0.33205502 0.33205502 0.1756384 0.05755958 0.05755958 0.08559372
#> [3,] 0.1325548 0.34916269 0.34916269 0.2703805 0.19188420 0.19188420 0.38029680
#> [4,] 0.2711698 0.24423836 0.24423836 0.4286640 0.16447880 0.16447880 0.30725641
#>           0.56       0.57       0.58       0.59        0.6       0.61
#> [1,] 0.3794983 0.48579419 0.07454393 0.38987434 0.58607742 0.25471103
#> [2,] 0.1485147 0.11025680 0.33205502 0.05298708 0.05755958 0.36803859
#> [3,] 0.2103605 0.33869429 0.34916269 0.35525329 0.19188420 0.01276348
#> [4,] 0.2616265 0.06525473 0.24423836 0.20188529 0.16447880 0.36448689
#>            0.62      0.63      0.64      0.65       0.66       0.67       0.68
#> [1,] 0.25471103 0.3794983 0.2937668 0.2937668 0.25471103 0.48579419 0.25471103
#> [2,] 0.36803859 0.1485147 0.1642404 0.1642404 0.36803859 0.11025680 0.36803859
#> [3,] 0.01276348 0.2103605 0.3291795 0.3291795 0.01276348 0.33869429 0.01276348
#> [4,] 0.36448689 0.2616265 0.2128132 0.2128132 0.36448689 0.06525473 0.36448689
#>            0.69        0.7       0.71       0.72      0.73       0.74
#> [1,] 0.07454393 0.38987434 0.25471103 0.07454393 0.2937668 0.38987434
#> [2,] 0.33205502 0.05298708 0.36803859 0.33205502 0.1642404 0.05298708
#> [3,] 0.34916269 0.35525329 0.01276348 0.34916269 0.3291795 0.35525329
#> [4,] 0.24423836 0.20188529 0.36448689 0.24423836 0.2128132 0.20188529
#>            0.75       0.76       0.77      0.78       0.79        0.8      0.81
#> [1,] 0.07454393 0.38987434 0.38987434 0.1253172 0.58607742 0.58607742 0.3794983
#> [2,] 0.33205502 0.05298708 0.05298708 0.1756384 0.05755958 0.05755958 0.1485147
#> [3,] 0.34916269 0.35525329 0.35525329 0.2703805 0.19188420 0.19188420 0.2103605
#> [4,] 0.24423836 0.20188529 0.20188529 0.4286640 0.16447880 0.16447880 0.2616265
#>            0.82       0.83      0.84      0.85      0.86      0.87      0.88
#> [1,] 0.25471103 0.38987434 0.2503025 0.1253172 0.2937668 0.3794983 0.2503025
#> [2,] 0.36803859 0.05298708 0.3459729 0.1756384 0.1642404 0.1485147 0.3459729
#> [3,] 0.01276348 0.35525329 0.1325548 0.2703805 0.3291795 0.2103605 0.1325548
#> [4,] 0.36448689 0.20188529 0.2711698 0.4286640 0.2128132 0.2616265 0.2711698
#>            0.89        0.9      0.91      0.92      0.93       0.94       0.95
#> [1,] 0.48579419 0.22685307 0.1253172 0.2937668 0.1253172 0.07454393 0.58607742
#> [2,] 0.11025680 0.08559372 0.1756384 0.1642404 0.1756384 0.33205502 0.05755958
#> [3,] 0.33869429 0.38029680 0.2703805 0.3291795 0.2703805 0.34916269 0.19188420
#> [4,] 0.06525473 0.30725641 0.4286640 0.2128132 0.4286640 0.24423836 0.16447880
#>            0.96       0.97       0.98       0.99
#> [1,] 0.07454393 0.07454393 0.07454393 0.22685307
#> [2,] 0.33205502 0.33205502 0.33205502 0.08559372
#> [3,] 0.34916269 0.34916269 0.34916269 0.38029680
#> [4,] 0.24423836 0.24423836 0.24423836 0.30725641

^{Created on 2023-02-22 by the reprex package (v2.0.1)}

But double check it, I really don't know enough about your specific data structures, their behaviours and your expectations. But it should give you an idea how to build an appliable function to use something like lapply().

safara · February 27, 2023, 1:03pm

Thank you very much!

If I want to repeat the above code but with different return period sets. for example my first return period was start from "2019-01-01", to="2019-11-30" , I want to repeat the same process but with period start from "2019-02-01", to="2019-12-30" , then repeat it again with another period start of Mach 2019 an so on . Do you have any idea how to do that ? I searched about this process and found it called rolling window but I could not figure out how to define the whole process as one function where I just need to change the return period.

system · March 6, 2023, 1:04pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.