I ran three model-building procedures with different parameters on the same sample and obtained the selection of my optimized hyperparameter for each outer fold (each of the analyses had 100 outer folds in my nested-cross validation). This gives me a contingency table, with the models as rows and the absolute frequencies for each of the possible hyperparameter categories as columns. For the sake of simplicity, let's call the rows day 1, day 2, and day 3 and let the columns be different ice cream sorts and let the 100 folds be the same sample of people buying ice cream on three different days:
As you can 'see', on in order of the days, the distribution becomes less 'spread out'.
I would like to measure this difference in variance, but I am not sure, which test to used. I came upon Chi2, McNemar and Cochrane's Q. Chi2 is not for repeated measures, McNemar only works for 2x2 tables and McNemar & Cochrane's Q are apparently designed for binary responses. Does anybody know which test might be appropriate for my contingency table?
I could not embed the image in my post, that's why I just pasted the link.
Right-click the image on the image hoster website to get the ful link (https://i.imgur.com/Hjt6RBO.png), then paste this into the image field. I made the change for you.
Even though we have some members here with statistics expertise I guess you could get more responses when posting this over at Cross-Validated: https://stats.stackexchange.com/
Awesome, thank you! :)
Good idea! I tried it already on https://datascience.stackexchange.com/, but this forum seems to be quite inactive. Maybe https://stats.stackexchange.com/ is a better forum for my question.