Modern machine learning research reminds me a lot of particle physics research though this is partly bias due to having only read one book on sociology of science. There is a lot of fitting of interpretation to empirical results (not to be confused with ML itself which is also the fitting of models) which is very different to other branches of CS.

It's interesting how many methods are used as a result of social forces as much as empirical results. Methods come in and out of vogue without much impact on quality. I've read a few papers doing ablation tests and there is a common pattern that much of common practice can be replaced with alternatives or even thrown out completely. I'm never sure how important things like dropout or batch norm or choice of activation function really are.

As a colleague just pointed out to me, some of practice may be shaped simply by the fact that researchers are in a hurry to get papers out and need to have a paragraph or two on interpretation of results. The paper gets accepted because the methods work but now the interpretation work is also published, a bit like a rider bill, even though it might not be justified.