In reply to Jon Dickens.

Hi Jon, in practice, there are many comparisons of different models on different datasets. For example, comparing metrics of different models on a validation dataset to find the "best" model, or comparing metrics of a model on different validation datasets from different time periods to see if performance holds out of period. These are done to understand model performance/stability on known datasets.

There is another situation where one does not have control over the datasets, rather the task is to assess quality of different datasets. For example, you might have a credit portfolio and a trusted risk model, and you may be asked to evaluate the quality of another portfolio. To the extent you believe the model is reliable, you can compare the score distributions of the datasets to understand their relative quality. This is done to evaluate the risk of different credit portfolios all the time in practice.