If subgroups with different prognoses are identified, was there adjustment for important prognostic factors and was there validation in an independent, "test set" of patients?
We often want to know if patients with certain characteristics will have a different prognosis. For example, are patients with an intracranial hemorrhage at increased risk of seizure? Demographic, disease-specific or comorbid variables that are associated with the outcome of interest are called prognostic factors. They need not be causal but must be strongly enough associated with the development of an outcome to predict its occurrence.
The identification of a prognostic factor for the first time could be the result of a chance difference in its distribution between patients with different prognoses. Therefore, the initial patient group in which the variable was identified as a prognostic factor may be considered to be a training set or a hypothesis generation set. Indeed, if investigators were to search for multiple potential prognostic factors in the same data set, a few would likely emerge on the basis of chance alone. Ideally, therefore, data from a second independent patient group, or a "test set" would be required to confirm the importance of a prognostic factor. Although this degree of evidence has often not been collected in the past, an increasing number of reports are describing a second, independent study validating the predictive power of prognostic factors. If a second, independent study validates these prognostic factors, it can be called a clinical prediction guide.
In the study we found, the investigators looked at patients with different stroke types and identified that patients in these groups had different risks of seizures. This was not tested in an independent group of patients to see if it holds true.
If the study fails any of the above criteria, we need to consider if the flaw is significant and threatens the validity of the study. If this is the case, we'll need to look for another study. Returning to our clinical scenario, the paper we found satisfies all of the above criteria and we will proceed to assessing it for importance.
- Was a defined, representative sample of patients assembled at a common (usually early) point in the course of their disease?
- Was patient follow-up sufficiently long and complete?
- Were objective outcome criteria applied in a "blind" fashion?
If subgroups with different prognoses are identified:
- Was there adjustment for important prognostic factors?
- Was there validation in an independent group of "test-set" patients?