Subgroup analysis for longitudinal data based on a partial linear varying coefficient model with a change plane

Abstract

Subgroup analysis has become an important tool to characterize the treatment effect heterogeneity, and finally towards precision medicine. On the other hand, longitudinal study is widespread in many fields, but subgroup analysis for this data type is still limited. In this article, we study a partial linear varying coefficient model with a change plane, in which the subgroups are defined based on linear combination of grouping variables, and the time‐varying effects in different subgroups are estimated to capture the dynamic association between predictors and response. The varying coefficients are approximated by basis functions and the group indicator function is smoothed by kernel function, which are included in the generalized estimating equation for estimation. Asymptotic properties of the estimators for the varying coefficients, the constant coefficients and the change plane coefficients are established. Simulations are conducted to demonstrate the flexibility, efficiency and robustness of the proposed method. Based on the Standard and New Antiepileptic Drugs study, we successfully identify a subgroup in which patients are sensitive to the newer drug in a specific period of time.