Evidence based on test content, response process and internal structure: Validity testing of the eHealth Literacy Questionnaire (eHLQ) in Australian community health settings Part 2 (Preprint)

Abstract

Digital technologies have changed how we manage our health and eHealth literacy is needed to engage with health technologies. Any eHealth strategy would be ineffective if the eHealth literacy needs of users are not addressed. Hence, a robust measure of eHealth literacy is essential for understanding these needs. Based on the eHealth Literacy Framework (eHLF) which identified 7 dimensions of eHealth literacy, the eHealth Literacy Questionnaire (eHLQ) was developed. The tool has demonstrated robust psychometric properties in the Danish setting, but validity testing should be an ongoing and accumulative process.

This study aimed to evaluate validity evidence based on test content, response process and internal structure of the eHLQ in the Australian community health setting.

A mixed-method approach was used with cognitive interviewing conducted to examine evidence on test content and response process while a cross-sectional survey was undertaken for evidence on internal structure. Data were collected at 3 diverse community health sites in Victoria, Australia. Psychometric testing included both classical test theory and item response theory (IRT) approaches. Methods included Bayesian structural equation modelling (BSEM) for confirmatory factor analysis, internal consistency and test-retest for reliability and Bayesian multiple indicators multiple causes (MIMIC) model for testing of differential item functioning (DIF).

Cognitive interviewing identified only one confusing term and this was clarified. All items were easy to read and understood as intended. A total of 525 completed questionnaires were included for psychometric analysis. All scales were homogenous with composite scale reliability ranging from .73 to .90. Intraclass Correlation Coefficient for test-retest for the 7 scales ranged from 0.72 to 0.95. A 7-factor BSEM using small variance priors for cross-loadings and residual covariances was fitted to the data and the model of interest produced a satisfactory fit (posterior productive P value=.49, 95% CI for the difference between observed and replicated Chi-square values=-101.40-108.83, prior-posterior productive P value =.92). All items loaded on the relevant factor, with loadings ranging from 0.36 to 0.94. No significant cross-loading was found. There was no evidence of DIF for administration format, site area and health setting. However, discriminant validity was not well-established for Scales 1, 3, 5, 6 and 7. IRT analysis found all items provided precise information at different trait levels except for one item. All items demonstrated different sensitivity to different trait levels and represented a range of difficulty levels.

The evidence suggests that the eHLQ is a tool with robust psychometric properties while further investigation of the discriminant validity among some of the scales is recommended. It is ready to be used to identify eHealth literacy strengths and challenges and assist the development of digital health interventions to ensure people with limited digital access and skills are not left behind.