Scalable multimodal assessment of the micro-neighborhood using orthogonal visual inputs

Abstract

AbstractThe features of the micro-location and in particular the micro-neighborhood that residents perceive on a daily basis have a considerable influence on the quality of living and also on housing prices. For automated valuation models (AVMs), the use of micro-neighborhood information would be beneficial, as incorporating additional spatial effects into the price estimate could potentially reduce the empirical error. However, measuring related features is difficult, as they must first be defined and then collected, which is extremely challenging at such a small spatial level. In this study, we investigate the extent to which the quality of micro-neighborhoods can be assessed holistically using multiple data modalities. We design a scalable approach using alternative data (images and text), with the potential to expand coverage to other urban regions. To achieve this, we propose a multimodal deep learning architecture that integrates both textual and visual inputs and fuses this information. In addition, we introduce a training strategy that enables a targeted fusion of orthogonal visual representations of the residential area within the model architecture. In our experiments, we test and compare different unimodal models with our multimodal architectures. The results demonstrate that the multimodal model with targeted fusion of the orthogonal visual inputs achieves the best performance and also improves the prediction accuracy for underrepresented location quality classes.