In this paper we propose and analyze a stabilization-free hybridizable discontinuous Galerkin (HDG) method in stress-velocity formulation for fluid-structure interaction based on Alfeld splits. A unified mixed formulation is employed for the Stokes equations and the elastodynamic equations. We use the standard polynomial space with strong symmetry to define the stress space, and use the broken
H
(
d
i
v
;
Ω
H(div;\Omega
)-conforming space of the same degree to define the vector space in a careful way such that the resulting scheme is stable without resorting to any stabilization. In particular, the proposed scheme addresses the pressure-robustness, which is known to be important for incompressible flows. The transmission conditions can be incorporated naturally without resorting to additional variables or Nitsche-type stabilization owing to the bespoke construction of the discrete formulation. To show the optimal convergence, we establish a new projection operator for the stress space whose definition accounts for traces of the method. Furthermore, the pressure-independence and the robustness with respect to fluid viscosity and the Lamé constants are investigated. We also show the characterization of the hybridization and the size of the global system is greatly reduced, rendering the scheme computationally attractive. Several numerical experiments are presented to verify the proposed theories.