The Generating Function of Ternary Trees and Continued Fractions

Abstract

Michael Somos conjectured a relation between Hankel determinants whose entries ${1\over 2n+1}{3n\choose n}$ count ternary trees and the number of certain plane partitions and alternating sign matrices. Tamm evaluated these determinants by showing that the generating function for these entries has a continued fraction that is a special case of Gauss's continued fraction for a quotient of hypergeometric series. We give a systematic application of the continued fraction method to a number of similar Hankel determinants. We also describe a simple method for transforming determinants using the generating function for their entries. In this way we transform Somos's Hankel determinants to known determinants, and we obtain, up to a power of $3$, a Hankel determinant for the number of alternating sign matrices. We obtain a combinatorial proof, in terms of nonintersecting paths, of determinant identities involving the number of ternary trees and more general determinant identities involving the number of $r$-ary trees.