The science manual is included in the latest Dev download zip file. Chapter 6 details the mixing model.
The original paper on this is
Hodges, B.R., Imberger, J., Saggio, A. & Winters, K.B. 2000 Modeling basin-scale internal waves in a stratified lake. Limnol. Oceanogr. 45, pp. 1603–1620.