Oh my goodness, @jbratt 
What a surprise!
(We worked together at a previous job.)
@DPaschall I haven't tried ELMo on the same datasets that we are using with ULMFit so I don't know if I can speak to a direct performance comparison at this point. However, we are doing something similar where we have a large dataset of domain-specific language, and then a quite small dataset of labeled data for the classifier. It's remarkable what good results we are getting!