I may never find the answer as to why the rest of the family stayed behind or why he moved, unless I am lucky to find letters preserved somewhere. It could happen, but I'm not holding my breath. Trying to figure out when he migrated is an easier question to tackle and find a decent answer to. I turned to the tax records of Knox County.
To do this I went to the Tennessee State Library and Archives. They have the microfilm of these records and a very helpful staff to point you in the right direction. The Luttrell family and its branches are large, and they had a terrible habit of using the same names over and over. When viewing the microfilm it seemed this had potential to confuse the officials too. There was a notation next to Hugh's father, James C. Luttrell, of "Jr." William Luttrell was the father of James, so he was not technically a "junior." However, James had an uncle named....wait for it.... James C. Luttrell. It didn't help that there seemed to be a few people named Hugh. It is enough to make your head spin.
Anyway, I finally figured it all out and the last record I could find Hugh listed on was in 1854. The records then skipped to 1856 and he is not there, nor in the following years. So it seems he left some time between the two tax records. To confirm this I will have to turn to the Missouri tax records. First I will have to figure out what county he was living in. I haven't found him on the 1860 census yet. That may take a little time.
So the point is; if your ancestor shows up in two different places consecutive census years and you want to figure out when he moved, look at the tax records. The government was very diligent about wanting their cut, even back then.