Nobody would believe Thomas
Because even though he was telling Thomas that he believed him, he knew that not many people outside of the family would have that same attitude. They would probably laugh his brother out of the building, and Bryan simply did not want to risk that. He cared about Thomas too much.