Certain factions within my organization are not trusting me but rather waging calculated offenses against me because they estimate my effectiveness as a leader is marginal because of my size.
I see no such deficiency.
Leaders throughout history have been of all sizes. Napoleon was 5 feet 6 1/2 inches tall. If the advantages of height were more significant than the disadvantages, then we would see every species continuing to enlarge without end. In fact, as height increases, the supportability of a body decreases, as supportability goes as the square of linear size divided by the cube, or resultantly the inverse of linear size. The square determines the strength of the frame to support its weight. The cube determines the volume, and therefore the mass, which must be supported by the slower rising area.
A man who understands the role played in warfare by intelligence will always fare better among men than a man who doesn't, on average.
Consider your world, gentlemen, but be comprehensive.