Coxcomb was once spelled cockscomb. The cock’s comb in question was the traditional jester’s cap, which which had a serrated red crest rather like the one on a rooster. A cockscomb was therefore a jester or fool, in the professional sense of that last word.
Around the middle of the sixteenth century, the word became a term for the head. Charles Kingsley used it three centuries later in his Westward Ho!, though it was by then essentially obsolete: “That slate descended on the bald coxcomb of Sir Vindex Brimblecombe, with so shrewd a blow that slate and pate cracked at the same instant, and the poor pedagogue dropped to the floor, and lay for dead.”
At about the same period (the 1570s), it came to mean a simpleton or fool, then a foolish, conceited, showy, vain person, “a fop; a superficial pretender to knowledge or accomplishments”, as Samuel Johnson defined the type in his Dictionary. Through the following two centuries it was a common term of disparagement for a man whose exterior display of fashion disguised the superficiality within. It was often qualified by adjectives such as “impertinent”, “arrant”, and “egregious” (in the negative sense of outstandingly bad).
George Thompson of New York University found a description in the Commercial Advertiser of June 1800 which suggested that the type flourished in America as well as in Britain, though the style was different: “Receipt to make a New-York COXCOMB. Take a creature of any age, from 12 to 21, cut off its hair close to its head, excepting a little bit resembling a small kitten’s tail, dress it in large white trowsers, and high top bootees and putting a lighted segar [cigar] in its mouth, set it a-walking among the ladies at Corres’ gardens. The more smoke it puffs in the faces of the company the better.”
So what started out as a name for a professional comic turned into one for a figure of fun.