A contrasting cardigan, even in a conservative color, makes a big difference in your visual profile.  I choose brighter colors to go along with the shirt or tie.  Pocket square then should match either shirt, cardigan, or the secondary color in the tie (not the primary color)
{}