Not all correspondences are equally arbitrary, and meaning is not entirely a human construction. Consider the association of the directions with the seasons. These are far from arbitrary. The east is the place where the sun rises, and as such bears a natural correspondence to all kinds of new beginnings, including the new beginning of spring. Likewise, the south is the place of greatest solar intensity, and hence a natural fit for summer, etc, etc. Or consider the correspondence of red to various hot, passionate matters, and blue to various cool, reflective ones. Again, not arbitrary. Red is the color of blood and generally of fire, while blue is the color of the sky and deep, clean water. These meanings are basically inscribed in nature.
Of course, not all correspondences are equally compelling, and once you start making large networks of correspondences there are often clusters and chains of association that could have been drawn in other ways. But these correspondences are often a more-or-less intimate part of a particular traditions, and adopting and working with a particular system of correspondences is a part of immersing yourself in a tradition, thus creating a context for that tradition to speak its riches to you.