The moss will hold moisture against the wall. Concrete, bricks, stone and mortar are all porous and will absorb moisture. When the embedded moisture freezes is will expand and can damage the building materials. Thats just plain physics. I like moss and I think it can look appealing growing on the ground and on tree trunks and what not. However, you are in CT and it does get very cold there in the winter. Moisture saturated masonry and freezing temperatures are not good and can lead to spalling of the masonry units. The moss should be removed.