Norfolk is a town in Litchfield County , Connecticut , United States . The population was 1,660 at the 2000 census. Norfolk is perhaps best known as the site of the summer campus of the Yale University School of Music, which hosts an annual chamber music concert series in "the Music Shed," a performance hall located on the Ellen Battell Stoeckel estate to the west of the village green. Norfolk also boasts important examples of regional architecture, notably The Norfolk Library (a Romanesque Revival structure by George Keller, 1888/9) and over thirty buildings, in a wide variety of styles, designed by Alfredo S.G. Taylor (of the New York firm Taylor & Levi) in the four decades before the Second World War.