The large main house where retreats and conferences are held was built during World War I by Dr. Horace Packard, professor of medicine and surgery at Boston University.  He turned to the East and was honored by the government of Japan for bringing Japanese students to the country to study.  The house was built with lumber imported from Japan, and the screen-like interior design of its walls and windows gives it an oriental architectural attitude: each space opens out into another.

A commitment to service soon opened other vistas.  During the 40s Mrs. Crawford, a wise financier who helped to save Dr. Packard's fortune during the Crash and Depression, and one of the first African-American women to attend an Ivy League college, ran an orphanage here.  In the 50s a group of Protestant clergymen redefined the Manse as a camp, the “Christian Fellowship”, to be used for conferences of renewal for Protestant churches.  They soon realized that the desired renewal would not occur without facing the core problem of Christian division, thus determining the destiny of Packard Manse as an ecumenical center, a place looking for the health of the whole household of faith.

Members of the community of Taiz, whose recovery of monasticism for Protestants was influenced by Catholic Benedictine and Cistercian spirituality, came to help the household of the Manse work out its mission to overcome the separation of Christians.  The sun porch was their place of prayer, marked now by the monks' gift of a stained glass window portraying the circle of Christ and his disciples.

With a re-writing of the corporate charter of the Manse, it became the first ecumenical center in the United States for which Protestants and Catholics together were co-responsible.  One of the earliest activities was a sustained Jewish-Christian Dialogue.

In the great social spirit of the 1960s and early 70s, the Manse sought to address other kinds of division.  It became the coördinating center in the Boston area of the civil rights and peace movements and the scene of a collective experiment.

In the late 70s the zeitgeist of the Movement passed and the household underwent a certain dispersion.  Thinking changed.

During the 80s and early 90s, those who remained at the Manse were absorbed for life's sake within some of the very systems it had been their part to challenge a decade earlier.  A profound awareness of the systemic injustice affecting the lives of women took root and sought and found positive expression in various forms of service consistent with the Manse's abiding concern for those at the margins of our society.

Throughout this time, the retreat center has continued to offer hospitality to groups as diverse as youth from inner city parishes, families, college and university men and women, people in recovery, the elderly, health care providers, individuals and families who are affected by HIV and related issues, immigrants, and others whose lives are in transition.