The Queen Anne Revival design of 2 O'Connor Drive incorporates several striking Dutch gables into the mansion's south and west façades, a feature more common in Jacobean architecture.

The house was built in 1885 from red brick made at the nearby Don Valley Pressed Brick Company (later renamed Don Valley Brickworks) for the facility's owner, business magnate John F. Taylor. It was designed by Scottish-Torontonian architect David Brash Dick. A founding family of East York, the Taylors emigrated to Vaughan, Upper Canada, from Staffordshire, England, in the 1820s. They also owned a paper mill, Todmorden Mills, and a brewery.

In 1903, the property was purchased by Robert L. Patterson – a manufacturer of print type who was presumably related to John F. Taylor's wife, Elizabeth Patterson – who christened it “Fernwood.” In 1930, the site was subsequently sold to the United Church of Canada and converted into the Ina Grafton Gage Home for the Aged, which was established with funds donated by the publisher Sir William Gage in memory of his wife, Ina Grafton Gage. 

In 2013, the Sisters of St. Joseph Toronto incorporated the building into the construction of their new residence, which was designed by Shim-Sutcliffe Architects. ERA Architects provided heritage consultation for the process, which included restoring original features using drawings and watercolour paintings from the Archives of Ontario.

• Architect - David B. Dick  <-- More buildings by this architect.
• Architect - Shim-Sutcliffe Architects  <-- More buildings by this architect.
• Architect - ERA Architects Inc.  <-- More buildings by this architect.