The Caldwell House was designed and built by G.D.Coleman in 1840-1841 Rev. J.M. Beurel, founder of the Convent of the Holy Infant Jesus, the Cathedral of the Good Shepherd and St Josephs Institution, purchased the Caldwell House at August 1852.
Between 1855 to1892, series of convent buildings were add on due to flourishing of the school. In 1903, a Chapel was added and consecrated in 1904, followed by adding new wings and classrooms for increasing students.
In 1983, the school was moved to the new premises and the secondary school block was demolished in 1984.
The remainder of the complex was developed into retail outlets and restaurants and is known as CHIJMES.