In the early 17th century, an eminent soldier, Paul Gore, came into possession of Belle Isle Castle. His descendant, Sir Ralph Gore built the first house on the island. His grandson, who was born at Belle Isle Castle in 1725, and was also named Sir Ralph Gore, extended the house and created a magnificent garden that reached to the lough shore. Sir Ralph was created Earl of Ross in 1772.
He died in 1801, leaving Belle Isle Castle to his only surviving child, Mary. She married an Englishman, Richard Hardinge, who sold Belle Isle in 1830 to Rev. John Porter for Â£68,000. The Porters expanded the castle and added the tower. The coach house was built in 1856, along with the estate offices and farmyard. Porter’s grandson expanded the Belle Isle Castle again in 1910, adding the gallery, more bedrooms, and the porch. In 1991 the Duke of Abercorn bought Belle Isle Castle Estate from Miss Lavinia Baird, the last member of the Porter family.