In the 11th century, the presence of a castle was responding to strategic concerns, control of a critical pathway: the road to La Roche-Bernard and Malestroit Ploërmel that connects the coast to the interior of the Duchy of Brittany .

In the 14th century, the family of Rochefort joins that of Rieux (to Redon) by marriage. The site and the castle became the property of the family of Rieux-Rochefort until the 16th century. It is the reign of this family that the city becomes prosperous and takes the face that we know today. The castle was destroyed for the third time when the French Revolution but ....

As in Pont-Aven, in the late 19th century, Rochefort was visited by many painters charmed by the old houses full of history. One of these painters, an American named Alfred Klots fell in love with Rochefort. It is he who will be the revival of the city. In 1907 he bought the castle ruins. After the war, he built a mansion from the sublime elements of the castle to Keralio Noyal Muzillac. By the 1930s, Rochefort became a popular site for its flowers, its architectural quality and its inns. Alfred Klots died in 1939. His son, Trafford, painter, too, will continue to come to Rochefort.

In 1976, the General Council of Morbihan bought the castle in an annuity to Mrs. Isabelle Klots after the death of her husband the same year. The castle has been open to visitors.