Construction of the Island Castle on one of the larger islands of Galvė Lake began in the second half of the 14th century; at the same time, modernisation of the Peninsula Castle took place. Initially, the Island Castle had elements both of enclosure and convent types; it had a U-form ducal palace and an outer ward. Presumably, the construction of the castle was begun by the Grand Duke Kęstutis and was concluded in the early 15th century by his son Vytautas. It was meant to be a fortress, and it was never conquered by any raiders.

In the second stage of construction works, the layout of the castle was slightly changed: between the two castle blocks a 9,2 x 9,6 m large and 6-storey high keep (donjon) was built (total height – 33 meters); a defensive wall with counterforts was built around the island. The castle was built in a Gothic style with some Romanesque elements. All the rooms had vaulted ceilings, window openings were decorated with profiled bricks. On the first floor of the right block there was the Great Hall; its windows were decorated with stained glass. The architecture of the Island and the Peninsula castles was for the first time described by a Flemish traveller Guillebert de Lannoy; he commented: “…the second castle is in the middle of the lake, in a distance of a cannon shoot form the first castle. It is completely new and is built from brick in French style.”

In the last stage of the building works, the ducal palace was separated from the outer ward by a fosse; a defensive wall with towers was erected around the outer ward and casemates built on the inside. In the southern and triangular casemates there were kitchens, on the lower floors of the western casemates there were stockrooms, and on the second floor of the casemates, servants lived. In the south-western tower of the castle there was a prison. The outer ward gained a form of irregular trapezium.

In the eve of the Battle of Grunwald, the expanded castle was a modern structure; the towers of the curtain wall were adjusted for the flank defence and had 15 cannons. After the defeat of the Teutonic Order at the Battle of Grunwald in 1410, the Island Castle became the residence of the Grand Duke. In the beginning of the 15th century, Trakai was a flourishing city, frequently visited by merchants, honoured guests, and foreign messengers, who were welcomed in the Great Hall of the Island Castle. In 1413, Benedictus de Macra, the messenger of the Emperor Sigismund assigned an arbiter in the argument between the Grand Duchy of Lithuania and the Teutonic Order regarding the Samogitian border, was received in the Island Castle. The King of Poland Jogaila (Jagiello), a cousin of Vytautas the Great, visited the castle for 13 times.

In the 16th century, having lost the military and residential functions, the role of the castle went down and the visits of Grand Dukes became increasingly seldom. The Lithuanian Metrics that had been kept in the castle was moved to Vilnius in 1511, and the Island Castle became a prison for noblemen.

After desolating invasion of Muscovy troops in 1655–1661, Trakai was plundered and burned down; the Island Castle was destroyed and remained ruined; the city did not revive and became a small provincial town.

In the 19th century, Romanticism ideas provoke interest in Lithuanian past, especially in the castles of Trakai. Artists painted the ruins of the Island Castle, made copies of the remaining fragments of the wall paintings, engineers and architects prepared restoration projects for the castle. Engineer B. Malewski prepared a project for partial restoration and conservation of the south-eastern tower of the curtain wall. Although the works accomplished were not large-scale, they were important for the preservation of the castle. From 1929 till 1941, restoration works were supervised by a Polish architect J. Borowski. During that period, the foundation of the stone-walls was reinforced, the corner buttresses partially restored, the facade of the castle cleared from ruins, the central tower strengthened, and the reconstruction of the Great Hall begun.

After the World War II, conservation, restoration and reconstruction works of the Trakai castles were continued by Lithuanian restorers. The reconstruction of the Island Castle started in 1953. The ducal palace was rebuilt under the project of the architect B. Krūminis in 1962; the reconstruction of the outer ward buildings was carried out under the project of the architect St. Mikulionis and was concluded in 1987. After the corner towers and the western casemates of the outer ward were rebuilt, the Island Castle that is a symbol of Lithuania and Trakai assumed the look it had in the 15th century. After the restoration of Independence of Lithuania, Trakai Island Castle again welcomes honoured guests; important state treaties were signed there.

In 1962, the Island Castle was conveyed to Trakai History Museum. Both permanent and temporary exhibitions are now open to the visitors; numerous concerts, festivals and other events take place in the Island Castle in the present days.