Masjid al-Haram was built by the first Caliph, Omar Ibn al-Khattab (634-644) and rebuilt, refurbished and expanded many times after that. As more people converted to Islam and made pilgrimages to Mecca, Omar expanded the mosque, demolishing houses surrounding the Ka'ba.

During the reign of his successor Caliph Uthman Ibn Affan (644-656), the prayer space was enlarged and was covered with a roof.

In 692, the Umayyad Caliph Abdul Malik bin Marwan conquered Mecca from Ibn Zubayr, the guardian of the holy site, he had the outer walls of the mosque raised, the ceiling covered with teak and the column capitals painted in gold.

His son, al-Walid (705-715), had the wooden columns replaced with marble columns and the arches decorated with mosaics.

Abbasid Caliph Abu Ja'far al-Mansur (754-775) added the minaret of Bab al-Umra.

Abbasid Caliph al-Mahdi (775-785) demolished more houses around the Ka'ba and rebuilt the mosque with the Ka'ba in the center in 777. Al-Mahdi also built three minarets around the wall at Bab al-Salam, Bab Ali and Bab al-Wadi of the mosque.
Mamluk Sultan Nasir Faraj bin Barquq (1399-1405) rebuilt the mosque after it was damaged in a fire.

The Ottomans sultans under the Sultan Selim II (1566-1574) ordered court architect Sinan to renovate the mosque. Sinan replaced the flat roof of the prayer hall with domes decorated with gilded calligraphy on the inside. New columns brought from the nearby Shams Mountains, were placed among the old columns to support the new roof.

Due to the damaging rains of 1611, Sultan Murad IV (1623-1640) ordered the restoration of the mosque and the rebuilding of the Ka'ba in 1629. The mosque was composed of a new stone arcade supported on thin columns, with inscriptive medallions between the arches. The floor tiles around the Ka'ba were replaced with new colored marble tiles and the mosque was equipped with seven minarets.

By the end of the Ottoman rule in the Hijaz during World War I, the external enclosure of the mosque measured 192 by 132 meters.

During the reign of Saudi King Abdul Aziz (1932-1953) the mosque was extended again requiring demolitions around the Ottoman mosque. The extension was built on two floors. Bab al-Salam and Bab al-Umra were renovated to match the style of the newly built Bab King Abdul Aziz on the southern façade. Four minarets were erected near Bab al-Umra and Bab al-Salam and the three older ones were refashioned.

The second extension sponsored by King Fahd (1982-present), consisted of a new wing and an outdoor praying area, both situated to the south east of the existing mosque. The new prayer hall is accessed through the monumental Fahd Gate at the southeast that leads to the Ka'ba.

Masjid al-Haram is the only mosque that has no qibla direction as worshippers pray facing the Ka'ba, in the central open courtyard. The mosque today covers an area of 356,800 square meters including the outdoor and indoor praying spaces and can accommodate up to 820,000 worshippers during the hajj period.