Masjid Haji Muhammad Salleh sits on piece of waqaf land within the bustling Central Business District. The mosque is flanked to the left by the elevated Keppel Viaduct and on its right by Mount Palmer on which sit two important shrines of Habib Noh and Habib Abdurrahman.
The mosque was built by Haji Muhammad Salleh a merchant from Batavia, now known as Jakarta, who lived in Singapore in the mid 19th to the early 20th century. Haji Muhammad Salleh was a friend of Habib Noh, whom he highly revered. Habib Noh frequented Mount Palmer to pray in seclusion and Haji Muhammad Salleh intended to build a surau there for his beloved friend to pray in ease however Habib Noh passed away before his wish was fulfilled.
Haji Muhammad Salleh donated a piece of land at the foot of Mount Palmer and began construction of a mosque to serve the visitors of the shrine. The mosque was completed in 1903.
The Jules Miniot Map 1861 shows that during the lifetime of Habib Noh, there was a mosque near the Parsee Lodge which was close to Mount Palmer. It is likely that Masjid Haji Muhammad Salleh had replaced this earlier mosque.
In the early years before land reclamation, the mosque was close to the sea. In 1987, MUIS upgraded the mosque at cost of $1 million. The upgrading was undertaken following a decision by the authorities not to demolish the mosque to give way to the contruction of the expressway, but to curve the expressway around it. The upgrading project was undertaken by MUIS in 1987 at a cost of $1 million. The new mosque was designed with a prayer space to accommodate 500 worshippers.