Historic Peninsula is the center of ancient Istanbul. Four major empires had been ruled from this point. As you can guess, there are many historic places on the Historic Peninsula like mosques, churches, palaces, etc. I tried to pick the most important ones, for the ones with limited time and I will be writing very briefly about them:

Hagia Sophia (Ayasofya: The Church of the Divine Wisdom): Built as a cathedral in 537, converted to a mosque after the conquest of Istanbul, now is being used as a museum.
Topkapi Palace (Topkapı Sarayı): Ottoman Empire was ruled from this palace for 380 years. Built in 1470′s. It is a museum now.
Mosques: Sultanahmet Mosque, Suleymaniye Mosque, Yeni Mosque
The Grand Bazaar (Kapalıçarşı): One of the world’s largest covered markets with more than 4,000 shops. Well known for its jewelry, pottery, spice, leather and carpet shops.
Basilica Cistern (Yerebatan Sarnıcı): This world-famous underground cistern was built in the 6th century and is the largest of several hundred ancient cisterns that still lie beneath the city of Istanbul; size of 143 by 65 meters, with capacity of holding 80,000 cubic meters of water.
Gulhane Park (Gülhane Parkı): One of the largest public parks in Istanbul. It was once part of the outer garden of Topkapi Palace, it is being used as as a public park since 1912. It extends from Topkapi Palace to Bosphorus, it is a nice alternative for a walk and a coffee break in a nice weather.
Beyazit Tower (Beyazıt Kulesi): It is an 85 meter tall fire-watch tower located in the courtyard of Istanbul University’s main campus on Beyazıt Square. It was built in 1828.
The Spice Bazaar (Mısır Çarşısı): It is the second largest covered bazaar after the Grand Bazaar. It stands next to Yeni Mosque. You can find many types of spices, herbs, nuts, Turkish delight, etc.

For more touristic places and detailed information, I recommend you to get map and a guide from the tourism information office which is next to the Basilica Cistern.