On the opposite side of Prospect Bay, beyond the village of Terence Bay is the peninsular community of Lower Prospect, on the shores of the Pennant Bay. The land is rugged, all sea and rocks. The original settlers were Irish, Foreign Protestants, German, and French-Swiss who were primarily fisher families. In 1851 John Brophy and family were amongst the first families living in the fishing village.

Between 1854 and 1866, when the Reciprocity Treaty between the US and Canada was in place, the American fishing fleet often called for supplies. Many beautiful islands off the coast of Lower Prospect are named after early settlers: Ryan's Island, Norris' Island, and Marr's Island.

In 1859 a school was built in Lower Prospect and at that time, the community was a thriving fishing village with a church, post office and stores. The first school master was James McCurdy.

In 1873 the Edmund Ryan family lived in the village and they were the first to become aware of the sinking of the SS Atlantic. As they sat down for breakfast, their dog came home, followed by a crowd of young men all soaking wet, some half dressed and without their boots. Scratched and bruised, some men were falling with exhaustion and painful cuts to their nearly frozen feet. When they reached the house, they rushed through the door, gathered around the hot stove and grabbed food, even the uncooked bread dough Mrs. Ryan had been preparing. They explained that they had just escaped from the wreck of the ship Atlantic and others were still on board needing help. Local boats involved in rescue operations were manned by Dennis and Frank Ryan, James Coolen, John and Benjamin Blackburn, James and Michael O'Brien, William Lacey, P. Dollard and J. F. Tooley.

William Chase, a local photographer, took the only child's picture who had been saved from the wreck. He was the 11 year old, John Hindley. Mrs. Clancy, a local lady, nursed him back to health.

In the mid 1880s fishing stages were prevalent in Lower Prospect. A stage included the sheds and wharf. Bait, lobster traps, eel cribs and other fishing gear were stored in the shed. Most local residents were fishers, however, there were also carpenters and those who commuted into the City of Halifax for employment. The Norris, Brophy and Blackburn families stayed after many moved away when the Prospect Road was completed.

Two notable descendents of the earliest settlers are Irene Ryan who starred in the television program, "The Beverly Hillbillies" and artist Joe Norris.