Most of the early settlers of Woody Island came from England. Samuel Williams, who was born on Woody Island in 1901 said his great-grandfather Samuel came to Woody Island around 1820 with his brother. It was around this time also that other Englishmen were settling on the island. One of these was Robert Barrett, who became a very prominent figure in early Woody Island.
An old man named George Stainer also moved from England and settled on the island, he claimed to be 115 years old and he still had a strong British accent when he died.
The other two men who moved to Woody Island in the 1800s were Sam Crab and Thomas Russell, neither of whom was married. Both men were fishermen, then one of them drowned and the other moved away. It seems that they did not move from the island before 1880, as old Mrs. Collins, a resident of Woody Island born in 1832, met Sam Crab on a footpath in 1880 and she describes his outfit that day as "white duck overhauls." Courtneys, Smiths, Husseys, Allens, Marshalls, Pearces, and Lockyers also inhabited the island sometime in the 1800s.
One time, old Mr. Stainer and his son William were sharpening an axe on a grindstone. Billy was turning the grindstone slowly and the old man was becoming impatient. The grindstone would creak with each revolution, and young Billy teasingly asked his father what the grindstone was saying. The old man answered without hesitation, "It's saying turn faster, Billy Collins Stainer, turn you buggar, turn!"