James Milson (25 November 1783 – 25 October 1872) was an early settler on the North Shore of Sydney, New South Wales, Australia.

He was born on 25 November 1783 at Grantham, Lincolnshire, England and died at the age of 88 on 25 October 1872 at Milsons Point, Sydney, New South Wales, Australia.

23-year-old James Milson arrived in Port Jackson (Sydney) on Albion on 19 August 1806 as one of the earliest free settlers in the Colony of New South Wales. His motivation for immigrating was the same as the many who followed him, the promise of free land. He was a native of Lincolnshire experienced in farming and was welcomed by the colonists, desperate for men with agricultural knowledge. He married in 1810 and subsequently raised a family of 6 children.

Milson did well in the Colony of New South Wales, and established a number of prosperous businesses, which included supplying ships with stone ballast, fresh water, and the produce of his dairy, orchard, and vegetable gardens.

In his own words, in the years before 1825 Milson had "principally resided in District of Parramatta" (more specifically in the area in the District of Parramatta then called the "Field of Mars"The "Field of Mars" stretched from the north bank of the Parramatta River and up to and including Pennant Hills, over to the west bank of the Lane Cove River and down to the North Shore of Sydney Harbour, and included today's Sydney suburb of Ryde. It comprised the whole of today's parishes of the Field of Mars and Hunters Hill in the County of Cumberland.).

In the early 1820s Milson settled in the District of Sydney in the vicinity of today's Jeffrey Street, Kirribilli, a Sydney suburb on the North Shore of Sydney Harbour. He was a prominent resident of the area for more than 50 years until his death in 1872 at home at "Gratham" in the modern suburb of Milsons Point in what was then called the Municipality of East St Leonards.

Milson and his sons and grandsons built several homes in the area: "Brisbane House" (James Milson), "Grantham" (James Milson), "Fern Lodge" (James Milson), "Wia Wia" (John Milson), "Elamang" (James Milson Jnr), "Coreena" (Alfred Milson, son of James Milson Jnr), and "Wayala" (Arthur Milson, son of James Milson Jnr). Milson's son-in-law William Shairp also built "Carabella". Fern Lodge (heritage listed) and Elamang (in the grounds of the Loreto Convent) were still standing in 2008.

Milsons Point, the headland into Sydney Harbour on which the north pylons of the Sydney Harbour Bridge stand, and the suburb on this headland to the west of Kirribilli, are both named after James Milson. A retirement village at North Sydney that includes a nursing home is also named after him.

In addition Milson Island on the Hawkesbury River (previously known as Mud Island, renamed in 1976), Milson Passage the passage in the Hawkesbury River between Milson Island and the southern bank (previously known as South(ern) Channel or The Gutter, renamed in 1976), and Milsons Passage the suburb on the southern bank of the Hawkesbury River opposite Milson Island (previously known as Prickly Point, renamed in 1995) are all named after Milson's son Robert Milson (1824–1886).

Surname In many records James Milson's surname is shown as "Milsom". "Milsom" was the surname by which James signed letters, in which legal proceedings were transacted, and which appears on early maps such as the 1840s map shown below. By the time that James died in 1872 he was known as Milson, the name by which he and his family are known today.

Family tradition In the words of the North Sydney Historical society: James Milson is a controversial figure in local history, having made many claims during his lifetime (mostly in attempts to claim land) which either cannot be substantiated, or in some cases have been positively disproved

There is a Milson family tradition which includes that: (1) James Milson was "promised" all the land on the North Shore of Sydney Harbour between Hulk (Lavender) Bay and Rainbow (Neutral) Bay by Governor Philip Gidley King. Despite Milson's wife's continuous urging, he did not press the Governor for a written-out grant. However, James Milson did not arrive in Port Jackson until after Philip Gidley King had ceased to be governor on 12 August 1806. James Milson arrived on Albion on 19 August 1806 during the period that Captain William Bligh was governor from 13 August 1806 to 26 January 1808 (until the military coup by the NSW Corps). James Milson's arrival and his marriage on 8 January 1810 were both events that occurred after Philip Gidley King ceased to be Governor of New South Wales. Therefore there is no way that Philip Gidley King could have made Milson a land grant, or that Milson's wife could have urged Milson to press Philip Gidley King for a written-out grant. Milson marriage occurred during the time that Lachlan Macquarie was Governor of New South Wales. Lachlan Macquarie was governor from 1 January 1810 to 1 December 1821. James Milson was instead given land by Governor Macqaurie. In James Milson's memorial to Governor Macquarie dated 5 June 1820 it was stated that James Milson was living with his family at Pennant Hills "following agricultural pursuits" on a 100-acre grant previously given to him by Governor Macquarie. (2) Milson was shown a plan delineating the land in his name and that this sufficed as legal ownership in the early colony. However there is no legal precedent for this assertion. (3) Milson built his house on "Milson's Point" shortly after his marriage in 1810. However in the years after his marriage James Milson lived at the "Field of Mars". in addition to the memorial of 5 June 1820 establishing residence for himself and his family at that time at Pennant Hills (in the "Field of Mars", district of Parramatta), in Milson's own words in the years before 1825 he had "principally resided in District of Parramatta". He did not build a residence at Milsons Point in the District of Sydney shortly after his marriage in 1810. (4) No actual acreage was stated and this fact weakened Milson's case in the subsequent legal dispute between himself and Campbell over title to the land that included Milson's Orchard. However this case did indeed mention acreage, and was not about who owned the land. Milson lost his case against Campbell based on non-payment of a lease agreement between himself and Campbell. The case mentioned that Milson leased a farm of from Robert Campbell. In this case Milson did not purport to own the land that he was leasing. Milson instead referred to Campbell's absence of proof of title which made it impossible to ascertain the boundary between the that Milson leased from Campbell, and the adjoining "very large portion of land" (acreage not stated in the case but known to be ) that Milson had received as a grant from Governor Sir Thomas Brisbane in August 1824. It is true, however, that Milson did dispute Campbells ownership of the land during the nearly 3 years from 1828 that he ceased paying rent. During this time Milson wrote to the Surveyor-General and Governor Ralph Darling in an attempt to have the land recognised as his own instead of Robert Campbell's. The reply that Milson received on 13 March 1829 was that he could not obtain the land as it was the property of Robert Campbell, and that the land that Milson owned (the adjoining 50 acres) was already measured and marked out.

Versions of this family tradition, or parts thereof, have been repeated as fact in many publications about the life of James Milson. The most common part of the family tradition that is repeated is that the court case between Campbell and Milson was a dispute over land title. Examples of repetition of the family tradition, or parts thereof, are found in the Australian Dictionary of Biography, the publication by Elizabeth Warne, the publication of R.H. Goddard, publications of the North Sydney Council and the North Sydney Historical Society, and the Dictionary of Sydney.

Newspaper report that contains the family tradition An example of a version of the family tradition follows from the Sydney Morning Herald of 19 May 1926.

Sydney Morning Herald,19 May 1926Evidence from documentation AN OLD FAMILY THE MILSON STORY (BY R. H. G.)