Old Filth by Jane Gardam

Sir Edward  Feathers, aka Old Filth (Failed in London Try Hong Kong), had a successful career as a judge and a decades-long marriage, but he never got over his traumatic childhood and his sense of abandonment. After years in the far East, he and his beloved Betty have returned to England to retire and he thinks back over the life he's led and the events that shaped him.

Old Filth is the first book in a trilogy. One book focuses on his wife's story and the other on his enemy, Terry Veneering. Parts of Old Filth are kind of funny but a lot of it is very sad, especially the part about the Raj Orphans, whose parents (stationed in the East) sent their kids back to England to live in foster care "for their safety."