Region: Middle Earth
I don't think it was unsafe per se, it's that they could not be rid of the effects. For Frodo it was also the sword wound he took, it would never properly heal unless he left. Essentially they could not live in proper peace unless they departed.
From Dwarf.. "Shippey discusses Tolkien's own experiences in the first world war, his and other returning vets' experiences in attempting to rejoin the civilized world, and how many of them felt such a re-acclimation to be impossible and had to "leave" once again. It doesn't have to be a one-to-one allegory, but the emotions and experience certainly seem to have contributed, in some way."
I quite like this, though I might wonder if it was a yearning for pre-war. There were massive cultural changes in Europe, the social order falling, the ahem.. pandemic.. so it may have more been a desire to return to things as they were - but it's a nice thought nonetheless