It's just an empty object for authors to use as a namespace so they can avoid any naming conflicts with SugarCube or with the browser. It's generally a better place to store functions and data than the global scope.
There's no real docs on how to use it because there's really not much more to it than that. For completeness, though, to use it in Story JavaScript:
setup.myFunction = function (message) {
UI.alert(message);
};
In TwineScript:
<<set setup.myVariable to 10>>