Don't want Users to have access to all the branches, how can I set that?

To do so, go to the Edit User page as mentioned above.

You will see fields “ Office ” and “ Secondary Office ” in the Edit form. Here you get the option to assign a selective number of branches or offices to the User.