The main considerations are sample quantity, darkness and viscosity. Smaller bore tubes obviously require less sample, however, larger tubes are generally easier to fill (and clean), especially as the viscosity of the sample increases.
If the bore of the sample tube is smaller than the beam diameter of your polarimeter, light will be lost by beam obstruction. In Optical Activity polarimeters the beam diameter is 4mm. Tubes with bores less than 4mm will partially obstruct the beam and, while not affecting the accuracy of the instrument, will reduce the optical density tolerance. Tubes with bores of 2.5mm or 1.5mm cannot therefore be considered suitable for dark, coloured or cloudy samples. A 4mm or 5mm bore tube is a good compromise to minimise the amount of sample, while remaining easy to fill and not obstructing the beam.