The buttons are made from ttk so they inherit their appearance from ttk styles. Therefore it's difficult to change their color.
You could use a tk button.
As an example, put this into a file in the scripts folder "test.tcl":
Code:
button $UI(toolbar).mybutton -text "Make selected red" -background red -command "ac3d set_col 2"
pack $UI(toolbar).mybutton -side left
When AC3D starts, it will run the script and put a button on the tool bar, which when pressed will set the color of anything to selected to red.
Not as pretty. Like you discovered, toolbar_add_button is the way to go for standard toolbar buttons