Is your dialog table set to use pixels or dbus? If its pixels then there's no scaling applied. If its dbu, it should use window's scaling.
With that said, this scaling issue has been brought up a few timesSmall ButtonsSmall UI
The best end-user solution would be to have two tables: one for smaller res, then one for HD/UHD resolutions