I've done a thing some time ago that may be relevant:
https://compostjs.github.io/compost/
It is fairly small composable DSL for composing visualizations - unlike grammar of graphics libraries (vega), it is more low-level and you construct graphical primitives yourself - but it handles mapping of units from "domain units" to pixels (where domain units can be continuous or categorical) - so the idea is to have something that is a sweet spot between low level drawing and high-level (Vega, D3, etc) libraries...