Rhetorical question?: how would your programming workflow change if you deeply believed that creating SCNs was cheap and easy? Perl leaned heavily on REGEX. REGEX was once thought to be hoary and problematic. Perl made it accessible. New kinds of things were invented when Perl started being widely used. T2t is “better” than REGEX, because t2t makes it easy to deal with patterns containing recursive nesting. CFGs, like YACC, make it possible to parse recursively nested text, but, CFGs need painful, full specification of too much detail. If you had a way to specify pattern-matching using tiny DSLs that didn’t cost a lot of time to build, how would your workflow change? ['t2t' leans heavily on PEG and esp. OhmJS] [SCN === Solution Centric Notation, essentially a nano-DSL, I think of Richard Feynman breaking away from mathematical/Gutenberg notation and inventing and using Feynman diagrams. Language affects thought, notation affects thought].

I do actually believe that creating SCNs is cheap and easy, but so far I express this belief mostly by creating s-expression-based SCNs in Lisp. I am aware of the limitations of that technique, but so far it has been a good compromise for me. The step to PEG gives more power but also comes with a higher cost, in particular if you consider editor support for your DSL, which with s-expressions basically comes for free. What I'd love to see explored is other wide-spectrum syntax frameworks than s-expressions. See this blog post by Shriram Krishnamurthi for a detailed discussion of s-expression as a framework for building DSLs (the label "framework" is mine though). In particular for non-textual syntax, which remains largely unchartered territory. Assuming we get to the point of people widely believing in SCNs being possible and desirable, the next interesting question is software system architecture design with SCN, i.e. putting them to good use with minimal undesirable side effects.

The goal should be to produce notations so concise that whole programs fit in one eye-full, so that traditional concerns, like editor support, don't apply any more. Writing "bigger" programs should mean writing less code in islands and layers. I first wrote t2t using diagrams (using the non-programming editor draw.io) based on the idea of pipelines and circuits that used OhmJS multiple times. I've since reduced it to a single js program to ease carrying it around. In my mind, the technique is powerful enough to build compilers with and to do non-traditional things like parsing diagrams and building notations that fit in less space on the screen. I've written a lot of Lisp macros, now PEG/OhmJS/t2t give me the power to do this with non-lispy languages. I was taught to build compilers as pipelines connected by little, custom SCNs for each pass - I found this to be mentally liberating allowing me to concentrate deeply on one task at a time. Parsing is easy, the other stuff is hard, I toss custom parsers around liberally.

See: Racket’s “Language Oriented Programming”

Ruby lends itself to DSLs and this pardigm is widely used in Ruby code.

@Guyren Howe Note that @guitarvydas was also asking for real parsers rather than DSLs that look like the language in which they’re embedded. That’s the thing with Racket’s approach: you can have the syntax of Pascal (or whatever) and the Racket tooling gives you some automatic goodies for things like syntax highlighting in your new syntax, &c. I haven’t seen that done as well by any other project. That said, I find DSLs with radically different syntax are usually not worth creating for quick things, and on balance prefer embedding the DSL in a language with a flexible syntax (that is, I’d rather just write some Lisp macros in most cases).

eval really treats "data as code", not "code as data".

"syntax is a view"

[Points that jumped out at me from the blog article that @Konrad Hinsen mentioned]

When you’ve lots of declarative machinery available, it’s nice to be able to write those declarations in a concise way. A Ruby on Rails controller might have code within it like:

has_one :user
has_many :logins
has_and_belongs_to_Many :account_groups

validates :user, presence: true

It’s nice when without much fuss, your language naturally supports expressing declarative truths in your program with close to optimal concision.

Is Ruby syntax able to come close to optimal for expressing these?

@guitarvydas Relevant for editor support: Emacs (haven't tried this yet). It would definitely be nice to have PEG support as ubiquitous as regex support.

@Konrad Hinsen Wonderful! Haha, it never dawned on me to look to see if emacs supported PEG (I shoulda known better :-). I will poke at it some more. Thanks!

Here’s an example of embedding a different syntax in a program: https://docs.racket-lang.org/algol60/ … and the source code for the parser for Algol 60 used in this implementation: https://github.com/racket/algol60/blob/master/parse.rkt And a friendly article on the approach: https://cacm.acm.org/research/a-programmable-programming-language/

@guitarvydas PEG support is a new feature in Emacs 30, though the library has been around as an add-on for a while (https://www.emacswiki.org/emacs/peg.el). As for language-oriented programming in Racket, that's definitely worth a look. The last link in @Jack Rusher's comment is a good entry point. Racket has become a modular compiler in which a new language is just another library.

The goal should be to produce notations so concise that whole programs fit in one eye-full,

yes. Maybe not entire programs, because of non-reducible amount of states in non-trivial programs, but yes.

"syntax is a view"

yes

I express this belief mostly by creating s-expression-based SCNs in Lisp.

yes. Not that many 'usual' text-based things are hard to cram into s-expressions. Trivial things are expressible with no extra work, but few extra parens. Larger but manageable things can be just string arg (that racket-algol example, or infamous SQL strings). But there is always just "flat s-expr with hundred args" approach, especially when you can express DSL's literal tokens in your fav LISP's literals: chars, strings, symbols, numbers, bools, keywords, vecs, lists, maps, sets. For example, clojure's

(let [a 1] a)

is a DSL too, since in the rest of the language same level tokens have independent scope: in

(+ x y)

-

y

does not have

x

in scope. But replace

+

with

let

and

x

with bindings vector, and suddenly

y

has in scope whatever is on the left:

(+ x y)

->

(let bindings y)

->

(let [y 1] y)

(ofc every macro is a DSL). I guess my point is you can ride s-expr macros pretty far even without reaching for multiple compilers, especially if DSLs are tiny. re: IDE support - this seems relevant: https://www.jetbrains.com/help/idea/using-language-injections.html#language_annotation