Hi everyone, I’ve been thinking about using hypertext instead of text files as the medium to represent programs and their executions. Programs contain a lot of cross references that are resolved only after the parser/runtime have had a go. IDEs also redo this parsing work to simulate the same cross references for easy editing. The idea is to embed these references in the medium itself. So, get rid of import statements, file boundaries and such. Each textual reference - function name, type name, any identifier really - would be link to the object it references. Unlike strongly structured environments, this allows some flexibility in representing partial and possibly invalid structures. I’m interested in implementations you know of that match this idea.

Your idea reminds me of Jonathan Edwards' Subtext system: https://www.subtext-lang.org/

Ah yes thanks for bringing that up. I’m familiar with Subtext and in fact using the same structure for both the execution trace and the program is influenced by Reifying Programming. The difference here is using hypertext objects as the underlying structure. Subtext tends to have more strict tree like editors.

I assume hyperlinks here would be just a mechanism to construct trees? I've long thought there should be a dual of

link

called `inline`: you don't need to click, you just see the contents of the destination in place. A combination of

link

and

inline

might subsume lots of ideas like Subtext and Brief ()

Thanks for the video - will check it out. Hyperlinks are not mechanisms just to construct trees. Even in basic cases references form DAGs and even cycles. eg here’s a trivial DAG:

class T: ...
def f1(a: T): ...
def f2(b: T):
  f1(b)

The

T

annotation in both function definitions link to the class definition. The

f1()

call links to the

f1

function definition. Any mutually recursive definition will form cycles. Note also the hypertext objects here are individual definitions and dont impose any language semantics. Not sure I understood the

inline

idea. Are you talking about transclusion?

Yeah, everything is in Xanadu.

Indeed I think to

inline

any refererence makes sense. Any hard link should be inline-able by the reader. I don’t think the writer has to decide if the link should be inlined or not.

There's advantages both ways. Writer deciding can construct experiences. I mostly care about when writer == reader, so the question is moot.

@shalabh One ingredient of your idea is extreme late binding. That's one of the principles of Smalltalk, and Smalltalk IDEs implement some of what is being discussed here, though only within the universe of the image. Example: Glamorous Toolkit lets you see the definitions of methods you call inline, which is very convenient. Your outline of metaprograms defining programs which again contain dynamically resolved references is pretty much what people call metaprogramming in Smalltalk. Pharo's relatively recent slot mechanism would allow implementing more general types of links, but I haven't seen this done. Smalltalk users wouldn't call their code "hypertext" but rather "object graph", as much of the code structure never gets expressed as text. But that seems a minor technical detail to me.

I very much like this idea, it's so intuitive that I just thought, geez why isn't it done like that already

@Konrad Hinsen interestingly I was thinking one aspect of code-as-hypertext is earlier than usual binding. References get bound to objects when written. This is even earlier than in plain text compiled languages (identifiers in a text file are not bound at all). However the bindings-as-written are not the final bindings. During the course of meta-program and program execution they get rebound. Eg a reference to an abstract type may get re-bound to a concrete version of that type. Evolution of these bindings can be represented in the same structure. Smalltalk / Newspeak etc are a key influence behind the above idea. In Smalltalk we write classes and their methods directly, not worrying about file boundaries. I was thinking why the method body is text and not hypertext? Maybe new versions of GT/Pharo do this better. GT still seems layered on rather than native.

Where do you imagine the hyperlinks going to, other definitions? Many of my apps have that ability, Teliva and some of the Freewheeling stuff, and I'm wondering if the ability to click on hyperlinks is all we're discussing.

Where do you imagine the hyperlinks going to, other definitions?

Yes and also derived definitions (eg objects created via meta programming) and even runtime traces can link back to the definition objects.

I’m wondering if the ability to click on hyperlinks is all we’re discussing.

Good question. Clickable links between source definitions exist in many environments. Even vscode lets me do that with most of my programs. However this is typically implemented as a layer on top of the underlying media (text files). The links may not match always actual semantics, but more importantly they do not navigate to derived definitions or connect runtime objects with definitions. In fact derived definitions are not typically available in the dev environment at all (smalltalk envs are the exception). The idea is to use hypertext as a unifying interaction model between the person and the system - for hand written definitions, derived definitions as well as runtime traces. The hypertext ux would include clickable links and any other features we want to include eg inline content.

I think I understand now. The runtime traces is hard to do performantly, but for everything else, yes this sounds like a nice reasonable goal. My stuff isn't it yet; you can't click on any function to go to its call, as a very basic example. The metaprogramming, I gather you want to be able to click on a macro call and see its expanded output. All this seems doable. I think I'm headed in a similar direction but wasn't aimed quite here. Now I might be. 🤔

FWIW: this idea fires a set of neurons in me, labelled “GOTO”. ## Synopsis GOTO v1 - assembler GOTO GOTO v2 - message passing GOTO v3 - CPS - Continuation Passing Style GOTO v4 - URL All versions of GOTO suffer from the problem of so-called “structured”-ness. ### Random Notes • Kinopio is GOTO v4 • Message Passing is my current sweet spot. • most current programming languages are based on the Synchronous Pattern, i.e. they handle GOTO v1 and GOTO v3 but fumble GOTO v2 and GOTO v4. • “come from” is just GOTO in reverse, i.e. backlinks ### Lessons From Org Charts ## Further Thoughts https://publish.obsidian.md/programmingsimplicity/2023-07-01-GOTO

The runtime traces is hard to do performantly

Agree. This could be opt-in via a debugging / “capture trace” mode. Hypertext that links back to definitions seems better than explicit print style debugging.

The metaprogramming, I gather you want to be able to click on a macro call and see its expanded output

Yes. I also think it would be nice to have the generated definitions available in searches. Derived definitions may also be produced from external schemas or import time execution.

I was thinking one aspect of code-as-hypertext is earlier than usual binding.

and then:

However the bindings-as-written are not the final bindings.

I am not sure I understand your bindings. Are they mutable or not? Or maybe mutable initially, then frozen? As for GT, yes, it's a new tooling layer on top of a standard Pharo object graph. The bindings presented in the UI thus reflect the current state of the system, but everything can change at any time.

I was thinking why the method body is text and not hypertext?

Names would then point to what? There is nothing but names and objects in the Smalltalk object graph (if you see literals as special names).

In fact derived definitions are not typically available in the dev environment at all (smalltalk envs are the exception).

And the big practical issue in Smalltalk is that you get only the derived definitions, with no provenance tracking unless the metaprogramming framework has its own mechanism for that. But then, that's not a fundamental issue, it's part of the tooling layer.

I am not sure I understand your bindings. Are they mutable or not? Or maybe mutable initially, then frozen?

This would be PL dependent but typically I imagine no in-place rebinding. However a form of rebinding happens by creating new versions of the hypertext objects. Eg the hand edited source objects could only be rebound by the user as part of editing. In the metaprogramming stage the system would create derived versions of these objects where some of the bindings have changed. For instance a binding to an abstract type in a source object may be resolved to a concrete subtype in the derived objects. So the PL semantics would also restrict what kinds of “rebindings” happen. Eg “constants” would be bindings that don’t change in any derived objects.

I think you might run into situations where you want a thing to hyperlink to multiple possible targets. The current binding yes, but sometimes also the previous binding, or the previous edit. Or the edit history. Does your point against

inline

apply also to the very idea of hypertext? Why does the writer need to create hyperlinks, can the reader just have tools for all these transforms? That's basically the world I live in today. My IDE takes me to a word's definition with F12, all its references with S-F12, all its callsites with C-k C-t. Perhaps we just need to do more of this? Now I want to link to an old thread of Lisp Machine screenshots, but it's sadly stuck in Twitter.

I think you might run into situations where you want a thing to hyperlink to multiple possible targets.

Yes. I imagine viewing the source and various derived objects as distinct hypertext objects where you are able to go back and forth and follow the chain of provenance.

Does your point against

inline

apply also to the very idea of hypertext? Why does the writer need to create hyperlinks, can the reader just have tools for all these transforms? That’s basically the world I live in today.

Good question. While it does bother me that the tooling overlaid links are an approximation (and poorer for dynamic langs generally), the bigger issue is the lack uniform navigation across derived definitions and source definitions. I want to see the rendered macro, the concretized generic method and so on. I want to deftly bounce between those and the original source definitions. BTW your earlier counterpoint of writer designed experience via

inline

is valid. I think the reader should still have the ability to customize it further (eg collapse an

inline

). A related point is that the reader doesn’t get to customize the text in text-file-as-code. For example if the writer chose to write

import pandas as pd

, the reader has no choice but to see

pd.foo

scattered across the file. If this was hypertext-as-code, it’s easier for tools to offer customized rendering, eg “show the fully qualified name everywhere”, or “show the short name, but blue for `pandas`” and so on. In theory this is possible with text-file-as-code but more challenging, given that tooling is trying to simulate the PL semantics to extract high fidelity references.

I want to see the rendered macro, the concretized generic method and so on.

Now that I understand your idea, I actually want more; I want to edit the rendered macro, or manually send it through a domain-specific transformer. I want those changes recorded, and when later I edit the upstream code, I want to be able to merge the changes made at different levels. Example use case: optimization. Quite often, optimization is in conflict with abstraction. If I get to edit both the abstract and the concrete code, I can have my cake and eat it.

Now that I understand your idea, I actually want more; I want to edit the rendered macro, or manually send it through a domain-specific transformer.

Hear hear! I think about a flow where you edit a derived object directly and the system shows how the change can be translated back to the source objects. There could be more than one possible source edit and the system would give you a choice. More importantly, it can show you the effect of that change on other derived objects.

Not sure that can be done, and if it can, I am not sure that's always what I want. Staying with the "rendered macro" case as a concrete situation, the change to the derived object can require a change to the input text, to the macro, or to both. In my optimization scenario, the most likely required change is adding a special case to the macro, making it harder to understand. I'd prefer keeping the special-case code attached to the special case.

Fair point. The way I think about separating optimizations from “core logic” is by using separate definitions for each. Both would be source definitions, ie hand edited. The core logic would specify the essential structure + logic and another overlaid definition would describe the memory layout and other “encoding specific” details. Typically even in high level languages the encoding is automatically derived. This requires us to mix core logic and optimization details. However this idea is still vague and easier said than done. I think of these definitions as “domain definitions” and “encoding definitions”.

Unison language perhaps went farther than others in early binding (by merkle tree hash) to specific definitions: https://www.unison-lang.org/learn/the-big-idea/ Dunno if they do anything in reified-execution direction.

Great idea @shalabh. I also do “edit-time binding”. The problem is that this line of thinking naturally leads to structure editing of code and that is both hard and taboo.

It’s less taboo in this forum 😄. It is hard. By using hypertext, I imagine the editing experience may be more fluid - somewhat similar to Notion but without formatting and more auto-linking.

While waiting for someone to come up with a really good implementation of structure editing, plain text editing with automatic structure highlighting plus structure-based operations is already very nice. Emacs modes for Lisp are a good example (paredit, rainbow parentheses, ...). Glamorous Toolkit goes one step further: code is re-parsed at every keypress, and small visual annotations are added for linking to whatever is appropriate. What's nice about this is that you don't have to learn the structural stuff if you don't want to. You can just work at the text level. It's opt-in.

It's not exactly what you're talking about, but html might actually work for aspects of this. https://futureofcoding.slack.com/archives/CCL5VVBAN/p1689083336878249 There's a part I didn't fully describe which is a system to swap out nodes with alternate representations which know how to turn themselves back into ast nodes. For example, replace all function calls to

my_color_picker

with

<input type=color>

<- can think of these almost as "visual/dynamic macros"