Sure, if 1. is the case it’s no problem I’m happy, the original abstraction is reused I don’t feel so bad, “it did worth it!“. But usually this is not the part I am afraid of. And also my point is, that it is only matter of time when reuse won’t be possible with these updated abstractions and I must just dump. Requirements changed so much and IN SUCH A WAY + IN SUCH AN ORDER, it is just literally EASIER to dump the existing (layers) of abstractions to get to the goal. Another point: knowing this happens all the time it feels wrong to think a lot about “complicated” abstractions which will help me in future. Earlier, I thought it is not a problem to sit and think a lot about these “partially good abstractions”, because I thought they will be useful. Also, reusing these “half baked” solutions seemed OK and justifiable. Now it feels like chasing a shadow and betting too much on catching it. 2.(b): it happens, I am OK with it. But here is the worst point: 2.(a): getting rid of the old abstraction(s). It is insanely hard! And it is weird why it is… Entangled code, changed meanings! Ill-evolved abstractions both in test and in production code! Whaaat?! I think that is the part we should do something about. “Half baked abstractions” feel like boundaries for safety (code reuse is also safety: “not to forget this case”-safety). But boundaries are rigid. It feels to me a “Sacrificing flexibility for safety”. Sure, abstractions also do have “meaning”, but this meaning alters over time (the dirty part, not the clean part) and sometimes the “stuff” is abstracted away is more like a fast changing “process” without real boundaries and not like an “entity” / unit / “noun” with clear boundaries. If it was a String its OK, it is a clean abstraction. My problem is with the “intermediate stuff” which are more like steps (which are more likely to change) in a process not a real entity like “AnIdiotAdapterFromXToYAfterZHappenedButBlah”. Abstractions feel declarative and rigid. Lack of abstractions feels imperative and flexible. So with all this what I am thinking about is this: The production code should be as flexible (dirty, ugly, duplicated, etc.) as possible, with SHALLOW (layers of) abstractions (shallow not equals low complexity). It feels a bit like “composition over inheritance” aka “fuck and duplicate over using defaults you wont be able to delete later + you have to duplicate later no matter what”. On the other hand it seems to be a good idea to have a nice set of integration tests with “full coverage” which tests DO SHARE a lot of abstractions for being able to describe “all the RAW cases”. (By raw I mean no/less intermediate stuff.) Maybe asserting on invariants in tests (and making these asserts reusable in test abstractions) is more maintainable then evolving and maintaining abstractions of production code. I might be 110% wrong. I am quite sure if I’d do this I’d pray for abstractions I did not introduce … Complexity does this all the time. The reason I feel its something else now is that its not complexity itself which ruins me (and my code) but the ORDER OF STEPS I get there.

I don't follow all the details, but you're right that this subject is interesting to me. 1. As @Scott Anderson pointed out above, not trusting abstractions seems not so much an extraordinary idea but more just a secret truth that we don't talk about in polite company. 2. Among hard-core TDD folks there's a school of thought that's the exact opposite of yours: make sure to keep tests entirely devoid of abstraction, to counteract any tendencies for complexity creep and over-abstraction in production code. I subscribe to this idea, though there's always a question of how far to take "no abstractions". Do we avoid all standard libraries? Do we skip the language and just program in machine code? My current rule of thumb is that I can use any abstractions from production code, but I am extremely selective about creating test-only abstractions. Since this means production code is often used for test setup, it's very important to be very explicit in each test about the component under test. 3. Maintaining control over abstractions shouldn't imply avoiding them entirely. Just to 'stay loose' mentally. Any abstraction can be questioned or taken out at any time. 4. At the risk of repeating myself, you should check out my notion of white-box tests: http://akkartik.name/post/tracing-tests. Rather than have tests run sub-components and make assertions on their behavior, I always run the whole program, emitting a trace of domain-specific events (facts deduced by the program), namespaced by different conceptual sub-components. Then different tests make assertions on the state of the trace. Fine-grained unit tests may focus on just traces pertaining to a specific namespace, while coarse-grained integration tests may look at a different namespace. Traces are an abstraction-busting device: tests no longer care about details of how facts are deduced, they only focus on what facts are deduced. And this permits radical changes to abstractions without needing to modify tests. 5. You should also check out my notion of layers: http://akkartik.name/post/wart-layers. Layers are an abstraction-busting device for gradually inserting code tastefully into a skeleton of a program. They're often useful for recapitulating a cleaned-up history of a program's evolution so that newcomers have access to always working copies of earlier, simpler snapshots. One thing I find myself using layers to do is track alternatives. When things were simple some sub-system was implemented one way. As features were added the old way got replaced. But it's still really useful for conveying the core idea. I keep the old implementation around in earlier layers, and replace it in later layers. That way the codebase continues to exercise both alternatives rather than let the old way accumulate rust in a git log somewhere. Anyway, I keep repeating myself about traces and layers, so apologies to those who are sick of hearing about them. But I really think they're super valuable ideas. Certainly they're 2 of the 3 best ideas I've ever had, and I don't expect to add to that list for the rest of my life.

@Pezo - Zoltan Peto I think you're hitting the nail on the head.. abstractions are necessary, but they are also close to impossible to get right, and can do great damage to software

@Kartik Agaram Thanks for your response, you have some really good points. Will check out your posts later as well!

This is a very Forth-like topic :)

@Dan Cook I don’t know Forth, can you give me more intuition on that? 🙂

I guess what he means is that abstracting (and re-inlining) are trivial operations in Forth.

I think this is the right link for my favorite talk on Forth:

What I meant is that Forth is meant to be so simple (compact, anyway) that everything about the program can be invented right there, just for that program. So everything you need to understand is immediately available and no extra layers of bloat. I haven't watched that video yet, but the description sounds like it's roughly about the same ideas

Yeah, though I'd say Forth's flexible abstractions stem more from the culture of the community than anything technical about the language.

ty

@Dan Cook "everything about the program can be invented right there, just for that program".. but why is that? Does terseness buy you that? A versatile set of built-in functionality? Does it work beyond simple programs?

Maybe most programs could be vastly simpler. MS Windows has enough code to fill enough textbooks that would stack up as tall as a sky scraper -- hundreds of millions of lines of code. And then Kay makes an operating system (STEPS) that has all the fundamental tools for authoring and web browsing, with only 20 thousand LOC.

I would agree that software can be vastly simpler, but there are some giant hurdles in the way a) the fact that (large) software creation is a extremely incremental process, where you add to the current structure rather than all at once, and of course b) humans tendency to overcomplicate.

Why 99%? Why not 30%? Does anyone really use more than that? The real problem is that each of us relies on a different 30%. We really need to move past talking in terms of proper nouns. Redoing MS Word is an insurmountable goal, partly because much of it serves the needs of MS rather than any users. But redoing text editing or typesetting, on the other hand. That's much more tractable.

because my though experiment is about how much simpler code can in theory be.. what the functionality being implemented is doesn't matter. The reason I don't say 100% is because that is an unreasonable expectation, since any software has a ton of idiosyncrasies due to its implementation details, and requiring all of those to be replicated would seriously reduce the ability to simplify. Or put differently, I expect a steep power law curve of code size as you approach 100%, none of which is of actual use to anyone.

Those are all proper nouns. My point is still not getting across.

What I am saying is that your point appears unrelated to my comments

Based on restructor you seem to think that the major sources of complexity are duplication. If only all the duplication could be removed, software would be 10x or 100x smaller. Is that kinda where your comments are leading?

So like for a company that can't replace an unnecessarily giant codebase because it's in active use my many parallel projects/products, but a vast simplification would greatly cut down the cost of maintaining it and the time it takes to add new features (and be able to do so in a sane manner). That exactly calls for a massive (albeit careful and piecemeal) refactor.

@Kartik Agaram no, my comment actually said that Restructor is NOT able to do such a refactoring.. I merely mention it as related to to thought experiment. The real refactoring ability needed to execute my thought experiment is beyond any current humans and current software, as all software design decisions would need to be re-evaluated in parallel

Yeah, there's a big difference between refactoring existing software that's in widespread use, and starting over from first principles. But yes, some software is so insanely large and complex (e.g. MS Word) that a refactor / restructure is just not humanly possible.

Thanks, that's some.. cool architecture 🙂 I'm failing to see how that translates to software though

Ok, I'll try to explain... This thread was about software architecture & size / complexity, and choosing abstractions and patterns to use to build it up, and whether to maintain or replace them. (Think of abstractions as the building materials or mode of construction) In both fields, decisions and thought around these things, are heavily skewed toward things that have nothing to do with the needs that the architecture is ultimately meant to address. You don't sell an arrangement of boxes fitting a certain way, nor layers of abstractions, nor (for either field) a specific framework or fittings for a specific framework. You sell a place to work or live or socialize -- carving out a box (even if done well) does not even begin to scratch the right level of thinking/goals for creating a "good" place to live, etc. It's the same with software (see "Screaming Architecture": https://blog.cleancoder.com/uncle-bob/2011/09/30/Screaming-Architecture.html). That is, good "software engineering", is about "good software" engineering -- which sadly, is typically far from the center of focus. There's often a response back to this that user experience is such a small piece, it's just the "skin" on the outside ... No no, it runs all the way deep, and if done right, can LITERALLY shape the architecture of software (again, see "Screaming Architecture" ... It's actually a lot deeper than that explains, but it points in the right direction) What I'm saying is that the exact same backwards thinking / disbelief that the architecture can (or should) be that tied to the user experience, exists in both fields. Bjarke's work not only shows EXACTLY how this can be the case, but also how much better things become: not just asthetically, but functionally; and not just for the user, but cost of building and maintenance, environmental impact, structural concerns, etc. all improve -- because the thing literally is, what it is. Software gets so tremendously large and complex, largely because so much of the scaffolding and architecture is so focused on all this incidental artificial stuff that has nothing to do with what the software is actually supposed to do or be. Unfortunately, all that incidental stuff is what programmers tend to think of as what the software "is" (and it's similar for physical architecture). But all that stuff is just the baggage "necessary" (or rather, elected) to create the user space / experience which is the whole reason for the software's (or building's) existence! Once you really get that (which doesn't happen after just a few hours of thought, because it's "new" and not "news"), then similarities are striking. Read execution in the kingdom of nouns and screaming architecture, and then relate to the physical architecture resources (all linked to previously), and I think it will all begin to connect. (Ironically, I struggle with conciseness. I might trim this back later)

@Kartik Agaram What you show in “A new way of testing” is I think a really similar implementation I’ve been thinking of, thanks for the pointer! I’ll also add that this “trace” based approach solves the problem of “how to test logging” implicitly, which is really useful if its needed. I also like that approach because I think its really shows to codes “real” nature which (to me) is about the control flow. Something which is more closer to the “deeper” control flow than to the abstractions “on the top of the control flow” seems more robust to rely on.

It's been a pretty good experience. Only a little taste is required to ensure I'm checking deep facts rather than brittle details.

That sounds great! 🙂 Stay around for a sec, I have another question and I’m so curious! 😄

Definitely seems worth trying! The key property of traces is that it's a versatile append-only data structure. There's definitely room to add more structure to it -- as long as there's always an escape hatch to strings. Hmm, it occurs to me that traces can be seen as a kind of inverse of @Tudor Girba's moldable programming. It consists of an extensible way to turn data structures into browseable UIs. Traces perhaps should be an extensible way to turn browseable UIs into data structures.

@Kartik Agaram an entry in the trace is an “event” right? Is it more / less than that? Maybe not just a plain event but with extra parameter information, right?

Well, it's open to interpretation. I think of it as an atomic fact of the domain. That seems mildly helpful to make sure I'm not fixating on internal events.

Great, thanks! 🙂

I haven't, but that is very much part of the goal: to replace expensive, heavyweight, flaky Selenium tests with robust, lightweight tracing tests at just the right level of abstraction.

Ah wait, by snapshot testing I did not mean screen capturing but like jest’s snapshot testing https://jestjs.io/docs/en/snapshot-testing

I see. Hmm, I'm not sure what problem this is solving. Why would you not write a regular test over the generated html parse tree?

@Kartik Agaram It’s not about the html part. It is just a standard use case, because it might be hard to come up with the tests when its about

visuals

. But ignore html for now. It’s like saying I don’t know “how the current implementation works” / “what are the abstractions and their relations in the current implementation”, but I’m sure it’s good and want to match my next implementation with the current one. I would not use this kind of testing alone, but for testing the glue between loosely coupled boundaries + the “log” nature of the traces seems a good composition. I could say: I don’t know what is the exact ordering between facts in the trace and it might change awkwardly, but for now I want to save the current implementation (via it’s trace), because I am sure it’s ok and I want to test against that later.

I see what you mean by snapshot tests. I suppose to could use them? 🤷‍♂️ But it seems more "black box" than I like 😅

Okok, just mentioned them to ask if you already thought about that use case so I can ask you how is it :)