Is there any hope for end-user programming when programmers themselves don't use programming to solve their own problems? For a long time, I've been asking myself the question about why more programmers don't use programming to solve their own problems, but it just occurred to me the implications of that to end-user programming. Does end-user programming ever have a chance of succeeding for non-programmers to solve their problems if programmers themselves aren't using programming to solve their problems? When I say programmers don't solve their own problems with programming, what I mean is, there is sort of a ladder of useful techniques to use programming to make programming itself easier. It starts with customizing your shell or your text editor by cut-and-pasting code snippets you find online, and progresses to writing your own customizations from scratch, to writing your own shell/text editor extensions, and finally to writing your own full programs to solve your own problems. I find it so odd that it's so rare for any of the programmers I know personally to progress beyond the first stage (some light shell/text editor customization by cut-and-pasting some code they found online). Since programmers are experts at programming, and they generally choose not to use solve their own problems with programming, what hope is there for end-users to use programming to solve their problems? Or is there something wrong with the lens I'm looking through here? Perhaps programmers are using programming to solve their own problems in a way I'm not seeing? (I.e., that aren't shell and text editor scripting)?

We first need to articulate what is meant by "programming." Without a coherent definition, we will be at a loss to come up with some modified form of it called "end user programming." My view on this is that "telling a computer what to do, later" is a reasonable enough definition, and from that it's easy for us to see that most things done using a computer are "programming-like" already.

I mean, many people work at companies on products they don’t themselves use. Part of the rise in the profession of Product Managers and Customer Success is to help bridge the gap to the end-users and their goals. Programmers ALONE sitting in a cave probably can’t create programming tools / interfaces / languages that accidentally empower end-users. I would agree there

They are not designed with that goal in mind, and are in fact actively hostile towards it

yeah I broadly agree with @Eric Casteleijn

Another way to look at this is to look at compelling examples from the past of end user programmable systems

thats why new mediums / platforms entirely are interesting. Like Dynamicland or VR / AR or some new platform entirely

Hypercard is a good example. It was great, and extremely popular in its own day. There are clones today -- why are they not as popular? My answer is because HC fit holistically into the OS of its day. It could do a large part of what the whole OS could do, but within its own environment

all good analogies. Here’s a more computing related one… Microsoft was investigated for anti-trust b/c of Windows dominance and their shenanigans with IE. Google ended up being more important b/c they were kinda the homepage for the internet after 2000. Then, Apple ended up becoming super important b/c of their control of mobile. But neither of these companies threatened Windows by fighting directly. They would have lost! Google tried to commodotize / trivialize the OS by focusing on the web (Search + chrome). Apple built a parallel platform that had nothing to do with Windows

The main reason I don't solve more of my problems with programming is that the tools for doing so are so clumsy that it's only worth it for really big jobs. There's an aspect of learnability (bash language is a nightmare) and of integration with the system (hooking most applications is a nightmare AFAIK, especially GUI)

Most of these comments are (interesting) observations about why it’s hard for end-users to use existing environments for programming. Any comments specific to why programmers don’t use them for programming? (E.g., they write code as their job, but don’t write much code to do their job more effectively.)

I'm probably not the only programmer who started off trying to use programming to solve things, but got burned because it was just way more efficient to do it manually.

@robenkleene it's the same reason

Yeah totally makes sense, things like Bash seem incredibly clumsy until you try to use something else, and then realize it’s the task itself that’s clumsy (my interpretation)

If you are a professional programmer, someone is paying you to do all that complicated stuff

I do a ton of this kind of thing on company time, I usually automate all the things everyone is doing manually. But the people who do this work are maybe 1 out of 10 programmers in my experience.

The task is usually more burdened by accidental complexity than actually clumsy. That's the system integration aspect a couple of us have mentioned. Doing everything in plain text was cute but causes a lot of friction when working with structured data, especially when you have to convert between formats.

@robenkleene can you clarify what you mean by:

For a long time, I’ve been asking myself the question about why more programmers don’t use programming to solve their own problems,

No @robenkleene I think you are right; most paid people are not doing it

what do you mean by “their own problems”?

But most paid people are working in environments that are clumsy, archaic, and in many ways hostile

problems b/c of programming / their code? Or like problems in their life (e.g. too much time spent running errands)

personally I don't write too many bash scripts because I'd rather shove a meat thermometer into my ear and wiggle it around. And if I'd rather to that, then I'm always up for just manually doing whatever the task is

yeah I do 0 bash scripting myself

Writing a bash script to do a huge find and replace programmatically, making scripts to quickly open projects they open over and over again, customizations, e.g., I always adding a shortcut to open hyperlinks from my text editor. It’s more lots of small things rather than one big thing.

Then I think "maybe there is some linux program that does this" and I'm off to the races trying to find a command name that makes sense, usually dropping vowels for no reason

Jeez, this thread is going fast. I'll wait for it to slow down, except to say: I don't understand @robenkleene's original question, and why it assumes customizing is not programming, and why extensions are not programming. Why is only writing new full programs considered programming? Most of us only modify existing programs in our day jobs. A founding ethos of programming is to do as little of it as possible. That's broadly true even in my vision of utopia.

I always think of @stevekrouse example he talked about a “email construction kit”, I keep thinking, wait a minute programmers already have that (Emacs) and they barely use it, and they already know how to program...

Writing a bash script to do a huge find and replace programmatically, making scripts to quickly open projects they open over and over again, customizations, e.g., I always adding a shortcut to open hyperlinks from my text editor. It’s more lots of small things rather than one big thing.

Now I see what you mean, kinda like ‘automate the boring things’.

I'm with @Kartik Agaram here -- configuration is also "programming"

These seem worth reflecting on: * https://xkcd.com/1205 * https://xkcd.com/1319

@Kartik Agaram Those are all programming! My point is mainly that they stay low on the hierarchy, and don’t progress to writing their own customizations. E.g., when we talk about end users programming, presumably that’s more than cut and pasting a few common snippets online, and that’s about as far as most programmers go with their own environment (while I agree that 100%) qualifies as programming.

Emacs is not more popular because it is designed to be a system that is text only

Re the xkcd, if we haven’t been able to make automation a worthwhile time investment for experts, then what chance is there we'll be able to make it worthwhile for non-programmers?

I think your question is a very good one actually

For me the answer is that nothing feels like a stable foundation on which to build. If somebody wants to pay me to spend a majority of my time fixing previous code I and others wrote, that’s fine, but there’s something deeply frustrating about it when you see the extent to which your tools get in the way of your own vision. I don’t want to live in the command line, but that’s about the only place I feel confident that what I build can persist.

... if we haven't been able to make automation a worthwhile time investment for experts, then what chance is there we'll be able to make it worthwhile for non-programmers?

"There never was much hope, just a fool's hope"... but for some reason we're all in this Future of Coding thingy anyway. I see this as the exact problem we're here to solve.

I actually do automate a lot of things (and I use Emacs). Mainly because the emacs makes it easy to do. For me, the important aspect is extensibility. In many cases there is an almost good solution but with one or two things I need to do differently. And too often the cost to modify the solution is about as hard as to reimplement it from scratch. Especially true for full-blown GUI and android apps. For emacs, you can always take a half-baked solution, modify its code on the fly and you're done. If I were to speculate about why others don't do the same and automate easy tasks, I would say that the tools have high learning curve. It took me quite a while before I started to feel proficient scripting Emacs. There is also an attitude problem—not all programmers know they can extend their editor and what's possible, so they don't even look for opportunities.

in particular, i think that the majority of the “old european cities” we’re living in we’re built for batch processes. most of the tasks i’d like to automate have to do with user input (the click a button example is really stark and appropriate). and that user input isn’t the start or end of the program. id really like pretty much any environment that gave me live insight into how eg keyboard shortcuts and button presses were flowing across the system, and similar insight into graphical primitives and responses. i can build data abstractions pretty much anywhere but gui and input processes always feel like i’m running a marathon with petulant snakes around my ankles

Another relevant xkcd; https://xkcd.com/974/ I think currently automation appeals to people who actually enjoy the problem solving aspect. The fact that it may or may not be massively more efficient is secondary. I wrote a python script to combine image files into 2x2 combined images of 4, because it was slightly cheaper to print those as "single" images for my wedding than use the company's own quarter size print option. Probably only saved a few quid in total but I enjoyed the task. This instinct is kinda orthogonal to people who like programming/computers. I know people who come up with elaborate schemes for making sandwiches/wrapping presents/stuffing envelopes etc en masse. These type of people are the ones who will jump on end user programming when it becomes easier. A lot of users though, aren't even using copy-paste, select all, find-replace etc yet

Maybe because we are programmers, we know the cost of creating/maintaining the things we make, so we rather spend our time doing other things. One quote that has stuck recently is, “In software, anything is possible, but nothing is free”. Even though, I can certainly optimize some of the things I want to do, I don’t want to pay the time and effort to do so.

@Garth Goldwater I will add that when I'm working on Smalltalk projects, I tend to make those kinds of buttons all the time, and just pop them out onto the desktop. I wish I could do it in macOS

@robenkleene I eventually came to believe that it’d take a pretty significant societal shift for it to be realistic now. The time that many of us base our dreams on (the days of hypercard and VB6) just doesn’t exist anymore - there are now endless distractions and more apps to do what you want than you could ever begin to look at. In the earlier days of computing there was just.. less stuff. And it was more obvious that if you wanted something maybe you’d put it together yourself. Even with the perfect toolset, most people aren’t aware that they even could change things and even when you tell them they can, they don’t know what to do with it. Fortunately if you take a longer view, there is one clear exception to that: contexts where you can use programming to build stories/worlds. Games like Minecraft, Roblox, etc, expose kids to programming in a way that isn’t “let’s do this faster,” but instead through the joy of making universes to explore. If I were going at this now, I’d be focused on building a path for the folks who grow up in those worlds to apply what they learned to the more mundane “adult” life.

to second @ibdknox in the early days, there weren’t these dominant platforms / models that everyone was already used to having. I think there’s optimism here if you look at other areas entirely where the experience & tooling is very different for an end-user vs a professional. Cooking is the best example I keep coming back. Home chefs / ’amateur’s use scaled down pots & pans. Industrial kitchens are more like factories and use giant equipment that an ‘end-user’ would never really consider buying. But both of these are 100% valid industries and crafts. There’s more content targeting end-user home-chefs than there is for elite chefs. In theory, restaurants could provide 99% of the meals that all humans need but cost, cultural traditions, etc are still big barriers. This has preserved home-cooking for literally centuries! If restaurants / central kitchens could provide all meals at the same cost as you making it, I predict we’d see a huge dip in home-cooking. Most people don’t make their own furniture anymore but pretty much everyone still cooks to some degree (even if you’re just microwaving food). Necessity is really the driver / mother of invention here. Home-cooking is still the best hammer for most people’s food problems

This just showed up on my feed and seems relevant. https://twitter.com/tayroga/status/1296538378255491072 Essentially programmers (and myself) do some programming for our own problems. But the systems we have aren't fundamentally designed for end user programability. So the burden is large. We do it more where some software has been designed for extension in some respects only - e.g write a plugin for your editor.

I like Alan’s heuristic of fist / hand of code, and then a “page of code”. If adding something takes more than these… the burden might be too much. Anyone (even kids) can quickly load code-context into their short-term memory and extend / modify software. Past that, you’re closer to a programmer

Speculation: The vision that Alan Kay describes in this talk hasn't been realized in mainstream software because it doesn't fit with how industrial societies see production as separate from consumption. We even use the economic link between the two (money flow, GDP) as the measure of collective wealth. People solving their own problems don't contribute to GDP, so their work has no value for the economy. If there is some truth to this, the good news is that industrial societies are slowly realizing that maximizing GDP isn't such a great idea. And in a way the FOSS movement is a first sign of this in the software world.

@Konrad Hinsen I agree but I think my comment from earlier adds another tidbit here: https://futureofcoding.slack.com/archives/C5T9GPWFL/p1599602445178200?thread_ts=1599588394.135900&cid=C5T9GPWFL There’s no economic story around end user programming. In my cooking analogy I give, making food yourself has pretty much always been cheaper than buying restaurant or mass produced food. There’s also been strong cultural ideals around cooking. But if it was ONLY the culture, I’d suspect very few people would cook.

When @ibdknox says we need a cultural change, he’s right. Our culture is utilitarian and wrapped in short-termism. This is reflected in computing as well.

I share all of your thoughts @Eric Gade and sentiments. There’s a pessimistic and an optimistic view, but the nature of the change of revolutions seems to be about the niches that adopt these ideas. The good thing with some of the areas I listed above is that I feel / think that those end-users are a bit more patient. They want to do more with less and they’re willing to try tools that are different but can enhance their workflow. There’s still the “integration with existing tools / workflow” problem though. The data science one is that ultimately any fancy EUP tooling still has to talk to a clunky big database. With art, people expect similar formats as the output (SVG, PNG, MP4, etc).

@srini Great examples! I'm particularly interested in the Hollow Knight story, if you happen to have a link or any other source I'd love to read more about it.

Perhaps expressiveness > utility

Lots of good contributions here... just one more comment on the "most people don't need more than ready-made apps" argument: that's really a cultural issues. People need food and shelter, plus whatever it takes to become a worthy and respected member of their society. Software is new, it appeared when our societies were already well into an industrial mindset of production and consumption. Other DIY activities, including cooking, have been part of our culture since long before the industrial era. One reason people cook at home is that they grew up seeing other people do it, and they have seen the advantages (economic, social, etc.) it brings. In a culture without end-user programming, it's not surprising that few people miss it.

@robenkleene https://twitter.com/AsherVo/status/1293653951309426688

Thanks! Made in Unity with Playmaker (which I'd never heard of) https://assetstore.unity.com/packages/tools/visual-scripting/playmaker-368 Just amazing!

The theme of the responses here seems to be: Only a small percentage of people are interested in using programming to improve their own workflow, but much more people are interested in using programming to build things to share. Which I think resolves my initial conundrum about programmers not using programming to solve their own problems: I was looking at scripting and customization, but what I probably should have been looking at is things like side projects and personal sites. Which to me seem much more popular than scripting/customization?

I would nuance this more. Clayton Christenssen / Jobs-to-be-done framework and all these other stuff from product management land (disclaimer: I used to be a PM) emphasize that people don’t care about your product. They care about making progress towards a problem they have and they may hire your product / service or another one based on how well it solves their needs. Cooking food solves many problems for people (cost to feed family, taste — restaurants can’t quite replicate home food taste for many, convenience once you know how to cook, feeling of self-sustenance, social — cooking for others, and probably 5 more). Getting food delivered is a worse proposition in many cases (except when you’re busy or money isn’t a concern or you want specific cuisine). Right now, most programming doesn’t solve problems people have today. It’s not ‘economically’ a better solution for anything for most people.

using programming to improve their own workflow, but much more people are interested in using programming to build things to share.

There ARE people who care about improving their workflow. Bankers and traders 🙂 they learn the shit out of Excel and use it to automate their workflow, check for errors, etc. B/c it literally saves them time and helps them make more money or get an advantage in the market.

Regarding bankers and traders, do you think a higher percentage of those groups customize their workflow with scripts than programmers customize their workflows? Sort of my starting theses is that for programmers, using programming to improve their own workflow is niche. I'd be very curious if there are other industries where using programming to improve their workflow is mainstream (for that industry).

no idea tbh, but I know a large % of them maximize the hell outta Excel. They really learn them well. Maybe programmers do this in other ways (customize their emacs setup, or their shell setup or w/e for little wins here / there).

Re "FOSS being counterproductive strategically" - this twitter has interesting takes: https://twitter.com/jonathoda/status/1104522585092481024 Hypothetically lets say what's holding us back is poor forms of composition and the existing ontology of computing, so we need to design new forms that scale up better. However to be adopted we must be compatible with the existing world and so "compose" with it - so we perpetuate the existing composition and ontology models. This affects both commercial and free/open source software. Might affect side projects more because of a greater need to 'fit in'.

yeah I saw Jonathan’s tweet when it went out and it really made me ponder! At work, we’re commercializing / stewarding this viz / BI tool called Apache Superset - https://github.com/apache/incubator-superset but right now the #1 reason people use Superset and leave PowerBI / Tableau is because “we’re free” lol. We haven’t leveled up yet on the value front

I’ve described it this way. I think of FOSS and perhaps establishment computing today as being like the medieval scholastics. The certainly produced a lot of work, even original work in some sense, but it was all confined to Aristotelian thinking, and their contributions were increments on rehashing Aristotle. For centuries.

Regarding:

I mean we have an eng team at work and they’re constantly looking for ways to improve the cloud deployment infrastructure, speeding up test suite, etc.

Personally, I've struggled to get other engineers interested in working on these things, but of course that's extremely anecdotal. I'd love to hear perspectives from others on this.

> I mean we have an eng team at work and they’re constantly looking for ways to improve the cloud deployment infrastructure, speeding up test suite, etc.

Personally, I’ve struggled to get other engineers interested in working on these things

I think when an org is big enough to have an internal products or platform team, they would work on these optimization/efficiency/infrastructure tasks.

In my experience, once an org reaches a certain size, there will be a dedicated team to maintain this stuff, but until then no one really wants to work on it. (Which I find so odd, because if there are any problems with it, if you're a dev, you're feeling those problems on a daily basis.)

I agree, based on working at a company with about 200 engineers. In a startup there is a lot of firefighting and pressure to launch / work on user-facing products, and infrastructure can be just good enough. When you have permission and management support (and even incentives) to make infrastructure really solid, then you can fix a lot of systemic problems. At the opposite end, you can go on infrastructure vision quests, making something that other devs don't want to use, etc.

It's probably been said in the other 109 comments, but more people would shell script if the tool names and interfaces weren't so incredibly unmemorable. Is anti-memorable a thing? (I just decided it is.) The interface kinda stinks. Git has an anti-memorable user interface, but by building GitHub and the GitHub desktop on top, it's now pretty easy to use without the fear of never seeing your code again. More people use git (i'm guessing) than shell scripts, even though its command line interface is irregular, dangerous, and unpredictable.

was just checking out some of bessemer’s recently released investment memos and i think an argument can be made that Shopify is actually a great example of an end-user programming success story: https://www.bvp.com/memos/shopify

yeah I agree. Shopify is a beast of a company and they aren’t interested in trying to centralize everyone onto a vague platform like Medium did. They embraced “the edges”

I might be missing it in the thread here, but one reason even programmers don't script things is that there's no good way to do it, a huge gap between interacting with a GUI and automating the interactions.

Coming back to this thread, I am wondering; what is that "programmer" category? Professional software developers, perhaps? Is that a sufficienly homogeneous group in their professional practices to discuss how much they develop software for their own needs?

My perspective: I did more "end-user programming" before I was a programmer, but knew enough programming to be useful. This was at an office job, where everything ran on Excel. I was doing lots of web scraping, data processing, making small tools for tedious tasks, etc. Then I got a job doing programming, and didn't have any material problems to solve with programming any more!

My experience definitely matches @yoshiki's (although I went the other direction, out of software development). The amount of menial, manual use of computers that goes on in the world is a travesty. I'm talking about stuff like physically typing filenames into a Word doc. It's long been recognised that a manager might send emails to her staff, instructing them to do some task on a computer, in a way that is almost pseudocode. e.g. "can you go through every sales report on the K drive for February and check if any of the unit codes have expired?". End user programming should be looking to eliminate the middle human from this kind of human->human->computer situation. This is the type of thing that SQL was meant to eliminate, but it didn't. I think the issue is solutions like SQL demand too much subservience in how information is managed, and they want it stored in a way that is different to how humans would naturally do it. The also demand labour up front (at the point of storing), for no immediate reward, which is always a foolishly optimistic thing to require. I have high hopes for a system that approaches the OS from the same perspective as the user, for example all text OCRed, structure automatically inferred from physical layout etc.

@Chris Knott Totally agree about the manager request example. I did a freelance job a couple of years ago for a non profit. They really just needed a specific way to process bank account information from several online services into an Excel template they used. The people working there knew exactly “what needed to happen,” but the tools on their system had no way for them to express it, and the services they were using were siloed from each other

Regarding:

one hypothesis is that programming tooling is already pretty well optimized for the output that the industry wants from programmers. Programmers are customers too: It only takes a small amount of dedicated people making tooling like IDEs, text editors, plugins etc for the majority to benefit from the care and thought put into these tools.

This is an interesting perspective, and I believe it's true to an extent, e.g., while almost every powerful application from Excel to Photoshop to DAWs has a cottage industry of extensions that surround it, it's certainly true that developer tools dwarf the others in quantity of existing customizations. But I still see a lot of contrary evidence. E.g., it still seems that most programmers, when they encounter a problem with a programmatic solution, they'll tend to choose a manual solution. The canonical example is automated testing. While automated testing has certainly become popular, it still seems to be in the "eat your vegetables" category, instead of something programmers just do naturally. Which is odd if you think about it, because manual testing is just that, manual, and programmers are master automators, so...? More personally, what I find most fulfilling about writing my own scripts and customizations is that I can make the software behave the way I want it to. When you're using someone else's customizations you're always at the mercy of the creator's decisions. I don't think that wishing your tools worked differently is rare, that's a sentiment that I feel almost every computer user has, but programmers are the only group that's empowered to change how their tools work today, and for the most part, they still choose not to.

I’ll challenge some of your assumptions here @robenkleene not sure if this is useful, but could be another perspective!: • Automating isn’t always better than manual. Clicking through a new UI sequence can give you the ULTIMATE end-user gut check. Things can still go wrong (caching, what have you) but less is likely to go wrong. You can build a burrito robot but unless you taste the burrito at the end with your own mouth, you just don’t know if everything worked as you thought it would. • Automation is often less concrete / tangible. Similar to the first 1, but automation is also more abstraction. Abstraction is complexity. Even if the automation script is something simple, there’s overhead now to maintain a list of automations. Perhaps we need better ‘automation interfaces’ where the overhead is brought to 0. Analogy: I don’t think about not eating cookies in my day to day, I just don’t keep cookies in the house. 0 overhead!

I agree 100% with both of these points. I'd be the first to admit that I like automating things because I don't like doing them manually, not because I think it's objectively better by any other metric besides my own personal preference.

this is a long thread… it could be interesting to fork and start a new thread and this time include like 5-10 concrete examples in your life / things you’ve seen other programmers do @robenkleene etc to spark the discussion!

I'd also say that "how can doing things manually still be better than automating them in 2020?" Because I believe that it's still better for many, possibly even most, cases is one of the central questions I'm grappling with.

@robenkleene: Nooooooo! 😄 It seemed like you were moving towards the light here:

I'd be the first to admit that I like automating things because I don't like doing them manually, not because I think it's objectively better by any other metric..

But then you immediately put your blind spot back on in the next comment:

how can doing things manually still be better than automating them in 2020?

That framing is only going to lead you in circles. As @srini pointed out above, and as I tried to say in the overflow thread ↪️: * The line between 'manual' and 'automated' is fuzzy on the computer. If I switch windows and type a command on the shell, I'm still making use of automation. Just less of it. * Adding levels of automation always has costs. If it seems always a good idea to you, just wait a few years. We can improve lots of things here, but it's just not a reasonable goal to aim for "adding levels of automation should always improve life". There will always be situations where doing something manual is simpler, faster, less alienating. Start developing some warm and fuzzy feeling for doing things manually. * Desire for automation is subjective to some extent, as you pointed out in your first comment. I'd say join me over here where the goal isn't automation but comprehension. Practice throwing kicks not because kicks are always a good idea, but just so you build up judgment on when to use a kick, and so you can do a lot more with a single kick when the moment arrives. (Movie recommendation: https://www.imdb.com/title/tt0061770)

Adding levels of automation always has costs. If it seems always a good idea to you, just wait a few years.

I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem. Correspondingly, I tend to choose to solve problems my approach is a good fit for. I do a lot of work with frameworks, where automated testing is more important. I tend to avoid issues in the UI layer, since manual testing is usually more efficient there. (I actually love working on design-system level UI stuff, but bugs that exist between the UI and the data layer are probably my least favorite thing.) I would love to hear more about this "love of doing things manually" though.

I would love to hear more about this “love of doing things manually” though.

I’m thinking if the environment you are in is introspective and malleable enough, the manual approach is just so easy

Sure... I agree that line can be fuzzy, but I guess I'm not sure how useful that distinction is, e.g., I'd put someone using a malleable environment / using automation on one side of the coin, but most programmers are still on the other side of the coin, where they're not using a malleable environment (or at least leveraging it), and they're not automating.

I think someone should do this as a real study

Also if anyone wants to share there definitions or examples of malleable environments (i.e., environments that are so efficient that they remove the advantages of automation) I'd love to hear about that

The most obvious thing that comes to mind is text editor keyboard macros

Ditto for doing things in Emacs lisp. Once you have the lisp made for it, turning it into a command is easy

Yeah, that's the first thing I thought of too. Keyboard macros seem pretty clearly automation to me? (And correspondingly, would be a feature most programmers don't use). It sounds like there's a concept of a malleable system that's not based in automation here, I'd love to understand what that is. (And I'd love to hear any more examples of both.)

While I object to needing your IDE to have thought of the things you want, JetBrains does a pretty good job of thinking of the things that my coworkers and I want, and I think we do leverage it to an extent that it is “automation”. Multi-cursor, regexp-replace and structural-replace are some examples that come to mind.

These text editor examples highlight something: good automation blurs the line between manual and automated action. Specifically, it makes the automation invisible by making it so easy it feels like manual work (this might just be my view of what you folks are talking about re malleable systems). Multi-selections are a great example. You can think of them as automating repetitive identical edits, but they feel like just cutting with a sharper knife, not programming a chopping machine. Possibly one of the key things that makes them feel manual is that you can see the changes as they're made. Not exactly a new idea (oh hai Bret Victor), but maybe a new perspective on why it's important. If every bulk edit operation on a doc or database ran in a transaction with a live preview of the changes, people might feel more comfortable playing with them.

but programmers are the only group that's empowered to change how their tools work today, and for the most part, they still choose not to.

@robenkleene I don't think it's so much that they choose not to, it's often that they don't need to.

@yoshiki Do you have any thoughts on preceding part?

When you're using someone else's customizations you're always at the mercy of the creator's decisions. I don't think that wishing your tools worked differently is rare, that's a sentiment that I feel almost every computer user has,

E.g., my observation is that developers tend to be frustrated with their tools (especially more experienced ones). Do you not find this to be the case?

Don't you also see experienced developers being more resigned to their tools? In fact, that's one way to diagnose all of us pushing back on you in this thread: we've been living with our chains so long that we've forgotten about them. That would certainly explain why we don't customize more. I had more customizations in 2001-2005, then I moved and changed a lot of stuff and just restarted my text editor settings from scratch. And they've never grown back to the sort of multi-megabyte state they were in then. Backing up to your previous comment to me (I wrote a response yesterday that the Slack app ate):

I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem.

I don't understand the distinction you're making here. At least, what you're saying sounds like it answers your earlier question, "how can doing things manually still be better than automating them in 2020?" Funny thing is, I do a lot to automate testing. In fact, one way to view my Mu project is as making the UI layer easy to test.

I would love to hear more about this "love of doing things manually" though.

When I do something manually I sleep soundly at night that I haven't created new tech debt for myself. I don't have something new I have to maintain, or try to read later to try to understand just what I was thinking. Manual labor can be therapeutic, like gardening. All these things have nothing to do with the state of the underlying system. They're just about the mess I make, and about taking care of my own state of mind. Sometimes I do things manually for a few days even when I'm sure I have to automate them eventually. Manual work keeps me close to the data and might give me some new insight. "Being the computer" helps me understand the problem before I try to solve it.

E.g., my observation is that developers tend to be frustrated with their tools (especially more experienced ones). Do you not find this to be the case?

I see pockets of people feeling frustrated and other pockets of people who aren't. It's difficult to summarize since I think the landscape is complex(like you said, experience plays a part). I agree though that not everyone is satisfied. I'll have to revisit this later when I have more to say.

On automation: it has two very different costs that need to be weighed against the benefits. One has already been cited: the initial effort to put automation in place. The other one is less obvious: a fading understanding of what is really happening. It's much more pronounced if you run someone else's automation, but also happens when you run your own code for a long time without studying it from time to time. And if then you have to change something, it can be difficult. So automation is an obvious win only if the two costs are low. Stuff like renaming hundreds of files: shallow but lengthy. The loop with a few-line body.

@Kartik Agaram I really appreciate these comments, some responses below:

Don't you also see experienced developers being more resigned to their tools? In fact, that's one way to diagnose all of us pushing back on you in this thread: we've been living with our chains so long that we've forgotten about them. That would certainly explain why we don't customize more.

This seems to be saying the same thing I am, experienced programmers are unhappy with their tools, but accept them (that's the way I'm interpreting "resigned" at least?) The question is just why. You've given some great reasons, but I'm not entirely convinced (not saying I disagree either, just not sure either way), but I think the best response would be to give some concrete examples. That's also something a couple of others have requested too, and I also think it's a good idea, so I'm going to start a new thread soon with examples.

I had more customizations in 2001-2005, then I moved and changed a lot of stuff and just restarted my text editor settings from scratch. And they've never grown back to the sort of multi-megabyte state they were in then.

I used off-the-shelf software for the first half of my career (2002-2010), mainly with the defaults, then started customizing after that. I realized it solved a really really big problem for me: I used to keep changing software because an alternative would solve some problem I have, but after using the new software I'd realize the old software did some other things better. So my total number of problems would always stay the same. Now that I customize, my total number of problems goes down. Now I think of applications like as a shell that I can customize to make it do the things that are important to me very quickly. And I'm so much happier as a computer user this way, because I'm not in a constant state of frustration because nothing works the way I want it to like I was before (this is is probably a core trait of customizers/automators, as well as contrarians in general).

> I think you're misunderstanding me here, I'm specifically saying I don't always think automation is a good idea, I'm saying that it's the way I prefer to solve problems. It's an inclination that has more to do with me, than it does the problem.

I don't understand the distinction you're making here. At least, what you're saying sounds like it answers your earlier question, "how can doing things manually still be better than automating them in 2020?"

To me these are two separate thoughts: In the first, I'd guesstimate automators are about 1/10 of programmers, I'm just saying I'm in that 1/10. The second part is just surprise that using a computer programmatically is still so difficult in 2020, given it's a problem so many people have worked on. Not sure there's a contradiction here? I think you're saying #2 leads to the 1/10 in #1, which I agree with. But I don't see why that would change my personal preference?

When I do something manually I sleep soundly at night that I haven't created new tech debt for myself. I don't have something new I have to maintain, or try to read later to try to understand just what I was thinking. Manual labor can be therapeutic, like gardening. All these things have nothing to do with the state of the underlying system. They're just about the mess I make, and about taking care of my own state of mind.

I understand this in theory, but I just don't experience using computers this way. This reminds me of how I feel when I watch Gray Bernhardt type, I find him amazing to watch, but he has an affinity for the mechanical act of typing that I just do not share. I find just imagining doing what he does myself exhausting. (Also, doing that many small mechanical motions relative to his output seems like a recipe for RSI to me.) There is just no way I'm ever going to use a computer by typing everything out the way he does, the "cost of a key stroke" is just higher for me than it is for him.

@robenkleene That really helps clarify things. I certainly agree that computers today fit better for people with a certain profile (who find certain things like typing cheaper to do), and that there are huge barriers to customizing them for other profiles. I think it analogous to the direction of a vector rather than its magnitude. The things you want are not that far from the current defaults, it's just that there's huge impassable mountains in the direction you want to take computers, whereas I[1] and Gary Bernhardt have the advantage of living in the direction of the flat plains. (And the poor laypeople are out in outer space with no service.) [1] Though I see myself moving closer to you over time, what with my recent RSI troubles.

I’ll just add that I think both approaches work, John Carmack seems like a use the defaults type (https://twitter.com/id_aa_carmack/status/1302651878065475584?s=21). Linus seems like a customizer (maintaining his own Emacs, and

git

seems like explosion of Bash scripts, at least when I’ve looked at it under the hood).

I'm noticing an analogy here with lifestyle design. A few years ago there was a movement towards minimalism until people realized that not everyone is rich enough to afford minimalism[1]. Similarly, I think when you see someone who says they just use the defaults, you're really seeing someone privileged to be better suited to their environment. [1] Arguably this pandemic has made us all poorer by forcing us to run less lean and maintain more inventory of more types.

I think the distinction is that the self-written software there is not something that produces a result. He's talking about trying to automate something that he can then forget about and rely on. That's completely different to cobbling something together that gives you active superpowers right now, for something that you can see or check worked.

True of the article in general, but this particular quote popped out at me for how much broader it was phrased.