Please, share your favorite resources about approaches to versioning of APIs, DB schemas, code libraries etc. Versions rollout, support, and deprecation still feel unsolved and ad-hoc.

Very true. That's why static linking (modular monoliths) are so much nicer to work with. Here is a fun idea: separately built pieces of code often lead to incompatibility and bugs when they exchange data whose schema has changed. Would it be possible to keep the data distributed but send code as messages (as opposed to keeping the code distributed and sending data as messages).

was hoping for approaches within traditional architectures 😅 e.g. https://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html but, say for SQL databases. For scenarios where 1 field is no longer enough for <feature> and second is added, but prev api version still needs to be online, etc.

TL;DR: versión 1 and 2 of a library should packaged as deferent dependencies that can be loaded into the runtime at the same time to make migrations possible earlier. https://jolynch.github.io/posts/semver_considered_harmful/

Libraries are the easiest case of all, especially pure ones. Just(?) make runtime names be different between versions, eg. don't reuse java class names between versions, so many of those could coexist within the same runtime. Not pure (with some singletons or other equivalents of shared state) are not so trivial though.

libraries -> semver APIs -> never deprecate, append only. Add route prefix v1/ v2/ etc. for breaking changes. I like the Google API Design guide, but also the Zalando one. DBs -> Expand - contract pattern -> Alembic or something similar to remember where we are. Zillions of articles on the expand and contract pattern, I feel that is well trodden best-practice, which also applies to infra changes.

semver sucks never deprecate api - not viable in the slightest

This is an overview of how we did hundreds of database migrations while minimizing client burden.

For APIs, I think server should return api version number and client should know how to use it. https://jarirajari.wordpress.com/2022/02/08/api-design-fundamentals/

This is my personal canonical text for web API versioning https://stripe.com/blog/api-versioning

Are you specifically looking for external-facing versioning? For internal, my favorite technique is live code generation with a compiled language, where I can quickly see what I’ve broken. If I didn’t have legacy architecture, and could follow that concept all the way through, I’d reach for Lamdera: https://dashboard.lamdera.app/docs/overview. Then all API changes and migrations become type-safe mappings. Obviously that doesn’t work for everything, but it’s powerful when you go all-in.

Libraries are the easiest case of all, especially pure ones.

Elm has spoiled me to everything else. 😊

Not only external-facing one. But external is the hardest, because it is a superset of all others, and has all the problems, smaller pieces might not have: shared global state across versions (db), unknown open set of clients spread across time (you never know if 4yo client will call some endpoint within next 5minutes), etc. (actually, what else?). On the day job, I need to figure out cheapest 80% solution to at least detect (if not enforce) backward compatibility breakage, balancing dev speed vs. support burden. In my after hours project (think unisonlang, but retrofit, not green field), I need to figure out how "library release artifacts" look like across the versions, so those would cooperate with the rest of similar libraries or projects. So any strategies, tactics, povs, tips and tricks, gotchas, usecases, – help, at least to map out the problem/solution space, before grounding onto particular tech stack.

expand and contract is quite clear https://www.prisma.io/dataguide/types/relational/expand-and-contract-pattern. You add expressibity (new structure), migrate the usage, then contract expressivity (remove the old). If you want to deprecate an API its the same thing except the migrating the usage involves telling every caller to stop using the old API which is not practical for public APIs. At Google it was common to find the V1 proto was transforming to the V2 proto and then just calling the internal v2 handler. v1 was never deprecated, but there was not 2 business logic handlers. Its not always possible to nest, depending on how bad you messed up the v1 design. I always work from the assumption you can never deprecate an API if it is public. Internal APIs => expand and contract. Google API guide prohibits removing fields from protos, and the modern gRPC removed

required

fields because they prevent future migrations. Every long lived proto message is stacked with deprecated fields, but they are at least costless on the wire. A fancy DB migration technology I keep staring lustfully at is https://pgroll.com/blog/introducing-pgroll-zero-downtime-reversible-schema-migrations-for-postgres but I have never used it (its docs even say it implements expand and contract).

“never deprecate API” worked very well for the team I used it on for a large project that had a high request load and a low error rate, for whatever that experience is worth Not saying you need to use it, but i think dismissing it out of hand is unwise

https://killedbygoogle.com/ :)) There are plenty of business situatoins where "never deprecate an api" is just not an option. Even from mechanical new-data-required point of view (legal, financial, regulation, gov id auth, etc.), not to mention all the flavors of "we can't afford to spend time on it, so iphone4 support goes bye bye". E.g. not everything has sensible (legally viable) defaults to upgrade v1 req to v2 req to return v2 resp. "you should have never screwed up v1" – not a strategy. "I've never had a use case where I needed to deprecate things" – not a tactic. "never deprecating things costs you nothing (or is always 1) affordable 2) by you)" – is just not true. "you screwed up your architecture, and now versioning is a mess. just file for bankruptcy, or start over" – rarely an option. -- anyways, reading through the linked articles, thanks!

the API is up tho https://developers.google.com/workspace/admin/reports/v1/appendix/activity/jamboard 🤣

some of "graceful" deprecation tactics: • public schedule way ahead of time https://developers.google.com/google-ads/api/docs/sunset-dates#timetable • (mobile apps) popups for some use cases (apis/workflows): "please upgrade an app", e.g. when payment processor needs more user info, and the only way to collect it in app - app store release, and reinstall

I wrote about our model here: https://notes.billmill.org/blog/2024/06/Serving_a_billion_web_requests_with_boring_code.html A more careful phrasing of “never deprecate API” in the sense I mean it here is “only make backwards-compatible changes, and if you ever want to remove a response field or API endpoint, deprecate it, wait a long time, and then remove it if metrics tell you nobody is using it” I don’t think that prevents having a v2 of your API, but the one I wrote about there is still on v1 7 years and 10s of billions of requests later

Nitpicking here, but I think this is very important within this topic: you can't always "only make backwards-compatible changes". (you, gov, payment processor) need a new field? – breaking change. This is not only skill or desire issue, it is not always within your (service) control. Yeah, you can try to preserve structure, but sooner or later the only sensible backward-compatible response you would be able to give is "sorry, no, upgrade". Yes, api stays online, yes, response adheres schema, but actual feature workflow this api call is part of – no longer useful because it is just a dead end. And I am not sure I would always(ever?) choose this dead end over 404 "api no longer available", as 404 at least does not give me hope "sorry no" is temporary (human is not always in the loop to read human readable error message to detect meaning change without response structure change, e.g. [200 "result"] vs [200 "sorry, no, upgrade"]).

The old APIs in financial sector are XML based and still work (excluding availability). What they do is start introducing modes, like we will respond with Y if you sent X in the request unless z was present. They are awful to work with but you can work around any problem with an additional logical layer of indirection.

More reading - I've quite enjoyed Phil's work over the years even though I haven't found it so applicable personally. https://apisyouwonthate.com/blog/api-versioning-has-no-right-way/

Tom, the qualifier to "The old APIs in financial sector ... still work" is "*some":* "some old apis still work". It is possible to overcome a lot of things, but just not all of them, e.g. due to (lack of) funding or just things being out of your control. When your transitive dependency disappears, what you gonna do? https://docs.stripe.com/payments/older-apis#deprecation-of-the-sources-api > We’ve deprecated support for local payment methods in the Sources API and plan to turn it off. Again: https://developers.google.com/google-ads/api/docs/sunset-dates#timetable Notice how google (you give as a never-deprecate-an-api example) has

v19

as the oldest available version in that table 🙂 (and yes, there were at least 18 major versions, and the

v1

is just 7 years old), and quote https://developers.google.com/google-ads/api/docs/sunset-dates#differences_between_deprecation_and_sunset > API endpoints for the sunset versions stop working after the sunset dates. The Google Ads API will throw an error if you try to access the API endpoints of the sunset versions. Meaning even top100 money bag in the world choses to shut down 1yo apis (v19 release February 26, 2025, shutdown February 2026)

I think Google ads is a special case

Bill, (some highlights are mine, some - original)

Given that we (tried very hard to) never make backwards incompatible changes to the database, the *app*s would check at startup that their database schema version number was greater than or equal to the database version number stored in the database, and refuse to start if they were not.

This was a general pattern: If an app encountered any unexpected or missing configuration, it refused to start and threw noticeable, hopefully clear, errors.

:) Everything is way easier for internal "network of apis and clients", simply because you can have dashboard with all mismatches. "Internal apis" - is an important qualifier to any "never do this" and "easy to do that". On the other hand: I use your api. You don't even know this. You shut down v1. My app does not start anymore. You never see this. What do I do? What do you do? :)

I use your api. You don’t even know this. You shut down v1. My app does not start anymore. You never see this. What do I do? What do you do? :)

It would be a long process of announce a deprecation a long time in advance, add warnings, eventually when it’s close to being shut off do announced intermittent blackouts, then finally shut it off. It would be extremely protracted and painful, but that was appropriate to the API we were building - it was a big commitment

(you, gov, payment processor) need a new [required] field [in the request]? – breaking change.

yes, absolutely! That would be a new endpoint, that is a consequence of this API design. You can add endpoints safely but not add a required field to them. This isn’t a gotcha, that was the design

"never deprecate" ≠ "finally shut it off" 🙂 "> new endpoint" is ok, but is not the point. the point is "current endpoint is de facto can't fulfill its purpose anymore (broken), whether you keep it online or not"

The topic of software evolution or stable interfaces reminded me of Zawinski's Law of Software:

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

I like to think of the chain of callers like a rubber band. As one side changes, it puts pressure on the other end to keep up. You can only hide the issue behind backward compatibility layers for so long until something breaks.