https://futureofcoding.org/ logo
#thinking-together
Title
# thinking-together
m

Matthew Linkous

06/13/2023, 4:38 PM
I’m been thinking about this concept of a “full-stack” database. I’m curious what connotations that invokes for folks here and what features you might imagine a futuristic, “full-stack” database would have? Bonus question: is it “fullstack”, “full-stack” or “full stack”?
w

Walker Griggs

06/13/2023, 4:48 PM
Full stack, in my mind, has a few connotations: level abstraction and accessibility. Two things that come to mind: • Designed to be used by all front-end, back-end, and edge applications. A cross between firebase and postgres, if you will. In many ways DynamoDB feels like an interesting abstraction: NoSQL built on a MySQL backend. So my S1 when I hear "full stack" is a realtime nosql abstraction built on a strictly relational db (with an escape hatch to access the latter). • A DB with bundled, native ORM that speaks wire protocol. Most ORMs I see are built as an afterthought. "Full-stack" evokes notions of accessibility from a language API perspective. This point is more of a personal feel and not a practical concern though
Bonus: I often see a micro-service deployed to translate between the business logic and the DB layers. That service often ends up being "endpoints on a stick" with a side of "validation and authentication" A "full-stack" db in my mind would alleviate the need for such a service. Whether that means exposing http endpoints and natively translating to the query language or omitting the query sql language layer all together and exclusively bundling the ORM + query language + wire protocol, I'm not sure. Your question is bringing a few thoughts on accessibility to the surface and I'm brain dumping
m

Matthew Linkous

06/13/2023, 5:09 PM
Very interesting! I like where your heads at.
e

Eli Mellen

06/13/2023, 5:21 PM
Have you looked into frontier?
I also wonder about array programming languages, especially k/q that come bundled with a database, so the distance between language and data and database is minimal
m

Matthew Linkous

06/13/2023, 5:22 PM
Mmm never heard of that but that sounds similar to how SmallTalk is part database
e

Eli Mellen

06/13/2023, 6:29 PM
yeah! a similar vibe for sure
k

Kevin Greer

06/13/2023, 6:56 PM
Makes me think of a few things: 1. https://en.wikipedia.org/wiki/Naked_objects 2. My own FOAM Framework, which was an actual database in earlier incarnations, but now abstracts away the underlying database

https://www.youtube.com/watch?v=S4LbUv5FsGQ

, 3. I don't have a reference but I once read about someone writing their whole web application in Oracle stored procedures. 4. IBM's OS/400 https://www.reddit.com/r/hackernews/comments/13603zt/ibm_as400_databases_all_the_way_down_video_2019/ 5. Gemstone, the Smalltalk Database: https://en.wikipedia.org/wiki/GemStone/S
m

Matthew Linkous

06/13/2023, 6:59 PM
Ah yes I've seen the IBM demo! So cool. Havent heard of FOAM before. Has that pattern / your framework unlocked any cool abilities for you? @Kevin Greer
Very cool. That sounds similar (at least on the database side) to how I understand Facebook’s ENT framework works
k

Kevin Greer

06/13/2023, 9:28 PM
Thanks for the ENT link. Yes, it looks similar, as do many other ORM and middleware systems, but the difference is that FOAM is much wider, in that it spans all the way from the back-end of the back-end to the front-end of the front-end, so spans several languages in the process. This also helps to avoid a lot of the glue code generally needed to create a complete system.
a

Andrew F

06/14/2023, 1:41 AM
To me it connotes trying to abstract over an unreliable network, and the inevitable frustrating failures resulting therefrom. :') But that's just me being bitter. Better would just be a database that's usable in programs running in any setting, and maybe even makes it easy to synchronize them. Couchdb/pouchdb come pretty close, if you squint.
k

Kevin Greer

06/14/2023, 3:11 AM
The problem with giving raw database access to clients is that you then have to duplicate business logic, which is a lot of effort and error prone, and the database's built-in authorization isn't usually sufficient, so you would also like that to be somewhere trusted and secure, not on the clients.
l

Leonard Pauli

06/14/2023, 5:42 AM
A complete-stack db is the logic, ui, action/history, and networking engine just as much as it is the data persistence one. Declarative/functional/constraint based, allowing for streamlined authorization. Platform agnostic, through exporters from itself to different environments, while retaining functionally and adding native interoperability. If a field, eg. profile_picture is added, literally through less keypresses than letters in that name; (auto suggestions, inc. concept type with acl, UI, performance assumptions, and more context for later suggestions/auto coding); the whole system reflects the change. Eg. on user posts, their profile, editable with upload + crop functionality... No need to update model/migration/validation(backend+frontend+orm)/authorization/endpoints/frontend api client/forms/multiple ui components/spec/documentation/... "It just works", still with the ability to change every detail down to the HDL architecture for the accelerator co-processor that will perform the upscaling if the uploaded picture is too lowres, used when exporting the app straight to custom silicon instead of, let's say as a p2p wasm webgpu webxr spacial app with a mesh of persistence servers that also syncs with the iPhone/Android/macos/linux/chatbot(whatsapp etc) versions deployed in the same click...
a

Andrew F

06/14/2023, 4:34 PM
@Kevin Greer not sure if you were responding to me, but I do strongly agree.. for the most part. The only exception is that with Couch, for my app, I'm pretty comfortable giving each end user their own database and letting them face the consequences of mucking it up. If they interacted with each other, it would get a lot more complicated real quick (my membership, billing, etc are in the standard relational setup).
m

Matthew Linkous

06/14/2023, 4:35 PM
@Andrew F I’ve always loved the idea of giving each org/user their own database instance. Doesn’t work for all apps but when you can have that separation it’s awesome. Schema migrations can be tricker if you literally have different instances of databases
w

Walker Griggs

06/14/2023, 4:49 PM
A complete-stack db is the logic, ui, action/history, and networking engine just as much as it is the data persistence one.
I completely agree here. This is maybe what I was getting out with "a full-stack db in my mind would alleviate the need for such a service [db layer logic translation]". Databases tend to be at the center of our systems and, for the most part, services are organized according to their data access patterns. It seems absolutely reasonable to me that a "full stack" DB would handle and centralize core business logic without the need for each consuming service to duplicate logic.
I’ve always loved the idea of giving each org/user their own database instance.
@Matthew Linkous , not a loft thought, but this reminds me of basic.xyz. I wonder what their team would have to say about this conversation 🤔
k

Kevin Greer

06/14/2023, 5:46 PM
We created issue database and gmail clients which stored most of the data localling in the browser in indexeddb and then just behaved as though that were the real database. Two-way updates were synchronized between the client and server when the client was online. This had two nice advantages: it allowed for simple offline support, and it offered excellent performance because of data locality. Here's an example of the kind of client-side performance we had:

https://www.youtube.com/watch?v=y9i4oW9dHHw

e

Eli Mellen

06/14/2023, 5:47 PM
@Kevin Greer was this a production system or a prototype?
I am really interested in this sort of architecture, but when I’ve explored deploying it to production, there was a significant edge case that ended up eating up a lot of lines of code to handle scenarios where users had limited device memory/couldn’t hold everything locally for some reason
the naive approach I tried was just to fail to a total REST api, sort of normal architecture, but I wanted to find a way to do it incrementally.
k

Kevin Greer

06/14/2023, 5:55 PM
The issue trackers (there were two) were production and the GMail client was prototype. We don't need to hold everything. If the user does a query, we return the answer, then if online issue the same query to the server and update the client DB, this would cause an MVC observer event and force the GUI to refresh if there was more data. Then you could have a janitor to purge local data not viewed for some time.
They're all open-source if you would like to see the code.
e

Eli Mellen

06/14/2023, 5:56 PM
I’d love that!
k

Kevin Greer

06/14/2023, 5:57 PM
I have a meeting in 3 mins, but will post links later.
Here's a screenshot of the GMail client compared to the Android version:
Here's one of the issue trackers (the other one is Google internal):
image.png
a

Alex Cruise

06/15/2023, 3:40 PM
EdgeDB is also aiming in this direction:
k

Kevin Greer

06/15/2023, 4:31 PM
Why is it called "Edge" DB? From the name I would have thought that it would run in the browser or on mobile clients, but it looks like a server DB based on Postgres.
w

Walker Griggs

06/15/2023, 4:50 PM
The name feels like a misnomer. It's a compelling offering but the docs don't make any suggestion to running "close to the client"; cdn, on-device, or otherwise. Could be wrong though!
m

Matthew Linkous

06/15/2023, 4:52 PM
Yeah if I had to guess I bet they pivoted but kept the name. EdgeDB’s query language is super cool but I don’t think they’re doing anything that breaks the Postgres mold regarding deployment.
k

Kevin Greer

06/15/2023, 6:27 PM
Apparently it doesn't have to do with the edge of the network but the edges between objects (ie. relationships) with something they call "materialized edges". (From what I read on reddit, but I didn't see that on their website.)
a

Alex Cruise

06/16/2023, 3:56 PM
I still Want To Believe™ in something like OODBs but I don’t think anyone’s figured out how to scale them
k

Kevin Greer

06/16/2023, 4:59 PM
What's your definition of an OODB? An object-graph, or something like an Object-Relationshional database where the individual tables contain objects instead of tuples and the relationships between tables are themselves represented in the objects rather than externally through implicit SQL joins? Or something else? And at what scale are you consider? I worked on this project https://www.pipelinepub.com/competitive-communications-market/breaking-benchmarks-with-microsoft-and-redknee.html/4 which scaled well using the Object-Relational model over-top of other data-stores. The Microsoft SQL Server part of that story is a bit fraudulent since the performance critical components weren't stored there, just historic records.
a

Alex Cruise

06/16/2023, 6:54 PM
hmm.. I care most about having the domain model represented well, being able to use both uni- and bi-directional relationships as pointers, not having to worry about FKs… I was going to say I don’t really care about inheritance, but it always comes up when I’m struggling with ORM too. One reason I’m not particularly sanguine about OODBs so far is I find I always need to be persnickety about the representation on the RDBMS side
It’d be nice to have an ORM that was more like a constraint solver I guess, where I could describe properties of both the OO and RDBMS sides, and have it find a match
8 Views