Every time I need to persist some simple data on a web app, I go through a dreaded “which database should I use?” process which – given I’m not building a global panopticon – is *always* overkill so I’ve decided to bite the bullet and write a very basic transparent (using Proxy) in-memory database for Site.js that flushes to JSON files on each write and is available on all your routes so you can use it without knowing you’re using a database. Working title: What Database? :)
So I’m throwing around ideas for the interface of the simple local persistence layer for Site.js.
Please feel free to chime in :)
Last time I hit this I just used SQLLite - it's file based, it can be memory based. Moderately powerful SQL engine.
@fjfish Yeah, SQLite is great. But still overkill for my needs. And for most Small Web needs, really.
Especially given that for any large-scale content, Site.js has Hugo bundled (which we can optionally expose for production use as well) and any interpersonal data should really be modelled via a peer-to-peer replicated system (I’m not ready to integrate such a system into Site.js yet but getting closer to it.)
@aral The answer is Postgres. Never write your own DB.
@deathpanels Sure, let me just add Postgres as a requirement to Site.js and… kill it in its cradle :)
Nothing’s a silver bullet. Evaluate the use cases and limits of what you’re building and make your design decisions accordingly.
Sometimes, that might mean writing your own (very basic) DB :)
(That said… never roll your own crypto.) :P
@aral Postgres can do json really well WITH indexes on column values.
I totally do that all of the time, databases are big honkin' beasts and I never feel like starting one...
When I outgrow `Fs.writeFile(JSON.stringify())` I move on to writing JSON objects line-by-line to a log, I call this ndjson (newline-deliniated json), which has a nice feature of being able to return to any historical state by reloading only part of the log.
@cjd Hey Caleb, yeah. I think we’re just conditioned to jump to some database by decades of centralised/“Big Data” (which is almost never big data) dogma.
As long as you don’t have circular references JSON.stringify and fs.writeFile are beasts when it comes to write performance for Small Web use. (I did try a stringify that handled circular references but that gets very slow very fast.)
@cjd (And yeah, append-only logs are great but a flush-on-write in-memory database fits my use case of wanting a transparent API better. As I see it, if you outgrow this database, whatever you’re building isn’t really Small Web.) :)
For me, every project is different, CryptPad is all append-only logs but that's not Small Web as much as it is Lazy Web, or NoSQL-without-the-hype.
I would be a lot more likely to use databases if they weren't garbage at their actual stated purpose (btw, stop by at the Q&A for my #APConf2020 lecture which is about this)
@aral Please don't. If you want to integrate it, use SQLite or maybe something like indexdb.
I stopped counting how often my "let's just write a json file somewhere" resulted in dataloss. From "oh the process went OOM and didn't finish to write the data" to "it turned off in the middle of it" if you want to save everyone a bad experience, use at least atomic writes.
@sheogorath I can see how you’d get data loss if you aren’t queuing your writes. But if you look at how modules like JsonDown, etc., are handling it, that’s basically what they’re doing also.
Why don't you pick something like SQLite and fix the issues you see until it serves you well?
Creating a new database is a lot of effort and in the end you will make a bigger contribution if you just help an existing project.
@aral Do you mean NeDB? BTW, what is your opinion on this? https://www.aljazeera.com/indepth/opinion/aligned-technologies-movement-200907090048112.html
@aral Maybe database is the wrong term? I think people strongly associate DB with SQL or some other query language. Will you be having search capabilities or is it a key-value store?
@alfonsomunozpomer You’ll be able to query it using something like this (not final, just playing around):
In your route:
@martyn Hmm, in general, I guess, yes. Trying to go for a much simpler interface though.
@jauntywunderkind420 Neat. (But I need this to be in-process.)
@aral so I just brainstormed some things, that imho still are relatively kiss.
Added one example of nested json that might add a tad of complexity, but then again maybe not if looked at well.
@humanetech Thanks, appreciate it. About to take a look now :)
@aral I added a follow-up regarding negations :)
@aral neat! What would be the limitations?
@edgaras In-memory, no circular references in your object graph (huge difference in speed between native JSON serialisation and the libraries I’ve been able to find that can handle circular references so leaving that out) and not for Big Data™ ;)
@aral I see, so the data is wiped with browser local memory? What about simple file-system solutions like SQLite (I know a bit old school) or similar?
@edgaras Ah, no, sorry, it’s in-memory as in the whole database is kept in memory (so must be < 1.4GB without tweaks) but it is written to disk as soon as possible after updates.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!