@mk @djsumdog That's true, it is annoying and don't forget to cleanup /var/lib/postgresql/ directoy as well. I have renamed older versions to <number>.backup so I can always restore it. If migration worked, then I delete those directories.
@mk Postgres changes the binary format between major releases.
The idea is that you don't migrate between major releases compulsively. It's not like opensshd, it's a language runtime with a large amount of data attached. You're supposed to be conservative about migrating.
> so what? every developer has to deal with this..write migration scripts and ship it with your upgrades..this is NOT a user problem.
No, no, the on-disk format of the data. You don't do a "migration script" for a filesystem, you release the new version of the filesystem and then everyone copies their shit over.
> yeah..people stop fucking upgrading and run decades old software..really great strategie.
Previous versions are still supported and get updates. This isn't a browser. What you are saying is bad. You don't auto-migrate a 10TB database.
You're not on a different version of ZFS. The version of ZFS hasn't changed in forever. You're adding new feature flags. Some of those flags will break compatibility when you upgrade the pool. That's why it's a manual process.
High availability is the default expectation for a database. Your local dev box is the exception. When you have large postgres servers (multiple TB), you need logical replications to your new replica set, promotion and failover. Postgres's core data model is not at all like a file system version feature flag update.
It's not some "enterprise" feature. I've done it on clusters as small as 4 nodes. It's what happens behind the scenes on those fancy "cloud" instances.
Flippant; the question was intended to get you to think instead of internet-arguing. The old versions are maintained, they have bug fixes, security patches. Same with the kernel: http://kernel.org/ still maintains 5.4, can you guess why?
i bet that the majority of them don't have any of your high-availabilty enterprise problems and just want it to fucking work without extra support costs.
enlyft industry research shows that PostgreSQL has a market share of about 7.73% compared to leading competitors MySQL, Microsoft SQL Server and Microsoft Access.
My entire career has been spent either working at or running that type of company.
> i bet that the majority of them don't have any of your high-availabilty enterprise problems
Literally every company I have ever worked at cares about this. Any hacker that gives a damn about his work cares about zero downtime. Maybe it doesn't matter in .de if your shit is broken but it sure as hell does here.
@mk @pistolero @djsumdog I can understand you both. On one hand, software X can migrate/upgrade its file structure (e.g. "binary format") by itself by running automatically some migration/upgrade script. These scripts can be bound to version numbers (when the upgrade takes place) so the user=admin isn't bothered by it. The "in-place" migration also forces the upgrade on the user=admin.
But it seems like #PostgreSQL seem to follow an other philosophy that says that administrators are the responsible persons and it should be in their full control when the migration/upgrade should take place. So they don't want to force the upgrade on the admin's decision-making progress and let him decide whenever it takes place.
So what would be the solution? A simple script like pg_upgrade.sh that the administrator can manually run or for those people like @mk is that the installer asks "Should pg_upgrade.sh run for your automatically?" and it is done then during upgrade.
@roland @mk It's not like the RDB or anything, semantics change, the APIs that extensions use change. There are plenty of community-developed scripts for simple upgrades of small clusters. FSE (not "enterprise") is 611GB on disk. When I did the upgrade to 16.2, it took a while: these upgrades have to be done deliberately. It's not quite as bad as Python 2 to Python 3 but it's closer to that than updating to a new version of some random application. The closer you get to the base levels of the system, the more careful you have to be, and for most uses of a database (the uses for which Postgres is designed), data loss and downtime are unacceptable, some query that was fast suddenly gets slow, you can't have that. And how do you even do that across distros? Maybe his distro should provide that script; Debian does if I recall correctly, but he's shoving everything into Docker containers, and Docker containers tend to offload the heavy storage stuff until they can't.
@roland @p @mk > It takes time to port #python2 code to #python3 style. That's why there was a 6 year deprecation notice put out 11 years ago. It's been EOL for 5 years. 😭
@RedTechEngineer @mk @pistolero @djsumdog Ops ... Slow lazy developers again. Hey devs, port your damn software! Even my software started with PHP 4 (!) and runs on PHP 8 now. Yes, runs. But with deprecated stuff like array() instead of [] and still with closing PHP tag ?>. Shame on me! 😂
@roland @mk @RedTechEngineer Well, Python 3 was a big change. All of my Ruby 1.8(!!) code still works in 3.4, most PHP stuff is the same, but in Python's case, it's got less to do with code, it's a coordination effort: lift the world without moving it.
Jython, for example, is still only compatible with Python 2.7. People that use Jython still have to write 2.7 code: if you want your Python code to talk to Scala, you have to write code that works on Python 2. So you write a library that bridges some code written in Java with some code written in Python and then someone else's Scala depends on it, you can't just wave a wand.
@roland @mk @RedTechEngineer Unlikely; if it was 1:1 then it wouldn't require porting to begin with. You could maybe have a bot attempt it, but the bot would have to decipher what the original author *meant* to do rather than what they actually did.
For example, Python 3 changed how strings were handled. Probably in most cases this is trivial, but sometimes when reading the code, a machine wouldn't be able to tell if it was intended to handle ASCII or UTF-8 or binary data. And that's one semantic change: a lot of libraries went away, so how do you automate that? Just dump the library code into the middle of the file, or select a new API? Can you automate the process of translating a web2py application into a Django one? A lot of these things you can't really automate: you need to make decisions about what to do, which requires you to understand why the code exists.
Yea, I was guessing you were talking about old simple, one task scripts. I did have to port some Python2 to 3 on a few projects. Some of it wasn't that bad, but there were some things that took a while.
I commend the efforts of the BeautifulSoup4 devs who had to deal with the entire SGML parser being removed.
The 90% number was hyperbolic. It does spit out what it can't auto fix, and I just hoped the process had gotten better now that most libraries/frameworks have been ported over. I haven't had to do any py2->3 porting in years either. 😋
> Yea, I was guessing you were talking about old simple, one task scripts.
I used to write those in Ruby unless I had a reason; nowadays they're in awk unless I have a reason (e.g., JSON). So I can sit down at a terminal in 1985 and be fine, ha.
> The 90% number was hyperbolic. It does spit out what it can't auto fix,
Yeah, I think, like...half the time it's easier to just put the old code on the left, an empty buffer on the right, and do a rewrite.
Really? I don't think I've written any Python 2 in at least 5 years. The package isn't even available anymore in most Linux distros. Pretty much every major library has been updated by now.
If you still have Python2 scripts 2to3 usually gets you 99% of the way there.
I bet Python 2 is probably still embedded in GIS programs too. ArcGIS was using VisualBaisc for Applications (VBA, a VB6 bastard stepchild) in 2012, and had added Python 2 support. Most of the professors refused to move from VBA to Python.
meh. I'll fine dealing with it. I have a container somewhere that can load two versions. Is there anything that really challenges PostegreSQL and MySQL/MiriaDB/variants in the FOSS relational database world right now?
mk
in reply to djsumdog • • •it's fucking retarded that postgresql isn't able to migrate their own fucking data -inplace- on major upgrades..
fuck this pos software
Roland Häder
in reply to mk • •/var/lib/postgresql/directoy as well. I have renamed older versions to<number>.backupso I can always restore it. If migration worked, then I delete those directories.mk
in reply to mk • • •how i upgrade all of my docker services (including non-postgres-databasesystems)
$ docker compose pull && docker compose up -d
easy...
postgres is the only piece of shit software that has me manually copying data around..
https://hedgedoc.satoshishop.de/ZHpbTrZTSAWVvU9WIpC42A?view
total fucking waste of time.
Postgres upgrade - HedgeDoc
hedgedoc.satoshishop.deRoland Häder likes this.
pistolero
in reply to mk • • •@mk Postgres changes the binary format between major releases.
The idea is that you don't migrate between major releases compulsively. It's not like opensshd, it's a language runtime with a large amount of data attached. You're supposed to be conservative about migrating.
Roland Häder likes this.
mk
in reply to pistolero • • •@p
"Postgres changes the binary format between major releases."
so what? every developer has to deal with this..write migration scripts and ship it with your upgrades..this is NOT a user problem.
"You're supposed to be conservative about migrating."
yeah..people stop fucking upgrading and run decades old software..really great strategie.
Roland Häder likes this.
djsumdog
in reply to mk • • •Roland Häder likes this.
mk
in reply to djsumdog • • •@p
the exception doesn't make the rule.
I BET that the absolute staggering majority of all postgres installations is single database for some opensource project like peertube/mastodon.
i don't care about your enterprise problems. go fuck off and buy a oracle subscription.
Roland Häder likes this.
pistolero
in reply to mk • • •@mk
> the exception doesn't make the rule.
It is *designed* for that use. If you're using it and you don't like how it's designed to be used, you probably shouldn't use it.
> i don't care about your enterprise problems. go fuck off and buy a oracle subscription.
https://freedos.org/
" title="
"> If you want to wade in the kiddie pool, here.
The FreeDOS Project
freedos.orgRoland Häder likes this.
mk
in reply to pistolero • • •@p
"It is *designed*" to be garbage software that can't even inplace upgrade itself...
it's not a feature, it's a bug.
you fucking idiot.
pistolero
in reply to mk • • •@mk
> so what? every developer has to deal with this..write migration scripts and ship it with your upgrades..this is NOT a user problem.
No, no, the on-disk format of the data. You don't do a "migration script" for a filesystem, you release the new version of the filesystem and then everyone copies their shit over.
> yeah..people stop fucking upgrading and run decades old software..really great strategie.
Previous versions are still supported and get updates. This isn't a browser. What you are saying is bad. You don't auto-migrate a 10TB database.
Also decades-old software is bad exactly why?
Roland Häder likes this.
mk
in reply to pistolero • • •@p
"You don't do a "migration script" for a filesystem, you release the new version of the filesystem"
here's how i upgrade my filesystem:
```
$ zpool upgrade hetznerBackupPool
This system supports ZFS pool feature flags.
Enabled the following features on 'hetznerBackupPool':
zilsaxattr
head_errlog
blake3
block_cloning
vdev_zaps_v2
```
Roland Häder likes this.
pistolero
in reply to mk • • •@mk
> here's how i upgrade my filesystem:
zfs didn't change its on-disk format.
mk
in reply to pistolero • • •@p
"zfs didn't change its on-disk format."
wtf are you talking about? you can switch compression on/off WHILE you actively write a fucking file to disk..
$ fio \
--filename=/pool/fiotest \
--sync=1 \
--rw=write \
--name=journal-test \
--size=10000m \
--rate=1m,1m
$ zfs set compression=off pool
$ zfs set compression=lz4 pool
Roland Häder likes this.
djsumdog
in reply to mk • • •You're not on a different version of ZFS. The version of ZFS hasn't changed in forever. You're adding new feature flags. Some of those flags will break compatibility when you upgrade the pool. That's why it's a manual process.
High availability is the default expectation for a database. Your local dev box is the exception. When you have large postgres servers (multiple TB), you need logical replications to your new replica set, promotion and failover. Postgres's core data model is not at all like a file system version feature flag update.
It's not some "enterprise" feature. I've done it on clusters as small as 4 nodes. It's what happens behind the scenes on those fancy "cloud" instances.
Roland Häder likes this.
mk
in reply to djsumdog • • •@p
"The version of ZFS hasn't changed in forever."
forever == 5 days ago
https://github.com/openzfs/zfs/releases
Releases · openzfs/zfs
GitHubdjsumdog
in reply to mk • • •Roland Häder likes this.
mk
in reply to djsumdog • • •@p
"The version of ZFS hasn't changed in forever."
i upgraded my pools on 2024-10-15 and since then there's at least one more pool-upgrade available.
what's your definition of "forever"
mk
in reply to djsumdog • • •@p
"You're adding new feature flags. Some of those flags will break compatibility when you upgrade the pool. That's why it's a manual process."
so what? downgrade doesn't work with you copying data..this isn't the topic we're talking about.
we're talking about UPGRADING you're software.
mk
in reply to djsumdog • • •@p
"High availability is the default expectation for a database."
the majority of people use relational databases without any of your enterprise bullshit.
so no. it's not.
mk
in reply to pistolero • • •@p
"Previous versions are still supported and get updates."
why would you waste developer time like that? this is even more retarded.
mk
in reply to pistolero • • •@p
"You don't auto-migrate a 10TB database."
again..a exception doesn't make the rule.
if you have these problems, export and import your shit manually, but don't bather every fucking body else with your <1% problem.
mk
in reply to pistolero • • •@p
"Also decades-old software is bad exactly why?"
don't upgrade and find out.
pistolero
in reply to mk • • •@mk
> if you have these problems, export and import your shit manually, but don't bather every fucking body else with your <1% problem.
Don't whine about `pg_dump $connection1 | psql $connection2` if you want to use the big boy software.
Roland Häder likes this.
mk
in reply to pistolero • • •@p
"big boy software"
it's dinosaur software..unfortunatly a lot of devs just didn't see the meteor yet.
pistolero
in reply to mk • • •@mk
> don't upgrade and find out.
Flippant; the question was intended to get you to think instead of internet-arguing. The old versions are maintained, they have bug fixes, security patches. Same with the kernel: http://kernel.org/ still maintains 5.4, can you guess why?
The Linux Kernel Archives
kernel.orgmk
in reply to pistolero • • •@p
"The old versions are maintained[..]can you guess why?"
because you're upgrade strategy is absolute garbage and people refuse to upgrade...
you're wasting TONS of developer AND user time with this.
pistolero
in reply to mk • • •mk
in reply to pistolero • • •"Your time doesn't matter."
this is what you're telling postgres users? majority small and medium sized companies..
---
Company Size Distribution
Small companies (<50 employees): 43%
Medium-sized companies: 42%
Large companies (>1000 employees): 15%
https://enlyft.com/tech/products/postgresql
no private usage counted.
---
i bet that the majority of them don't have any of your high-availabilty enterprise problems and just want it to fucking work without extra support costs.
PostgreSQL commands 7.73% market share in Database Management System
enlyft.comRoland Häder likes this.
pistolero
in reply to mk • • •@mk
> majority small and medium sized companies..
My entire career has been spent either working at or running that type of company.
> i bet that the majority of them don't have any of your high-availabilty enterprise problems
Literally every company I have ever worked at cares about this. Any hacker that gives a damn about his work cares about zero downtime. Maybe it doesn't matter in .de if your shit is broken but it sure as hell does here.
Roland Häder likes this.
mk
in reply to pistolero • • •@p
high-availibility != downtime.
you'd have less downtime with a fucking databasesystem that's able to inplace-upgrade itself.
wtf are you talking about?
Roland Häder likes this.
Roland Häder
in reply to mk • •@mk @pistolero @djsumdog I can understand you both. On one hand, software X can migrate/upgrade its file structure (e.g. "binary format") by itself by running automatically some migration/upgrade script. These scripts can be bound to version numbers (when the upgrade takes place) so the user=admin isn't bothered by it. The "in-place" migration also forces the upgrade on the user=admin.
But it seems like #PostgreSQL seem to follow an other philosophy that says that administrators are the responsible persons and it should be in their full control when the migration/upgrade should take place. So they don't want to force the upgrade on the admin's decision-making progress and let him decide whenever it takes place.
So what would be the solution? A simple script like
pg_upgrade.shthat the administrator can manually run or for those people like @mk is that the installer asks "Should pg_upgrade.sh run for your automatically?" and it is done then during upgrade.pistolero
in reply to Roland Häder • • •Roland Häder likes this.
RedTechEngineer
in reply to pistolero • • •> It's not quite as bad as Python 2 to Python 3
I refuse to believe you don't still have something relying on python 2.
like this
Roland Häder and pistolero like this.
Roland Häder
in reply to RedTechEngineer • •RedTechEngineer
in reply to Roland Häder • • •> It takes time to port #python2 code to #python3 style.
That's why there was a 6 year deprecation notice put out 11 years ago. It's been EOL for 5 years. 😭
like this
Roland Häder and pistolero like this.
Roland Häder
in reply to RedTechEngineer • •array()instead of[]and still with closing PHP tag?>. Shame on me! 😂RedTechEngineer
in reply to Roland Häder • • •like this
Roland Häder and pistolero like this.
Roland Häder
in reply to RedTechEngineer • •foointobarto avoid deprecation removal.pistolero likes this.
pistolero
in reply to Roland Häder • • •@roland @mk @RedTechEngineer Well, Python 3 was a big change. All of my Ruby 1.8(!!) code still works in 3.4, most PHP stuff is the same, but in Python's case, it's got less to do with code, it's a coordination effort: lift the world without moving it.
Jython, for example, is still only compatible with Python 2.7. People that use Jython still have to write 2.7 code: if you want your Python code to talk to Scala, you have to write code that works on Python 2. So you write a library that bridges some code written in Java with some code written in Python and then someone else's Scala depends on it, you can't just wave a wand.
Roland Häder likes this.
Roland Häder
in reply to pistolero • •pistolero likes this.
pistolero
in reply to Roland Häder • • •pistolero
in reply to Roland Häder • • •@roland @mk @RedTechEngineer Unlikely; if it was 1:1 then it wouldn't require porting to begin with. You could maybe have a bot attempt it, but the bot would have to decipher what the original author *meant* to do rather than what they actually did.
For example, Python 3 changed how strings were handled. Probably in most cases this is trivial, but sometimes when reading the code, a machine wouldn't be able to tell if it was intended to handle ASCII or UTF-8 or binary data. And that's one semantic change: a lot of libraries went away, so how do you automate that? Just dump the library code into the middle of the file, or select a new API? Can you automate the process of translating a web2py application into a Django one? A lot of these things you can't really automate: you need to make decisions about what to do, which requires you to understand why the code exists.
like this
Roland Häder and djsumdog like this.
RedTechEngineer
in reply to pistolero • • •like this
Roland Häder and pistolero like this.
djsumdog
in reply to pistolero • • •Yea, I was guessing you were talking about old simple, one task scripts. I did have to port some Python2 to 3 on a few projects. Some of it wasn't that bad, but there were some things that took a while.
I commend the efforts of the BeautifulSoup4 devs who had to deal with the entire SGML parser being removed.
The 90% number was hyperbolic. It does spit out what it can't auto fix, and I just hoped the process had gotten better now that most libraries/frameworks have been ported over. I haven't had to do any py2->3 porting in years either. 😋
like this
Roland Häder and pistolero like this.
pistolero
in reply to djsumdog • • •@roland @RedTechEngineer @mk
> Yea, I was guessing you were talking about old simple, one task scripts.
I used to write those in Ruby unless I had a reason; nowadays they're in awk unless I have a reason (e.g., JSON). So I can sit down at a terminal in 1985 and be fine, ha.
> The 90% number was hyperbolic. It does spit out what it can't auto fix,
Yeah, I think, like...half the time it's easier to just put the old code on the left, an empty buffer on the right, and do a rewrite.
pistolero
in reply to RedTechEngineer • • •Roland Häder likes this.
djsumdog
in reply to pistolero • • •Really? I don't think I've written any Python 2 in at least 5 years. The package isn't even available anymore in most Linux distros. Pretty much every major library has been updated by now.
If you still have Python2 scripts
2to3usually gets you 99% of the way there.Roland Häder likes this.
pistolero
in reply to djsumdog • • •@roland @RedTechEngineer @mk
> The package isn't even available anymore in most Linux distros.
I don't know what distros you mean, but I have both installed on this machine, at least.
pistolero
in reply to pistolero • • •Roland Häder likes this.
djsumdog
in reply to pistolero • • •pistolero likes this.
djsumdog
in reply to djsumdog • • •like this
Roland Häder and pistolero like this.
BSD/r000t
in reply to mk • • •I can't hear you guys over my penis being so amazingly spectacularly enormous on account of my use of sqlite3
@p @djsumdog
Roland Häder likes this.
djsumdog
in reply to mk • • •mk
in reply to djsumdog • • •Roland Häder likes this.
Roland Häder
in reply to mk • •