Does anyone know a battle-tested tool that would help with (almost)online migrations of postgresql servers to other hosts? I know it can be done by manually, but I'd like to avoid that
PG's internal wal-level replication? primary to a read-replica, and then switch the read to become the primary. You'll have a bit of downtime while you stop connections on the original server, switch the new server to primary, and update your app config to connect to the new server.
I believe that's a pretty standard way to provide "HA" postgres. (We use Patroni for our HA setup)
Why would you do 50k+ connections if they can't be active all at once either way? Unless you have 50k+ cores and IO beefy enough not to get overwhelmed due to that.
You can have as much connections as you want, but you'll have to trade it for having lower work mem numbers, which hurts performance. Traditional advice is to keep it below 500 per PostgreSQL instance (I'd say physical host).
I've ran dozens of micro services handling thousands of requests per second with a total connection limit of around 200 of which most was still unused - all without any server-side pooler.
> After the commit, the Connection object associated with that transaction is closed, causing its underlying DBAPI connection to be released back to the connection pool associated with the Engine to which the Session is bound.
I expect other reasonably sane libs for working with transactional databases do the same.
So, if you are doing pooling correctly, you can only run out of available connections if you want to have a lot of long running transactions.
So, why would you want every of your 50k frontends keep an open transaction simultaneously?
Because there's an overhead to make a connection, authenticate, set the default parameters on the connection, etc. I've never seen a framework that closed db connections between requests.
Of course, the better design is to write a nonblocking worker that can run async requests on a single connection, and not need a giant pool of blocking workers, but that is a major architecture plan that can't be added late in a project that started as blocking worker pools. MySQL has always fit well with those large blocking worker pools. Postgres less so.
As I said, you can return the connection to the connection pool.
From the perspective of keeping the number of open connections low it doesn't really matter if you close it or return to the pool, because in either case the connection becomes available to other clients.
I might not be understanding what you're pointing out here. It sounds to me like sqlalchemy is talking about a pool of connections within one process, in which case releasing back to that pool does not close the connection by that process to the database. Parent comment is talking about one connection per process with 50k processes. My comment was that you don't need that many processes if each process can handle hundreds of web requests asynchronously.
If you are saying that a connection pool can be shared between processes without pgbouncer, that is news to me.
The most common design for a Web app on Linux in the last 20 years is to have a pool of worker processes, each single-threaded and ready to serve one request. The processes might be apache ready to invoke PHP, or mod-perl, or a pool of ruby-on-rails or perl or python processes receiving the requests directly. Java tends to be threads instead of processes. I've personally never needed to go past about 100 workers, but I've talked to people who scale up to thousands, and they happen to be using MySQL. I've never used pgbouncer, but understand that's the tool to reach for rather than configuring Pg to allow thousands of connections.
I don't think this is correct. The difference between well optimized code and unoptimized code on the CPU is frequently at least an order of magnitude performance.
Reason it doesn't seem that way is that the CPU is so fast we often bottleneck on I/O first. However, for compute-workloads like inference, it really does matter.
While this is true, the most effective optimizations you don't do yourself. The compiler or runtime does it. They get the low-hanging fruit. You can further optimize yourself, but unless your design is fundamentally bad, you're gonna be micro-optimizing.
gcc -O0 and -O2 has a HUGE performance gain. We don't really have anything to auto-magically do this for models, yet. Compilers are intimately familiar with x86.
While the compiler is decent at producing code that is good in terms of saturating the instruction pipeline, there are many things the compiler simply can't help you with.
Having cache friendly memory access patterns is perhaps the biggest one. Though automatic vectorization is also still not quite there, so in cases where there's a severe bottleneck, doing that manually may still considerably improve performance, if the workload is vectorizable.
I was bullied by 3 separate bullies in my school years, every time it ended when I reached some deep rage in me and hit back.
One time I've thrown the bully down the stairs, other time I've hit him with a chair and the last time it was when the bully stole my phone, started going through the photos my girlfriend sent me and commenting them inappropriately - I've smashed his face so hard, the glasses flew like 4 meters away. I was short and overweight, they've never expected the amounts of power I could generate when in rage mode.
So from my limited, anegdotal experience - it works. It also shows other people that they can stand up to the bully and the bully is not some all-powerful being that has to define their present and future.
Porn is just the justification. It's easy to find something repugnant on whatever streaming video site and then start with the "protect the children" nonsense.
Backward countries being backward. The main flaw of modern liberal societies is that parts of them have stopped believing that liberalism is indeed progress. All hail the moral police and long live cultural relativism or whatever its currently trendy post-structural reconstruction is.
I think most countries that do this also block/redirect the major DoH providers like CloudFlare or Google. Of course, you can always hide your DoH traffic by going to other servers or worse case using an HTTP proxy and avoid that.
There are even countries that MITM all HTTPS traffic, and your choices are to install the government MITM root certificates into your trust store, or not use HTTPS.
> There are even countries that MITM all HTTPS traffic, and your choices are to install the government MITM root certificates into your trust store, or not use HTTPS.
Are there? When Kazakhstan announced they were going to do this, all the major browser vendors blocked their CA... so they backed down. What other countries do this and get away with it?
I'm in the UK; my ISP hijacks dns requests on port 53 so nope, none of that works. They're not alone doing this https://en.wikipedia.org/wiki/DNS_hijacking#Manipulation_by_...
For the most part this is not noticeable; but addresses to a bunch of my _work_ stuff don't resolve on whatever hacky dns replacement they offer, if I'm not on the work vpn.
They also block port 853 (so no DoT), and https to well-known dns servers; so you can't use DoH to google, but others may work.
If you're on a vpn they never see the traffic, you can also bypass them using a pihole with unbound to proxy dns to a DoH server - as long as they haven't blocked it.
Ironically the corporate vpn I use also hijacks dns (but locally only), which bypasses all the ISP issues but makes debugging work DNS problems awkward
Comcast/Xfinity does that in the USA, at least if you use the newer modem/routers that they provide. If you use your own router you can still set your own DNS provider. DoH is a workaround for web browsing.
The UK government IPs show up on our ban lists often for illegal theft of service, and CVE scans. Have you tried a Bind9 relay with iodine/vpn tunnels for local transparent network traversal across the hostile sandbox?
i.e. obfuscate the traffic using the hijacking DNS servers themselves.
the isp blocks/redirects the traffic outside my network. so if you just try to send normal udp/tcp port 53 externally, it won't get there. This is why I mention a pihole; by setting my dns server to something on my local network and then having that use DoH I can get past the block. I can't configure every device to use eg DoT or DoH directly, but I usually can configure their port 53 nameserver, directly or via DHCP
the vpn provider, it's just a split tunnel thing; since that is a local process, yes they can hijack it. Originally when we switched to our current vpn provider it didn't even let us use localhost or loopback dns, but we needed that for the way we use docker in development, so now it's just anything except those being redirected.
I configure my router to divert all UDP/53 to my pi hole. The advertising industry hates this type of behaviour, but it means ever an IoT device using hard coded dns (rather than what I tell them from my dhcp or nd settings)
This is a feature. That some people choose terrible ISPs is a trivial problem to avoid, far easier than avoiding terrible user agents which are beholden to their advertising masters.
They absolutely can and some do. The destination UDP port number of a UDP packet traversing the core network of an ISP can be inspected and acted upon as one pleases.
Unless it is tunneled over an binary obfuscation layer, and wrapped in a purposely weakened cryptography to booby-trap their parser.
There is also the global satellite uplinks... so its ultimately a pointless game to keep people ignorant, that is unless they plan to follow people around like a hot-air balloon villain from Pokemon Go. lol =3
They can do anything unless constrained by cryptography. I assume it just means redirecting all port 53 traffic which 99% of time will be DNS regardless of IP.
You choose an isp with those features that’s on you. It’s not like the UK is a backwards country with a monopoly of one or two ISPs for a given location.
I had just switched to this one when I discovered the problem, so was under contract for the next couple of years, and it's not like they advertise this as a feature where you'd have made that choice beforehand. Also, I didn't just need "an ISP" I needed a high speed connection and at the time my previous provider said they didn't offer that to existing customers, while the handful of others appeared to only offer 1/10 of the speed I wanted or only offered it bundled with tv/sport packages (I don't watch tv)
Since then City Fibre completed their rollout and I'm no longer an existing customer with BT so now I _do_ have a choice.
But bigger picture here: I mentioned my setup on a thread where a country is mandating all of their ISPs do this. Sometimes you don't have a choice.
Virgin Media. At the time I switched I needed more bandwidth for work - dealing with multi-gigabyte blobs all day; I was with BT, but BT wouldn't let me upgrade to a gigabit fibre connection, and the City Fibre network which is now everywhere wasn't yet in my street.
Many ISP will also auto-redirect un-allocated domain names to their own websites. Others will ban most inbound connections with a port under 1000 to prevent self-hosting/video-surveillance users.
Annoying if you are trying to bring up a remote domain server, and thinking WTF while checking things out in dig. lol =)
An easy solution would be for Google to host their DoH endpoints on the same domain(s) as their regular service, so that governments can't block DoH without blocking all of Google or YouTube. Using a dedicated domain like that, they're just begging to be blocked.
I wonder if DoH requests can be easily proxied? So if I set up https://www.mydomain.com/dns-query on a U.S.-based cloud server and proxy_pass all requests to Google or Cloudflare, and point my browser at my server, will it work?
Iodine will obfuscate the traffic using the redirected DNS hijack servers themselves.
Perhaps someone will put a configured wifi router image together over Christmas holidays for demonstration purposes... because it is fun to ignore tcp drop DoS too.
Tunneling well-obfuscated traffic is easier than most imagine... and IDS technology will fail to detect such things without an OS OSI layer snitch. =3
> An easy solution would be for Google to host their DoH endpoints on the same domain(s) as their regular service
That's not how that works. DoH resolvers need an IP address, not a domain name. Sure, Google could host DoH on www.google.com, www.youtube.com, etc. but most users are not going to be savvy enough to find those IPs and use them.
Then again, perhaps users savvy enough to try to use DoH to bypass these blocks would also be fine with this.
> most users are not going to be savvy enough to find those IPs and use them.
Very few people configure DoH on their own. It's up to the DoH-enabled client software (mostly browsers) to obtain lists of resolver IPs and keep them up to date.
If Cloudflare, for example, really wanted to make their DoH traffic indistinguishable from other HTTPS traffic, they could literally host DoH on any domain or IP under their control and rotate the list every now and then.
Unless your local router tunnels the DNS traffic via other means. The clients may see slightly higher latency, but for <16 host hotspots it would be negligible.
It is quite easy for example, to bonce traffic through a reverse proxy on a Tor tunnel, and start ignoring spoofed drop-connection packets (hence these bypass local DNS, tunnel to a proxy IP to obfuscate Tor traffic detection, and exit someplace new every minute or so.) This is a common method to escape the cellular LTE/G5 network sandbox.
Ever played chase the Kl0wN? Some folks are difficult to find for various reasons.
It sounds like you're working with a model in which most users are conscious that they're very offended or inconvenienced by censorship, and want to research technical means of circumventing it. I wish that were true, but I doubt it's nearly as common as your intuition suggests.
Except the lists often change every minute, and some types of proxies are just a compromised script/page sitting on commercial, private, and government servers.
> Only a very small percentage of people will be savvy enough to attempt to circumvent these bans.
There are several one-button vpn/proxy+tor apps for unrooted phones already, and they are dodgy on a good day. =3
Chinese government couldn't have cared less about that "impact" -- even if only less than 1% of Wikipedia content mentions Chinese government at all, they are going to block the Wikipedia website.
Bulk of those blocked requests are to Launch Darkly, Datadog and Stripe.
I'd expect commenters on tech-oriented site to understand how uBlock Origin works... or at the very least how to check the browser console instead of fearmongering.
My company looked into paying Sourcegraph many times in the past, but they were prohibtively expensive every time we checked.
It's 49 USD per user per month for Code Search, like what the hell man?
It's more than twice as expensive as Github Enterprise.
Almost twice the cost of Gitlab Premium.
At some point it was 100USD per month per dev, I also remember it being "Starts from 5k USD per year", you can find some quotes for that in old submissions regarding Sourcegraph going open, closed, open and closed again.
That's so often the case. I was recently looking at supply chain security / SBOM software. "Coincidentally", 3 different vendors with 3 very different products quoted us the exact same annual price for the features we wanted, and that price was on the order of magnitude of "hire someone to do this manually full-time".
There are IMO too many companies that have no tier between Free and Enterprise. I understand the desire to focus on a small number of whales, but can't help feeling like that's leaving money on the table from all the smaller companies who'd be willing to pay something in the middle.
