Too many abstraction layers
I worked at a place where a site had to show the number of members, but for some reason it didn’t show the correct number. The reason was the query to the database was wrong, I corrected it in five minutes and the boss told me to put numbers manually, increasing them until we could display the correct query result because apparently the problem was there a long time before I started working there and it would make a bad image for the company if it was so easily corrected.
On other project a friend of mine told me about things were so slow and getting slower with time. Database didn’t have keys, indexes, nothing you should expect a relational database to have.
I haven’t worked in a while because of my health issues and I can assure you I’d be happier getting paid for watering plants than working with computers.
Has everyone forgotten the art of creating or using performance profiling tools
I was in a meeting with a senior software engineer, who makes a shit ton more than me. While sharing my screen he saw flame graph I had for one program I was working on. They asked “what is that?”, I was taken back, and said it’s for looking at which functions are taking more time. Then I was asked “where did you find it?” I was shell shocked, didn’t know how to answer that. Similar things have happened before where I taught them certain things, and they go “great find”, like dude, I didn’t find it just now. That’s basic knowledge if you work with database.
I am not even a developer hire, I’m self learned, and because the company doesn’t have many developers they gave me some of the tasks and I’m doing it because it’s more fun than my job responsibilities, and when I don’t have enough tasks.
You see, performance is rarely a talking point in retrospectives unless its egregious. If you say a piece of compute is struggling the solution is to always buy more compute. It is rare that anyone would think of optimization unless you’re writing libraries.
Or if its employee facing only let it get slower and slower to the point where it takes a minute to update one record
Which is exactly why western AI companies are bankrupting themselves building data centers while China is easily keeping pace with a fraction of the resources. They’re not magic. They just do basic things like make things more efficient.
I wrote a program in Python to read some huge Json files. It took forever to finish. Turned out I could optimize it with some simple changes, which reduced 10 minutes to 10 seconds (not literally, just figurative) and made a difference in huge files only. Later I rewrote the program in Rust. And you know what happened? It was slower than the Python version. Only after understanding what I was doing and making right changes made the Rust version faster.
Most of the time its the design of the code or libraries that slows down. The language might play a huge role in performance, but it depends on what type of code and application it is.
I actually wanted write a blog post about this… if I still have the project data I might explain what happened. BTW this had nothing to do with Ai.
As an OPS guy you know what really grinds my gears? Developers not paying attention to shit like this.
We have a job in production that is a powershell script calling a powershell script calling a java applet. Now you may already be thinking “what is this abomination?” And you’re right, but that java application is set to DEBUG. IN PRODUCTION. So every time it runs it generates 1gb of logs.
Do they fix it? No of course not because if it’s not broken don’t fix it. But Jesus Christ we have people developing without understanding fundamentals like “YOU DONT LEAVE DEBUG ON IN PRODUCTION”.
yeah, sometimes it feels like the same people complaining about python speed are the ones writing “C” code in python.
What about the fact that runtime startup alone auto-disqualifies python for all short-lived process use-cases?
Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?
“languages are just tools” and “use the right tool for the job” were already stupid catch phrases. Pretending languages don’t matter is a big further step into this shallow faux-intellectualism masquerading as wisdom.
And we didn’t even get to low level stuff yet.
What about the fact that runtime startup alone auto-disqualifies python for all short-lived process use-cases?
While Python’s start-up time is a lot slower than a corresponding compiled program, in most cases it’s not slow enough to matter even if the process itself is short-lived:
$ hyperfine 'python3 -c "pass"' Benchmark 1: python3 -c "pass" (412 runs) mean ± σ min … max Wall Time 7.1 ms ± 0.2 ms 6.7 ms … 8.4 ms Memory 9.4 MiB ± 0.1 MiB 9.1 MiB … 9.5 MiBIf you are running a small program so often that this kind of start-up time matters, then the loop should probably already be implemented in the program itself. In line with the article’s thesis
Yeah nah, Lua runs circles around Python both at runtime and especially at startup. I’ve made me some desktop automation scripts in Lua to call from Alfred, I had to recheck that I didn’t leave static test data in there instead of calling the script, because the results were appearing instantly.
Recently had to develop a Python project, the tests took at least ten seconds to start each single time. This is while I made over a hundred tests for my module and re-ran them after every change.
Recently had to develop a Python project, the tests took at least ten seconds to start each single time. This is while I made over a hundred tests for my module and re-ran them after every change.
Python is dog slow, but if your tests are taking 10 seconds to start (not even run!), then something is catastrophically wrong with your setup. For comparison, for one of my projects, it takes a total of 12 seconds to run nox on a clean slate, which involves setting up venvs for five major versions of Python, installing the package and its dependencies in each, and finally running the 1500 test test-suite five times
It wasn’t my project. Afaict it’s a standard pytest setup with the Litestar framework (unused in my tests) and an in-memory mock Postgre transaction with like five records per test, so choose for yourself which part of this is ‘catastrophically wrong’. However, I can indeed agree that most of it is trash.
Same approach with dozens upon dozens of tests worked snappily as heck in PHP, haven’t yet tried it with my beloved Lua.
Then its the Test environment that is slow, not Python.
Lua also doesn’t provide everything Python does and has a totally different use case.
Oh really? Well guess what, that’s a huge benefit for Lua because its entire total size is around 500 KB. It’s loaded into memory and executed start to finish faster than you blink. If you need a ‘standard library’, just use Penlight. As I already mentioned, vast majority of libraries for Lua are written in… Lua itself, because it’s so goddamn fast. I’m using Fennel, the Lua Lisp that’s transcribed on-the-fly, and it’s still faster than Python. Lua is great for interactive tasks like desktop automation, e.g. with Alfred, Hammerspoon (which latter uses Lua throughout), or for scripting in Termux on the phone where Python would kill the memory immediately.
Lua’s problems with the absence of popular libraries aren’t because of any language issues (even the lack of
nullcan be dealt with), but solely due to its non-adoption in the masses. It is very popular as an embedded language in gaming, though, and that’s not a coincidence, because it fits that niche precisely because of its small size and extraordinary speed.Well guess what, that’s a huge benefit for Lua because its entire total size is around 500 KB.
Filesize doesn’t matter, if the functionality is missing. Lua is just not a direct comparison and replacement for Python. Penlight isn’t standard library. In example if I want to write a commandline script, I better off doing it in Python. The language is much better to me, has better features, has a standard library builtin and is even installed in all Linux distributions.
Lua is meant to be embedded into other applications, not meant to be used on its own.
Lua’s problems with the absence of popular libraries aren’t because of any language issues (even the lack of null can be dealt with), but solely due to its non-adoption in the masses.
Wrong. This is a design choice, because Lua isn’t meant to be used by its own. The embedded environment is meant to provide the functionality. In example in Neovim they provide their own functionality embedded within the editor. I can’t use that script on a different environment.
Or the fact that you can’t do concurrency especially across the language barrier from another language without multiple python runtimes orchestrated with a queue and IPC because of possible process-wide global state and similar concerns/assumptions for any python library dependency that is not all native across its own dependency chain?
This is very out of date. I haven’t written Python for over 4 years and even back then, I was using multiprocessing.Pool for pretty much all parallel processing that I ever needed. Sure, it’s doesn’t fit every use case for concurrency, but for simple stuff it’s perfectly fine.
Not only that, it’s one of the simplest parallel processing APIs I’ve ever used:
from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': with Pool(5) as p: print(p.map(f, [1, 2, 3]))It’s not outdated, and maybe you didn’t get what I was talking about, but probably because I didn’t word it well. So let me quote someone else instead:
While pure Python code should work unchanged, code written in other languages or using the CPython C API may not. The GIL was implicitly protecting a lot of thread-unsafe C, C++, Cython, Fortran, etc. code - and now it no longer does. Which may lead to all sorts of fun outcomes (crashes, intermittent incorrect behavior, etc.)
This covers the layers below python, but still doesn’t fully cover the process wide restrictions (it’s not just about thread safety, global initialization and resources are very relevant too). It also doesn’t cover the exact same problem which also exists for any layer calling into python from above, which also relies/relied on the GIL for synchronization. This means that you can’t effectively, confidently, and safely scale python without scaling the number of python runtime instances, each in its own full OS process.
The fact that
multiprocessingbecame popular, instead of just scaling/parallelizing with threads, is itself an extension of the GIL/no-GIL circus. But its not the primary point here.So not only the proposition (which someone referenced in another sub-thread) to move everything into python is in and of itself a self-defeating argument, since it proves a language being a problem (through forcing language lock-in). But even within a language (python), not all problems are solvable still.
Note that this is not all theoretical for me. In one of my previous projects, I had to call into a python library (which didn’t have an adequate alternative) from rust code (through pyo3). And I was forced to scale with multiple python runtimes, each getting one request/message at a time.
And we still didn’t get to any low-level details yet 😉
Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.
I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.
I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.
I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases (which is what the intellectually deficient OP tried to argue against), then I’m not sure you’re qualified to opine on the matter (hint: spawning a thread per task is already too much overhead for many workloads). And I don’t mean any disrespect to you here. Your replies were far from the worst in this thread.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases
That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using
multiprocessing.Pooland getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
At least we agree on something!
Both matter…
I’ve been vibecoding pyqt based python apps lately. The performance even one of the so called “slowest” languages can do is more than fine. Not using Electron, or actually optimizing matter more than the language itself, most of the time anyway.
Nah, software is slow because it runs a web engine for GUI.
I’d say a bit of both. People who are fine with having a DOM worth of overhead don’t give me much confidence they are selecting algorithms well, which can absolutely affect performance and efficiency regardless of what language they are written in.
Hell, even with the optimal data structures and algorithms selected, how you set them up can make a huge difference.
Like I once had essentially a data translator that, in order to be generic, used keys in a dictionary to replace the hardcoded data structure that needed to be updated any time the spec changed, even if we didn’t care about the new values.
The first version used an array of dictionaries and was slow as fuck. Each line of data would get its own dictionary and just parsing the data into this scheme took a couple minutes IIRC. Since the data was guaranteed to have the same keys each line (we were worried about changes between versions, not lines), I changed it to a dictionary of arrays, where each line would have the same index across the dicts, and it brought parse time down to several seconds. Huge difference even though big O notation would have simplified to the same expression for the two methods.
But if you pick something with a worse big O expression, it gets worse on a whole other level.
Noticeable speed? Sure. But I am so pissed if a fucking todo list app is a 100mb download and consumed 300mb memory when running, all because this fucker bundles and loads a whole browser engine to render itself.
I therefore do think tech stack matters as well.
What if the language has a bad design?
Then use it poorly and it’s a double negative.
Ship in the minimum amount of time possible, by any means, to be first to market. And this means poor design and smooshing together all kind of fat general-purpose frameworks and libraries to get something out the door.
Sure, it’s slow. Google’s landing page is 4 MB for a small graphic, a text form, and some buttons. But what are you going to do? Not use it? They won. They don’t have to fix shit.
And now with LLMs they can produce bloat faster than ever and pay even less attention to speed or quality.
“The goal was never quality software.”
I’d add that most of the time nobody actually measured anything. at work we inherit slow sites all the time and the first guess is always “the stack”, then you profile it and it’s one query running per item in a loop or something like that. I had the same with my own side project, was sure I needed a bigger server and after measuring the problem was somewhere else completely. measuring first is boring but it’s cheaper than any rewrite I guess
Profiling complex systems can be quite difficult. That said, I believe in the adage “Profile twice, optimize once.”, adapted from the old carpenters adage of “Measure twice, cut once.”
Slowness is rarely essential, but I can be far away from where you expect it.
That’s what Open Telemetry is for, honestly, specifically tracing. I’m in the process of adding tracing for all of our web handlers, http client calls, database transactions/individual queries, and asynchronous jobs. Should give us a pretty good view into the performance of the system, and allow us to monitor for things like performance regressions without any dedicated profiling needed (though we certainly could add additional profiling information through Otel down the road). Since I know you’re a Haskeller (recognize your handle), the
hs-opentelemetryecosystem is quite solid and has integrations with a lot of libraries in the wider ecosystem, likewai,persistent,postgresql-simple,conduit, and a bunch more. I haven’t needed to write too much of our own instrumentation, aside from our job system since that’s a bespoke system.Yeah we used hs-opentelemetry at my last paid position. The company was an MS shop, tho so we were using ODBC to connect to MS SQL Server which doesn’t have great in-process profiling. But, at least we could figure out what quer(y/ies) were running long and then we could use the MS tools to look at the plan and fiddle with that. It was also a small org, so it’s not like there was a lot of internal services in one flow – basically everything either stayed in the Web Yesod app or in the IVR app. All our profiling could have been instrumented process profiling instead of having to use otel at all.
So, I haven’t seen how easy/difficult it is to actually use otel across multiple internal service calls to satisfy one external request.
yeah that matches what I see, most of what we run at work is one app and one database, so the slow query log plus looking at the plan gets you like 90% of the way. I never really had the multi service case where otel pays off, I guess at some point it does, but for a small team it felt like a lot of plumbing to end up finding the same n+1 query lol
I’d argue a combination of bad design and/or bad implementation, which i would consider different things.
It’s a difference that becomes more pronounced the larger the team grows.
A somewhat reasonable design can be absolutely destroyed by a poor implementation of that design
