In contrast to mixing ETL within a single program, Airflow and shell scripts using GNU Parallel can handle retries, concurrency, logging, etc

Am I reading your message wrong? It appears to say that a single program cannot handle retries, concurrency, logging, etc.

Before getting to overly integrated technology we first must take a step from the completely disjointed technology. I’m arguing for the first steps, not for going all the way to the other extreme.

Love it. Especially for exploring maps together with my wife.

You actually support my point by referring to manuals. The manual for find is more that thousand lines of text. Should I read it before first use and memorize all the peculiarities forever? Or should I check it every time I use find to see if there are comments specific to the current scenario? Both alternatives seem complex. Remember how I am complaining about unnecessary complexity? When using an alternative in a programming language I don’t have to bother with all that: I get a file handle and pass it to delete method.

Many commenters on the internet say “you should read the manual”. I doubt they follow this advice. I won’t take you as an example because you claimed no expertise. But whenever someone says these words, ask them to compare find -delete and find rm without looking into manual, see what they say.

Remember how I was saying that good integration is difficult? Well, if you just “naively” integrate with rm then you get issues like “too many parameters passed to rm” or “there is a vulnerability if you manipulate filesystem while find is still running”. These are among the reasons a well-integrated -delete exists, as “impure” and “complex” as it is.

Have you considered the reasons this option was added despite this seemingly obvious fact?

You know there are colors beyond black and white? There’s a spectrum of alternatives.

I’m arguing for a little more singing and a little more dancing well-integrated solutions. Like the -delete parameter to the find tool is the step in the right direction. Or do you think that this parameter is superfluous and good old rm is perfectly fine?

I only objected to the use of “only” in the comment above. I’d like to object to “never” as well.

I am a big fan of ISO8601, and I only objected to your use of “only” in the comment above.

Many commenters assume I propose do-all be-all monsters. Looks like I failed to explain my idea of custom-made programs for specific use-cases.

People are supposed to be skilled

Software is supposed to make simple things simple, while complex things should be possible. I’d expect iterating over a list of files to be simple, but everyone has to remember to include special handling of whitespaces in filenames. That’s just one example.

I mean one often thinks they don’t apply and then is surprised by breakage. Or worse: it goes unnoticed. How many problems have happened because someone hasn’t considered whitespace in filenames?

If I understood your idea correctly, my proposal is the opposite of what you mentioned. I argue for more of better integrated software that does less.

So, it is me “missing the difference between professional software development and a QnD script” or the whole software industry? ;)

ignore all of the failure modes and edge cases except those that a script is going to directly run into

I’ve seen lots of cases of catastrophically wrong expectations of what the script will run into. Both for casual personal scripts, and professionally written scripts. I dare to claim that the majority of script authors cannot properly estimate this.

…or your file manager is smart enough to do numeric sorting

People slowly learned to use their keyboards for proper typography?

Initially I had some words in the post about the philosophy opposing some other approaches, and I can guess that there were some Drupal-like examples at the time. Is it possible that the approach was taken to the extreme and the baby was spilled with the water?

More complex user problems are then solved by combining multiple simpler components.

What I am trying to say is that this solution is very likely to be poor, not taking into account various failures and edge cases.

Well, my post describes the experience of a developer using these tools. “User” here is supposed to mean the person who combines tools to build a more complex piece of software.

In my experience, programming languages offer more control than just connecting inputs and outputs, and it’s easier to get right