WYGIWYG

  • 0 Posts
  • 111 Comments
Joined 2 years ago
cake
Cake day: September 24th, 2024

help-circle



  • Maybe. I’m not entirely sure on that.

    If you just look at the number of red and blue, sure.

    Then discount the blue that are overred areas who are mainly there for -war -gas -foodcost. How many of them are going to be honest and vote for their people, those people who would definately NOT vote to dump Electoral college because it has served them well for ages.

    Then discount DINO sleepers sitting in Congress who look and talk blue then flip when they are needed?

    Then in the same venn circle in many cases, take the remaining blueish moderates who can be bought by your friendly neighborhood oligarch.

    We’re blue on paper using binary counting; politics is not a binary sport.





  • You’re absolutely correct, 100% It’s entirely scriptable. And I have done that work many, many times in the past 15 years

    The interesting differences:

    I didn’t write the ansible jobs, it did to my specifications, i just had to review them. It can generate an Ansible job that would take me 30 minutes to properly write/document in less time than it takes me to review its output.

    It knows how to do AWS AMI and RDS images without details. If Amazon changes something and it errors making the ami, it will, without intervention, find amazon docs and update it’s process.

    My playbooks are essentially Word documents (md tho) telling it to make an AMI backup and disable notifications using the monitor server who’s name and IP are documented the neo4j. I don’t tell it where the logs are, it knows. It’ll fire up a Cloudtrail to hunt something down if it needs it.

    The diffs on the health checks aren’t binary. If it sees something it can’t explain, it will go back further in time, inspect other nodes, even start going down the debugging route for the app, (as long as I give it docs on how it should debug). It doesn’t miss a line. It doesn’t blindly ignore things. I was doing a rebuild on a server the other day, and during the post-health check, it noted that a certain error on this node is present on 5 other nodes and that the most common cause is a minor config issue. I asked it to diagnose; it found a config file from the old image that we never touch. I’ve been running with that issue on 6 servers for years. By my hand alone, I’d never have gone back, researched it and fixed it. It’s truly trivial, but it’s also things like that I don’t have the time to invest in that make this so useful.

    And it goes through this level of detail for every node. It only bugs me if it finds something outside of scope that’s unexplainable.

    I don’t give it any passwords or hard config. It queries Amazon or Neo4j, which are kept up to date programmatically.

    The real terror is I can do this on several different projects at once. As my documentation, guardrails, and automation get better, I’m getting multiple times my own hours’ work done every day, and I was really big on automation and self-healing stuff before this ever came along.

    Could the node classifier go bonkers and wipe out everything? It could. But I’m careful not to just cron up a bunch of jobs and take a nap. Probably the next step would be to make some overseers to verify backups are being made and it’s waiting between jobs.








  • I don’t have a known good resource, but you should look into K3S. https://docs.k3s.io/

    The documentation is very dense, I fear. I would probably go hit youtube search for “k3s cluster” and find a beginner that doesn’t annoy you too badly.

    The general path you’ll have to do is:

    Translate Configuration: Convert your docker-compose.yml settings into Kubernetes YAML manifests

    Install k3s

    Map Storage and Secrets. Replace local Docker volumes with PersistentVolumeClaims (PVCs) and .env files with Kubernetes Secrets and ConfigMaps.

    Deploy and Verify: Apply your manifests using kubectl apply and verify routing through the built-in Traefik ingress.

    Then you can install and add a new node.

    depending on how you set it up, it’ll move the containers around for you automatically.

    Note: all your k3s nodes need to be very close network wise. you can dump one at a friends house and make a cool geo-diverse setup, they need REALLY low latency



  • Yeah, there are a bunch of ways to tackle that. Chunk-Based Content-Addressable Block Storage (that weird thing where they put a folder of 0-A the folders under that with hex-based naming and small chunks of data wouldn’t be horrible keeping update sizes managable. The real problem there is dealing with Syncthing conflict files. The monobinary is either new or old. Maybe a local-only CBC with deltas delivered by syncthing so they can overwrite each other and a rollback/history option.

    P2P anything is complicated. Something as damning and imporant as password management makes it so much worse.


  • Anthropic enterprise models are very strong at following runbooks and managing infrastructure. To frame this properly, I spent about a month documenting the environment and creating runbooks. I can currently do things like the following:

    Follow the steps below to roll over this cluster. {

    Choose a server in this cluster by ID in numeric order (this server)

    Use Ansible+vault to create a backup AMI of (this server)

    Use Ansible+vault to collect health data on (this server) according to the server health runbook

    Use Ansible+vault to disable notification for this server

    Use Ansible+vault to preform a full kernel update and reboot the vm according to the update runbook.

    Use Ansible+vault to collect health data on this server according to the server health runbook post update

    Compare the pre and post health and server alarms for (this server), if there is an issue, pause, SOS notify oncall

    Use Ansible+vault to enable notification for this server

    }

    Each runbook contains exact steps required, how to verify success, each one of them is able to pull the E-stop, freeze on that step and SOS notify.


  • Completely valid. I myself am still on Bitwarden, waiting for the other shoe to drop, but my family is already entrenched in it.

    I think if I have to move them again, I might seriously consider forking keepass, adding the features I use in bitwarden.

    I do not look forward to dealing with IOS tho.