Next I run #syncthing on my laptop, and on my #homelab #intelN100 #n100 mini pc / server that runs in the cupboard and is very #lowpower I run #proxmox and this also has a #samba share which allows any other network devices to see the media.
With syncthing running, I always have two copies of the media, but for backup I was using #rclone to send an encrypted copy to #googledrive - which I am in the process of switching over to #nextcloud running on #hetzner
3/4
Aha. The first issue was that for some reason, I did not install kube-vip on 2 of the 3 CP nodes. So there was only ever one host holding/serving the k8s VIP. I'm also thinking about getting some load balancing going for the k8s API. Right now, whoever holds the VIP for the API via kube-vip gets all of the requests, if I understand it correctly. Perhaps I could improve the stability by load-balancing the VIP. That's of course not possible with ARP mode, so some more reading necessary.
Okay, from initial investigations it looks like the crash this morning at 10:17 AM was due to kube-vip failing to do its leader election due to timeouts when contacting the k8s API, and consequently the k8s API IP going offline. That wreaked havoc in the cluster for a bit. I'm still not 100% whether the I/O overload I'm seeing on the CP node was created by the k8s API going down, or whether it caused the API to go down.
The new control plane on my Pi 4 is not really stable. I woke up to one sealed Vault instance because it crashed in the middle of the night, and then an hour ago a lot of my services went down for a period, possibly due to I/O overload on one of the CP Pis. Need to dig deeper into what happened there now.
This weeks stupidity… I made a mini 5u HomeLab for my HomeAssistant and PiHole boxes that sit inside an Ikea storage crate.
Includes a Pi5 16GB, Pi 3b+, Pi Zero W2 and a Pimoroni Badger for stats.
#ikea #ikeahack #homelab #minilab #homeassistant #pihole #network #3dprinting #3dprinted #raspberrypi #pimoroni #diy #maker @geerlingguy
Interesting. Just switching off the three k8s control plane VMs on my expansion server brought the power draw of my Homelab down by 20 Watts:
And the machine isn't idle either, there are still two k8s worker VMs running on it.
It's done. I've just shut down the k8s control plane node VMs after the Pi4 nodes finally had everything running.
The migration is done.
Now onto the cleanup list.
Bruh, I might've wasted my time learning how to passthrough a GPU to an #LXC container on #Proxmox (as well as mount a SMB/CIFS share) and write up a guide (haven't been able to test yet, cept with the latter) - all by doing some seemingly magic #Linux fu with some user/group mappings and custom configs, if it turns out that you could actually achieve the same result just as easily graphically using a standard wizard on PVE.
It's 4am, I'll prolly try to find time later during the day, or rather evening (open house to attend at noon), and try using the wizard to 1) Add a device passthrough on an LXC container for my #AMD iGPU (until my #Intel #ArcA380 GPU arrives) and see if the root user + service user on the container could access it/use it for transcoding on #Jellyfin/#ErsatzTV, and 2) Add a SMB/CIFS storage on the Proxmox Datacenter, tho my #NAS is also just a Proxmox VM in the same cluster (not sure if this is a bad idea?) and see if I could mount that storage to the LXC container that way.
#Homelab folks who have done this, feel free to give some tips or wtv if you've done this before!
Project "Talos Most of the #HomeLab Things" has taken another step closer to fruition. The new NAS head machine arrived yesterday so I put in an HBA and a 10Gbit card and installed #TrueNAS.
This morning I swapped the ZFS array from its old home (nibbler) to the new NAS head (morbo), imported the pools and set up NFSv4 exports. After mounting those shares on nibbler my #Jellyfin LXC booted right up.
The next step is to convert Jellyfin into a #Kubernetes workload with an NFS-backed PVC. After I've got that working for everything but transcodes I'll be able to pave nibbler with Talos and get transcodes back, then work on the rest of the media stack.
Future steps are to pave hypnotoad and lrrr with Talos, put TrueNAS on the backup machine (crushinator) and maybe put control plane nodes into TrueNAS VMs.
I think if we could manage to bottle the post-"Oh god, I think I just nuked my storage cluster" feeling of elation when it works again, we might have a really potent new drug on our hands.
Cositas en mi homelab:
- HomeAssitant (automatización del hogar)
- Jellyfin (media player)
- Nextcloud (fotos, calendario, contactos, RSS, documentos, Kanban, ...)
- Searxng (buscador agregador de otros)
- LibreTranslate (traductor LLM local)
- Forgejo (gestor de repositorios)
- Motion (detector de movimiento para cámaras IP)
- Prosody y Synapse (XMPP para video llamadas, chat, etc.)
- Y más cositas para gestionar lo anterior (Gatus, Nagios, Grafana, Prometheus, Graphite, Puppet, ...)
#HomeLab
The cluster is back now. No data loss occurred. Everything surprisingly kept running or came back very fast.
Also, I now know how to do manual surgery on a Ceph monmap.
On the positive side: My MONs are now all located on the right host.
And if I wasn't such an impatient git sometimes, this could have certainly been accomplished without shaving off 5 years from my life expectancy.
Now excuse me while I return right back to my fainting couch.
2/2
Okay people. Listen to me: When Ceph Rook says it's not safe do switch off the MON you want removed, at least check the Ceph quorum. Make sure that that MON is not the last survivor. Only then do whatever you think is necessary to get the MON deleted.
Because otherwise, you will end up with a Ceph cluster without a quorum, and you will experience things like "not being able to get any output from ceph -s".
1/x
New blog post!
In which I talk in detail how I deployed Nextcloud locally in my homelab using LXD and NextcloudPi.
I go through the whole process of setting up LXD, importing the NCP image, installing, configuring, and scheduling backups for my own personal cloud.
https://stfn.pl/blog/67-deploying-nextcloud-locally-with-lxd/
Okay dumb idea. Since colocation in Germany is expensive and I live moderately close to the Netherlands.
Is the situation any better over there? #homelab
How do y'all deal with #homelab machines that occasionally fail to boot? I have no SSH (it didn't boot). I have no remote management (it is an Intel NUC...).
So I have to grab a monitor to debug, but it lives on the opposite end of my house from any monitor, and its a pain...
Alright, enough time on the fainting couch, time for a new attempt. This time first increasing the number of MONs and then waiting for the cluster to settle before going down again.
After a re-setup of my #homelab server with #truenas I finally have #nextcloud up and running again.
In the past I used Nextcloud Photos to backup my phone but now moved to #immich. Now finally can also move all my #googlephotos to my private cloud 
Just triggered the export of the photos ;-)
Again one small stepp away from google and co!
Wooooooh. Ceph is a good piece of software. And so is Rook. Because if they just bluntly did what I just told them to do, i would be so royally screwed. I would probably be questioning my choice of hobbies right now.
Pro tip: The wrong way to migrate MONs is to just change the node selector to remove two of the old MONs and add the two new ones. That is the absolute wrong way of doing it.
Now excuse me while I get acquainted with my fainting couch.
