So, SourceHut is not hosted in anyone's cloud. I own all of the hardware outright and colocate most of it in a local datacenter.

I just built a new server for, and boy is she a beat. It cost me about $5.5K as a one-time upfront cost, and now I just pay for power, bandwidth, and space, which runs about $650/mo for *all* of my servers (10+).

Ran back of the napkin numbers with AWS's price estimator for a server of equivalent specs, and without even considering bandwidth usage it'd cost me almost TEN GRAND PER MONTH to host JUST that server alone on AWS.

AWS is how techbro startups pile up and BURN their investor money.

Running all of on AWS at our current scale would probably cost me more than $250K/year

@sir My employer's literally bought buildings to be as close to exchanges as possible and I'm pretty sure that was significantly cheaper (and way faster!) than the second-best that you can get with cloud services.

@sir That's crazy. I've been contemplating hosting my Fediverse servers on a bunch of Raspberry Pi's to cut hosting costs. Unclear how feasible it would be yet, but clearly a different scale!

@alex you should do this if you're prepared to lose all of your data when all of your SD cards fail

@alex friends don't let friends store important data on SD cards

@sir Yeah I was just thinking about I/O speed too. I'd dedicate one Pi to the database and connect a proper SSD to it. The rest of the nodes could be configured to fall back to other ones if they fail.

@alex then one rpi acting as a load balancer? Tbh I feel like this is definitely going to end in tears

@sir Yeah that's what I was thinking. It's similar to the setup I'm already using in my cloud host: 1 load balancer, 1 db server, and a bunch of disposable app servers. If the load balancer or db node fail it would be very painful. Maybe I could use a more reliable machine for just those nodes.

@alex tbh I don't trust raspberry pis with anything close to production workloads and I don't let them within sight of important data

@sir @alex I personally use a NAS running a ZFS mirror and NFS that I use for anything running on a Pi.

@sir @alex the site is hosted on a pi cluster ( ). Also the raspi4 was supposed to get support for network booting with a firmware update at some point, which would at least eliminate SD cards as a failure point

@6a62 @alex would also probably be okay to boot to a read only SD card and maintain no local state

@sir @6a62 @alex how would you even split mastodon into "disposable app servers?"

@passenger @alex @6a62 oh, I was mentally extrapolating this discussion to my plans for an ARM cluster for I still wouldn't use rpis to run mastodon instances under any circumstances

@passenger Here's a nice reference showing the different services that comprise Mastodon. Sidekiq, Streaming, and Rails can each have any number of processes and can be split across any number of nodes

@sir @6a62

@passenger @sir @6a62 Wtf is up with all those pgBouncers everywhere, what's the source of that image, is it serious? I think you only need one pgBouncer.

@alex @passenger @6a62 actually it's generally advisable to have one pgbouncer at each server and one pgbouncer at each client

@alex @passenger @6a62 at least for HA setups, or as close to HA as one can get with pg

@sir @passenger @6a62 Any more you could tell me about that? I'm pretty new to sysadmin and trying to understand why more pgbouncers is better

@alex @passenger @6a62 on the server allows you to load balance between different pg servers on that host, for example if doing a major version upgrade or something. On the client, it allows you to failover to the secondary server without extra work

@sir @6a62 @alex I have been running my Pi 3 4 node cluster using Docker swarm and NFS for all data storage for about a year and a half without having any SD card failure. All for personal use though not public. So far so good.

But I most definitely have had data corruption issues in the past. Maybe it's just me but for over a decade now I have held the opinion that SD cards are some of the most unreliable storage I have ever used. And I used zip disks...

@Kalvo @6a62 @alex all of my microSD cards are less than one year old. I've seen dozens of microSD cards fail

@sir @Kalvo @6a62 @alex

But have you seen dozens of microSD cards fail at the same time (blackmantappinghead.jpg)? Also if you don't write to them SD cards are great.
@sir @alex @6a62 @Kalvo

>that card
I don't even want to ask... Is that SD card hardware raid?

@lain @passenger @sir @Kalvo @6a62 I feel that it's more like inverse whack-a-mole but I laughed

@passenger @sir @alex @6a62 And if I never turn my computer on it's completely fail safe 😜

@sir Fair, I've had Pis burn out several times. But they're cheap AF. It would be really awesome if these little single board computers could be legitimately used for production, it would level the playing field.

@alex @sir why not use something from Pine64? I heard they have better I/O.
the Raspberry Pi is not the only SOC after all.

@grainloom @alex I sent them an email asking for a quote on a pine64 cluster but I haven't heard back. @PINE64?

@sir @grainloom @alex @PINE64 they're probably slammed with the Pinebook Pro and PinePhone launches. Want me to cross post this toot to Twitter for ya?

@klardotsh @grainloom @alex @PINE64 I'll probably ask about it when I see them at FOSDEM too

@sir @grainloom @alex @PINE64 context: I tend to get hella fast responses on Twitter relative to all other platforms

@sir @grainloom @alex

We're working on our own cluster of RockPro64 boards to power our infrastructure. Something like this might become available in the future.

In the meantime, we have the SOPine Clusterboard: 28 cores and 14GB of RAM in a compact package for just barely over $300. :)

@PINE64 @grainloom @alex was looking into that but deemed it unsuitable for my use-case. Would buy a similar cluster to the one you've been blogging about in a heartbeat and use it to bring ARM CI to the masses :)

@sir @grainloom @alex

We're planning to have the bigger of our two clusters available for use as a compilation box for developers. 😊

To be fair, the hardware architecture of our cluster(s) isn't that complex. It's basically a whole lot of RP64 boards connected together by Ethernet, and data being accessed from a NAS setup. There's some custom fabrication to fit it in a 2U case, but that's not necessary if you're not putting it in a data center.

@PINE64 @grainloom @alex the fact that I am putting it in a datacenter is why I deemed the other approach non-suitable for my needs

@sir @grainloom @alex

The rackmount case is likely to stay a one-off for us. There's not really enough market of people wanting to put together $1000+ ARM cluster setups to justify the manufacturing/setup costs to sell them. The network backplane board could be something which may be sold if people want it.

@PINE64 @grainloom @alex for the record my price rance was well above that figure

@PINE64 @grainloom @alex there are really not a lot of great choices out there for rack mounted ARM setups

@sir @PINE64 @grainloom @alex could try and get your hands on some ThunderX2 stuff

@sir @alex How do you even store data reliably for clusters? Right now I have glusterfs running duplicated across two SD cards, which I understand to be the sketchiest setup ever. Daily backups to B2 + a big HDD works for my usecase, but means that one specific storage node always needs to be up.

@njha @alex ZFS with 3 or more hard drives (ideally at least 5), frequent ZFS snapshots, and frequent off-site backups, with periodic backup integrity tests and prod restoration dry-runs

@alex What's your opinion of the PICO cluster?
This is a x10 Pi4 edition, but there are others with different boards and a different number of boards.


@alex @sir on old pi models Bs as pictured here, you'd be hard pressed to run much. Better off with half the number is Pi 4Bs

Sign in to participate in the conversation

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!