Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home

Buff Hardback posted:

If its supermicro, you can use IPMIView and not have to fight with Java web start stuff

It's ASRockRack, but on a lark I downloaded this anyway. It does connect! Event log, sensors, users all work, just not the fuckin console lol.

Adbot
ADBOT LOVES YOU

lampey
Mar 27, 2012

Rooted Vegetable posted:

Oh now I want it more. How do I hide it in a condo? (I refused to take no for an answer)

Screw it to the bottom of an ikea lack side table

BlankSystemDaemon
Mar 13, 2009



Jysk is Danish and not as widely available (though still in large parts of Europe, judging by the amount of non-Danish reviews on their site), but
I bought this from them, to store a server and disk shelves in.
Drilled a few holes to put four fans in, as well as a hole in the back because the rack-lock ears on the server couldn't be removed, and meant it was 4mm too wide.
Put a hole in the bottom of the right-most compartment, along with a PDU and cable ties, a switch for the light in the left-most compartment, and a 240V to 5V variable AC adapter I had lying around, so I can adjust fan speeds if necessary.
Server and disk-shelves are in the left-most compartment - which leaves just enough room on top for a UPS which I need to source (electrical grid is extremely stable in Denmark, there's only been a blackout twice in the last decade).
I plan on getting 1-2 cm sound-deadening foam to pad the inside with, should reduce the sound somewhat.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

I have a Synology RS3617xs that is used mainly as a backup target for Veeam. About every 4-5 months I will get an error on a drive, and I never hear from it again.

WD is a pain to RMA a drive with lately. If it's still marked 'healthy' in the system, is it okay to just leave it in there? These are all WD Enterprise 3TB drives

Dear user,

An I/O error occurred to drive 7 in SYN01, but the drive is working properly now after several retries. The error might have been caused by bad sectors. We recommend backing up your data immediately to ensure data integrity. Afterward, please go to Storage Manager > HDD/SSD > S.M.A.R.T. to run the Extended Test and refer to the test results for the recommended actions.

Crunchy Black
Oct 24, 2017

by Athanatos
I used to think that IPMI was an absolute requirement for anything in the rack but I couldn't find a good Threadripper board that had it for a reasonable price so I've gotten a lot better with the terminal/ssh. In general, it's made me a much better computer toucher even if its mostly a hobby.

Volguus
Mar 3, 2009
Currently my gateway is a normal consumer level PC in a rack-able case. I don't know if the MB has IPMI or not, I'll have to check, though I don't quite know even what to look for in the BIOS. For the future I'll definitely look for that feature in whatever equipment I buy, though I presume that most if not all of the enterprise stuff comes with it. Luckily I don't need physical keyboard or monitor that often, maybe once or twice per year, but when I do i don't have any other choice. PiKVM looks promising as a fairly cheap and capable solution.

Enos Cabell
Nov 3, 2004


Anyone have a quick tl,dr on how IPMI works? I'm pretty sure my MB supports it (Gigabyte GA-7PESH2) but I've never tested it out.

e: guess I can google for myself, looks like I have a project for this evening https://www.servethehome.com/gigabyte-server-motherboard-ipmi-walkthrough-part-1/

BlankSystemDaemon
Mar 13, 2009



Crunchy Black posted:

I used to think that IPMI was an absolute requirement for anything in the rack but I couldn't find a good Threadripper board that had it for a reasonable price so I've gotten a lot better with the terminal/ssh. In general, it's made me a much better computer toucher even if its mostly a hobby.
I mean, IPMI is a requirement for anything in a rack, but not for the reason you're saying. If a FreeBSD machine with 32 threads has a lot of, say, 128.N it means that it's got more than four times the number of processes in its runqueue compared to the number of threads, and if you've got the scheduler interactivity configured to keep processes on active threads without deferring, you'll find that logging in via ssh is impossible.
What you can do, instead of rebooting the system (because there's nothing really wrong with it, it's just busier than a one-legged person in an rear end-kicking contest), is use out-of-band management to connect to the console via serial-over-LAN over vKVM and from there either start killing processes, or more reasonably, use cpuset(1) to restrict all processes to 31 of your threads, so that you suddenly have a whole thread free to do whatever you want with, and the system can continue chugging along or you can ssh into it and investigate why everything's blowing up (is it just a production workload, are you having IOPS issues getting to your storage, or is there a fork bomb happening - these are all questions you can't necessarily know the answers to if you reboot, without replicating the scenario).

Volguus posted:

Currently my gateway is a normal consumer level PC in a rack-able case. I don't know if the MB has IPMI or not, I'll have to check, though I don't quite know even what to look for in the BIOS. For the future I'll definitely look for that feature in whatever equipment I buy, though I presume that most if not all of the enterprise stuff comes with it. Luckily I don't need physical keyboard or monitor that often, maybe once or twice per year, but when I do i don't have any other choice. PiKVM looks promising as a fairly cheap and capable solution.
Usually very very little about out-of-band management gets configured via the firmware, you usually have to access the cards own Option ROM after the firmware is finished loading and POST has been completed (this is true, even if it's not actually a separate card, as it'll appear to be attached to the PCI bus in either case).

Your best bet is to look for a block diagram of the motherboard in question.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Enos Cabell posted:

Anyone have a quick tl,dr on how IPMI works? I'm pretty sure my MB supports it (Gigabyte GA-7PESH2) but I've never tested it out.

e: guess I can google for myself, looks like I have a project for this evening https://www.servethehome.com/gigabyte-server-motherboard-ipmi-walkthrough-part-1/

The page you linked to there is a pretty decent intro to it. You can either utilize the tools talked about there, or grab a copy of IPMIView from Supermicro. Either way, your MB is probably defaulted to having IPMI enabled out of the box, but otherwise you might have to drop into BIOS via traditional kb/monitor once to enable it. After that, it'll show up on your network as a separate device, and you enter the appropriate IP for your server (IPMIView has a scan function that'll just check your entire network for compatible IPMI devices). Then you're in the interface and you can do whatever from there. Both the Gigabyte web-UI and IPMIView should let you be able to do basic stuff like interact with BIOS settings, act as a KVM, and "plug in" media (.iso/.img files) so you can install stuff as if you'd stuck it in there on a USB drive.

Honestly it's pretty intuitive--the biggest challenge is usually just connecting in the first place, and honestly even that's pretty easy. Minor note that, depending on your settings, IPMIView may not want to save the connection settings under Win10, which is a little obnoxious.

Raymond T. Racing
Jun 11, 2019

Enos Cabell posted:

Anyone have a quick tl,dr on how IPMI works? I'm pretty sure my MB supports it (Gigabyte GA-7PESH2) but I've never tested it out.

e: guess I can google for myself, looks like I have a project for this evening https://www.servethehome.com/gigabyte-server-motherboard-ipmi-walkthrough-part-1/

Since you have a 7PESH2, I have to ask, what's your boot media? are you running off a USB drive? If so, I strongly suspect you are eventually going to have that flash drive get completely killed.

We didn't discover it until after JDM got mad at our group for not liking the way he ran things and kicked us out, but we strongly suspect there's a voltage issue on all 7PESH2s that will fry any USB devices connected to it given enough reboots.

GreatGreen
Jul 3, 2007
That's not what gaslighting means you hyperbolic dipshit.
Hey guys, I need to buy some drives for a security camera storage pool at a local business. The company who sells the security camera setup typically uses Western Digital Purple drives with their stuff, but I've been hearing chatter about Seagate Skyhawk Surveillance drives and wanted to ask you guys about them. They're 7200rpm as opposed to the WD's 5400, and they have a workload rating of 550 TB per year.

However, last I checked (admittedly several years ago), Western Digital was the king of platter drives and Seagate was pretty shaky in terms of reliability. Has Seagate stepped up their game? Would you guys recommend Seagate Skyhawk Surveillance drives or Western Digital Purple drives for a company's security camera setup?

Enos Cabell
Nov 3, 2004


Buff Hardback posted:

Since you have a 7PESH2, I have to ask, what's your boot media? are you running off a USB drive? If so, I strongly suspect you are eventually going to have that flash drive get completely killed.

We didn't discover it until after JDM got mad at our group for not liking the way he ran things and kicked us out, but we strongly suspect there's a voltage issue on all 7PESH2s that will fry any USB devices connected to it given enough reboots.

drat! I guess that explains how I've gone through 2 Unraid USB boot drives, both killed after a reboot. That's wild, I thought I just had a lovely batch since the two that failed were from the same multipack. Super annoying, but since I reboot about once a year on average I guess I'll just have to live with it until I completely redo this system.

bobfather
Sep 20, 2001

I will analyze your nervous system for beer money

GreatGreen posted:

Hey guys, I need to buy some drives for a security camera storage pool at a local business. The company who sells the security camera setup typically uses Western Digital Purple drives with their stuff, but I've been hearing chatter about Seagate Skyhawk Surveillance drives and wanted to ask you guys about them. They're 7200rpm as opposed to the WD's 5400, and they have a workload rating of 550 TB per year.

However, last I checked (admittedly several years ago), Western Digital was the king of platter drives and Seagate was pretty shaky in terms of reliability. Has Seagate stepped up their game? Would you guys recommend Seagate Skyhawk Surveillance drives or Western Digital Purple drives for a company's security camera setup?

I have had 2/4 of 4 TB Purples begin to develop bad sectors just outside of their warranty period. Low sample size, though, so it could just be my bad luck. I replaced the purples with a larger Seagate Ironwolf that has been trucking along recording Blue Iris footage with nary a problem for the past few years.

IOwnCalculus
Apr 2, 2003





Volguus posted:

For the future I'll definitely look for that feature in whatever equipment I buy, though I presume that most if not all of the enterprise stuff comes with it.

The vast majority of my experience is with Supermicro, but it seems like IPMI went from "option that only some people checked" to "standard unless you custom-order 10,000 servers and want us to remove it to save $0.50 per server" around Sandy Bridge to Haswell. It's really rare to see an actual server without it that's newer than that.

Hughlander
May 11, 2005

The Milkman posted:

Yeah this. I just wish it was more like what I was under the impression it was (something generic I could connect to with any ol VNC-ish client) instead of what it is (semi-proprietary interface that in my case relies on a crummy and increasingly out of date java web start widget)

I updated the Bios of my SuperMicro to get HTML5 IPMI client. Would do 100 times again.

Raymond T. Racing
Jun 11, 2019

Enos Cabell posted:

drat! I guess that explains how I've gone through 2 Unraid USB boot drives, both killed after a reboot. That's wild, I thought I just had a lovely batch since the two that failed were from the same multipack. Super annoying, but since I reboot about once a year on average I guess I'll just have to live with it until I completely redo this system.

We were looking at hacky ways of getting around it, either wiring up a separate USB port running on PSU 5v instead of motherboard 5v or doing some sort of kickstart boot using a PCIE USB card (SSD plugged into SATA port boots, loads the ROMs need to boot off that USB card, kicks it over to USB), but by the time we had diagnosed the issue, pretty much everyone with one had switched away from it.

Hughlander posted:

I updated the Bios of my SuperMicro to get HTML5 IPMI client. Would do 100 times again.

Which motherboard did you do this on? I’ve got an X9 and would kill for HTML5 IPMI.

BlankSystemDaemon
Mar 13, 2009



Buff Hardback posted:

We were looking at hacky ways of getting around it, either wiring up a separate USB port running on PSU 5v instead of motherboard 5v or doing some sort of kickstart boot using a PCIE USB card (SSD plugged into SATA port boots, loads the ROMs need to boot off that USB card, kicks it over to USB), but by the time we had diagnosed the issue, pretty much everyone with one had switched away from it.


Which motherboard did you do this on? I’ve got an X9 and would kill for HTML5 IPMI.
The trick is to find a SuperMicro motherboard with your generation of AST chip and see if it's got an updated firmware. All AST chips of a given revision should be the same.

Hughlander
May 11, 2005

Buff Hardback posted:

Which motherboard did you do this on? I’ve got an X9 and would kill for HTML5 IPMI.

X10SL7-f, I went to Firmware revision 3.72.00 which gave IPMI 2.0 and the HTML5 viewer. BMC Firmware https://www.supermicro.com/support/resources/bios_ipmi.php?vendor=1 seems to be the correct link. Standard warning this this could brick your server if done wrong / yatta yatta.

Raymond T. Racing
Jun 11, 2019

BlankSystemDaemon posted:

The trick is to find a SuperMicro motherboard with your generation of AST chip and see if it's got an updated firmware. All AST chips of a given revision should be the same.

Hughlander posted:

X10SL7-f, I went to Firmware revision 3.72.00 which gave IPMI 2.0 and the HTML5 viewer. BMC Firmware https://www.supermicro.com/support/resources/bios_ipmi.php?vendor=1 seems to be the correct link. Standard warning this this could brick your server if done wrong / yatta yatta.

drat, looks like the X10 is using the AST, whereas my X9 is using the WPCM450 which doesn't appear to have a version using HTML5.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

Buff Hardback posted:

drat, looks like the X10 is using the AST, whereas my X9 is using the WPCM450 which doesn't appear to have a version using HTML5.

Yeah, X9's and older get no love. IPMIView is functional, but certainly not as slick as what you can get out of some of the newer ones.

Crunchy Black
Oct 24, 2017

by Athanatos

BlankSystemDaemon posted:

I mean, IPMI is a requirement for anything in a rack, but not for the reason you're saying.
Agreed on almost all points, just making sure folks understand ssh exists and its a cool thing to know. Having IPMI is ALWAYS preferable to not, and my rack is just out in the garage. [Its worth pointing out that the machine I was talking about is my F@H rig and not my freenas, which is a Haswell Xeon with IPMI. :) ]

GreatGreen
Jul 3, 2007
That's not what gaslighting means you hyperbolic dipshit.

bobfather posted:

I have had 2/4 of 4 TB Purples begin to develop bad sectors just outside of their warranty period. Low sample size, though, so it could just be my bad luck. I replaced the purples with a larger Seagate Ironwolf that has been trucking along recording Blue Iris footage with nary a problem for the past few years.

Cool to know, thanks for the info. :)

movax
Aug 30, 2008

Hughlander posted:

X10SL7-f, I went to Firmware revision 3.72.00 which gave IPMI 2.0 and the HTML5 viewer. BMC Firmware https://www.supermicro.com/support/resources/bios_ipmi.php?vendor=1 seems to be the correct link. Standard warning this this could brick your server if done wrong / yatta yatta.

Wait, what?!?! I upgraded the BMC on my X11SSL-CF in December and I swear it's still a Java-based POS. Definitely going to check again now...

madsushi
Apr 19, 2009

Baller.
#essereFerrari
Just updated my X10SRL-F to 3.90 and have that HTML5 goodness. Thanks for the tip!

Hughlander
May 11, 2005

madsushi posted:

Just updated my X10SRL-F to 3.90 and have that HTML5 goodness. Thanks for the tip!

Glad it helped someone !

BlankSystemDaemon
Mar 13, 2009



Crunchy Black posted:

Agreed on almost all points, just making sure folks understand ssh exists and its a cool thing to know. Having IPMI is ALWAYS preferable to not, and my rack is just out in the garage. [Its worth pointing out that the machine I was talking about is my F@H rig and not my freenas, which is a Haswell Xeon with IPMI. :) ]
Sure, for day to day operation, I ssh in - but then I live most of my computing time on the terminal with tmux.

Rooted Vegetable
Jun 1, 2002

This brings me to yet another point, SSH Public Key Management. What is the usual approach the context of consumer NAS/Home Servers for this?

I've got a mixture of devices around the place, one Linux sofa warrior laptop, a VM for shitposting, a work windows laptop, gaming windows VM, Pi for a jumpbox. For the devices that support it (Unraid, explain yourself again!) keeping SSH keys neatly managed between all of them is kind of a hassle if you follow the "private key never leaves the device" approach. If I went to add each public key to every other device, that would take ages.

Long story short: Want to sync public keys between all of those? The simplier and easier to manage the better.

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)

Rooted Vegetable posted:

This brings me to yet another point, SSH Public Key Management. What is the usual approach the context of consumer NAS/Home Servers for this?

I've got a mixture of devices around the place, one Linux sofa warrior laptop, a VM for shitposting, a work windows laptop, gaming windows VM, Pi for a jumpbox. For the devices that support it (Unraid, explain yourself again!) keeping SSH keys neatly managed between all of them is kind of a hassle if you follow the "private key never leaves the device" approach. If I went to add each public key to every other device, that would take ages.

Long story short: Want to sync public keys between all of those? The simplier and easier to manage the better.

*stares in ansible*

Hadlock
Nov 9, 2004

Can you just use one pub/priv keypair for everything? Or split off the work laptop and your private computers and use just two keypairs? Ideally you're not logging into your work machine from your personal machine but that's up to you I guess

If you settle on a single key pair for your private machines then you don't have a traveling salesman problem of keeping your key pairs up to date

If you truly don't care about security add ~/.ssh/ to dropbox, then write a cron job that runs rm ~/.ssh/authorized_hosts, then for each .pub, cat *.pub >> authorized_hosts and chmod 700 the new auth hosts file, then set the cron job to run every 10 * * * *

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)
Write an ansible inventory file with all the hosts you want to connect to and create a playbook with something like this:

code:
- name: Set authorized key taken from file
  ansible.posix.authorized_key:
    user: "{{ ansible_user }}"
    state: present
    key: "{{ lookup('file', '/path/to/.ssh/id_rsa.pub') }}"
Congrats, now you can put devops in your linkedin.

Raymond T. Racing
Jun 11, 2019

Just Yubikey GPG/SSH key, problem solved.

Rooted Vegetable
Jun 1, 2002

Matt Zerella posted:

*stares in ansible*

Now I'm considering learning it, and puppet, just for the sake of it. It does kind of seem like an overreaction (of the best kind)...

Hadlock posted:

Ideally you're not logging into your work machine from your personal machine but that's up to you I guess

I'd stop if they actually cared.

Hadlock posted:

If you settle on a single key pair for your private machines then you don't have a traveling salesman problem of keeping your key pairs up to date

Well yes true and considered but in part I was wondering if there was a more simple way I wasn't thinking of... actually that reminds me about something for the next one.


Hadlock posted:

If you truly don't care about security add ~/.ssh/ to dropbox, then write a cron job that runs rm ~/.ssh/authorized_hosts, then for each .pub, cat *.pub >> authorized_hosts and chmod 700 the new auth hosts file, then set the cron job to run every 10 * * * *

This exact text got me thinking about the setup for Ubuntu Server which includes a simple text prompt to import from GitHub (2fa protected audited account etc). I started wondering if, at least on Ubuntu, I could automate that somehow. This lead me to this forum post and this Ubuntu blog about ssh-import-id which could come in handy. Keep in mind at least the GitHub account is secured with MFA and a strong password. Just that but with Cron or something else suitable e.g. update on login.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
I do this stuff for a living and I'd avoid learning Puppet and Chef frankly given how they're slowly dying off. If you're not doing containers everywhere there's really only two sane options for machine configuration management both professionally and at home going forward - Ansible or Salt.

Other options include doing way overengineered stuff for home like

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)
Please for the love of god don't learn puppet.

If you want state management use salt. If you just want an easy way to bash poo poo at a bunch of different servers, use ansible.

Jeff Geerling has an excellent book on ansible and even did a large web series on using it and he's basically the Ansible master.

H110Hawk
Dec 28, 2006

Hadlock posted:

Can you just use one pub/priv keypair for everything? Or split off the work laptop and your private computers and use just two keypairs? Ideally you're not logging into your work machine from your personal machine but that's up to you I guess

If you settle on a single key pair for your private machines then you don't have a traveling salesman problem of keeping your key pairs up to date

SSH supports CA's. Sign your keys and send your CA everywhere. Never think about it again.

BlankSystemDaemon
Mar 13, 2009



Rooted Vegetable posted:

This brings me to yet another point, SSH Public Key Management. What is the usual approach the context of consumer NAS/Home Servers for this?

I've got a mixture of devices around the place, one Linux sofa warrior laptop, a VM for shitposting, a work windows laptop, gaming windows VM, Pi for a jumpbox. For the devices that support it (Unraid, explain yourself again!) keeping SSH keys neatly managed between all of them is kind of a hassle if you follow the "private key never leaves the device" approach. If I went to add each public key to every other device, that would take ages.

Long story short: Want to sync public keys between all of those? The simplier and easier to manage the better.
ssh-copy-id is your friend, it exists to copy your public key as the first thing you do when you connect to a new server.

Matt Zerella posted:

*stares in ansible*
If we have to make things complicated, I'm tempted to say that the proper way to do it is (optionally split-horizon) DNS with (self-hosted?) CA, DNSSEC, and SSHFP Records - because there's no such thing as overkill.

Rooted Vegetable
Jun 1, 2002

BlankSystemDaemon posted:

ssh-copy-id is your friend, it exists to copy your public key as the first thing you do when you connect to a new server..

This is what I ended up doing. I spent 5 minutes running round my machines and utility VMs getting Public Keys to add to GitHub, then another few running that command + restarting sshd (or the whole machine because who gives a poo poo).

However when I add a machine or key later I've got the whole performance all over again.

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)

Rooted Vegetable posted:

This is what I ended up doing. I spent 5 minutes running round my machines and utility VMs getting Public Keys to add to GitHub, then another few running that command + restarting sshd (or the whole machine because who gives a poo poo).

However when I add a machine or key later I've got the whole performance all over again.

Hence, ansible.

Rooted Vegetable
Jun 1, 2002

Matt Zerella posted:

Hence, ansible.

Alight I'll head off and learn the enterprise machine inventory configuration tool

Adbot
ADBOT LOVES YOU

Axe-man
Apr 16, 2005

The product of hundreds of hours of scientific investigation and research.

The perfect meatball.
Clapping Larry

Rooted Vegetable posted:

Alight I'll head off and learn the enterprise machine inventory configuration tool

Hey devops is hot right now! Good resume fluff.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply