Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
HPL
Aug 28, 2002

Worst case scenario.
Oh no, I forgot to compile the TCP stack into Linux!

Adbot
ADBOT LOVES YOU

evol262
Nov 30, 2010
#!/usr/bin/perl

Tab8715 posted:

FTP wasn't ever designed to be secure. That's where SFTP comes into play.

FTPS :eng101:


SFTP is part of openssh.

jre posted:

If you are spending any measurable amount of time configuring stuff in containers, you're doing it wrong.
I'm not saying that's not true. I'm saying it's depressingly common, and "you're doing it wrong" doesn't even register with shops which have conflated "configuration as code" with "my configuration management system is the source of all knowledge".

jre posted:

These same devs are (not) managing the security of their vms because "devops"

Yes, but with higher visibility in many cases, because they're running a "normal" system which can be managed/scanned/etc.

jre posted:

The other thing thing that docker can do that vms can't is get better cpu utilisation in :yaycloud: platforms. Its incredibly expensive to run under utlitsed vms in aws / rackspace etc just for the purposes of isolation. You can bang lots of containers onto an ECS cluster with much lower spend. This is actually the use case that made me go from what's the point of docker to :vince:

Yes, you can. Though I'd probably still be looking at kubernetes or Mesos as an addition.

I'm not saying VMs are better or containers are better. I'm saying they have different use cases, which we know, but the hype machine is real, and we have a lot of customers who tried and failed to use openstack because of hype, and are trying (and failing) to use containers well, because of hype. "You should use containers" is easy. Building containers is easy. Container best practice is hard, because the management model is similar enough to :yaycloud: that shops don't "get it", and some forget that it's still a very single purpose real system. How many security failures have been due to not updating " normal" systems which can be managed just by yum/sccm/whatever? It gets worse when your "servers" are seen as applications the dev team builds.

I'm happy to talk about best practice. I'm not happy to be painted as defending the crappy things people do just for pointing out that it happens.

jre
Sep 2, 2011

To the cloud ?



evol262 posted:

we have a lot of customers who tried and failed to use openstack because of hype

Right, because of "hype"

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

jre posted:

Right, because of "hype"
Get enough alcohol in me and I'll talk about running 1,000+ Nova servers in production :suicide:

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Tab8715 posted:

Do you also complain that your Honda Civic doesn't tow, go through snow or dirt roads well? The bank picked the wrong tool for the job, that doesn't make FTP bad.

He's not complaining about the Honda Civic, he's complaining about the tow company using Honda Civic's.

evol262
Nov 30, 2010
#!/usr/bin/perl

jre posted:

Right, because of "hype"

I meant they "needed" openstack because of the hype around it, but couldn't make their company's development model work with it. The same would apply to AWS/GCE/whatever, but a lot of institutional customers (airlines, banks, etc) are unwilling to use any kind of public cloud, so running their own was the option.

A depressing amount of decision makers really adhere to the fluff pieces in tech rags, and they hear about cloud, containers, hybrid cloud, nosql, server-side JavaScript, or whatever, and push it. Then they look at their shiny new thing and wedge all their crufty old poo poo on it, and it doesn't "work" like their old environment used to.

That's not the fault of openstack, docker, or anything else, but it's a good (and often ignored) lesson about not jumping on the bandwagon, and this is where I'm going with the "config management inside containers" thing, because that'll happen instead of using packer/lorax/whatever.

Not defending openstack, which is OK for its use model, but requires a team just to get it running and keep it managed without falling over. But openstack failing for these customers is largely due to their development model and operations teams, not openstack.

It's openstack's fault when your 1000 compute node cluster falls over because "everything must pass through MySQL instead of the message queue we require"

It is not openstack's fault when your 10 node POC fails because your application's resilience depends on a highly available pet, and this is the majority of failed deployments I see.

The same caveats apply to containerizing everything just because. Your development model needs to work with it.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
Look guys, Windows 98 is a great operating system. It's the most recent thing my doctor's PACS software will run on so

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

What's the hip and cool way for monitoring/graphing SNMP data (from a ubiquiti router if it matters) nowadays? I've used cacti and mrtg in the past, but, to be honest, they're pretty ugly and I'd like something with a cool, modern web UI.

evol262
Nov 30, 2010
#!/usr/bin/perl

Thermopyle posted:

What's the hip and cool way for monitoring/graphing SNMP data (from a ubiquiti router if it matters) nowadays? I've used cacti and mrtg in the past, but, to be honest, they're pretty ugly and I'd like something with a cool, modern web UI.

Grafana plus your choice of graphite+collectd or influxdb. Or the opennms grafana plugin if you're masochistic enough to run opennms

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

evol262 posted:

Grafana plus your choice of graphite+collectd or influxdb. Or the opennms grafana plugin if you're masochistic enough to run opennms

Oh yeah, this looks good.

Why would I choose one over the other when it comes to graphite/collectd or influxdb?

jre
Sep 2, 2011

To the cloud ?



Thermopyle posted:

Oh yeah, this looks good.

Why would I choose one over the other when it comes to graphite/collectd or influxdb?

Graphite has huge amount of support & tooling but has an awful file based storage format which limits how far you can scale it

Influx has better storage format but is new & flaky and doesn't have as much tooling support

jre fucked around with this message at 22:26 on Apr 16, 2016

Cidrick
Jun 10, 2001

Praise the siamese

Suspicious Dish posted:

do you want a job doing exactly that same thing but for a fancy startup running ubuntu 15.10

also we dont have logging yet because logging sucks afaict

Not unless you went back to RedHat which I'm assuming you didn't since otherwise you wouldn't be running Ubuntu 15 :spergin:

(I Have Opinions about Ubuntu in the enterprise Linux world but I'm not going to start that here)

jre posted:

Graphite has huge amount of support & tooling but has an awful file based storage format which limits how far you can scale it

Influx has better storage format but is new & flaky and doesn't have as much tooling support

Graphite is great if you can use a better storage backend driver for it that isn't whisper files, because that's a nightmare to try and scale. Influx is supposed to be good, but the Graphite Guy at my shop claimed it was too immature to use and there was developer drama which he thought would really hurt its long-term viability. Cyanite looks interesting since that uses Cassandra, but it's still super new and I haven't heard of any shops actually using it yet. If anyone ends up trying it out I would love to hear your experiences with it.

jre
Sep 2, 2011

To the cloud ?



Cidrick posted:

Graphite is great if you can use a better storage backend driver for it that isn't whisper files, because that's a nightmare to try and scale.
There isn't a non awful backend for it. :smith:

quote:

Cyanite looks interesting since that uses Cassandra, but it's still super new and I haven't heard of any shops actually using it yet. If anyone ends up trying it out I would love to hear your experiences with it.
It was buggy as gently caress and randomly returned wrong / missing / no data when I last tried it . Even if they could make it work properly it would still suck because graphite doesn't support dimensions.

quote:

Influx is supposed to be good, but the Graphite Guy at my shop claimed it was too immature to use and there was developer drama which he thought would really hurt its long-term viability.
Influx looks amazing, but the the repeated massive non backward compatible changes they keep making to storage engine give me the fear. Also no-one is using it any kind of scale yet.

I'm in the process of migrating from a huge graphite infrastructure to opentsdb. Apart from hassle of dealing with hbase it's waaaaaay better.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe

Cidrick posted:

Not unless you went back to RedHat which I'm assuming you didn't since otherwise you wouldn't be running Ubuntu 15 :spergin:

(I Have Opinions about Ubuntu in the enterprise Linux world but I'm not going to start that here)

Nah, still at Endless. Working on a large content management pipeline. I'm not a fan of Ubuntu either.

We need a better mesos cluster. We tried to set ElasticSearch/Kibana for logging but that really sucked (each like in a traceback became its own log entry). We have grafana but we haven't got it to measure inside containers/cgroups yet. Everybody says "monitoring and alerts" but nobody can tell me what that means or what software to use. Nobody is sure how to properly secure some endpoints into the cluster while leaving others exposed. For data volume management we found this tool called Flocker, which, when failing, fills up our disk with 4GB of log files. Also, each node in a cluster requires a unique SSL certificate, so it's difficult to do autoscaling for more nodes. Docker also fills up our disks (no GC of unused layers), Mesos and Marathon barely have any error handling (if a container fails to start, it will just try again it instantly with no backoff until the end of time).

I'm unsure how people actually set the up in production and we need somebody to help us with that.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!
I'm trying to setup a system wide default for fonts under Gnome. Ideally I'd like the computer to associate e.g., "Monospace --> Liberation Mono", "Sans Serif --> Liberation Sans", and "Serif --> Liberation" so that random programs asking for "Sans 12" font will use "Liberation Sans 12" and so on. I also want to configure font hinting and antialiasing as well.

What's the best way to do this? I'm confused because it looks like I can alter the following files:

/etc/dconf/local.d/local.key
/etc/X11/Xresources
/etc/fonts/font.conf

It needs to be system wide so gsettings set ... doesn't work. My first thought is to edit the dconf file and from my testing it works as expected, but this wiki page talking about setting the hinting and antialiasing got me confused because hinting shows up as a setting as well when I list all the keys available in gsettings and they recommend editing the fonts.conf file.

RFC2324
Jun 7, 2012

http 418

Boris Galerkin posted:

I'm trying to setup a system wide default for fonts under Gnome. Ideally I'd like the computer to associate e.g., "Monospace --> Liberation Mono", "Sans Serif --> Liberation Sans", and "Serif --> Liberation" so that random programs asking for "Sans 12" font will use "Liberation Sans 12" and so on. I also want to configure font hinting and antialiasing as well.

I don't work much with the GUI, but wouldn't this be easiest accomplished by symlinking the font files you want to alias?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

Also, each node in a cluster requires a unique SSL certificate, so it's difficult to do autoscaling for more nodes.
Do they need to be issued from a well-known CA? If not, have them generate their own and write the public portion into etcd or something.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
no, but they have to be unique per-node. why would etcd help us?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

no, but they have to be unique per-node. why would etcd help us?
"Generate a new certificate at startup" seemed like such an obvious solution I figured there had to be some reason it wouldn't work on its own

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
well, not from a well-known CA, but Flocker makes you have your own CA inside the cluster. so the certificates have to be signed by the master node key

https://github.com/clusterhq/flocker-openssl might explain the authentication strategy

also, it's really dumb

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

RFC2324 posted:

I don't work much with the GUI, but wouldn't this be easiest accomplished by symlinking the font files you want to alias?

You mean like just symlinking the actual FontB over FontA?

e: I'd prefer not to do that because it doesn't seem like the "right" way to do it. I think I figured out how to set the default "Sans Serif" font and so on. This one is edited in /etc/fonts/conf.d somewhere.

Boris Galerkin fucked around with this message at 10:45 on Apr 18, 2016

Xenomorph
Jun 13, 2001
Is anyone familiar with the changes made in winbind going from Samba 3.6/4.0/4.1 to 4.2/4.3?

I had what I thought was a simple config that should work in 3.x and 4.x, but it's not working as expected in 4.2/4.3.

"wbinfo -g" lists all groups.
"wbinfo -i username" lists info on that specific username.
"wbinfo -u" lists nothing in 4.2/4.3 (but lists all users in 3.6/4.0/4.1)

It's connecting to an AD server and querying it just fine. I have 'winbind enum users = yes' in my config.

Experto Crede
Aug 19, 2008

Keep on Truckin'
So to follow up on the docker thing, my friend whose VM node I help manage wants quassel (remote IRC client) and gitlab, but wants them on separate VMs for the sake of isolation. I think this is extremely wasteful, though I guess I can see why he wants them isolated.

Would this be the sort of situation where containers might come in handy? Obviously there's not much overlap between the two but I'm not sure how truly isolated things in containers are? How badly can something go down in a docker container without affecting other things?

dpkg chopra
Jun 9, 2007

Fast Food Fight

Grimey Drawer
I have a bunch of text files that are named in YYYYMMDD.Txt format (so today would be 20160420.Txt).

Each file is basically a log that contains a timestamp and and a unique ID, each value is separated by tab delimiters.

So for example, 20160420.Txt has the following values:

DATE        TIME   ID
20160420   0135   123456
20160420   0240   234567
20160420   1252   345678


I need to extract all the Unique IDs present in those files, but only on those files from the last 6 months.

The catch is that I can't use the mtime because all the files were recreated again in the past week (ie: the mtime does not correspond with the filename).

Is there any way I can do this with grep/find/sort?

dpkg chopra fucked around with this message at 17:58 on Apr 20, 2016

VikingofRock
Aug 24, 2008




Ur Getting Fatter posted:

I have a bunch of text files that are named in YYYYMMDD.Txt format (so today would be 20160420.Txt).

Each file is basically a log that contains a timestamp and and a unique ID, each value is separated by tab delimiters.

So for example, 20160420.Txt has the following values:

DATE        TIME   ID
20160420   0135   123456
20160420   0240   234567
20160420   1252   345678


I need to extract all the Unique IDs present in those files, but only on those files from the last 6 months.

The catch is that I can't use the mtime because all the files were recreated again in the past week (ie: the mtime does not correspond with the filename).

Is there any way I can do this with grep/find/sort?

Probably, but you're better off IMO just writing this as a script in a more fully featured language. I'm sure this is relatively simple in Python, for example.

RFC2324
Jun 7, 2012

http 418

VikingofRock posted:

Probably, but you're better off IMO just writing this as a script in a more fully featured language. I'm sure this is relatively simple in Python, for example.

Pretty sure this would be equally simple in bash or python. The hardest part is parsing the filenames to figure out the dates, once you do that(in bash) you would just do an 'cat filename.txt|awk '{print $3}' |grep -v "ID"'

Personally, I would just move all the required files to a separate folder(cp 2016*.txt tepmfolder && cp 201512*.txt tempfolder && cp 201511*.txt tempfolder) and then run the above on *, assuming this is a one off task.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

RFC2324 posted:

Pretty sure this would be equally simple in bash or python. The hardest part is parsing the filenames to figure out the dates, once you do that(in bash) you would just do an 'cat filename.txt|awk '{print $3}' |grep -v "ID"'

Personally, I would just move all the required files to a separate folder(cp 2016*.txt tepmfolder && cp 201512*.txt tempfolder && cp 201511*.txt tempfolder) and then run the above on *, assuming this is a one off task.
The date math, i.e. "which files are from the last six months?" is a lot harder in Bash than some other language, and you'll probably need to shell out to a Perl one-liner or something to do the validation correctly anyway.

If this is a one-off and you can figure it out in your head and pass it as an argument, that's great too though. Lexically-sorted dates are great.

RFC2324
Jun 7, 2012

http 418

Vulture Culture posted:

The date math, i.e. "which files are from the last six months?" is a lot harder in Bash than some other language, and you'll probably need to shell out to a Perl one-liner or something to do the validation correctly anyway.

If this is a one-off and you can figure it out in your head and pass it as an argument, that's great too though. Lexically-sorted dates are great.

Yeah, my whole thing assumes its a one off. I started trying to work out how to have it figure the dates out dynamically after I posted and got a headache, so if its a repeating issue, python/perl would doubtless be vastly superior.

E: http://stackoverflow.com/questions/6099795/bash-script-to-find-old-files-based-off-date-in-file-name This might help if you don't know anything other than bash scripting.

RFC2324 fucked around with this message at 20:32 on Apr 20, 2016

Powered Descent
Jul 13, 2008

We haven't had that spirit here since 1969.

Ur Getting Fatter posted:

I have a bunch of text files that are named in YYYYMMDD.Txt format (so today would be 20160420.Txt).

Each file is basically a log that contains a timestamp and and a unique ID, each value is separated by tab delimiters.

So for example, 20160420.Txt has the following values:

DATE        TIME   ID
20160420   0135   123456
20160420   0240   234567
20160420   1252   345678


I need to extract all the Unique IDs present in those files, but only on those files from the last 6 months.

The catch is that I can't use the mtime because all the files were recreated again in the past week (ie: the mtime does not correspond with the filename).

Is there any way I can do this with grep/find/sort?

This is a job for awk!

code:
for i in *.txt; do awk '{if (substr(FILENAME,0,8) > 20151020){print $3}}' $i; done
Or to leave out the first (header) line in each file:
code:
for i in *.txt; do awk '{if (NR>1){if (substr(FILENAME,0,8) > 20151020){print $3}}}' $i; done
Awk is your friend, love the awk. Awk has a wonderful plan for your life.

For those wanting to learn the ways of the Force, this page is the Jedi Master that instructed me: http://www.grymoire.com/Unix/Awk.html

e: Quick explanation of the code, for those new to awk. We run awk on every .txt file in the directory and then look at the first 8 digits of the filename. If those characters as an integer evaluate as greater than 20151020 (six months ago today, more or less, depending on your definition), then we print the third field (the unique ID) of every line in that file.

e2: Fixed the post in which I managed to write perfectly good awk code but messed up how many months are in six months...

Powered Descent fucked around with this message at 03:32 on Apr 21, 2016

dpkg chopra
Jun 9, 2007

Fast Food Fight

Grimey Drawer
Managed to figure out the mtime thanks to a ready made for loop from the guys in stackexchange that worked perfectly since they pointed out that I could just modify each file's mtime using the filename itself.

Thanks for your answers as well!

Now I need to figure out how to extract the unique values out of each one, but that should be the easy part since I can actually use -mtime now.

Edit: this was the FOR loop in case anyone was curious:

code:
#!/bin/bash
for x in *.Txt;
do
        dd=${x%.Txt}
        touch -t "$dd"0000 "$x"
done

Powered Descent
Jul 13, 2008

We haven't had that spirit here since 1969.

Ur Getting Fatter posted:

Now I need to figure out how to extract the unique values out of each one, but that should be the easy part since I can actually use -mtime now.

Pipe your desired files into:
code:
awk '{print $3}'

dpkg chopra
Jun 9, 2007

Fast Food Fight

Grimey Drawer

Powered Descent posted:

Pipe your desired files into:
code:
awk '{print $3}'

Thanks! This + sort -u worked perfectly.

RFC2324
Jun 7, 2012

http 418

Ur Getting Fatter posted:

Thanks! This + sort -u worked perfectly.

You could have also used uniq to avoid re-ordering them, just as a note.(Always more than one good way to do a thing in unix :) )

IAmKale
Jun 7, 2007

やらないか

Fun Shoe
Is there a recommended touchscreen laptop model if I'm going to roll with Ubuntu or Mint as my daily driver OS? This Asus I've been using has been alright but there are quirks here and there that I think are due to a lack of fully-baked Linux drivers for a couple of features.

poxin
Nov 16, 2003

Why yes... I am full of stars!
Anyone have experience with ZFS? With the latest Ubuntu supporting it, I'm going to move over from FreeNAS to my own configuration on Ubuntu Server. God I hate FreeBSD.

So I'm looking to create some VMs either with virtualbox, proxmox(openvz/kvm) or the like. Is it okay to setup a zpool and put the VM disk image on that pool or will it run into problems long term? Not sure the best way to go about setting that up. I believe I remember reading using a CoW filesystem has issues with virtualization images or containers on it.

evol262
Nov 30, 2010
#!/usr/bin/perl
I wouldn't worry about it.

gourdcaptain
Nov 16, 2012

poxin posted:

Anyone have experience with ZFS? With the latest Ubuntu supporting it, I'm going to move over from FreeNAS to my own configuration on Ubuntu Server. God I hate FreeBSD.

So I'm looking to create some VMs either with virtualbox, proxmox(openvz/kvm) or the like. Is it okay to setup a zpool and put the VM disk image on that pool or will it run into problems long term? Not sure the best way to go about setting that up. I believe I remember reading using a CoW filesystem has issues with virtualization images or containers on it.

I dunno how it works on ZFS, but BTRFS does let you disable CoW on a file or folder basis if you're worried about it causing problems with VM images or such. Does lose you a lot of the filesystems benefits for those files, though, in Btrfs's case.

ItBurns
Jul 24, 2007

IAmKale posted:

Is there a recommended touchscreen laptop model if I'm going to roll with Ubuntu or Mint as my daily driver OS? This Asus I've been using has been alright but there are quirks here and there that I think are due to a lack of fully-baked Linux drivers for a couple of features.

I have an XPS13 and it works with Ubuntu 16 LTS. It's not the touchscreen model however, so that's still a question mark for compatibility but everything else is solid.

ToxicFrog
Apr 26, 2008


poxin posted:

Anyone have experience with ZFS? With the latest Ubuntu supporting it, I'm going to move over from FreeNAS to my own configuration on Ubuntu Server. God I hate FreeBSD.

So I'm looking to create some VMs either with virtualbox, proxmox(openvz/kvm) or the like. Is it okay to setup a zpool and put the VM disk image on that pool or will it run into problems long term? Not sure the best way to go about setting that up. I believe I remember reading using a CoW filesystem has issues with virtualization images or containers on it.

Not on Ubuntu, but I've been running ZFS on SUSE for a few years now.

If you want VM disk images, you may want zvols, virtual block devices backed by the zpool, rather than files inside the ZFS dataset. Either should work fine, though.

Adbot
ADBOT LOVES YOU

Tigren
Oct 3, 2003

IAmKale posted:

Is there a recommended touchscreen laptop model if I'm going to roll with Ubuntu or Mint as my daily driver OS? This Asus I've been using has been alright but there are quirks here and there that I think are due to a lack of fully-baked Linux drivers for a couple of features.

What are you having issues with? I have an XPS13 with touchscreen and run Fedora 23 with little issue.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply