The Puppet Questions MEGATHREAD - Lose your fucking mind!

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Puppet Questions MEGATHREAD - Lose your fucking mind!

«‹›2 »

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

What is Puppet?

In theory, Puppet is a configuration management tool that lets you setup any number of machines in a consistent, reliable manner. You declare the state you want your machine to be in (e.g. should have package X, service Y should be running, use this config file for it) and it all magically comes into existence at the press of a button!

In reality Puppet is staggering amount of hacks written on top of other hacks that defies reason and common sense at every turn. I strongly suspect it began life as a series of Perl scripts written by non-programming sys-admins that were eventually ported to Ruby by an unpaid intern. It's just bad decision followed by bad decision and then more bad decisions on top of that to make the previous bad decision work.

Surely you exagerate

Here are some fun facts about puppet:

A array with one value is just the value... unless of course you're calling it from within a Ruby template, in which case it's an array again. Oh unless of course the Ruby code is an actual type provider implementation, in which case it might be the array, the single value, a single string concentrating each array value and joining them with spaces, or the first value of the array even if it has more than one value.
Puppet keeps a comperhensive catalog of the current state of the system, tracking all the resources it's managing and their last applied state etc. You might think that this means when removing or moving a resource, puppe tknows exactly what is no longer needed and removes it, but you would be hilariously wrong - puppet immediately forgets about the previous state ad it is up to you to clean it up. If you renamed some service that takes up a shared reasource (say, a port binding) - it will simply make a new service with the new name while leaving the old service there, and then poo poo itself when it turns out the port is already taken.
Nothing, nothing in puppet can be defined more than once. Not variables, not resources. This becomes especially interesting when you have a resource class (which is a set of definitions that can be repeated under different names) that depends on a shared single resource (e.g. a number of nginx vhosts, all depend on a single base nginx install + configuration). There is no clean/safe way of having each instance declare that shared dependency. (ensure_resource exists yes, but it is decievingly prone to extremely hard to debug edge cases because puppet's execution model is basicaly nondeterministic outside of the requirements graph).
Speaking of which, Puppet has a huge and complicated dependency graph system much of it constructed implicitly behind your back. For example, a file inside a directory "requires" that directory, so when you tell puppet you would like both of those things to exist it knows to make the folder first and the file second. But what if you tell puppet you would like to get rid of those things instead? Well file requires folder, so obviously we have to deal with the folder first, but oh we can't, there's a file in it! There is absolutely no way of reversing the dependency graph when trying to clean up previously deployed resources (remember, as mentioned above, you ALWAYS have to do this manually - even thoguh puppet has all the information it needs to do it, it just doesn't). It is usually easiest to just ssh and clean poo poo up manually or re-provision the machine and start from scratch.
If you see puppet code, you might (not unreasonably) conclude puppet has the usual scalar values - strings, bools, numbers etc. Puppet actually just has one scalar type: string. Numbers, undef, keyword-looking things - these are all strings. If you write Ruby code against it, you just sort of have to type-sniff everything. Isn't that fun?

Why does anyone use this awful thing?

Because it was the first of its kind to come around and thus is the oldest and has most of the 3rd party modules and support. Other tools have since come along (Chef, Ansible, Docker depending on how you look at it) and although I haven't had much time to look at them in-depth (too much of our poo poo runs in puppet, switching it all out is out of the question at this point), I cannot imagine they could posisbly be worse. Simply by not designing their own hosed-up declarative dialect from lala-land that makes the PHP type system look like Hindley-Milner, they are sure to be decades ahead.

So if you are considering Puppet to get a grip on your configurations and streamline deployment environments, don't. Leave this thread and forget Puppet exists - look at any of those other options I mentioned.

If on the other hand you were a dumb poo poo like me and already sunk an regeretable number of hourse into this piece of grabage, this is the thread to post your questions and regeret the sitaution you find yourself in.

Mrs. Wynand fucked around with this message at 22:55 on Jul 29, 2014

# ? Jul 29, 2014 22:43

Adbot: ADBOT LOVES YOU

# ? Apr 26, 2024 13:05

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

I'll start!

What is the loving secret incantation to get newpoperty to just leave the property value the gently caress alone??

Here's a somewhat easier to read gist:

https://gist.github.com/rubyruy/b44389f68becd46e8975

My custom type:

Ruby code:

Puppet::Type.newtype(:aws_security_group) do
  @doc = "Manage AWS Security Groups"
  newparam(:name)
  ensurable
  newproperty(:description)
  newproperty(:vpc)
  autorequire(:aws_vpc) do
    self[:vpc]
  end
  newproperty(:tags)
  newproperty(:authorize_ingress) do
  end
  newproperty(:authorize_egress) do
  end
end

How I use it:

code:

aws_security_group {'default': 
    ensure => $ensure,
    vpc => "$name-vpc",
    authorize_ingress => [
        {
            protocol => 'any',
            ports => [],
            sources => ['default']
        },
        {
            protocol => 'tcp',
            ports => 22,
            sources => ['0.0.0.0/0'],
        },
    ],
}

Despair, this is the despair part, right here:

Ruby code:

# running apply noop
 
#Notice:
#  .. /Aws_security_group[default]/authorize_ingress: 
# current_value 
{"protocol"=>"any", "ports"=>[], "sources"=>["0.0.0.0/0"]},
#should be 
{"protocol"=>"any", "ports"=>[], "sources"=>["default"]} 
# HI, where is my second array element pretty please? 
 
# Ok, what if we do this?
newproperty(:authorize_ingress,  :array_matching => :all) do
end
 
# Notice: 
# .../Aws_security_group[default]/authorize_ingress: 
# current_value 
{"protocol"=>"any", "ports"=>[], "sources"=>["0.0.0.0/0"]},
#, should be
{"protocol"=>"any", "ports"=>[], "sources"=>["default"]} {"protocol"=>"tcp",
 "ports"=>"22", "sources"=>["0.0.0.0/0"]} 
 
# WTF is this? Why isn't it an Array? Where are the []s and the separating comma?
# Is it actually a string? (btw, yes, yes it is)
# Why?
# WHY PUPPET, WHY?

# ? Jul 29, 2014 22:43

feld: Feb 11, 2008; Out of nowhere its.....

Feldman

Mr. Wynand posted:

So if you are considering Puppet to get a grip on your configurations and streamline deployment environments, don't. Leave this thread and forget Puppet exists - look at any of those other options I mentioned.

The last conference I was at we had a BoF discussion and someone brought up configuration management. Someone said "what should you use" and everyone said in unison "ansible".

Even the guy who maintains puppet packages for an OS which I will not name

Bring your puppet questions here, but if you're adventuring into the world of configuration management don't implement puppet.

feld fucked around with this message at 22:51 on Jul 29, 2014

# ? Jul 29, 2014 22:49

pram: Jun 10, 2001

Yeah ansible owns.

# ? Jul 30, 2014 02:57

alo: May 1, 2005

Ansible does indeed own.

Irregardless of the config management/deployment toolkit wars, does anyone have any stories of taking an environment with no config management and turing it around? I have a few areas where I've been able to implement stuff, but a whole bunch of legacy pets remain. What's the best way you've made progress while still keeping ~~legacy stuff~~ people happy. So what were your first steps, where did you start, what kinds of things were harder than expected, where did you begin to feel that ${conf_mgmt_system} wasn't adequate?

My only wish is that ansible would dump the silly yaml syntax. Give me curly brackets or even python syntax.

# ? Jul 30, 2014 04:09

Dr. Jackal: Sep 13, 2009

I have worked with 3 separate companies (start ups) turning their infrastructure/deployment from mess of shell scripts (or nothing) into a <5 minute scratch-to-production using Chef.

I'd start with gathering all the shell scripts and making them easier to user, then shoving the CM tool inside the shell scripts, then you can move the entire shell scripts into the CM tool, and then finally start merging the CM recipes/playbooks into a single set of (environment) cookbooks.

The hardest part is always the orchestration, that's where you have to have the Database, Cache Servers, etc coming up before the application stack, or better yet how about sharding, replication and failover handling. They usually resolve (heh) down into problems with DNS, NTP, and your choice of lock/state storage (etcd, consul, zk, redis).

TBH I chose to go the CM route (and drop my shell scripts) because it felt more natural and it was easier to convince non-sys/tech-ops people to learn more about the infrastructure and the implications of their programming decisions when it was a mess of scripts in a language they were familiar with (no idea how/why kids are coming out of college without a decent amount of scripting experience).

Ansible is going down the wrong route at the moment (in my opinion). As a person who watched Chef go through the entire problem of "oh my nginx is better than yours" and "I need to fork your entire cookbook so I can make this adjustment", Ansible should be trying to expose people to write entire playbooks in python instead of the amalgamation of yaml that they are using. This way you can turn a playbook entirely into a library for commonly shared infrastructure components. E.g. How to install a package, choosing a supervisor, or a sysvinit replacement should be up to the user not the playbook.

# ? Jul 30, 2014 10:14

ahmeni: May 1, 2005; It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.; Grimey Drawer

Puppet is fantastic if you're like me and pretend you don't know anything better exists. We're doing an Ansible spike as a secret part of a new platform being brought up and I'm pretty excited to make the transition. I'm at least glad most of our stuff is declared in yaml via hiera though so it won't be too bad doing the eventual migration. I'm going to be so happy when I can have two server types both declare that they want a goddamn package installed without writing a custom manifest for a single package or a dumb mess of if not defined.

ahmeni fucked around with this message at 12:07 on Jul 30, 2014

# ? Jul 30, 2014 12:01

Novo: May 13, 2003; ^{Stercorem pro cerebro habes}; Soiled Meat

Surprise: Puppet is a functional language which specifies a graph under constraints that seem odd to people who want to think of configuration management as shell scripts come hell or high water. I will definitely check out Anisble since it seems to be getting popular, but unfortunately it simply doesn't do to replace your configuration management system based on trends.

Novo fucked around with this message at 21:41 on Jul 30, 2014

# ? Jul 30, 2014 21:38

Docjowles: Apr 9, 2009

At my company we're all-in on SaltStack for both config management and remote execution/orchestration. If this thread is going to be general config management chat and not just Puppet I'd be happy to talk Salt with you nerds.

# ? Jul 30, 2014 21:54

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

I figured out my problem. See, when you do this in a puppet custom type:

Ruby code:

Puppet::Type.newtype(:whatever) do
  newproperty(:yourmom) do
    describe "a lovley lady"
  end
end

What actually transpires in that newproperty block is this bane of my loving existance ruby "feature" called class_eval. It is one of those things about ruby that nobody should use unless they have an exceedingly good reason, but nooooo, everyone wants to be cool and make "DSLs" which means funny blocks that highjack your execution scope to do something which could have been done perfectly fine using boring old classes and inheritance and instantiating things the normal way.

So you're basically defining a a special subclass of Puppet::Property in that block. That method is actually Puppet::Property::describe and it will modify that subclass in-place to do whatever. Which in and of itself wouldn't be so bad (though still very stupid) except for the fact that you don't actually know if Puppet::Property is the base of your hosed-up inline class because that newproperty "helper" switches it out dynamically depending on which options you pass it which is just such a terrifyingly weird and obtuse thing to do for absolutely no reason.

Anywho, so what it comes down is that the default Property implementation actually makes a whole fuckload of assumptions about data validation and formatting and so forth. For example, it assumes values will only ever be strings, hashes or arrays of strings. Arrays of hashes? Who could have predicted such a thing could exist, we aren't theoretical physicists here!

SO, to handle the highly exotic and unsual functionality of "passing a property value to ruby umolested", we must do the following:

Ruby code:

class YouHorriblePieceOfShit < Puppet::Property
    def should
        # JUST RETURN IT, TRY NOT INJURE YOURSELF WHILE DOING SO
        return @should
    end

     def insync?(is)
        # You would thing the value mangling would be contained to the 
        # the function whose purpose is, ostensibly, to perform any 
        # sort of value transofmrations before returning them right?
        # Ahaha, of course not, it's spread all over the loving place
        # and every method in the Property interface has a at least 4 
        # different execution paths for the hard-coded assumptions about
        # not leaving your loving values be.
        @should == is
    end
    def should_to_s(newvalue)
        # Including the thing that prints values out for the CLI tools
        # for some reason, jesus christ
        newvlaue.inspect
    end
    def is_to_s(currentvalue)
        currentvalue.inspect
    end
end

Now we simply do:

Ruby code:

Puppet::Type.newtype(:whatever) do
  # Why use the language's regular inheritance mechanisms when we can 
  # make it an undocumented method on our undocumented "helper"?
  newproperty(:yourmom, :parent=> YouHorriblePieceOfShit)
end

And ta-loving-da, we did it! We passed a value into a function! :woop:

# ? Jul 30, 2014 22:34

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

Now for my next riddle:

The puppet resource man page posted:

code:

puppet resource [-h|--help] [-d|--debug] [-v|--verbose] [-e|--edit]
  [-H|--host <host>] [-p|--param <parameter>] [-t|--types] <type>
  [<name>] [<attribute>=<value> ...]

...
* --param:
  Add more parameters to be outputted from queries.

Now to me, this would suggest you can use the --param flag to display additional params (like, read-only params - not to be confused with properties - properties are states we can monitor and change, params are not). Well, whatevre the hell it is, it is deifinitely not that. Nothing I set it to seems to change anything about the output. What do you think it is? Perhaps it's like art interpretation. I think all opinions should be equally valid here.

# ? Jul 30, 2014 22:42

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

Novo posted:

Surprise: Puppet is a functional language which specifies a graph under constraints that seem odd to people who want to think of configuration management as shell scripts come hell or high water. I will definitely check out Anisble since it seems to be getting popular, but unfortunately it simply doesn't do to replace your configuration management system based on trends.

Nothing wrong with declarative execution - declarative is awsome, but it is definitely one of the more subtly difficult things to implement. There is a reason Haskell is developed primarily by people holding multiple PhDs. You know who probably can't be trusted with getting it right though? People who can't implement a basic loving array without making GBS threads the bed.

I just don't get what is going on there, like, ok, you're a novice programmer, fine, you know? I would never poo poo on someone just for that, but what is with this constant over-reaching for doing things the most novel and difficult way possible? Like, they are about to add loops to the language - great, but they won't just be adding loops, oh no - they will be implementing generic lambdas - and loops will simply be a method that takes a lambda, you know just like Ruby's each-blocks! Just, you know, maybe learn to walk before you try to run? Especially when you just fell flat on your face for the past 10 miles because you can't take a step without trying to eat your own shoes?

# ? Jul 30, 2014 22:52

evol262: Nov 30, 2010; #!/usr/bin/perl

Mr. Wynand posted:

Now for my next riddle:

Now to me, this would suggest you can use the --param flag to display additional params (like, read-only params - not to be confused with properties - properties are states we can monitor and change, params are not). Well, whatevre the hell it is, it is deifinitely not that. Nothing I set it to seems to change anything about the output. What do you think it is? Perhaps it's like art interpretation. I think all opinions should be equally valid here.

The param is a type attribute.

Yes, this really, really terrible naming. Especially because it doesn't actually refer to the individual instance of a resource. Just the metaclass it's based on. Go look at the type reference and pick one (vlan, augeas, whatever) to get a list of params you can use for that type.

Config management general?

General response from someone who has to write custom puppet types and providers: avoid if it you possibly can. If you need to do anything beyond what Puppet's DSL provides for your environment, run for the hills (Chef, Salt, or Ansible). Only muck with types/providers if you need to write providers for end-users and you're F5 or NetApp or someone.

On that note, I end up looking this any time I need to do anything new with providers because they've probably already done something similar to what I want to do.

# ? Jul 30, 2014 23:02

feld: Feb 11, 2008; Out of nowhere its.....

Feldman

alo posted:

Irregardless of the config management/deployment toolkit wars, does anyone have any stories of taking an environment with no config management and turing it around? I have a few areas where I've been able to implement stuff, but a whole bunch of legacy pets remain. What's the best way you've made progress while still keeping ~~legacy stuff~~ people happy. So what were your first steps, where did you start, what kinds of things were harder than expected, where did you begin to feel that ${conf_mgmt_system} wasn't adequate?

I have ansible at work and it doesn't do much simply because we have WAY too many systems doing different tasks. If we had more similar systems it would be easier to implement, but we manage most servers directly. However I am able to push out mass changes to system configs (ldap, dns, ntp, etc) or change passwords of local users en-masse. Servers that are touched regularly now have their configs in svn, but getting people to actually check in changes is like pulling teeth. I regularly run "svn status" and shame people. It's surprising how many people don't even grok the concept of tools like svn/git/cvs/etc.

As far as ansible itself: nobody else at work really knows it is there and there hasn't been much time for educating.

Now, as far as setting it up?

Step 1: write a script that adds an ansible user with password disabled or a random password, creates the proper authorized_keys file, and permits the user to sudo.

Step 2: use something like clusterssh or whatever tools you're comfortable with and get this onto all your servers, NOW. Do you have a template you build servers from? Put it on there so every new server is bootstrapped.

Step 3: write a playbook that manages common config files as a test -- maybe something like your resolv.conf.

Step 4: push to one server as a test. worked? good.

Step 5: push to all servers

Step 6: come up with more things to control

Step 7: rinse, repeat

Soon you'll have some playbooks/recipes/what have you and when you get the dreaded "oh man I have to do that on ALL the servers" task you're bootstrapped and ready to start using it.

Getting people to commit their changes to svn or the like and having ansible pull down from your repo and restart services would be a good improvement as well. Do you have a ticketing system? Hack the svn email python script to automatically insert a diff into the ticket if they supply the correct syntax -- I use TT#12345 as the format and it can be anywhere in the commit log.

Suddenly work is like magic, and people are blown away at how "simple" documenting changes is.

edit: Step 8 try to find time to review the changes people are committing... because... thats how you prevent disasters.

feld fucked around with this message at 23:09 on Jul 30, 2014

# ? Jul 30, 2014 23:05

Novo: May 13, 2003; ^{Stercorem pro cerebro habes}; Soiled Meat

evol262 posted:

General response from someone who has to write custom puppet types and providers: avoid if it you possibly can. If you need to do anything beyond what Puppet's DSL provides for your environment, run for the hills (Chef, Salt, or Ansible). Only muck with types/providers if you need to write providers for end-users and you're F5 or NetApp or someone.

Puppet is most definitely not a way to use Ruby for configuration management. You should be wringing every last ounce out of defining custom types before you consider writing your own implementations.

Novo fucked around with this message at 23:42 on Jul 30, 2014

# ? Jul 30, 2014 23:37

evol262: Nov 30, 2010; #!/usr/bin/perl

Novo posted:

Puppet is most definitely not a way to use Ruby for configuration management. You should be wringing every last ounce out of defining custom types before you consider writing your own implementations.

The problem is that defining those custom types is also about as fun as writing RPG.

Also, I work for Red Hat and I get to write providers for some of our products which cannot possibly be done with extant code, and so I spend time writing Ruby so end users can use Puppet and not worry about the implementation details. But implementing providers and types is terrible, terrible, terrible.

# ? Jul 30, 2014 23:42

Novo: May 13, 2003; ^{Stercorem pro cerebro habes}; Soiled Meat

Agreed, the main problem with Puppet is that the language is only tolerable to the extent that it provides clean abstractions for everything, and now you have two problems as they say.

# ? Jul 30, 2014 23:47

Comradephate: Feb 28, 2009; College Slice

pram posted:

Yeah ansible owns.

Does ansible do multi-master yet? Because SSH doesn't really scale, and everybody hates fireball mode.

E: Also we're still on puppet 2.7 and I hate it to death with fire and AIDS.

# ? Jul 31, 2014 00:59

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

evol262 posted:

The param is a type attribute.

Yes, this really, really terrible naming. Especially because it doesn't actually refer to the individual instance of a resource. Just the metaclass it's based on. Go look at the type reference and pick one (vlan, augeas, whatever) to get a list of params you can use for that type.

I don't understand what you mean by any of that.

I know a param is, in general, a thing that is like a property but not measurable - usually you use it to configure how something should run instead of how something should be - e.g. augeas's force=> true.

In the context of my question though, that is an option passed to the puppet resource command which can list resources. It looks like it is intended to allow you to list specific params instead of just properties for each resources - except it's not doing that.

After some code diving, I still think that is the intention:

(comments mine)

Ruby code:

# From the command line handler:
  text = resources.
      map { |resource| resource.prune_parameters(:parameters_to_include => @extra_params).to_manifest }.
      join("\n")
# ok makes sense, let's look at prune_parmeters

def prune_parameters(options = {})
    properties = resource_type.properties.map(&:name)
    # we clone ourselves so we don't just modify things in-place right?
    dup.collect do |attribute, value|
      if value.to_s.empty? or Array(value).empty?
        delete(attribute) # Lol nope, this deletes poo poo in place. 
        # Only reason they used  dup because because deleting during iteration will usually break ruby
        # Hopefully nobody else was using this instance!
      elsif value.to_s == "absent" and attribute.to_s != "ensure"
        delete(attribute)
      end

      parameters_to_include = options[:parameters_to_include] || []

      # SO - delete anything that isn't a property UNLESS it's on our 
      # whitelist, right? That's what I thought it was for!
      delete(attribute) unless properties.include?(attribute) || parameters_to_include.include?(attribute)

    end
    self # see? in-place
  end

Except as far as I can tell, properties never ever make it to that point.

See, the way those resources are gotten is this:

Ruby code:

Puppet::Resource.indirection.search( key, {} )

What this does is look up resource instances in the RAL. And as far as I can tell, resources come out of their provider like this:

Ruby code:

 {:name=>"somename", 
 :provider=>#<Puppet::Type::Aws_ec2_instance::ProviderApi:0x007fbdbaef9ca8 
    @property_hash={
        :name=>"somename",
        :someproperty=>"itsvalue",
        :ensure=>:present,
        :someparam=>"paramvalue" # THIS IS IS SET BY THE PROVIDER AS A PARAM ONLY 
                                          #- it's not a property, this is a param! This is what I hope to get back!
        }>}

But by the time they come out of the RAL (and remember, these are supposed to be the same instance of the same class!) they look like this:

Ruby code:

{
    :name=>"somename",
    :someproperty=>"itsvalue",
    :ensure=>:present
}

First of all, how can the same loving class have completely different internals during its own lifespan? But second of all, params are simply dropped by this point. If there is a way for the provider to set params, I cannot fathom what it might be. Which I guess might make sense - after all, the point of properties is that they are NOT the state of our system, so they wouldn't go into the RAL right? But then what the gently caress is that --param option for??

The whole loving codebase is just such an unmitigated disaster. The current maintainers seem to be just as flabbergasted by it as I am:

Ruby code:


  # @todo What to resource? Which one of the resource forms is prroduced? returned here?
  # @return [??? Resource] a resource that WHAT???
  #
  def to_resource
  end

  # The title attribute of WHAT ???
  # @todo Figure out what this is the title attribute of (it appears on line 1926 currently).
  # @return [String] the title
  attr_writer :title

  # The noop attribute of WHAT ??? does WHAT???
  # @todo Figure out what this is the noop attribute of (it appears on line 1931 currently).
  # @return [???] the noop WHAT ??? (mode? if so of what, or noop for an instance of the type, or for all
  #   instances of a type, or for what???
  #
  attr_writer :noop

  # @todo original = _"LAK 2007-05-09: Keep the model stuff around for backward compatibility"_, why is it
  #   both here (instance) and at class level? Is this a different model?
  # @return [???] model is WHAT?
  attr_reader :model

This is just a small sample - there are hundreds of these throughout the code.

Oh yeah and just as a bonus, I also found this:

Ruby code:

resource = Puppet::Resource.new("blond", "Bambi", :parameters => {
  :admits_to_age        => true,
  :admits_to_dying_hair => false
})

Yeah this totally wasn't written by some greasy neck-beard with opinions of himself far beyond his actual ability, oh no.

# ? Jul 31, 2014 01:08

evol262: Nov 30, 2010; #!/usr/bin/perl

Mr. Wynand posted:

I don't understand what you mean by any of that.

I know a param is, in general, a thing that is like a property but not measurable - usually you use it to configure how something should run instead of how something should be - e.g. augeas's force=> true.

In the context of my question though, that is an option passed to the puppet resource command which can list resources. It looks like it is intended to allow you to list specific params instead of just properties for each resources - except it's not doing that.

After some code diving, I still think that is the intention:

(comments mine)
Ruby code:
# From the command line handler:
  text = resources.
      map { |resource| resource.prune_parameters(:parameters_to_include => @extra_params).to_manifest }.
      join("\n")
# ok makes sense, let's look at prune_parmeters

def prune_parameters(options = {})
    properties = resource_type.properties.map(&:name)
    # we clone ourselves so we don't just modify things in-place right?
    dup.collect do |attribute, value|
      if value.to_s.empty? or Array(value).empty?
        delete(attribute) # Lol nope, this deletes poo poo in place. 
        # Only reason they used  dup because because deleting during iteration will usually break ruby
        # Hopefully nobody else was using this instance!
      elsif value.to_s == "absent" and attribute.to_s != "ensure"
        delete(attribute)
      end

      parameters_to_include = options[:parameters_to_include] || []

      # SO - delete anything that isn't a property UNLESS it's on our 
      # whitelist, right? That's what I thought it was for!
      delete(attribute) unless properties.include?(attribute) || parameters_to_include.include?(attribute)

    end
    self # see? in-place
  end
Except as far as I can tell, properties never ever make it to that point.

See, the way those resources are gotten is this:
Ruby code:
Puppet::Resource.indirection.search( key, {} )
What this does is look up resource instances in the RAL. And as far as I can tell, resources come out of their provider like this:
Ruby code:
 {:name=>"somename", 
 :provider=>#<Puppet::Type::Aws_ec2_instance::ProviderApi:0x007fbdbaef9ca8 
    @property_hash={
        :name=>"somename",
        :someproperty=>"itsvalue",
        :ensure=>:present,
        :someparam=>"paramvalue" # THIS IS IS SET BY THE PROVIDER AS A PARAM ONLY 
                                          #- it's not a property, this is a param! This is what I hope to get back!
        }>}
But by the time they come out of the RAL (and remember, these are supposed to be the same instance of the same class!) they look like this:
Ruby code:
{
    :name=>"somename",
    :someproperty=>"itsvalue",
    :ensure=>:present
}
First of all, how can the same loving class have completely different internals during its own lifespan? But second of all, params are simply dropped by this point. If there is a way for the provider to set params, I cannot fathom what it might be. Which I guess might make sense - after all, the point of properties is that they are NOT the state of our system, so they wouldn't go into the RAL right? But then what the gently caress is that --param option for??

The whole loving codebase is just such an unmitigated disaster. The current maintainers seem to be just as flabbergasted by it as I am:
Ruby code:
  # @todo What to resource? Which one of the resource forms is prroduced? returned here?
  # @return [??? Resource] a resource that WHAT???
  #
  def to_resource
  end

  # The title attribute of WHAT ???
  # @todo Figure out what this is the title attribute of (it appears on line 1926 currently).
  # @return [String] the title
  attr_writer :title

  # The noop attribute of WHAT ??? does WHAT???
  # @todo Figure out what this is the noop attribute of (it appears on line 1931 currently).
  # @return [???] the noop WHAT ??? (mode? if so of what, or noop for an instance of the type, or for all
  #   instances of a type, or for what???
  #
  attr_writer :noop

  # @todo original = _"LAK 2007-05-09: Keep the model stuff around for backward compatibility"_, why is it
  #   both here (instance) and at class level? Is this a different model?
  # @return [???] model is WHAT?
  attr_reader :model
This is just a small sample - there are hundreds of these throughout the code.

Oh yeah and just as a bonus, I also found this:
Ruby code:
resource = Puppet::Resource.new("blond", "Bambi", :parameters => {
  :admits_to_age        => true,
  :admits_to_dying_hair => false
})
Yeah this totally wasn't written by some greasy neck-beard with opinions of himself far beyond his actual ability, oh no.

I'll have to dive into it a bit when I get a chance this week. Phone posting. I think you're misunderstanding the intention, though.

The representation is not the class. You'd need to use reflection to really look at the instance.

-param someparam doesn't let you set arbitrary parameters. They must be present in the actual resource (type attribute) without an instance. e.g. in the base, uninitialized type.

I've never tried this particular option, though, and I'd have to look at it.

Also, please never look at any decade-old codebase with a team that's turned over multiple times with no major rewrites and castigate the developers because they're afraid to refactor parts which might be important but nobody knows.

Yes, it's bad. They also know that. That's why there are todos. But it happens.

# ? Jul 31, 2014 06:36

feld: Feb 11, 2008; Out of nowhere its.....

Feldman

Mr. Wynand posted:

Ruby code:

resource = Puppet::Resource.new("blond", "Bambi", "m'lady", :parameters => {
  :admits_to_age        => true,
  :admits_to_dying_hair => false,
})

Comradephate posted:

Does ansible do multi-master yet? Because SSH doesn't really scale, and everybody hates fireball mode.

E: Also we're still on puppet 2.7 and I hate it to death with fire and AIDS.

I'm not sure what a multi-master environment would be like. Are you saying that puppet or something else has the ability for multiple servers to manage/control these changes you're pushing out and it can spread the load among them to make the changes apply quicker to your entire infrastructure?

That sounds cool and all but we all know what happens when you start layering on complexity...

feld fucked around with this message at 15:28 on Jul 31, 2014

# ? Jul 31, 2014 15:25

Docjowles: Apr 9, 2009

Yes. Most config management tools have the ability to run multiple masters for both load balancing and redundancy. If you have a lot of servers exclusively managed by Puppet you don't really want to lose the ability to interact with them all when your only master takes a poo poo and you have to rebuild it.

# ? Jul 31, 2014 16:23

Comradephate: Feb 28, 2009; College Slice

feld posted:

I'm not sure what a multi-master environment would be like. Are you saying that puppet or something else has the ability for multiple servers to manage/control these changes you're pushing out and it can spread the load among them to make the changes apply quicker to your entire infrastructure?

That sounds cool and all but we all know what happens when you start layering on complexity...

Yes it's complex, but Ansible can't successfully talk to as many servers as we have, so it's a non-starter.

Also what DocJowles said - if a puppetmaster or puppet API server falls over, puppet slows down a little, but it doesn't stop.

If I had the time I could make it so that if a puppetmaster falls over, puppet will just shoot it and build a new one.

Ansible can't save itself from failure, because if your ansible server falls over, your config management doesn't exist anymore.

# ? Jul 31, 2014 21:25

deimos: Nov 30, 2006; Forget it man this bat is whack, it's got poobrain!

Saltstack is another one to look at from the same generation as Ansible. It's a little bit more involved to get it started into your infrastructure (but not much) but it has multi master and while halite (the web interface) is no Ansible Tower, it's at the very least functional for quick views into the infrastructure.

# ? Jul 31, 2014 22:30

Mrs. Wynand: Nov 23, 2002; DLT 4EVA

evol262 posted:

-param someparam doesn't let you set arbitrary parameters. They must be present in the actual resource (type attribute) without an instance. e.g. in the base, uninitialized type.

I hit refresh by mistake and lost my post, but I think you're thinking of the resource property setting args, which are something separate from this. I just want it to return an additional param that isn't strictly speaking a property.

Well actually I ended up just using a custom read-only property and I'm giving up on discovering what the gently caress --param is used for here.

quote:

Also, please never look at any decade-old codebase with a team that's turned over multiple times with no major rewrites and castigate the developers because they're afraid to refactor parts which might be important but nobody knows.

Yes, it's bad. They also know that. That's why there are todos. But it happens.

Oh I wasn't castigating the current developers at all - I was commiserating with them. I mean obviously now a whole team of people has to somehow make a tasty dish out of this poo poo-sandwich, and my prayers are with them.

The original developers though who let this out into the world to begin with? Less empathy there.

# ? Jul 31, 2014 22:30

tango alpha delta: Sep 9, 2011; Ask me about my wealthy lifestyle and passive income! I love bragging about my wealth to my lessers! My opinions are more valid because I have more money than you! Stealing the fruits of the labor of the working class is okay, so long as you don't do it using crypto. More money = better than!

Thank you for this thread. Our developers spin up their PIE boxes with Puppet.

tango alpha delta fucked around with this message at 09:27 on Aug 1, 2014

# ? Aug 1, 2014 09:20

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

We're using Foreman+Puppet to deploy and manage our machines. I don't know if it's puppet being terrible or our implementation being terrible, but every time I've had to write a Puppet module for something of ours, I end up having to modify some module 10 levels away that just assumes that resource X is always going to be the same no matter what.

Also a CUPS server is un-Puppetable, gently caress that so hard.

# ? Aug 6, 2014 06:20

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

From the top. I know you're ranting, but you might give some people the very wrong idea about how this poo poo works.

Mr. Wynand posted:

A array with one value is just the value... unless of course you're calling it from within a Ruby template, in which case it's an array again. Oh unless of course the Ruby code is an actual type provider implementation, in which case it might be the array, the single value, a single string concentrating each array value and joining them with spaces, or the first value of the array even if it has more than one value.

Completely and totally untrue. An array with one value is an array with one value. When you declare a resource, the name attribute (the thing right after the first {) can be a string or an array of strings. If it's an array of strings, a resource is created with the specified attributes for each value in the array.

Mr. Wynand posted:

Nothing, nothing in puppet can be defined more than once. Not variables, not resources. This becomes especially interesting when you have a resource class (which is a set of definitions that can be repeated under different names) that depends on a shared single resource (e.g. a number of nginx vhosts, all depend on a single base nginx install + configuration). There is no clean/safe way of having each instance declare that shared dependency. (ensure_resource exists yes, but it is decievingly prone to extremely hard to debug edge cases because puppet's execution model is basicaly nondeterministic outside of the requirements graph).

Variables can be defined more than once if the variable is defined within a different scope. This is good when you're sharing a global namespace with a bunch of third-party module authors whose code you don't control.

There are at least two good ways of handling this shared dependency:

Move that dependency into a class which is included by everything that needs the dependency. Since classes are singletons, this behaves exactly how you would expect.
Use the defined() function to check whether a resource is already defined or not. (This should be used with extreme caution, and has essentially a single use case: declaring the shared dependency inside a single defined type.)

Mr. Wynand posted:

If you see puppet code, you might (not unreasonably) conclude puppet has the usual scalar values - strings, bools, numbers etc. Puppet actually just has one scalar type: string. Numbers, undef, keyword-looking things - these are all strings. If you write Ruby code against it, you just sort of have to type-sniff everything. Isn't that fun?

I have no idea what this even means. Puppet's type system has some weird casting behavior, but the notion that everything is a string is completely false unless you're quoting all of your variables, which explicitly casts them into strings like any other language.

What is loving annoying is that older versions of Puppet (0.25?) did have different typing behavior when dropping Puppet variables into templates, so if you're supporting 4-year-old versions of Puppet then you do need to do all kinds of crazy things like you describe.

Mr. Wynand posted:

Other tools have since come along (Chef, Ansible, Docker depending on how you look at it) and although I haven't had much time to look at them in-depth (too much of our poo poo runs in puppet, switching it all out is out of the question at this point), I cannot imagine they could posisbly be worse.

Ahahahahaha, if you only knew the pain that the users of these tools feel.

Vulture Culture fucked around with this message at 15:35 on Aug 6, 2014

# ? Aug 6, 2014 15:30

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

I find myself writing a lot of basic modules that consist of init.pp, conf.pp (to copy over a single config file), pkg.pp (to install the relevant package), and service.pp (to run the relevant service). Is there some kind of workflow that starts with those basic files so I don't spend all my time manually copying them over? My original thought was a git repo that I could "copy" without forking (which I guess I can do with a bare clone and a mirror) but maybe there's a better way? I downloaded gepetto (the puppet IDE, basically Eclipse with puppet support) but that might be over complicated.

# ? Aug 7, 2014 03:05

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

FISHMANPET posted:

I find myself writing a lot of basic modules that consist of init.pp, conf.pp (to copy over a single config file), pkg.pp (to install the relevant package), and service.pp (to run the relevant service). Is there some kind of workflow that starts with those basic files so I don't spend all my time manually copying them over? My original thought was a git repo that I could "copy" without forking (which I guess I can do with a bare clone and a mirror) but maybe there's a better way? I downloaded gepetto (the puppet IDE, basically Eclipse with puppet support) but that might be over complicated.

https://github.com/garethr/puppet-module-skeleton

# ? Aug 7, 2014 05:24

cowboy beepboop: Feb 24, 2001

Dr. Jackal posted:

Ansible is going down the wrong route at the moment (in my opinion). As a person who watched Chef go through the entire problem of "oh my nginx is better than yours" and "I need to fork your entire cookbook so I can make this adjustment", Ansible should be trying to expose people to write entire playbooks in python instead of the amalgamation of yaml that they are using. This way you can turn a playbook entirely into a library for commonly shared infrastructure components. E.g. How to install a package, choosing a supervisor, or a sysvinit replacement should be up to the user not the playbook.

I've seen this a bit with the playbooks on Galaxy, which is annoying because I had high hopes

# ? Aug 11, 2014 13:28

swampcow: Jul 4, 2011

Mr. Wynand posted:

[*] Nothing, nothing in puppet can be defined more than once. Not variables, not resources. This becomes especially interesting when you have a resource class (which is a set of definitions that can be repeated under different names) that depends on a shared single resource (e.g. a number of nginx vhosts, all depend on a single base nginx install + configuration). There is no clean/safe way of having each instance declare that shared dependency. (ensure_resource exists yes, but it is decievingly prone to extremely hard to debug edge cases because puppet's execution model is basicaly nondeterministic outside of the requirements graph).

You can only define a resource once, but you can declare it in multiple places with virtual resources: https://docs.puppetlabs.com/guides/virtual_resources.html . That would solve your example problem.

Also, if you use 'include-style' declaration of classes, you can declare them in multiple places. If you use Hiera, you can pass parameters with the 'include-style', which avoids the only-declare-once weakness that the 'resource-style' declaration has.

# ? Sep 5, 2014 05:01

stubblyhead: Sep 13, 2007; That is treason, Johnny!; Fun Shoe

Are there any issues surrounding letter case in agent hostnames? I'm loving around with Puppet for the first time, and had a bunch of trouble getting an agent test to run successfully. I kept getting 403 forbidden errors when the node tries to get its own node definition, and as near a I could tell it was trickling all the way down to the deny all rule at the bottom. The hostname was in all caps, but the node gets added as all lowercase. I tried again with a different new node using all lower-case in the hostname, and it worked perfectly.

e: Looks like there kind of is. All the certs get created with the hostname in lowercase, but puppet.conf says to use the hostname in uppercase. Presumably it can't find that cert when it tries to authenticate, so the whole thing fails. :bravo:

stubblyhead fucked around with this message at 00:18 on Oct 14, 2014

# ? Oct 13, 2014 23:40

dunkan: Jul 10, 2006; rage is everywhere

swampcow posted:

You can only define a resource once, but you can declare it in multiple places with virtual resources: https://docs.puppetlabs.com/guides/virtual_resources.html . That would solve your example problem.

Also, if you use 'include-style' declaration of classes, you can declare them in multiple places. If you use Hiera, you can pass parameters with the 'include-style', which avoids the only-declare-once weakness that the 'resource-style' declaration has.

This! While it is true you can only define it once, if you define it as a virtual resource you can realize it where you need it, and override values.

Some things I have found useful in puppet:

- Use an external classifier. Puppetdb, your special DB, whatever. Trying to classify every host in code is painful fast.
- Write generic modules that do specific things when possible. Make them customizable by inputs. It's more work setting up the first module, but in the long run it's worth it.
- Abuse https://forge.puppetlabs.com/. Rarely do I find something that is exactly what I need, but often times you can get an idea or direction to go with. People do cool, stupid and crazy poo poo. Just like everywhere else on the internet.

# ? Oct 14, 2014 04:09

Senso: Nov 4, 2005; Always working

stubblyhead posted:

e: Looks like there kind of is. All the certs get created with the hostname in lowercase, but puppet.conf says to use the hostname in uppercase. Presumably it can't find that cert when it tries to authenticate, so the whole thing fails.

Weird, I have the same problem with a new Puppet 3 install and with all nodes I add (unless I add "allow *" at the bottom of auth.conf). My node hostnames are all lowercase, the certs are all lowercase too, but if I keep the default auth.conf, I'll get this for all client nodes:

code:

Error 403 on SERVER: Forbidden request: <hostname>(<IP>) access to /catalog/<exact same hostname> [find] at :119

Never had this problem at my previous company and I can't find anything about it. It's weird that a default install on Debian 7 doesn't let any of my nodes authenticate, even though the certs are generated and signed, etc.

# ? Nov 10, 2014 23:38

stubblyhead: Sep 13, 2007; That is treason, Johnny!; Fun Shoe

Senso posted:

Weird, I have the same problem with a new Puppet 3 install and with all nodes I add (unless I add "allow *" at the bottom of auth.conf). My node hostnames are all lowercase, the certs are all lowercase too, but if I keep the default auth.conf, I'll get this for all client nodes:
code:
Error 403 on SERVER: Forbidden request: <hostname>(<IP>) access to /catalog/<exact same hostname> [find] at :119
Never had this problem at my previous company and I can't find anything about it. It's weird that a default install on Debian 7 doesn't let any of my nodes authenticate, even though the certs are generated and signed, etc.

That is pretty weird. I actually opened a bug with them about it, and they rather unceremoniously closed it without comment. 119 I think refers to the last line in your auth.conf which is the default "deny all" rule. Basically it's saying it went all the way through the file without finding anything else that matched. If all your stuff is in the same case though I don't know what else to say. What does puppet.conf say on the client nodes?

# ? Nov 11, 2014 00:12

Senso: Nov 4, 2005; Always working

stubblyhead posted:

That is pretty weird. I actually opened a bug with them about it, and they rather unceremoniously closed it without comment. 119 I think refers to the last line in your auth.conf which is the default "deny all" rule. Basically it's saying it went all the way through the file without finding anything else that matched. If all your stuff is in the same case though I don't know what else to say. What does puppet.conf say on the client nodes?

Yeah, that's the default deny all rule.

My client configs are typically like this:

code:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
server = <hostname>
pluginsync = true

In the end, I'm allowing only the subnets we're using but it's still frustrating, especially since I had this problem right at the beginning, after doing "apt-get puppet" on the master and client nodes.

# ? Nov 11, 2014 15:52

luminalflux: May 27, 2005

What's a good resource to learn modern puppet if I've spent the last 3 years in chef-land?

# ? Nov 11, 2014 19:29

evol262: Nov 30, 2010; #!/usr/bin/perl

luminalflux posted:

What's a good resource to learn modern puppet if I've spent the last 3 years in chef-land?

Start here, because modules and hiera are the future.

# ? Nov 11, 2014 20:39

Adbot: ADBOT LOVES YOU

# ? Apr 26, 2024 13:05

swampcow: Jul 4, 2011

evol262 posted:

Start here, because modules and hiera are the future.

The link you provided is the old way of writing modules. The params section is obsoleted by hiera and automatic parameter lookups, for instance. Also, puppetlabs is now teaching that you shouldn't use inheritance.

# ? Nov 11, 2014 21:56

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Puppet Questions MEGATHREAD - Lose your fucking mind!

«‹›2 »