Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Bhodi posted:

You've got it backwards. Bitbucket is now the on prem stash replacement, called the same as hosted. Has a few new features, we upgraded last month.

good luck, Bitbucket has been hot garbage lately. We're seeing 1 to 2 issues a week with issues around the API right now. Last Friday when you hit the merge button in the GUI it would go into an eternal spin and sometimes merge or sometimes not. If you are doing a lot of code reviews I would suggest you get fisheye/crucible to do those on as the bitbucket pull request page only works well with smaller pull requests from my experience.

Adbot
ADBOT LOVES YOU

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Has anyone here used Spinnaker and has thoughts on it? Our system is a Jenkins master right now that supports 3 lower environments for dev, QA, and our staging environment and then 4 production regions in AWS. We manage around 2,000 servers for micro-services but are growing that number daily it seems. Our Jenkins master does about 16,000 jobs a day and uses a in-house jslave script to make sure we have enough executors in each environment. With the permissions needed to deploy this is a scary scenario. What I'm considering doing a proof of concept on is hiding Jenkins behind Spinnaker and using it just like the demo video sets up with Spinnaker being our front end and a lot of user access removed.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



I just got told to halt our Jenkins 2 upgrade process due to our merger closing early. Anyone dealt with splitting Jenkins that might have some life experiences or best practices? We have a single box in AWS and run about 250 slave executors. 4000+ jobs and growing daily. We average around 80 jobs concurrent and the system is starting to die slowly on me.

Definitely use job dsl and jenkinsfiles. We're in the middle of converting that over while gettthe no rid of our chained freestyle jobs for pipelines.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



We use a c4.2xlarge for the master. I should say it's 250 slave executors on other standalone Jenkins slave boxes. I'll look up data tomorrow and post it. I can also toss up a script that checks for jobs on master if you need any easy one to run and get s report back on a timer.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Hughlander posted:

We use a 8xl for about a similar load with the jvm tuned for large heaps. We also stop nightly for a clean backup.

What type of tuning do you have in place? I've only owned this beast for about 6 months and it has been mostly trying to figure out what the gently caress the last guy did (no notes) and stabilizing the platform. We're rebooting once a week right now to stop ~bad things~ from happening. I'm learning Java/Groovy slowly but am still a bit fuzzy on making sure it is setup right.

My current Java options look like this (Java 1.7):
code:
JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC 
-javaagent:/opt/newrelicjava/newrelic.jar -Xms8192m -Xmx10240m -XX:PermSize=8192m -Dorg.apache.commons.jelly.tags.fmt.timeZone=America/New_York"
We also have a script that runs GC every 10-15 minutes or so.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Jenkins has a new blog post up for GC tuning on large instances. I"m going to put it on our test server and throw some load at it.
GC tuning blog post

Anyone here going to re:invent? I'm looking forward to doing GameDay this year.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



I’m not at a computer but you should be able to stash it in pipelines and grab it back out between steps

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



the talent deficit posted:

i just wrapped up a big project consulting on a failed migration from self hosted to k8s-on-aws. i think they would have probably succeeded if they'd done self hosted k8s (or openshift) or just straight self hosted to aws but where they ended up was a huge mess

i'm reserving judgement on aws eks until it's actually ga but i think right now k8s on aws is a mistake unless you need to also support self hosted k8s or you are starting from something easily ported to k8s (like all your applications are already running in docker in production). almost every legacy project is going to have a hard enough time moving to aws rds/aws ec2/aws ecs/... without also throwing docker and k8s into the mix

Do you have a debrief of what went wrong? There is so much kubernetes running on AWS that they made a service specifically to help more people use it on AWS. Our TAMs were happy they pushed the kubernetes service so fast because it was blowing up their time from enterprises.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Vulture Culture posted:

And yet the only goon I've run across in Hangops or the Rands Leadership Slack rarely posts here anymore

I enjoy Rands but I lurk the gently caress out of it.

Adbot
ADBOT LOVES YOU

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



Docjowles posted:

Embrace the giant red box of security shame whenever you click Manage Jenkins

:siren: Warnings have been published for the following currently installed components :siren:

(list of all currently installed components follows)

Do you want to adopt this plugin?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply