|
Snapshots are great. They play a really important role in any continuity strategy for all the reasons mentioned and more. But one issue I have around calling them backups is that there's very, very rarely any effective privilege separation around them. Someone with access to trash the root data store, accidentally or maliciously, also has access to trash the snapshots. Even on an old-rear end 2006-vintage LSI SAN running a metro mirror, having the password to one unit doesn't mean that you're able to trash volumes on the replica.
|
#
?
May 18, 2016 01:45
|
|
|
|
| # ? Apr 24, 2024 09:53 |
|
|
Roargasm posted:Snapshots create a constantly growing delta file. If you snapshot a database you can fill up a drive and kill the host at a random time in the future. You also can't selectively restore from a snapshot, you have to roll back the whole thing. VMware snapshots don't really deserve the name. Array level snapshots or even things like VSS or filesystem level snapshot features can create instant recovery points without performance overhead and with the ability to selectively restore from snapshot or even create space efficient read write clones to work off of the snapshot. VMware snapshots should be called a rolling journal or something else, because they are bad. When I talk about snapshot as backup I'm limiting that to well implemented COW snapshots. keseph posted:A wheel is not a car, even if it's an extremely important component of most cars. :caranology: I will however fully admit to having a biased perspective because part of my job is to get called in when someone fucks up their "snapshots are my database backups". Your analogy is almost correct here, but the issue is that the backup is not the car. The data availability strategy is the car, and backups are the wheels, or the axles, or whatever you want to use in this analogy. You will likely need multiple backup copies to satisfy your data availability requirements, and these copies may leverage different technologies including snapshots, traditional backup, disk to disk replication, disk to tape replication, backup to cloud, WORM storage for archiving....there are a ton of options and they are all a "backup" but they are not, individually, a comprehensive strategy. This is why I hate the "snapshots are not backups" dictum, because it creates a false impression that if you're taking a "good backup" (i.e. writing it elsewhere) then you've done it right and that's the end of the conversation. When really that's the very beginning of the conversation. H110Hawk posted:RAID is not backup. Snapshots are not backups. A backup is something which cannot be destroyed by the catastrophic failure of the primary data. RAID and Snapshots provide resiliency to faults and convenient point in time views of your data. They will not save you if the raid controller shits the bed and writes garbage data into half of your COW map. Ideally a backup can withstand the catastrophic failure of the primary broadest likely failure domain, which is generally the building being destroyed by fire or natural disaster. Raid and snapshots are not equivalent. Raid isn't backup because it doesn't provide any recovery from data loss. More generally "A backup is something which cannot be destroyed by the catastrophic failure of the primary data" is a useless definition. I could carve all of my data on stone slabs and bury them in an unmarked grave and it would meet that definition but probably wouldn't qualify as a backup unless you really wanted to be argumentative. A backup provides the ability to recover data that was lost. That loss can be logical (someone deleted data, someone corrupted a guest filesystem) or physical (my array caught fire, my datacenter fell into a sinkhole) and your backup strategy should involve taking backups that take your business needs into account. Perhaps you don't NEED to recovery from a local disaster? We've met with some customers whose response to "do you have an off-site backup?" was "if something happens that takes out the primary array and our backup repository then the building is probably gone and we're done as a business anyway". Their backup strategy is going to be different than someone whose data has value and who needs to plan for business continuity. Likewise, writing data off to some stable media elsewhere incurs it's own costs. If I'm a hospital IT department I'm probably not going to pitch a solution that save all of our x-rays to a tape library with a media changer. If it takes hours to recovery a x-ray that was accidentally deleted or lost due to some corruption that's probably not sufficient to make it useful. How quickly the data needs to be available is a critical component when discussing backup methodology and the blithe "it's a backup or it's not" response misses the point entirely. If it takes so long to recover that the data is useless by the time it is recovered then you do not really have a backup, you have an archive. RTO matters and snapshots are an excellent method of achieving low RTO for some data types. Likewise, RPO matters. How useful would your companies financials database from a week ago be? From a day ago? A traditional backup that needs to meet an hourly RPO on a large sized database will require a significant investment in backup infrastructure. A snapshot backup can often do it very easily. If the database needs additional recoverability that the snapshot backup won't provide then create additional backup copies leveraging different technologies. Maybe array replication. Maybe CommVault or VEEAM. Maybe application level replication with different backup methodologies at each end. Maybe the application has it's own data protection and availability features, like deployed "backupless" Exchange DAGs. But the important question isn't "do you have backups" it's "what are your data availability needs and how can you meet them with the tools at your disposal." I mean, some people say that it's only really a backup if it resides at a site that is physically disconnected from your network, or from any network, because people are a SPOF as well. As we've seen before with some cloud companies, a single malicious user or hacker who can get elevated rights can wipe out your local site repository, your remote site repository, and all of your live data. Does that mean that all of those backups weren't really backups? It's a silly and arbitrary line to draw, and it deflects from the more fruitful conversations you can be having about how to make sure your data is available when you need it to run your business.
|
|
#
?
May 18, 2016 02:14
|
|
|
It's risk management. There are values to using snapshots as a part of an overall backup strategy as long as it's not your backup strategy. Most people saying "snaps aren't backup" are generally conveying the point that if you're only doing snaps then you'll probably run into a number of unrecoverable scenarios. Some involve array failures, some involve security breaches, some involve human error. I have plenty of customers that have pretty tight needs and they still depend on external devices as the "oh poo poo!" catchall. Snaps are just the first line of defense for them. A backup is a backup until it's not there anymore so you try to have a couple layers of backups. I'd rather go back to last week's financials than start trying to re-assemble from spreadsheets scattered over dozens of laptops.
|
|
#
?
May 18, 2016 03:07
|
|
|
All this talk about snapshots reminded me that before I got let go from my last job, I was looking into why some of our customers had year-old snapshots. I bet they're still there, too!
|
|
#
?
May 18, 2016 03:16
|
|
|
To Nipplefloss' point, the RTO and RPO may require a snapshot restore to meet. We did a tabletop DR scenario a few years ago and it quickly became obvious that relying on tape would be the end of our business. In the scenario, a plane crashed into the building next door to our primary iseries during our end of day processing (we are right next door to the airport, so it's a reasonable scenario). In the scenario, there is no access to the area until the NTSB clears it and power is shut off to the block. Our iseries goes offline due to the generator running out of fuel, and we have to restore from the previous day's backup. Every transaction has to be recreated. It simply is not feasible. All of our non-iseries servers were up and running with no lost data within 8 hours in our scenario, because we use san to san replication. The core banking data however, was a day old and it would take too long to figure out what transactions needed to be reentered. Customers would simply lose faith in our business.anthonypants posted:All this talk about snapshots reminded me that before I got let go from my last job, I was looking into why some of our customers had year-old snapshots. I bet they're still there, too!
|
|
#
?
May 18, 2016 03:39
|
|
|
Speaking of backups I just spent the entire night restoring 90gb of SQL server databases for our production application. This is after personally insisting I take over backup operations from the development team last week. (They were appending full backups every 4hrs all to a single .bak file that had chewed up all disk space and was sitting pretty at 500gb+ by the time that debate was resolved) Thank god I did. About 8min of data loss - versus God knows what would have been.. What a gently caress of a night. At least the development team is convinced the new backups work 
|
|
#
?
May 18, 2016 03:45
|
|
|
anthonypants posted:All this talk about snapshots reminded me that before I got let go from my last job, I was looking into why some of our customers had year-old snapshots. I bet they're still there, too! If these are VMware snapshots the cure may be worse than the disease. If they've grown sufficiently large deleting them could incur serious down time.
|
|
#
?
May 18, 2016 03:48
|
|
|
NippleFloss posted:If these are VMware snapshots the cure may be worse than the disease. If they've grown sufficiently large deleting them could incur serious down time.
|
|
#
?
May 18, 2016 03:55
|
|
|
All night for a 90gb restore? You got guys chiseling the platters by hand? Snaps are so awesome for quick backup and back out plans. I was upgrading our SQL cluster to SP3 this past Sunday.. I could have taken an hour to take a diff of all our DBs (around 6tb worth). Instead, I just took the cluster offline, snapped the data/log volumes, and then brought it back online on the patched node. If the SP3 upgrade went south, I just take it offline again and revert the snap in 10 seconds.
|
|
#
?
May 18, 2016 03:56
|
|
|
bull3964 posted:All night for a 90gb restore? Tell me about it. Most of that was spent getting to a point the development team was happy with, and their application operational - which requires development team engagement as the databases are brought back Hence:
|
|
#
?
May 18, 2016 04:02
|
|
|
Since we're on backup chat can anyone recommend some good resources for baby's first backup and disaster recovery job? It's something I've never had to deal with before my new job and now I'm supposed to be the one taking it over. We have a system in place but it probably hasn't gotten the maintenance it needs and of course there is no real good documentation on where we stand. I know the details will vary greatly depending on the needs of the organization but I'm admittedly clueless on a lot of the basics.
|
|
#
?
May 18, 2016 06:39
|
|
|
.
Chickenwalker fucked around with this message at 03:02 on Mar 1, 2019 |
|
#
?
May 18, 2016 06:46
|
|
|
Chickenwalker posted:Unifi v3 was dead simple. v4 changed a lot of stuff but should be pretty easy as well. I installed it on a headless Linux VM. They also sell a $70 cloud key you can plug into a POE switch that runs the controller software from there. I bought one of those for home but haven't used it yet. As far as a runs, pretty sure just one or 2. It's a small place so just one ap should do the job. I'm mostly charging for the configuring and securing of the network. I'm running a unifi ap at home, and am used to the interface (running on a Linux vm with the Web gui enabled) which is what made me think of it for this small application.
|
|
#
?
May 18, 2016 09:07
|
|
|
Managers who are late for their own meetings 
|
|
#
?
May 18, 2016 10:09
|
|
|
Thanks Ants posted:Managers who are late for their own meetings This is the story of my life. And not just a little late, more like 30 minutes late until you track them down then they berate you for wasting half an hour when you could have been working.
|
|
#
?
May 18, 2016 13:19
|
|
|
GnarlyCharlie4u posted:This is the story of my life. And not just a little late, more like 30 minutes late until you track them down then they berate you for wasting half an hour when you could have been working. Point out that that same argument could be used against the meeting itself.
|
|
#
?
May 18, 2016 13:40
|
|
|
Walked posted:development team engagement What's that?
|
|
#
?
May 18, 2016 13:48
|
|
|
I need to look into an MDM solution. Initial use case will be for android-based scanners, but we would like to extend it to company issued cell phones/byod devices down the road. What solution do people use, what do you like/dislike, and what is the per-device license cost? The baseline has been set at a sister company with Wavelink Avalanche, $58/device license + $12/yr maintenance. However, they went with the easiest solution and didn't really do any real research, they just went with what the scanner vendor suggested. e: also has to be able to be hosted on-premises, global parent company has a cloud ban. 
devmd01 fucked around with this message at 13:55 on May 18, 2016 |
|
#
?
May 18, 2016 13:52
|
|
|
rafikki posted:Point out that that same argument could be used against the meeting itself. "Well I was gonna work on it, but then we had this meeting..."
|
|
#
?
May 18, 2016 13:57
|
|
|
psydude posted:What's that? That's where you yell at the development team because they made what you told them and not what you want.
|
|
#
?
May 18, 2016 14:15
|
|
|
I thought it was when the development team demands a platform upgrade and then doesn't provide anybody to test during the change window.
|
|
#
?
May 18, 2016 14:36
|
|
|
Thanks Ants posted:Managers who are late for their own meetings
|
|
#
?
May 18, 2016 14:55
|
|
|
Recently, a manager scheduled a meeting for a couple hours later that same day. My teammate and I got to the conference room and sat there for a good 15 minutes. Finally I texted the organizer to ask wtf. "poo poo I forgot and went to lunch, let's meet when I get back" This was YOUR meeting, and you scheduled it like 2 hours ago! How did you forget this?
|
|
#
?
May 18, 2016 15:55
|
|
|
Me: two APs didn't come back up after the electricians recabled them. can you have them unmount the APs so I can troubleshoot? Client's PM: no, the APs are your responsibility. if they failed that's your fault, you need to take care of it. Me: uh ok, do you have a stepladder I can use? PM: no. I borrowed a stepladder from a friendly tradesman and took the APs down. Here's what I found:  That's ceiling tile dust. He used the phrase "if they failed that's your fault" verbatim which I find, uhh, interesting.
|
|
#
?
May 19, 2016 00:07
|
|
|
uniball posted:Me: two APs didn't come back up after the electricians recabled them. can you have them unmount the APs so I can troubleshoot? Ahhh good old "dust" in the ports. I ran into a weird one at a client, some sort of dust (I assume brake dust) had gotten into a biscuit jack, replaced cabling etc etc, everything tone tested fine when we sent a tech, after a bit of dicking about the tech he finally got down and looked in the port with a flashlight and it has all this black dust inside, he blew it out and everything worked again.
|
|
#
?
May 19, 2016 00:22
|
|
|
Xirrus?
|
|
#
?
May 19, 2016 00:26
|
|
|
Thanks Ants posted:Xirrus? Aruba IAP-225s.
|
|
#
?
May 19, 2016 01:01
|
|
|
Crazy stuff, I've not seen a 'normal' AP need 56v before.
|
|
#
?
May 19, 2016 07:57
|
|
|
Microsoft wants me to install something called fiddler2 to capture some trace logs. Any of you heard of fiddler2 before? Is it a widely used thing that I just don't know about?
|
|
#
?
May 19, 2016 13:41
|
|
|
Coredump posted:Microsoft wants me to install something called fiddler2 to capture some trace logs. Any of you heard of fiddler2 before? Is it a widely used thing that I just don't know about? Yeah, it's pretty widely used. It's a standard part of our development build and our QA uses it to test API calls as well. It's a great tool for capturing, replaying, or crafting HTTP(S) traffic.
|
|
#
?
May 19, 2016 13:51
|
|
|
Coredump posted:Microsoft wants me to install something called fiddler2 to capture some trace logs. Any of you heard of fiddler2 before? Is it a widely used thing that I just don't know about? Some of our guys use it for similar stuff, I don't know if that makes it widespread but there ya go.
|
|
#
?
May 19, 2016 13:53
|
|
|
Coredump posted:Microsoft wants me to install something called fiddler2 to capture some trace logs. Any of you heard of fiddler2 before? Is it a widely used thing that I just don't know about? Yes, it is a network traffic analyser and a useful tool. Pretty widely used in network and web design troubleshooting. Meanwhile... IT has finally broken me and I am on my way out. Long story short last November I left somewhere I had worked at for over 5 years for a number of reasons that always come up when you have worked in one place for a while and a couple that were just lovely management that took me for granted. Through someone I used to work with I got talking to some people at where I work now and was sold on the job I eventually took with them and have been working there for the last 6 months. However during that time it turns out the job I was sold was all make believe because they wanted me in the door for a couple of specific skills I have that are hard to find. This means that after I cleared up their backlog of stuff for that system and finished a bunch of half done projects I was sitting around with them trying to find things for me to do as they expected their backlog to take years to clear not a couple of months. So I have been pushed into doing some project management work that I despise because I hate having to be on the hook for other peoples bullshit and now it turns out one of the things that I implemented for them to help with release quality and SOX auditing has IT and Business people complaining since they cannot make changes to production systems on the fly anymore which was one of the main points of implementing a source control/continuous integration system to begin with. The new CIO who started after me is the most Dilbert of C level people you can imagine with a side of cronyism to go with it. She has basically decreed that my work is impeding the business and instead is bring in someone from her previous job to redo it. Now what I have implemented is pretty much exactly what anyone that uses source control and continuous integration for Salesforce will be using as there are not many choices with the issues with development speed being down to the IT staff being very poor technically and the business not being able to plan for more than 2 days ahead. No matter what you implement there will be overhead but of course time to hire in someone else who will probably change from Bitbucket and Bamboo to GitHub and CumulonusCI and call it a day. So I was sold a job that doesn't exist, now doing work I hate doing, being undermined by the CIO so she can get her friends jobs and the final nail being that I have 3 different reporting managers in the 6 months and between them I can count the conversations I have had on one hand as I work remotely. I am done with just configuring other peoples work for another group of people who don't know what they want for a third group who just want to coast through life doing the bare minimum. I am done with working with technical people who are just really bad and when you try to help them improve get all defensive, I am done with managers who know nothing about how anything works but make decisions on costs, timelines and staffing without talking to those who do know first and finally I am done working in huge companies where I have to deal with the politics and bullshit. In two weeks I start at a small company of 22 people that was spun out from a large university in Boston developing an application for Salesforce to manage student study plans and communication between students and lecturers as the lead Developer/Architect. I finally get to make something instead of just being the code janitor for other peoples crap, I get to institute what passes as code complete and the processes for smooth and stable releases. This is what I wanted to do from when I was 9 years old hammering away on a C64 but due to circumstances ended up in IT roles for over a decade (you don't turn down a job straight out of University making 10k more than everyone who graduated with you). It is a huge change and yeah there is some anxiety about a lot of factors but I needed a change because my current work situation was making me unhappy.
|
|
#
?
May 19, 2016 14:24
|
|
|
bull3964 posted:If you have a 100% changeover in data, your storage snap could hit the full size of the volume, but that's it. It won't lead to the hilarious VMWare situation of a 1tb snapshot for a 30gb machine. If you had the maximum number of snapshots (32), each containing 100% changes, you'd be at ~1TB. The max size of a single VMware snapshot is the size of the underlying base disk.. If you've ever seen this in a production environment, someone should be fired.
|
|
#
?
May 19, 2016 14:28
|
|
|
Congrats on getting out and getting into something you want to do. Best of luck.
|
|
#
?
May 19, 2016 14:30
|
|
|
I just watched (and heard) a howitzer fire and sat in a 5 million dollar military helicopter.
|
|
#
?
May 19, 2016 20:22
|
|
|
It gets old when said howitzer is 100m from your wooden shack and is constantly running missions at night (almost all of which are illum because ain't nobody giving clearance to start lobbing 155 rounds at bad guys).
|
|
#
?
May 19, 2016 20:30
|
|
|
Got an interview for a Network Engineer position Monday! Given the duties, the title should really be something like Junior Network Admin but hey, I'll take it. 
|
|
#
?
May 19, 2016 22:21
|
|
|
Coredump posted:Microsoft wants me to install something called fiddler2 to capture some trace logs. Any of you heard of fiddler2 before? Is it a widely used thing that I just don't know about? Fiddler is very common, alongside burp suite and others. The thing I don't like is you cannot give it a custom CA for fully transparent MITM. Charles proxy does give you that. (Though that may have changed on fiddler, or be an "enterprise" option I forget. Been a year.)
|
|
#
?
May 19, 2016 22:56
|
|
|
Update on the been on staycation since Monday from the old job and start the new one next Wednesday at the University. This is the first real vacation that I have had for more than 2 days in 5 years and I have done dick all except paint models, play video games, and clean and it is everything I thought it could be. I get to take my fiancée to the city tomorrow for fancy drinks and dinner. life is good right now. No more on call. Did I mention this? Higher Ed, I return to your loving bosom. Your benefits and free masters degree awaits.
|
|
#
?
May 20, 2016 02:18
|
|
|
|
| # ? Apr 24, 2024 09:53 |
|
|
Holy poo poo, no on call? Blessed are thee. I took today and tomorrow off because gently caress work and I need to burn some off to not max out. I played Rimworld and Uncharted 1 most of the day. Most relaxed I've felt in a while
|
|
#
?
May 20, 2016 04:11
|
|
