Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Old Binsby
Jun 27, 2014



Hey anyone here know any/been an Exchange Premier Field Engineer? Over the past years I've met a couple in passing that I was really impressed by and it seems like they have an awesome job. It's not like I know these guys very well (or I'd be spewing words at them instead of the SA Exchange thread), only met them a couple days/hours at a time. They were all super knowledgeable, helpful and confident about their advice and it kind of made me want to also do their job. Recently, I worked a while with someone from another company who had been a PFE for a couple years and while he was hugely more experienced (like 10 more years of work experience), we would almost be equally knowledgable on the problems we encountered planning/performing a migration and were both relied on to give advice to the org we worked for. This made me wonder if, maybe, I could apply. But I might also be hugely overestimating myself hence this stupidly long question. I could ask that ex-PFE but because he's pretty close with my manager it might get complicated and or weird if word got out I'm applying elsewhere.

So, what I'd love to know is 1) how hardcore should my knowledge be before applying? I've done about 3 years of Exchange ('10-'16 + Online) and migrating them back and forth and planning deployments sometimes. Apart from that a bit of Lync/SfB Server and a LOT of Powershell and general MS domain stuff but I rarely do super esoteric things 2) Is support for Exchange 07 still big thing? It's mentioned as either a plus or necessary in the ad (it's vague) but so is 'a proficiency in PowerShell 2.0' so I don't know about that.

I know being comfortable around technical as well as non-technical people and adjusting your way of explaining things is also a must, but that is something I have to be/do already. I don't do a lot of teaching currently, but I do instruct coworkers in small groups or one on one and give presentations to coworkers sometimes so that might help? Anyways any input would be appreciated.

Adbot
ADBOT LOVES YOU

Old Binsby
Jun 27, 2014



incoherent posted:

Clutter? Spam for the europeans? You can do some pretty amazing things through transport rules, but sometimes using them feel like i'm putting lazy dev technical debt on a new card.

Nah clutter is worse/better. It's a folder where Exchange can/could* put stuff it considered unimportant. The junk mail folder is basically worthless (saying this too politely) without putting some serious effort into configuring additional policies and 3rd party filtering software

Clutter is the opposite of that because it machine-learns which kinds of mail you tend to read and which you ignore. After a while you would start to receive 'less' mail, because the rest would be deposited in a separate folder. How very modern, I hear the Gmail crowd yawning (from the year 2013) but hey! It worked more or less ok and was a decent first iteration. There were some technical problems with false positives, but those weren't too bad. It could prove (darkly) humorous in some instances too, like posted above or the time I was sent to see what caused the row between a pretty bad team lead and his team. He was giving them a hard time for not informing him of some random thing or other while the team obviously said they sent him a message. It turned out that they did, he just never ever read messages coming from their mailbox.

So it was fine except when it was not. There, usually at least one of the three factors that recur like clockwork in companies new to rapid releases of their end user software happened

1. Unaware IT guys who miss/ignore/underestimate new feature announcements cause they haven't been on the O365 platform that long and didn't need to pay v close attention to new features when patching their on-premise servers.
2. Confused users, (several of whom disregard those notifications anyway because they're busy or 'don't understand this technical crap'
3. The big one was that there was also not a lot of post deployment notification. The clutter folder was kind of hidden just above the junk mail but below all user-made content. It lacked a huge arrow and some bold, yellow marked words screaming: 'SUP BUDDY - ALL THE CRAP MAIL YOU ALWAYS IGNORE IS NOW OVER HERE ------>>>

People calling IT support in a panic ('mail is disappearing') in numbers I didn't really expect either made Clutter an option to turn off by default, no matter how well it worked in other cases. Apart from a few exchange MVPs singing its praise I heard similar from a lot other freelance/consulting colleagues. I have to add that I mainly work at biggish, old-fashioned, IT-follows-business shops. At some point they'll probably get used to the iterative releases MS are doing, but a lot of these folks are wishing they could go back to a place in time where this would bother them only when they upgraded their server version to the next (skipping a few and possibly postponing change FOREVER!). Alas!

*These days I thought it got replaced on all tenants but apparently that's yet to be completed. The iteration is called Focused Inbox (like the one on the Outlook for Android/iOS app) and it's a big improvement mainly because it tackles problem no. 3 above. The algorithm may or may not work better but at least you get a way to access both kinds of mail and a fairly obvious notification that tells you about messages that arrive into the inbox not currently selected. All in the center of your screen, too.

E. hosed around a bunch because I goddamn love effortposting

Old Binsby fucked around with this message at 20:59 on Mar 8, 2017

Old Binsby
Jun 27, 2014



Sounds like you need to get rid of that domain asap. I'd consider putting an autoreply on these addresses that your ISP hosts informing customers that you're going to be using your other, cloud-ready domain exclusively in X weeks/months. After that time, your problems are gone.

Otherwise a lot of mail servers support​ 'simple' address rewriting where you swap out one domain for another but IIS probably doesn't. You might not want to increase your dependency on these kind of wobbly forwarding mechanisms however, they have a tendency to fail at the worst possible time. Like in a coupele months (depending on how that smarthost you mentioned is set up) when a bunch of these solutions are gonna crash and burn:
https://blogs.technet.microsoft.com/exchange/2016/03/29/important-notice-for-office-365-email-customers-who-have-configured-connectors/

Old Binsby
Jun 27, 2014



Eschatos posted:

I was hoping to avoid that. They've been pretty much useless in the past. Ah well.


Afraid that's not the case here, checking Junk, Clutter and running an all folders search in Outlook were one of the first things I tried.


Worth a shot.

Views have always been kind of problematic for outlook, sorting by category could gently caress up unread counters in outlook 2010 even in the very last version I worked with (6 months ago)

The same goes for the search folders that are starred by default at the top of owa/outlook. Sorting, searching and unread counters can behave unpredictably esp. in cached mode so try disabling that maybe

Old Binsby
Jun 27, 2014



Migrating to something with actual thought put into high availability at an earlier point than post-deployment sounds like the better move here, probably cheaper as well. Not an option, I guess?

Old Binsby
Jun 27, 2014



In that case go for it, seems like you're well aware nothing great will come from this solution. I think managing the clients expectations of a HA solution is always important but especially in cases like this it's 100% required to get a written signoff on acceptable data loss during failover

Old Binsby
Jun 27, 2014



AlternateAccount posted:

OMFG, I was sleepy and changed a user's send/receive limits to 100. KB. For some reason I assumed the field reflected MEGABYTES in the management panel. Good thing a few people got kickbacks when email this (important) user saying I CAN'T! HE CAN ONLY RECEIVE 100K!!

Much shame. Total disgrace. Details are important.

Yes that sucks and it's a great way for Powershell to save your day since there are builtins for MB, GB, TB etc. Stick MB without a space behind any numeric and it turns into a magic postfix operator with an output of 1048576 (defined as 2^20). Same for the others. I don't know how this works exactly but it's kind of neat sometimes.

It's a fun rite of passage for any Exchange admin to accidentally apply a super dumb limit to an entire database and/or perform some arbitrary action on the first 1000 mailboxes by mistake because you hosed up some get-mailbox somewhere. Although I've done that more than a single time probably

Old Binsby
Jun 27, 2014



If you have no experience and no documentation pulling off a migration like that smoothly will be challenging, I've seen plenty of people stumble that had many years' experience doing Exchange administration. It's not terribly complex, but experience with common pitfalls is invaluable especially for co-existence while migrating. I agree with thanks ants, moving to Exchange Online would be preferable if your organisation is willing and able to do that, especially since it appears there was no one really dedicated to herding a bunch of mail servers until you got that job. Exchange 2010 supports fully featured hybrid migration scenarios so that's not really a road block.

If you can't, then you'd better prepare for reading a lot of technet pages. First of all regarding sizing, things used to be a bit more complicated than they are now because all roles are co-located (except for the separate Edge role). It was a good idea to do it previously but you have no choice in the matter these days. For a quick and dirty shortcut, you can do a performance analysis of your current org and see if there's any bottlenecks. Then take the number of mailbox servers you have, put them on latest gen hardware (or virtual hosts) and add 25% of everything (storage speed, capacity, RAM, CPUs, whatever) that seemed to be causing bottlenecks. Now you're good to go

The pro move is obviously using the calculator but that's kind of tricky and pretty dependent on a few magical parameters. When service providers give you a size estimate for a new environment, a couple iterations until the numbers look 'right' and some of the above guesstimating is usually involved. For typical user profiles, it's rare that an average user hits anything above the lowest volume and message sizes that you can choose. You probably don't need more than 1 DAG (unless you're in >2 datacenters), though you will have to create a new one for Exchange 16 - I don't think you can put two versions of server in the same DAG. Virtualization should not be an issue, as long as you are aware that Exchange databases really dislike disk latency so don't put them on a shared controller or at least give them high priority access. You can test your storage using the JetStress tools for CYA purposes or if you mistrust your virtualization/storage. Databases: 3 might be right for around 500 users though probably on the low side. I like to aim at around 500GB, keeping ~10% free space overhead depending on typical growth. Other things to consider: you can do incremental/full backups on a cycle or use a lagged copy for a backupless solution. It sounds like that may already be in place, judging by your 3-copy setup, but it might not be.

Old Binsby
Jun 27, 2014



I said databases?

Old Binsby
Jun 27, 2014



Dlp rules are a combination and also result in a bunch of predefined transport rules, iirc. You're not supposed to gently caress with them outside of the security center but you can at least tell they were implemented by looking there. If its arm likely you'll find whatever you did over there too but the rollout of new portals can be slow. You might wanna check the old portal as well if you've ever used that. An unlikely option, but still v confusing is that your security roles were revoked cause you then miss several options

Old Binsby
Jun 27, 2014






You're right, the minimal HCW is best for smallish hybrid deployments with little to no fancy cross-prem configuration. But to answer your original question: for as long as your users are DirSynced (i.e. managed on-prem using whatever tools) you will need to have an Exchange hybrid server on-prem. It can be really minimal, but there will be an Exchange installation somewhere to do the user management with. It's scenario 2 from this Technet page. Judging by your wording maybe you figured that out already - sorry for repeating in that case.

In any case, the hybrid server isn't that bad to maintain. The typical associations that come with an Exchange server (cumbersome, a bitch to patch/fail over, finicky about resources) aren't applicable. It should host no production mailbox databases (it sort of can but it invalidates the hybrid license), it does not really need HA - you can reboot it during a move without breaking stuff - and it can run on basically whatever resources you have lying about. A samsung smartphone 2 generations back has enough raw power probably. The license is free, you need to request one but the same license key that the page where you get one spits out is the same each time. You run the hybrid connection wizard, fix a few warnings or errors that invariably occur and you're ready to start testing. From Exchange 2010 the migration isn't any more of a hassle than from newer ones since staging/executing/checking on migration batches is done from the Exchange Online web interface once hybrid connectivity works. Though obviously PowerShell provides a bit more options.

Old Binsby fucked around with this message at 21:03 on May 16, 2017

Old Binsby
Jun 27, 2014



anthonypants posted:

Exchange Server is not a prerequisite for Azure AD Connect.

yeah that first sentence is meant to read 'as long as your Exchange users are dirsynced from on-prem', but you get what I mean

Old Binsby
Jun 27, 2014



NevergirlsOFFICIAL posted:

But that's not really true either. You can do a cutover migration, and THEN implement dirsync, without Exchange on prem.

While you could, as far as I'm aware you'd lose the ability to edit Exchange properties of users in Exchange online since the on-prem AD becomes the source of authority for them after AADconnecting/dirsyncing. Then you'd either need that exchange server or ADSI edit. to change even simple things like email addresses. You'd need to run the first part of the Exchange setup every 3 months anyway because potentially every CU can extend your schema. Those CUs are the only source of the schema updates you need to keep your on-prem AD compatible with Exchange Online.

I don't think there exists a supported configuration of dir/aadsynced exchange online users without an on-prem Exchange server.

Old Binsby
Jun 27, 2014



Huh, I had no idea that this was done very much at all. If this thread can attract two examples within an hour I'll have to adjust that view. Using 3rd party user management tools to edit AD might also make skipping the hybrid server more feasible?

Anyway, I guess I'm a little predisposed to not do anything strictly described as unsupported in documentation. Most places I work for have MS support contracts that make following their guidelines non-optional. Though I wouldn't skimp on this particular subject regardless, the minimal hybrid install is so minimal it isn't worth the hassle imo.


E. ^^^^ definitely. I'm not advocating doing it the other way around, just commenting on whether it eliminated the need for a hybrid server

Old Binsby fucked around with this message at 17:48 on May 18, 2017

Old Binsby
Jun 27, 2014



Thanks Ants posted:

Do you have a TechNet article re: that being unsupported? Would help me out when I come up against resistance.
The Exchange Team blog mentions it and here is the TechNet page (ctrl+f ADSIEDIT for the relevant bit)



e. fixed broken link

Old Binsby fucked around with this message at 19:35 on May 18, 2017

Old Binsby
Jun 27, 2014



Depending on the scale, running Exchange on a DC is a disaster in and of itself. However you're not complaining and for small scale operations it can be feasible so I won't harp about best practices again, haha

Provided the two exchange servers are in two domains with a mutual trust (or even a one way in the right direction), migrating does not need .pst exports. You can configure them so that moving a mailbox feels rather similar moving them between databases, in exhange 13 with the new-moverequest or new-migrationbatch cmdlets. I'm on mobile right now but it's pretty decently documented on technet. If no one else does I'll expand later

E. Eh this meeting I'm in isn't going anywhere so I got you that link. This is the high level description of what you want to do, it should get you in the right direction. https://technet.microsoft.com/en-us/library/jj150543(v=exchg.150).aspx

A trust isn't even strictly necessary, it's been a while since I last did an on-prem to on-prem migration so I stand corrected there. Actually I just finished one but that was ~9TB of .psts - there were very specific reasons for that and I hope you never ever run into those. Unless you're moving <30 mailboxes using the built-in migration tools has many advantages, including queueing and better error handling. Also, it's good practice for an online move if the banking industry ever learns to deal with fear of the cloud

Old Binsby fucked around with this message at 10:33 on May 23, 2017

Old Binsby
Jun 27, 2014



First one sounds like a corruption of the address cache in outlook but if this issue persists across multiple clients I'd suspect Exchange. Easiest to move that mailbox between databases before trying anything else. Or run repair-mailbox against it but that's more effort if it's a small mailbox. Moving fixes some forms of corruption though mailbox store corruptions are becoming exceedingly rare in exchange these days. This guy has a similar issue: https://social.technet.microsoft.com/Forums/en-US/cad29f8f-e2ee-4976-9e54-981c51f4827e/outlook-mail-send-issue?forum=officeitproprevious
(Sending an email is very very similar to creating an appointment for exchange)

Second issue: maybe this? Doesn't sound familiar.
https://support.microsoft.com/en-us...leted-attendees

Old Binsby
Jun 27, 2014



Yeah you definitely need some form of log management because of IIS alone and Exchange can be a bit verbose as well. Your especially need to be mindful of disk space if you've got the message queue logs and database in the default directory on a mailbox server ( ExchangeInstalldir/transportroles/data/queue I think) since low disk space will cause the server to to delay or drop incoming messages due to backpressure at a disk space utilization of 90% and it won't restart full operation until you've brought it down to <80% again. Yeah 90% is really high but it happens and I think it is below the default critical value in SCOM and PRTG

Some people symlink the Logging directory to a different drive, which is kind of ugly and doesn't catch all logs anyway. All exchange services have a .config xml file that you can use to change the default log locations, but that's also error prone since I believe those files can get overwritten during CU installations. Better to just something like the script above. There's also a registry entry to change the number ETLs that are kept for trace logging, those files in particular are quite large so I always turn it down some at the very least if disk drive space is plentiful enough to leave the rest be. In Exchange 2016: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\16.0\Search\Diagnostics\Tracing

Old Binsby
Jun 27, 2014



rotaryfun posted:

Would this run on 2012r2 server with exchange 2010?

Also, I have a dag observation server that is being decommissioned. Anyone know of any documentation to migrate that task to another server?

It would probably run but the paths are for exchange 2013. You can easily check with treesize or something like that where your biggest folders are (that only contain .Log files and that aren't your database transaction logs), put those in there. For starters though, moving inetpub cause it tends to grow huge and the message queue database for safety off of C:\ entirely (both to separate disks) is an easy way to go. I don't remember whether exchange 10 does the same amount of trace logging by default, you can also curtail that with the regkey above.

e.

About moving a file share witness (FSW) if that's what you mean: unless you assign another Exchange server that is not a DAG member for the DAG you're witnessing, you need to
1) grant the Exchange Trusted Subsystem security group local admin rights on the new server
2) use Set-DatabaseAvailabilityGroup -WitnessServer [ServerFQDN] -Witnessdirectory [C:\whatever\FSW] -Identity [DAG]
to tell the DAG where the new witness server is. That's it. It creates the folder, share, permissions etc automatically that way. If you are moving the witness to another Exchange server (for example a CAS server without databases) you can skip step 1. I believe this is preferred, but these days a lot of people run multirole installations which prevent doing things this way.

Windows firewall needs to be open to file- and printer sharing on the new witness server for this to work. The new server can't be a DAG member that hosts mailboxes, but you can do crosswise FSW (ie DAG1 hosts the witness for DAG2 and vice versa). A single server can be witness to multiple DAGs as well, just keep separate folders for each DAG. Some of the other requirements are being in the same AD forest and not using a DFS-fileshare for the FSW folder location. The documentation for this is reasonably complete, so here is where I'd start looking.

Old Binsby fucked around with this message at 11:19 on Jun 9, 2017

Old Binsby
Jun 27, 2014



Out of your plan I'd skip step 6 because unless your thoughts and prayers go out to those fallen while taming it, Ex10 is bad and should be forgotten. I'd add 'double-doublecheck the coexistence with legacy exchange' if you're not already coexistent successfully. There's page on technet or this blog. It's so easy to forget a minor detail and gently caress up all your client connections, I've seen it happen twice the past months and with a lot of people breathing down your neck looking up/setting all those urls etc correctly loving suuuuuucks .

Also: just run the HCW once a week for the heck of it. It's easy, fast and sometimes it does a world of good

Old Binsby
Jun 27, 2014




It is! Now if only a custom cancellation message were added I'd throw out my own kludgy EWS-based script for doing this in bulk

Old Binsby
Jun 27, 2014



If you can export the cert along with the private key or if you have/make a copy of the cert+key someplace else (like your pki) go for it. If anything fails, try to see if you can fix it easily and if you cant you could reinstall the cert and troubleshoot. Only places this causes much trouble is in 100% uptime no breakage allowed environments. In that case check out at least

-The event viewer might show certs are still in use for certain services and shows warnings for invalid certs being used
- The verbose protocol logs for receive, send connectors definitely do, they'll be in the context/data fields of TLS connections and pop/imap (if logging is enabled).
-IIS cert bindings (don't edit things there though, exchange overwrites most iis settings periodically)

When the assigned services field is empty though, probably not a lot going on with them.

Old Binsby
Jun 27, 2014



AlternateAccount posted:

OK, I can accept all of that.

Webmail it is. OWA is pretty decent, but there's still going to be a lot of wailing and teeth-gnashing. Is what it is, I guess.

I'm pretty glad OWA works on phones now, and it's something you'll have to deal with on an iPhone apparently

But (This might be obvious to any web dev or general programmer)

what I don't understand is how it got to be that the app is so much faster and better than OWA. The mobile outlook app renders a few predefined standard UI elements, sticks text in them based on a few HTTP-requests to an Exchange web service and back, filling them lazily. This seems pretty similar to how a browser does things, there might be parts shared between each as well for all I know (this is what Microsoft aims at for all Outlooks if I'm not mistaken). Yet if they're both API-consuming web-implementations of an email client instead of deeply integrated MAPI tools - why the diference in speed and functionality??


devmd01 posted:

And that's almost a wrap on killing the on premise 2010 environment, nothing but an internal smtp relay exists for on prem/hybrid services, everything else routes through proofpoint to O365 inbound and outbound.

Congrats! On-premise or cloud can both be good but Ex10 never is.

Old Binsby
Jun 27, 2014



True but exchange online is pretty bad at filtering so there is a point in just disabling spam/malware protection there and using something else

Old Binsby
Jun 27, 2014



A Pinball Wizard posted:

I'm hoping there's someone here who's a MAPI wizard and can possibly help me make sure I'm not X/Ying myself into a corner. I am not an exchange guy, so if I'm using the wrong terms for something I apologize.

A user logs into our terminal server farm to use our LOB app. The TS has Outlook 2013 and the mail client built into our app, which I have very little info on beyond "it uses MAPI". E-mail works fine in Outlook, and she can send mail and access the address book from our app, but the app won't load her Inbox. I'm pretty sure the app is encountering an error and just silently failing instead of reporting it.

While MAPI is extremely relevant in Exchange still, it's hard to get to a point where it is the cause of something loving up. I'd guess this LOB app, if it's written in the past couple years, will probably be using EWS to make a connection over https to Exchange and do its thing. You can try to figure this out through documentation or by wiresharking and checking for connections to [url]https://[/url][exchange server name or load balancer FQDN]/EWS/exchange.asmx. Maybe, I never tried, you can proxy them through fiddler to see the contents too.

If it does use EWS, you can see IIS logs that it generates on the server. Though you don't have access to that, those logs would be useful to see if any requests are coming through at all and what their contents are at a high level. You can use the EWS Editor locally to simulate that connection and see if you can connect through the tool. If you can, chances are that yes: the app is silently loving up. If not, I'd be happy to post further thoughts

Old Binsby
Jun 27, 2014



Oscar Wilde Bunch posted:

When you create a new profile it does seem to give a little bit before it stops working right, but that time period can be like 30 minutes.

The folder shows Up to Date, Connected to: Microsoft Exchange. I can't think of any GPO's that would intersect.

WAPs don't come into play, network path is pretty uncomplicated. Object location is that same as everyone else, main complainer is neither a delegate of or to someone else. We don't have public folders anymore, no pst files.

We have people affected that were part of the original O365 migration, and some that are only a few months old.

You mention Outlook 2016, how do other versions behave? You can go down as low as 2010 connecting to O365, though you lose a few features here and there. Might be interesting to see if there are quirks that only manifest in the newer versions so you have somewhere to look. You are on the latest patch level of Office, or should be when troubleshooting hard to find stuff like this. But try the latest build, maybe even insiders if you have it, and see if it goes away because that has been the solution to quite a few of hard cases I encountered.

That a new profile gives temporary relief is an indication that something might be messing with profiles after they're made. Which antivirus do you run, and can you arrange a testing situation without antivirus for an affected user? By that I mean find a fresh machine where the regular AV isn't even installed because disabling doesn't always actually do the job entirely.

Is there anything in either of these locations in the registry?
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\16.0\Outlook\Cached Mode
HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Cached Mode

Last one, sorry for the barrage of questions, you can move mailboxes between databases in Exchange Online, though it's not widely know/done. Did you try that? If you're on prem you definitely should but since you mention O365 support I assumed Online. You can simply enter into an Exchange Online Powershell session:
code:
New-MoveRequest -Identity [user principal name]

Old Binsby
Jun 27, 2014



Oscar Wilde Bunch posted:

The only issue with downgrading now is that they won't support anything below 2016 anymore, as I think 2013 rotated out in ?June?. We're Exchange Online. We've tried moving the mailboxes between databases. We've tried machines with no AV period and it doesn't seem to do anything.

There's nothing in those settings, though I'm going to try out with the policy to change the default cached mode timings, as it seems that my problem essentially some kind of failure in that area. It's like the client just forgets to ask if there's anything new out there, unless it prompted to do so, or some arbitrary time has passed.

Outlook 2010 is interesting because I thought that was the quickest way to check out the difference between connections via the old protocol (MAPI over) RPC over HTTP and the newer MAPI over HTTP. It turns out later versions can also still do that. Troubleshooting while still supported, hurray! Done by adding the following REG_DWORD with value 1:

HKEY_CURRENT_USER\Software\Microsoft\Exchange\MapiHttpDisabled

It's a real long shot but you never know? The connection status screen (I'm guessing you know this but for completeness' sake) is opened by ctrl+right clicking the outlook icon in the system tray near the clock, the protocol should read RPC/HTTP instead of the default 'MAPI/HTTP' after the change above. If you always saw RPC/HTTP that's a red flag too, but you or Microsoft would have noticed that. Are request, fail, latency numbers all equal between good/bad clients? Also, you mentioned looking quite deep into the MAPI protocol and finding nothing. But did you see periodic communication between client and server at all? O365 support should also be able to analyze the troubleshooting logs that you can enable. Did they do so and find nothing? You can always try yourself by pulling the ETLs through tracerpt but it's hard to predict if that'll work properly.

These kind of issues are fun in a way when they stump the supplier support teams as well. But your support engineer ghosting you, never replying at all is pretty bad. If you have a technical account manager, try raising the issue with them. They might involve different engineers or move this toward another support team (i.e. Outlook instead of O365), although you're probably already at the right guys. Otherwise: Did you capture any network traces? Significant differences in the number of timeouts or retransmissions could point to a network issue after all.

e. typo

Old Binsby fucked around with this message at 20:59 on Jul 11, 2017

Old Binsby
Jun 27, 2014



Internet Explorer posted:

Why in the world would someone be using Bounce Address Tag Validation (BATV) in YTOOL2017?

They might be on Old Software? It is the year 2017 but I run into it mostly on that. This very year I decomissioned ForeFront TMG and Protection For Exchange servers someplace. Mainstream support ended ~two years and as far as I can tell neither were receiving spam nor malware signature updates anymore. This was apparently not that big of a problem. Luckily, Forefront did do backscatter protection so they were safe from that!


Moved them to EOP for email which introduced BATV as a 'new feature' in... 2015. So it's not dead but there are better ways to check where mail came from and whether you should accept the NDR these days.

(effortpoast coming up - tldr: what email security is Good????)
I always feel a little bad moving small-medium sized clients to Exchange Online Protection. I get why they do it, it's super easy to sell for MS (cause it's free with most enterprise licenses ). And honestly, not much will change is the customer is coming from old MS email filtering services but otoh it's mediocre to bad, in my opinion. There's too little granularity so I'm nearly always either blocking OR letting through way too much without a good middle ground. There's no integration with SIEM-like security software - which is usually also kind of convoluted stuff but at least it gives the actual security guys easy access to mail-related events. But who cares, as long as it stops the spam and malware. It doesn't really do that too well either, though... While Exchange is kind of my thing, email security isn't so any practical knowledge is welcome. What's the Good Email Security Solution these days?

I know only one alternative well enough to say anything about that, which is Forcepoint (formerly WebSense). I kind of like this but would not wholeheartedly recommend. Mostly because it is actually good at catching malware and spam and has pretty detailed rules for all sorts of scenarios built in. At least, comparing side by side during a PoC deployment I've seen it is noticably better at filtering spam and recent viruses than an EOP test instance (which is a low bar). Not super expensive, has a sandbox service to check executable content. Downsides are that, at least on Windows Server, it has no real command line interface for deployment, management nor automation as far as I could tell, which is a no go to me (you could clone xml configurations but... meh). It's also a terribly convoluted GUI that feels like several dozen mergers ago, there were 12 seperate programs that worked well but now someone superglued them together. Also you end up with a locally hosted management web service, on a box with literally 3 kinds of web servers, 3 different database engines, all sorts of weird prerequisites. It's kind of ugly.

Also used symantec message labs for 2 days once, which was awesomely slow and featureless. It might be better now? Is proofpoint as good as all the marketing says?

Old Binsby
Jun 27, 2014



anthonypants posted:

Mandatory TLS 1.2 is pretty good, imo.

Yes. you're right. That's not what I meant, but that's a good idea (until someone VITAL TO OUR BUSINESS, LITERALLY INDISPENSABLE can't do TLS 1.2 on his vintage Notes server).

Make me happy, share your favorite email malware/threat/spam detection software. Or least favorite, that's cool too.

Old Binsby
Jun 27, 2014



devmd01 posted:

While the security team does the daily administration of Proofpoint, I'm pretty satisfied with it from the perspective of both an end-user and email admin.

We get a digest every couple of hours listing everything in quarantine, and you can release it then and there from a link in the email if need be. Takes the "can you check the spam filter" helpdesk request and throws it right the gently caress out the window. The only thing I've seen slip past are some very well crafted it marketing emails, but then again my email has only existed for 3 months so it's not on the real nasty spam lists yet.

Implementation was relatively easy, though ymmv depending on how complicated your mail environment is. I put it in place in conjunction with ripping out our legacy on-premise environment, so our implementation time took a bit longer than an already stable setup.

I always like the idea of users releasing and reviewing mail in junk quarantine themselves but in practice, doesn't it open you up to users shooting you and themselves in the foot opening bad things because they think it's legit? Some spearphishing/regular old phishing stuff these days are really well done, they trip up most anyone I know. Not too proud to admit being fooled once myself by an internal security audit by some white hat firm which was reallywell done -- they got some kind of fuckup out of 90% of my fellow IT dudes (the remaining 10% prob survived because they ignore email entirely and only communicate through Slack/skype f b)

Old Binsby
Jun 27, 2014



OpaqueEcho posted:

I have a minor issue that is hopefully easy to solve.

In order of easiness to check
- open an Exchange powershell session. Check output for Get-ExchangeCertificate for each internal server (use the -server parameter or you just get local output from your powershell proxy) Do you see the thumbprint/friendly name from your screencap? Get the output for Get-SendConnector. Check if the server's still listed as a source somewhere and whether had a dedicated connector that got corrupted or wasn't removed.
- if not, you check the computer cert stores on remaining exchange servers, maybe it's still there somewhere
- if it's not there, it can hardly be bound to any service in IIS either but you should check there if it's still configured somewhere whatever means
- If it's not, check your Edge servers if you have them, the edge subscription can give weird errors if an internal host is decommissioned without removing the server from EdgeSync first. (though usually on the Edge, not the LAN). It'll show up in Get-Queue as a nexthop for a queue and also in the event viewer if removal is done improperly with errors die to sync failures
- check for the cert on your reverse proxies, if applicable

Hope you're getting somewhere by that time, but if not I'd be happy to think of a few other suggestions. This is where I'd start.

E.

Submarine Sandpaper posted:

New-MailboxImportRequest things

This cmdlet consists nearly entirely of gotchas and very little regularity. Few things I've stumbled over (this is all here, but kind of vaguely worded):
-targetrootfolder and sourcerootfolder are 1 folder, can't be a deeper tree, so they're always at the top level of the mailbox. Can't use wildcards in them. You do not need to use these if you're merging entire mailboxes
-includefolders and excludefolders also don't take wildcards. They accept a collection of strings as input, which may be a one or more trees of folders but they must be written out and not wildcards. You can try to do this with get-mailboxfolderstatistics but it won't be a pretty one-line thing
-the quotes thing above has merit but there's more to it than that. You can definitely nest quotes of the same type, just need to escape them. The # might also need the escape character, the accent grave (`) does this in powershell. I used the localized folder names exactly once and it was kind of bad I remember, so I'm not sure.

Old Binsby fucked around with this message at 23:19 on Aug 2, 2017

Old Binsby
Jun 27, 2014



NevergirlsOFFICIAL posted:

I'm on a fresh profile. Furthermore: when the user herself cancels an old meeting, it lets her, and then she gets bounceback e.g.

pre:
 This message could not be sent. Try sending the message again later, or contact your network administrator. 
You do not have the permission to send the message on behalf of the specified user. Error is [0x80070005-0x0004dc-0x000524].

Did you remove the X500 address value from the AD proxyaddresses attribute or did you empty the ExchangeLegacyDN for the user?

Either way, a lot of weird poo poo starts happening when you start removing these addresses due to ~~legacy routing protocols~~ in exchange. This blog goes into it a bit.

You can probably fix it by re-adding the previous (not their current) on-prem LegacyExchangeDN attribute for the user as an X500 address to the proxyaddresses field (prefix it with 'X500:') but I don't have enough information to be sure here and you might have had a good reason to remove it (just read you also don't have it anymore, though most likely you can extract the old X500 address from the NDR).

So probably the smartest thing to do is wait for the Exchange servers to update the GAL or force an update, and then clearing your clients local address cache (starting from a clean profile is fine). Alternatively, you can have them connect in online mode, without cache so you don't have to wait for that or use OWA to take the client out of the equation. If that doesn't work, remove permissions and re-add them, especially Send on Behalf (through the the outlook Delegate window, calendar permissions are likely to not work otherwise) due to that being an AD backlink attribute which break when you migrate a user with a SID history and/or [...]. It's kind of messy, not going to post thousands of words unless anyone is actually interested .

It's inevitable you run into this kind of problems during migrations which is why MS won't support cross-prem permissions and/or send on behalf permissions after migrations.

e.

OpaqueEcho posted:

Thanks for the reply and help! I ended up resolving the issue by rebuilding the Arbitration mailboxes.

Huh, cool. Something to keep in mind if I ever run into weird decommisioning issues.

Old Binsby fucked around with this message at 19:00 on Aug 23, 2017

Old Binsby
Jun 27, 2014



Thanks Ants posted:

IIRC the Outlook app used to be a non-Microsoft application, and would store the credentials for your Exchange account in their , sync mail to that, then push it down to your app. I have no idea how it functions now. Our recommendation for iPhone users is to use it over Mail.app.

It still does that, though I'm unsure if it currently stores all your and mail credentials. It needs significant parts of it anyway for focused inbox functionality so if you can't have cloud-hosted anything don't use the Outlook app

E. Eh I'm actually not a 100% sure that was the reason, come to think of it. But anyway it does still store your stuff on servers other than your own

Old Binsby fucked around with this message at 19:23 on Sep 19, 2017

Old Binsby
Jun 27, 2014



It's not Outlook. The organizer himself deleted the meeting in his calendar without sending out the cancellation, which is possible as long as you are the owner (like when it's your own calendar) and/or have full access permissions. On a resource mailbox this is typically not the case, you only get to add/delete stuff by sending it invitations and cancellations which is impossible once you remove your version of the invite. The admin will likely remove them with the ediscovery thing above or simply by granting himself full access

Old Binsby
Jun 27, 2014



Ranter posted:

Outlook 2016 doesn't let you delete without sending cancellation, unless you have different steps to the ones I provided earlier with the screenshot that I could perform. 99% of users including this one don't have full access permission to room resources. Are you suggesting that with Outlook 2016, the workflow I shared earlier is different if you have full access permission to a room resource? I don't think the client checks your permissions to the attendees folders and displays different GUI options before you send anything.

And no, the admin meaning administrative assistant who we give elevated permission to room resource calendars. I am the exchange admin, technically.

Eh you're right I misremembered. You can't delete a meeting without letting the rest of the invitees know if you're the organizer. The only way you can delete a meeting without letting people know you're doing so is when you're an invitee. I don't think this has changed much over the years


e. what tripped me up is that usually, the room is the invitee and not the organizer so you should be able to throw out the meeting without much hassle as long as you're an admin with full access permissions. You're right that the organizer shouldn't be able to throw out the meeting on his end without letting the room know he cancelled.

Old Binsby fucked around with this message at 10:33 on Oct 2, 2017

Old Binsby
Jun 27, 2014



goobernoodles posted:

Nope. I suppose I could set up a second machine for her, but... bleeeggghhhhh... I think I'll give the MAPI session increase a shot.

Anyone have a really, really good understanding of calendar permissions and ExFolders? I've stumbled onto some calendar sharing permission problems. From powershell, it appears problem users should be able to access certain other users' calendars as they have availabilityonly permissions explicitly set, as well as the default permissions being availabilityonly for the calendars. However, they get an error about the calendar not existing or not having permissions.
Both on-prem to on-prem and with on-prem <-> o365 mailboxes. Although our o365 consultant claims it should be fully functional, everything I've read thus far indicates that a) non-explicit permissions do not get migrated and b) cross premises permissions are only work when using full access permissions. Also, while our handful of migrations had no errors, digging into it now, it looks like we have a bunch of skipped items for corrupt ACL's. This ought to be fun............

You're right about that last bit: cross prem permissions are a crapshoot (and they're mostly not supported) but Full Access work most of the time. I've had weird permission issues crop up while migrating, most significantly that whatever is in the GrantsSendonBehalfTo attribute may be a SID on-prem or it might be the SID history attribute for a user. Once you migrate the users where the latter is used for send on behalf delegation, there's no chance those permissions still work afterwards. If authenticated users can't see free/busy data when the default user permission is set to allow that, there's probably some hybrid connectivity issue

The folder limits you're running into with the veep/assistant suck to troubleshoot. I remember something about Outlook keeping less/no folder connections open for archive mailboxes. Is it an option to dump a bunch of old mail in there?

Old Binsby
Jun 27, 2014



Agrikk posted:

I'm running into a certificate problem that I can't untangle.

I have a server called MX1.int.mycompany.com and it has a wildcard cert for OWA access assigned for *.mycompany.com. External access to mail.mycompany.com/owa works fine, but internal access via outlook throws up a security alert because the name on the cert (*.mycompany.com) doesn't match the name of the server (mx1.int.mycompany.com) due to me following MS best practices on using subdomains.

I can't find a place where I can assign a cert for external virtual directories and a different cert for internal access.

Can anyone help?

Exchange with wildcard certs is fraught with peril and you've stumbled upon a common pitfall. Although your issue isn't Exchange-specific, quoting http://www.ietf.org/rfc/rfc2818.txt

quote:

If more than one identity of a given type is present in the certificate (e.g., more than one dNSName name, a match in any one of the set is considered acceptable.) Names may contain the wildcard character * which is considered to match any single domain name component or component fragment. E.g., *.a.com matches foo.a.com but not bar.foo.a.com. f*.com matches foo.com but not bar.com.

assigning two different certs isn't supported btw, you can manually do that in IIS but the settings might not persist and they'll definitely revert when you apply a CU. Your split DNS unfortunately needs SAN certificates. It isn't a terrible practice to not do split brain though

Old Binsby fucked around with this message at 14:04 on Nov 25, 2017

Old Binsby
Jun 27, 2014



NevergirlsOFFICIAL posted:

Alternately is there anyway I can say that only domain joined workstations can use fat client

You can make it harder for fat clients to get in by not publishing the default autodiscover DNS-records internally, which is kind of fiddly but domain-joined will lookup the custom location via AD in the Service Connection Point Exchange publishes. Enforcing kerberos or certificate based authentication (bit of a pain) and disallowing NTLM/basic everywhere except OWA/ECP cuts down on workgroup devices getting in but that's independent of client type. Outlook Anywhere/ActiveSync may still be an option for non-domain joined machines of course

Agrikk posted:

I've managed to figure out the certificate issue using an ELB in front of the edge servers that avoids using the subdomain. It's kludge, but it works.

The next issue I have is that DAG doesn't seem to be working properly, or my understanding of an Exchange DAG is wrong (both are equally possible).


I have MX1 and MX2 with Database1 and Database2 running in a DAG using a share on Witness1 as a quorum. It appears to be set up right as I see data in the share, and each exchange server has two database directories. MX1 was the original Exchange server and I added MX2 and Witness1 when I built the cluster.

I have User1 associated with Database1 with the primary host on MX1 and User2 associated with Database2 on MX2. When I shut down MX2, everything for User2 breaks (I can't connect to exchange through OWA or Outlook to see if email is being delivered properly) but User1 operates normally. Similarly, when I shut down MX1, everything for User1 breaks. However, when MX1 is down, User2 cannot send or receive email.

Isn't the whole point of a DAG to create redundancy, so that if a member server goes down the cluster stays up and keeps client access running?

What have I missed that is causing the dependency of User2 on MX1 as well as User2/User1 not being able to reach email when MX2/MX1 is down?

Bold part: yes, mostly italic: no, the client access proxying exchange does can be done with no DAG at all but you're right in the general sense. However DAGs don't provide 100% hands-off always-online databases resilient out of the box. The DAG keeping a second copy is primarily useful if you need to bring down a host in a controlled manner: You can switch over database to its passive copy, drain client connections and advertises a new host by not renewing them on the server going down and the user won't notice a thing while you do maintenance. Whether a DB will switchover automatically when a host goes down is dependent on the DatacenterActivationMode and probably their activation policy and a couple things I forget right now because it's late. Anyway yess, DAGs provide HA but with some caveats, it's always better to fail over in a controlled way

Users dropping connectivity when you drop a hot while their mailbox database server is still up means that they were being proxied through the client access service on the machine you killed. Set-ServerComponentState before or after will probably help you there, though be aware that undrained servers will usually leave a bit of a messy impression on the users when you simply say 'gently caress it, serverwideOffline and lunch time'. Probably going to be a few PW prompts etc, maaaaybe some Outlook profile corruptions.

also to make sure I completely resemble the MSFT community support people here are some mediocre documentation links on DAC mode (Technet) you probably found on your own already

Old Binsby fucked around with this message at 02:31 on Dec 6, 2017

Old Binsby
Jun 27, 2014



goobernoodles posted:

My understanding is that it's best practice to retain an on-prem Exchange server. Apparently you lose a lot of powershell functionality without the on-prem Exchange server, or so I've heard. I'm not sure how much truth there is since the one of the dudes this came from had a hard time grasping that I had a newer powershell module for AzureAD and that the commands he was sending me had been depreciated. We're also very early in our migration (10/260 or so) and I haven't yet started working on shifting user provisioning/update/term scripts to utilize O365 commands instead of on-prem.


It's not just PowerShell you're losing, the GUI interface uses the same underlying services and you simply can't edit a lot of user attributes any more if you remove all Exchange servers from a domain where the users are still in an on-prem AD. If you have a hybrid deployment already, the limits of that show up fairly quickly if you commit to doing any and all management on users/mailboxes through the online shell (or https://outlook.office365.com/ecp). You can still edit on-prem objects through adsiedit or the attribute editor in dsa.msc though.

NevergirlsOFFICIAL posted:

If you’re big enough shop that you need to do new-remotemailbox a lot and in batches then yeah keep on prem Server. If you’re just doing onesie-twosies like less than once a month just do it all in cloud.

This is also true I guess but I've never really found the place where it makes sense to do that. To keep an entire Exchange server on-prem for a really small company, maybe a 10-20 mailboxes (half of which aren't really used anymore) seems like overkill, I agree. But at small places without a lot of cutting edge/homebrew stuff going on, it's more often than not possible to move to the online suite entirely so you can eliminate the on-prem server that way. At smallish organizations where Exchange rarely needs actual work done on it, the combination of inexperience and management tools with little to no warning mechanisms is especially potent at making a bigger mess than would have been possible through the native Exchange PS. The smallest hybrid installation takes 1 core and 8 GB (maybe 4 even?) RAM, probably the only 'role' where you can get away with underprovisioning that if need be. It has no cluster to worry about, can be autopatched/rebooted with no effort, license is free(or rather, included with online licensing). I never regret having one around. The installation, maybe a firewall opening, adding 2 dns records and a 3rd party cert is <a day of work but doing it under stress of poo poo hitting a fan and needing one ASAP is awful.

Then you have the larger shops, i.e. anywhere you might have an enterprise agreement or another type of licensing/support contract with MS. At those places you don't really have a choice because they're not going to support the artisanal editing of attributes, it's bothersome to do for many users + most managers want you to BEST PRACTICE any practice in sight (or at least run not too far afoul of the backup plan called MS Support while they're paying for it).

Adbot
ADBOT LOVES YOU

Old Binsby
Jun 27, 2014



it is, if you’re unlucky you get to wait even longer. I did a few extended message traces a couple months back when they broke OME message routing, an MS engineer was working with us while those traces ran, they clearly took ages. He really didn’t like waiting 5-10 hours, swearing up and down it used to be much less. I don’t remember such a time but he said he’d take it up with the product group or whatever they’re called now because potentially waiting an entire workday is insane. So yea this is still what is,

Old Binsby fucked around with this message at 23:39 on Dec 20, 2017

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply