|
Apparently posted (and then soon deleted) by a Facebook SRE with the Reddit username "ramenporn"quote:As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC). There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures. Someone else said: quote:There are various non-FB fallback measures, including IRC as a last-ditch method. The IRC fallback is usually tested once a year for each engineer. quote:I just heard from a contact that the fallback/backup IRC is also down. 
|
#
¿
Oct 4, 2021 20:33
|
|
|
|
| # ¿ Apr 27, 2024 21:31 |
|
|
SREs always love to telling people not to "Thundering herd" but they do not care of, "Blundering Nerd"
|
|
#
¿
Oct 4, 2021 22:30
|
|