Excellent! This helps @Bisgaard, Dustin (WaTech)<mailto:dustin.bisgaard@watech.wa.gov>! On the Pierce County side this changes how we will approach notification to the State. 1. We will no longer be reaching out for RFO unless all 4ea peers were down at the same time for over 10min at any given time. 2. Since it automatically generates a ticket on your side, how would you suggest that you notify you all when we see that all 4 peers were down over 10min? * The NOC e-mail address alone and leave the support@watech.wa.gov<mailto:support@watech.wa.gov> out of the communication as to not duplicate tickets? * Some other person or distro group. SS911 has BGP visibility......for those that DO not that leverage IGN connectivity through Pierce County and SS911 BGP sessions, we will keep those notified EVEN when all peers do go down for less than 10min as often users are on during that 30s or 9min window and are important enough to warrant an ANSWER as to WHAT HAPPEDED? You will already have it. Reginald Osborne Pierce County, WA Network Engineer Ph# 253-798-8838 From: Bisgaard, Dustin (WaTech) <dustin.bisgaard@watech.wa.gov> Sent: Thursday, March 12, 2026 10:52 AM To: Reginald Osborne <reggie.osborne@piercecountywa.gov>; WaTech mi Support Center <Support@WaTech.wa.gov>; WaTech DL NCC Operations <NOC@Watech.wa.gov> Cc: ITNetwork <ITNetwork@piercecountywa.gov>; inctpc-list@list.cityoftacoma.org; mwood@fifewa.gov; Network Operations <netops@southsound911.org> Subject: Re: SDC/QDC BGP flap Pierce County - CS0238387 Hi Reginald, Yes our 4 routers where your BGP peering lands do reflect the bounces on our side. I see 90 second outages for you and SS911 at the same time. Both the vendor and carrier report that they saw no outages and there weren't any maintenance events that would affect this. For anything under about 5 minutes, this answer is typical. Usually, any bounces up to 5 minutes is extremely hard for them to find. In my experience if it's a chronic issue of short drops it still takes the vendors a lot of time and energy to find the issue. Our monitoring system will create a ticket after 10 minutes of downtime, which gives us something to track and to work with. This period of time is a good length to washout customers rebooting their routers, or other short term events; with the 10 minutes it gives us a high probability that it's an actionable outage caused by either power issue or vendor issue. We do admit that if it is a chronic short term bouncing the monitoring system does miss those, but then we rely on customers to report that. Vendor's reporting "no faults" or "no issues found" or "cleared in testing" are all valid legal responses. I hope this sheds light into our thinking and operations around drops and outages, If there are additional questions please ask them. Thank you, -- Dustin Bisgaard | MSCIA Washington Technology Solutions Senior Data Network Specialist Network Services Division (360) 902-2292 | Office Need help? Contact us through our Customer Portal<https://watech.servicenowservices.com/> or 360-586-1000 | 1-855-WaTech1. Legal Notice: Under Washington's Public Records Act, all emails sent by or to me on this state-owned email account may be subject to public disclosure. ________________________________ From: Reginald Osborne <reggie.osborne@piercecountywa.gov<mailto:reggie.osborne@piercecountywa.gov>> Sent: Thursday, March 12, 2026 9:47 AM To: Bisgaard, Dustin (WaTech) <dustin.bisgaard@watech.wa.gov<mailto:dustin.bisgaard@watech.wa.gov>>; WaTech mi Support Center <Support@WaTech.wa.gov<mailto:Support@WaTech.wa.gov>>; WaTech DL NCC Operations <NOC@Watech.wa.gov<mailto:NOC@Watech.wa.gov>> Cc: ITNetwork <ITNetwork@piercecountywa.gov<mailto:ITNetwork@piercecountywa.gov>>; inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org> <inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org>>; mwood@fifewa.gov<mailto:mwood@fifewa.gov> <mwood@fifewa.gov<mailto:mwood@fifewa.gov>>; Network Operations <netops@southsound911.org<mailto:netops@southsound911.org>> Subject: RE: SDC/QDC BGP flap Pierce County - CS0238387 External Email I have some questions that will hopefully save me time in the future on whether or not to even send these message out. * If you look at boxes that host the IP addresses you see below....we have them as SDC1 / SDC2 / QDC1 / QDC2 do they indicate the same thing that our box is indicating? * If not, that is strange....... * If so, then are we stating by closing the ticket that we no longer want to investigate further on why? * How long should a BGP outage be for us to collectively try to determine why? Reginald Osborne Pierce County, WA Network Engineer Ph# 253-798-8838 From: Bisgaard, Dustin (WaTech) <dustin.bisgaard@watech.wa.gov<mailto:dustin.bisgaard@watech.wa.gov>> Sent: Thursday, March 12, 2026 9:39 AM To: Reginald Osborne <reggie.osborne@piercecountywa.gov<mailto:reggie.osborne@piercecountywa.gov>>; WaTech mi Support Center <Support@WaTech.wa.gov<mailto:Support@WaTech.wa.gov>>; WaTech DL NCC Operations <NOC@Watech.wa.gov<mailto:NOC@Watech.wa.gov>> Cc: ITNetwork <ITNetwork@piercecountywa.gov<mailto:ITNetwork@piercecountywa.gov>>; inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org>; mwood@fifewa.gov<mailto:mwood@fifewa.gov>; Network Operations <netops@southsound911.org<mailto:netops@southsound911.org>> Subject: Re: SDC/QDC BGP flap Pierce County - CS0238387 Good morning Reginald, The vendor and carrier report they don't see any outages on their side, and they didn't have any maintenance events that would have affected this circuit. We'll resolve this ticket at the end of today. Please let us know if you have any questions or concerns. Thank you, -- Dustin Bisgaard | MSCIA Washington Technology Solutions Senior Data Network Specialist Network Services Division (360) 902-2292 | Office Need help? Contact us through our Customer Portal<https://watech.servicenowservices.com/> or 360-586-1000 | 1-855-WaTech1. Legal Notice: Under Washington's Public Records Act, all emails sent by or to me on this state-owned email account may be subject to public disclosure. ________________________________ From: Bisgaard, Dustin (WaTech) <dustin.bisgaard@watech.wa.gov<mailto:dustin.bisgaard@watech.wa.gov>> Sent: Thursday, March 12, 2026 8:08 AM To: Reginald Osborne <reggie.osborne@piercecountywa.gov<mailto:reggie.osborne@piercecountywa.gov>>; WaTech mi Support Center <Support@WaTech.wa.gov<mailto:Support@WaTech.wa.gov>>; WaTech DL NCC Operations <NOC@Watech.wa.gov<mailto:NOC@Watech.wa.gov>> Cc: ITNetwork <ITNetwork@piercecountywa.gov<mailto:ITNetwork@piercecountywa.gov>>; inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org> <inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org>>; mwood@fifewa.gov<mailto:mwood@fifewa.gov> <mwood@fifewa.gov<mailto:mwood@fifewa.gov>>; Network Operations <netops@southsound911.org<mailto:netops@southsound911.org>> Subject: Re: SDC/QDC BGP flap Pierce County - CS0238387 Good morning Reginald and Pierce County, Your ticket number is CS0238387. I'll reach out to the vendor and see what they saw, but I am certain that this was related to the wind storm across the state last night. More information to come. Thank you, -- Dustin Bisgaard | MSCIA Washington Technology Solutions Senior Data Network Specialist Network Services Division (360) 902-2292 | Office Need help? Contact us through our Customer Portal<https://watech.servicenowservices.com/> or 360-586-1000 | 1-855-WaTech1. Legal Notice: Under Washington's Public Records Act, all emails sent by or to me on this state-owned email account may be subject to public disclosure. ________________________________ From: Reginald Osborne <reggie.osborne@piercecountywa.gov<mailto:reggie.osborne@piercecountywa.gov>> Sent: Thursday, March 12, 2026 5:31 AM To: WaTech mi Support Center <Support@WaTech.wa.gov<mailto:Support@WaTech.wa.gov>>; WaTech DL NCC Operations <NOC@Watech.wa.gov<mailto:NOC@Watech.wa.gov>> Cc: ITNetwork <ITNetwork@piercecountywa.gov<mailto:ITNetwork@piercecountywa.gov>>; inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org> <inctpc-list@list.cityoftacoma.org<mailto:inctpc-list@list.cityoftacoma.org>>; mwood@fifewa.gov<mailto:mwood@fifewa.gov> <mwood@fifewa.gov<mailto:mwood@fifewa.gov>>; Network Operations <netops@southsound911.org<mailto:netops@southsound911.org>> Subject: SDC/QDC BGP flap Pierce County External Email At 1:27am last night all of our BGP sessions here at Pierce County went down and came back up about 30s later for both SDC and QDC. Just looking for RFO? CLLI TACMWAVU05W / VE-TAC2705 - 17497713 Handoff Circuit ID: ETH/002/1019273 Virtual Circuit ID: ETH/002/1019289ETH/002/1019305 [cid:image001.png@01DCB216.494278F0] Reginald Osborne Pierce County, WA Network Engineer Ph# 253-798-8838