nanog mailing list archives
Re: Recent trouble with QUIC?
From: chris <tknchris () gmail com>
Date: Fri, 25 Sep 2015 19:49:00 -0400
This reminds me of something I ran into where I came to a similar conclusion. We had a customer who used google ad and docs products very heavily and all of a sudden they started getting captchas on accessing any google property. When we reached out to google we were told that they were "blacklisted" based on suspicious search queries or some kind of query manipulation that they believe was caused by malware. We search high and low internally and could not find anything and asked them to provide specifics about what they saw and they would not and then we tried to monitor network traffic we realized that google had just implemented SSL search as a default so we could not easily inspect the search traffic without putting in infrastructure that could do MITM and allow us to inspect (which we also suspected doing this could have serious blowback) At the end of the day the customer was extremely frustrated because they used google apps for their entire business and google insisted it was on their end but we couldnt not get any factual evidence and we would have had to do some really questionable things to try to go at debugging it on our own. TLDR, customer eventually bailed on all their google products because it scared them and reaching a human at google through regular channels was near impossible except through mazes of filling out forms and waiting 24hrs per email response. Even when we were able to connect with a fellow googler on nanog who tried to be helpful even though he wasnt on the right team we still got nowhere This is really the dark side of the "cloud" (no pun intended), when a company makes some kind of change or an event occurs with no communication and it backfires. Even the most basic advanced notifications or just having proper support available when a change occurs can be more important than the technical aspects. chris On Fri, Sep 25, 2015 at 7:20 PM, Ca By <cb.list6 () gmail com> wrote:
On Friday, September 25, 2015, Cody Grosskopf <codygrosskopf () gmail com <javascript:_e(%7B%7D,'cvml','codygrosskopf () gmail com');>> wrote:a) yes, 56,000 students and any on Chrome failed. I immediately blocked quic and told users to restart Chrome. Luckily the fallback to good ol'tcpsaved the day. b) I had this issue a few months ago and it subsided quickly Google reports it's an issue in this version of Chrome and the nextversionwill have a little smarts to automatically re initiate the connectionwithTCP automatically without having to disable quic.I remained very disappointed in how google has gone about quic. They are dismissive of network operators concerns (quic protocol list and ietf), cause substantial outages, and have lost a lot of good will in the process Here's your post mortem: RFO: Google unilaterally deployed a non-standard protocol to our production environment, driving up helpdesk calls x% After action: block udp 80/443 until production ready and standard ratified use deployed. And. Get off my lawn. On Wed, Sep 23, 2015 at 5:01 PM, Sean Hunter <jamesb2147 () gmail com> wrote:Hi all, I work for a 2500 user university and we've seen some odd behavior recently. 2-4 weeks ago we started seeing Google searches that wouldfailfor ~2 minutes, or disconnects in Gmail briefly. This week, and particularly in the last 2-3 days, we've had reports from numeroususersoncampus, even those who generally do not complain unless an issue hasbeenongoing for a while. Those reports include Drive disconnecting,searchesfailing, Gmail presenting a "007" error, and calendar failing to create events. In fact, the issue became so widespread today, that the campus paper is writing about it as a last minute article before they're weekly publication's deadline this evening. (Important in our little worldwherewe try to look good.) We aren't really staffed or equipped to figure out exactly what'shappening(and issues are sporadic, so packet captures are difficult, to say the least), but we found that disabling QUIC dramatically and immediately improved the experience of a couple of users on campus. We'rerecommendingvia the paper that others do so as well. What I'm curious about is: a) Has anyone here had a similar experience? Was the root cause QUIC in your case? b) Has anyone noticed anything remotely similar in the last few weeks/days/today? We're an Apps domain, so this may be specific to universities in theAppsuniverse. If anyone has any useful information or hints, or if someone fromwould like more information, please feel free to contact me, on or off list. Thanks for reading and have a great night everyone! Happy Wednesday!
Current thread:
- Recent trouble with QUIC? Sean Hunter (Sep 23)
- Re: Recent trouble with QUIC? Benson Schliesser (Sep 23)
- Re: Recent trouble with QUIC? Ca By (Sep 23)
- Re: Recent trouble with QUIC? Roland Dobbins (Sep 23)
- Re: Recent trouble with QUIC? Mike Meredith (Sep 24)
- Re: Recent trouble with QUIC? Todd Underwood (Sep 24)
- Re: Recent trouble with QUIC? Cody Grosskopf (Sep 25)
- Recent trouble with QUIC? Ca By (Sep 25)
- Re: Recent trouble with QUIC? chris (Sep 25)
- Re: Recent trouble with QUIC? Stephen Satchell (Sep 25)
- Re: Recent trouble with QUIC? Sean Hunter (Sep 25)
- Re: Recent trouble with QUIC? Matthew Kaufman (Sep 25)
- Re: Recent trouble with QUIC? Mike Hale (Sep 26)
- Re: Recent trouble with QUIC? James Bensley (Sep 26)
- Recent trouble with QUIC? Ca By (Sep 25)
- Re: Recent trouble with QUIC? Saku Ytti (Sep 27)
- Re: Recent trouble with QUIC? Lyle Giese (Sep 27)
- Re: Recent trouble with QUIC? Matthew Kaufman (Sep 27)
- Re: Recent trouble with QUIC? Saku Ytti (Sep 27)
- Re: Recent trouble with QUIC? Cody Grosskopf (Sep 28)