<!DOCTYPE html>
<html>
<head>
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
</head>
<body><div style="font-family:menlo, consolas, monospace;">You can consider it a lucky outcome versus a full guest os crash. File I/o and network I/o related operations are also done with never mind expecting crypto to hold up<br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;">One solution for example fedora has packages to use the hosts time instead the vm. FreeBSD seems to work just fine and syncing to host’s time.</div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;">On Mon, Dec 18, 2017, at 20:39, Isaac (.ike) Levy wrote:<br></div>
<div style="font-family:menlo, consolas, monospace;">> Hi All,<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> A bit OT from the pit of internet hell, but perhaps of interest to folks<br></div>
<div style="font-family:menlo, consolas, monospace;">> here: This weekend AWS has been doling out a disruption of service of<br></div>
<div style="font-family:menlo, consolas, monospace;">> the worst kind, clock skew insanity. And when I say insanity, I mean<br></div>
<div style="font-family:menlo, consolas, monospace;">> true madness.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> (For those who don't know me, this loathsome cloud infrastructure is<br></div>
<div style="font-family:menlo, consolas, monospace;">> something I'm paid to use, not tech I think is great or even acceptable<br></div>
<div style="font-family:menlo, consolas, monospace;">> for many uses, and I'm not engaging any "lets argue the value of the<br></div>
<div style="font-family:menlo, consolas, monospace;">> cloud" here today.)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Is anyone else experiencing the clock/drift issue and have interesting<br></div>
<div style="font-family:menlo, consolas, monospace;">> notes to share?<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> BIZARRE:<br></div>
<div style="font-family:menlo, consolas, monospace;">> Clocks drifiting up to 7-9min. Clocks drifting so fast that ntpdate and<br></div>
<div style="font-family:menlo, consolas, monospace;">> rdate can't even "set the time"*.<br></div>
<div style="font-family:menlo, consolas, monospace;">> Clocks drifting past ~5min window means that cryptographic network<br></div>
<div style="font-family:menlo, consolas, monospace;">> operations in our world fail outright, (ssl/tls and http services).<br></div>
<div style="font-family:menlo, consolas, monospace;">> Driftfile worthless- the drifting appears non-determinstic, we have<br></div>
<div style="font-family:menlo, consolas, monospace;">> found no apparent pattern on analysis.<br></div>
<div style="font-family:menlo, consolas, monospace;">> New instances coming up with clocks that are *years* in the past. ntpd<br></div>
<div style="font-family:menlo, consolas, monospace;">> freak out when trying to handle that at boot.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> First, we thought the problem was skew, so we put in the ntpdate run<br></div>
<div style="font-family:menlo, consolas, monospace;">> ahead of ntp start- that settled things for a bit. Then 90min later,<br></div>
<div style="font-family:menlo, consolas, monospace;">> hosts were drifting past 5min- NTP was reporting offsets of between<br></div>
<div style="font-family:menlo, consolas, monospace;">> 3k-45k and jitter of between 2k-60k on the *second and subsequent<br></div>
<div style="font-family:menlo, consolas, monospace;">> polls*.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Just to keep systems functioning, we've got a cron job running every<br></div>
<div style="font-family:menlo, consolas, monospace;">> 15min (ironic) to restart ntpd.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> AWS ACKNOWLEDGEMENT:<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> AWS is infamous for burrying outages in marketing material, so not a lot<br></div>
<div style="font-family:menlo, consolas, monospace;">> to go on here. Look, all green:<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://status.aws.amazon.com/<br></div>
<div style="font-family:menlo, consolas, monospace;">> We have loose ack from AWS, mostly in the form of other customers<br></div>
<div style="font-family:menlo, consolas, monospace;">> posting to AWS forums from their support tickets, like this:<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://forums.aws.amazon.com/thread.jspa?messageID=819947<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Furthermore, AWS support contracts have nasty NDA's precluding customers<br></div>
<div style="font-family:menlo, consolas, monospace;">> from sharing information from support tickets. Therefore, companies<br></div>
<div style="font-family:menlo, consolas, monospace;">> like mine cannot get much from support- because we'd be in breach of<br></div>
<div style="font-family:menlo, consolas, monospace;">> contract for merely telling our own customers about an AWS outage- let<br></div>
<div style="font-family:menlo, consolas, monospace;">> alone any technical details they'd provide. So, companies like mine<br></div>
<div style="font-family:menlo, consolas, monospace;">> can't get technical support contracts from AWS. (Of course I can neither<br></div>
<div style="font-family:menlo, consolas, monospace;">> confirm nor deny if this is the case for my employ).<br></div>
<div style="font-family:menlo, consolas, monospace;">> No worries though, after living with AWS technical support elsewhere,<br></div>
<div style="font-family:menlo, consolas, monospace;">> it's abysmal and nearly useless anyhow.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> USERLAND EFFECTS OF THIS INSANITY:<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> We don't see things happening which would indicate CPU cycles are being<br></div>
<div style="font-family:menlo, consolas, monospace;">> affected, just userland notions of time. So, this makes 2 distinct<br></div>
<div style="font-family:menlo, consolas, monospace;">> problems we see:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - Applications which rely on time, e.g. "do this at that time" are<br></div>
<div style="font-family:menlo, consolas, monospace;">> completely hozed. Less noticable with cron, totally happening with our<br></div>
<div style="font-family:menlo, consolas, monospace;">> own apps.<br></div>
<div style="font-family:menlo, consolas, monospace;">> - As mentioned above, cryptographic operations are so compromised they<br></div>
<div style="font-family:menlo, consolas, monospace;">> outright fail when the clocks drift up over 5 min.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> RANT ON THE PARADE OF THE AMATEUR, (possible root cause, AWS lit up some<br></div>
<div style="font-family:menlo, consolas, monospace;">> chronyc!)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Looks like some fool decided they can do better than ntpd, specifically<br></div>
<div style="font-family:menlo, consolas, monospace;">> for AWS. Named 'chrony' or 'chronyc' on some platforms.<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-the-amazon-time-sync-service/<br></div>
<div style="font-family:menlo, consolas, monospace;">> Some of the mind-blowingly bad decisions in here:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - deploy/announce an AWS-custom NTP daemon just weeks before Christmas<br></div>
<div style="font-family:menlo, consolas, monospace;">> shopping crunch! (What could possibly go wrong.)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - deploy/announce an AWS-custom NTP daemon in the first place, (Ask<br></div>
<div style="font-family:menlo, consolas, monospace;">> PHK, he makes NTP look easy!)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - keep using the NTP protocol, but abandon existing software, /facepalm<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Now here's where it gets even more interesting,<br></div>
<div style="font-family:menlo, consolas, monospace;">> <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html>,<br></div>
<div style="font-family:menlo, consolas, monospace;">> where we learn:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - "The Amazon Time Sync Service is available through NTP at the<br></div>
<div style="font-family:menlo, consolas, monospace;">> 169.254.169.123 IP address for any instance running in a VPC. Your<br></div>
<div style="font-family:menlo, consolas, monospace;">> instance does not require access to the internet, and you do not have to<br></div>
<div style="font-family:menlo, consolas, monospace;">> configure your security group rules or your network ACL rules to allow<br></div>
<div style="font-family:menlo, consolas, monospace;">> access...."<br></div>
<div style="font-family:menlo, consolas, monospace;">> That's right- beyond userland config massaging, they appear to have<br></div>
<div style="font-family:menlo, consolas, monospace;">> forced global whitelisting of UDP to that single IP address across your<br></div>
<div style="font-family:menlo, consolas, monospace;">> hand-built VPC ACL's. (What could go wrong there.)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> I don't think chronyc itself is the problem, but that they are smoking<br></div>
<div style="font-family:menlo, consolas, monospace;">> crack over there at AWS.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> So, as my team hobbles along today, does anyone else have any anectodal<br></div>
<div style="font-family:menlo, consolas, monospace;">> stories to share on this one?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - comment on the mechanics of cryptographic operations and time?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - root causes?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - any peek into actual technial detail, (kernel/hypervisors/drift?)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - impact to the GDP?<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Best,<br></div>
<div style="font-family:menlo, consolas, monospace;">> .ike<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> _______________________________________________<br></div>
<div style="font-family:menlo, consolas, monospace;">> talk mailing list<br></div>
<div style="font-family:menlo, consolas, monospace;">> talk@lists.nycbug.org<br></div>
<div style="font-family:menlo, consolas, monospace;">> http://lists.nycbug.org/mailman/listinfo/talk<br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
</body>
</html>