<!DOCTYPE html>
<html>
<head>
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
</head>
<body><div style="font-family:menlo, consolas, monospace;">Once Time is off by years or even just months ntpdate may fail due to ip problems, more obvious ntp over tcp. Yes date is the only command to bring it close enough so ntp can work. The ironic or chrony 15 mins ntpdate may just time out and never correct.</div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
<div style="font-family:menlo, consolas, monospace;">On Mon, Dec 18, 2017, at 20:39, Isaac (.ike) Levy wrote:<br></div>
<div style="font-family:menlo, consolas, monospace;">> Hi All,<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> A bit OT from the pit of internet hell, but perhaps of interest to folks<br></div>
<div style="font-family:menlo, consolas, monospace;">> here: This weekend AWS has been doling out a disruption of service of<br></div>
<div style="font-family:menlo, consolas, monospace;">> the worst kind, clock skew insanity. And when I say insanity, I mean<br></div>
<div style="font-family:menlo, consolas, monospace;">> true madness.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> (For those who don't know me, this loathsome cloud infrastructure is<br></div>
<div style="font-family:menlo, consolas, monospace;">> something I'm paid to use, not tech I think is great or even acceptable<br></div>
<div style="font-family:menlo, consolas, monospace;">> for many uses, and I'm not engaging any "lets argue the value of the<br></div>
<div style="font-family:menlo, consolas, monospace;">> cloud" here today.)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Is anyone else experiencing the clock/drift issue and have interesting<br></div>
<div style="font-family:menlo, consolas, monospace;">> notes to share?<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> BIZARRE:<br></div>
<div style="font-family:menlo, consolas, monospace;">> Clocks drifiting up to 7-9min. Clocks drifting so fast that ntpdate and<br></div>
<div style="font-family:menlo, consolas, monospace;">> rdate can't even "set the time"*.<br></div>
<div style="font-family:menlo, consolas, monospace;">> Clocks drifting past ~5min window means that cryptographic network<br></div>
<div style="font-family:menlo, consolas, monospace;">> operations in our world fail outright, (ssl/tls and http services).<br></div>
<div style="font-family:menlo, consolas, monospace;">> Driftfile worthless- the drifting appears non-determinstic, we have<br></div>
<div style="font-family:menlo, consolas, monospace;">> found no apparent pattern on analysis.<br></div>
<div style="font-family:menlo, consolas, monospace;">> New instances coming up with clocks that are *years* in the past. ntpd<br></div>
<div style="font-family:menlo, consolas, monospace;">> freak out when trying to handle that at boot.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> First, we thought the problem was skew, so we put in the ntpdate run<br></div>
<div style="font-family:menlo, consolas, monospace;">> ahead of ntp start- that settled things for a bit. Then 90min later,<br></div>
<div style="font-family:menlo, consolas, monospace;">> hosts were drifting past 5min- NTP was reporting offsets of between<br></div>
<div style="font-family:menlo, consolas, monospace;">> 3k-45k and jitter of between 2k-60k on the *second and subsequent<br></div>
<div style="font-family:menlo, consolas, monospace;">> polls*.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Just to keep systems functioning, we've got a cron job running every<br></div>
<div style="font-family:menlo, consolas, monospace;">> 15min (ironic) to restart ntpd.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> AWS ACKNOWLEDGEMENT:<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> AWS is infamous for burrying outages in marketing material, so not a lot<br></div>
<div style="font-family:menlo, consolas, monospace;">> to go on here. Look, all green:<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://status.aws.amazon.com/<br></div>
<div style="font-family:menlo, consolas, monospace;">> We have loose ack from AWS, mostly in the form of other customers<br></div>
<div style="font-family:menlo, consolas, monospace;">> posting to AWS forums from their support tickets, like this:<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://forums.aws.amazon.com/thread.jspa?messageID=819947<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Furthermore, AWS support contracts have nasty NDA's precluding customers<br></div>
<div style="font-family:menlo, consolas, monospace;">> from sharing information from support tickets. Therefore, companies<br></div>
<div style="font-family:menlo, consolas, monospace;">> like mine cannot get much from support- because we'd be in breach of<br></div>
<div style="font-family:menlo, consolas, monospace;">> contract for merely telling our own customers about an AWS outage- let<br></div>
<div style="font-family:menlo, consolas, monospace;">> alone any technical details they'd provide. So, companies like mine<br></div>
<div style="font-family:menlo, consolas, monospace;">> can't get technical support contracts from AWS. (Of course I can neither<br></div>
<div style="font-family:menlo, consolas, monospace;">> confirm nor deny if this is the case for my employ).<br></div>
<div style="font-family:menlo, consolas, monospace;">> No worries though, after living with AWS technical support elsewhere,<br></div>
<div style="font-family:menlo, consolas, monospace;">> it's abysmal and nearly useless anyhow.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> USERLAND EFFECTS OF THIS INSANITY:<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> We don't see things happening which would indicate CPU cycles are being<br></div>
<div style="font-family:menlo, consolas, monospace;">> affected, just userland notions of time. So, this makes 2 distinct<br></div>
<div style="font-family:menlo, consolas, monospace;">> problems we see:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - Applications which rely on time, e.g. "do this at that time" are<br></div>
<div style="font-family:menlo, consolas, monospace;">> completely hozed. Less noticable with cron, totally happening with our<br></div>
<div style="font-family:menlo, consolas, monospace;">> own apps.<br></div>
<div style="font-family:menlo, consolas, monospace;">> - As mentioned above, cryptographic operations are so compromised they<br></div>
<div style="font-family:menlo, consolas, monospace;">> outright fail when the clocks drift up over 5 min.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> RANT ON THE PARADE OF THE AMATEUR, (possible root cause, AWS lit up some<br></div>
<div style="font-family:menlo, consolas, monospace;">> chronyc!)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Looks like some fool decided they can do better than ntpd, specifically<br></div>
<div style="font-family:menlo, consolas, monospace;">> for AWS. Named 'chrony' or 'chronyc' on some platforms.<br></div>
<div style="font-family:menlo, consolas, monospace;">> https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-the-amazon-time-sync-service/<br></div>
<div style="font-family:menlo, consolas, monospace;">> Some of the mind-blowingly bad decisions in here:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - deploy/announce an AWS-custom NTP daemon just weeks before Christmas<br></div>
<div style="font-family:menlo, consolas, monospace;">> shopping crunch! (What could possibly go wrong.)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - deploy/announce an AWS-custom NTP daemon in the first place, (Ask<br></div>
<div style="font-family:menlo, consolas, monospace;">> PHK, he makes NTP look easy!)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - keep using the NTP protocol, but abandon existing software, /facepalm<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Now here's where it gets even more interesting,<br></div>
<div style="font-family:menlo, consolas, monospace;">> <http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html>,<br></div>
<div style="font-family:menlo, consolas, monospace;">> where we learn:<br></div>
<div style="font-family:menlo, consolas, monospace;">> - "The Amazon Time Sync Service is available through NTP at the<br></div>
<div style="font-family:menlo, consolas, monospace;">> 169.254.169.123 IP address for any instance running in a VPC. Your<br></div>
<div style="font-family:menlo, consolas, monospace;">> instance does not require access to the internet, and you do not have to<br></div>
<div style="font-family:menlo, consolas, monospace;">> configure your security group rules or your network ACL rules to allow<br></div>
<div style="font-family:menlo, consolas, monospace;">> access...."<br></div>
<div style="font-family:menlo, consolas, monospace;">> That's right- beyond userland config massaging, they appear to have<br></div>
<div style="font-family:menlo, consolas, monospace;">> forced global whitelisting of UDP to that single IP address across your<br></div>
<div style="font-family:menlo, consolas, monospace;">> hand-built VPC ACL's. (What could go wrong there.)<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> I don't think chronyc itself is the problem, but that they are smoking<br></div>
<div style="font-family:menlo, consolas, monospace;">> crack over there at AWS.<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> --<br></div>
<div style="font-family:menlo, consolas, monospace;">> So, as my team hobbles along today, does anyone else have any anectodal<br></div>
<div style="font-family:menlo, consolas, monospace;">> stories to share on this one?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - comment on the mechanics of cryptographic operations and time?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - root causes?<br></div>
<div style="font-family:menlo, consolas, monospace;">> - any peek into actual technial detail, (kernel/hypervisors/drift?)<br></div>
<div style="font-family:menlo, consolas, monospace;">> - impact to the GDP?<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> Best,<br></div>
<div style="font-family:menlo, consolas, monospace;">> .ike<br></div>
<div style="font-family:menlo, consolas, monospace;">><br></div>
<div style="font-family:menlo, consolas, monospace;">> _______________________________________________<br></div>
<div style="font-family:menlo, consolas, monospace;">> talk mailing list<br></div>
<div style="font-family:menlo, consolas, monospace;">> talk@lists.nycbug.org<br></div>
<div style="font-family:menlo, consolas, monospace;">> http://lists.nycbug.org/mailman/listinfo/talk<br></div>
<div style="font-family:menlo, consolas, monospace;"><br></div>
</body>
</html>