[talk] Embargo/Foundation questions, Intel CPU/speculation bugs

[Isaac (.ike) Levy]

Appreciate your time and thoughtful response GNN, 2 big questions remain, specific to FreeBSD:

- What benefit have the vendor NDA's had for the FreeBSD project (particularly Intel)?

- What negative effects have vendor NDA's had on the FreeBSD project?


These are the only two questions that really matter, more commentary for anyone with time to spend on it:

Worth me stating ahead of my commentary (for folks on list who don't know me or GNN), I have profound respect for GNN's epic work on FreeBSD, from core Networking, to comprehensive OS internals, his code touches us all in some way we cherish.  We're all better off because of GNN's publishing and educational work, as well as the time he's spent toiling for the Foundation- the list goes on and on.

Just wanted to state this up front so it's clear in this discourse that I'm aiming to keep this thread and any disagreement focused on constructive understanding of the issues, and all our common aim to keep working for *BSD systems to be as strong as ever.
Even though I'm a mere mortal, I'll certainly say that GNN and myself are every bit as invested in *BSD thriving, as so many of us are here.

On Mon, Jun 18, 2018, at 5:24 AM, George Neville-Neil wrote:
> Comments in line...
> 
> On 18 Jun 2018, at 3:59, Isaac (.ike) Levy wrote:
> 
> > Hi All,
> >
> > I'm sure folks have been tuned in to the fragmented security news of 
> > the week: Intel and more speculative execution security flaws.  (If 
> > not, check the URL footnotes below).
> >
> > With that, questions for list, genuinely looking for some logical 
> > answers in favor of the NDA/embargo on critical fixes we've seen in 
> > recent months:
> >
> > - What do folks think about these Intel NDA-covered embargo, (or any 
> > vendor for that matter)?
> > The embargoed information really doesn't seem to be doing us all any 
> > good at all here as users.
> > I'd really love to hear any constructive reasons why the NDA's are 
> > necessary or good for any project developement?!  How does this help 
> > the world?
> >
> > - Specific to FreeBSD, and the Foundation: I've been a huge fan of the 
> > FreeBSD foundation since it's inception.  To separate assets from the 
> > FreeBSD codebase was certainly born in the SCO era, with the the USL 
> > vs. BSDi era in mind, etc... this part of the mission of the 
> > Foundation has always made a lot of sense to me- separating legal 
> > assets from the codebase for the longevity of the codebase itself.  
> > The Foundation has done some excellent work, from handling donations, 
> > coordinating conference attendance, project grants, developer hardware 
> > and donations- and of course supporting the amazing RELENG work as a 
> > J.O.B. job for Glen Barber (et. al.).  This has produced amazing 
> > things for FreeBSD.
> >
> > Yet, what are the *boundaries* of the mission of the FreeBSD 
> > foundation?
> > Does the foundation even facilitate vendor relationships and shepherd 
> > NDA's for developers, or am I just totally mistaken here?
> 
> Indeed we do.  In fact one of my first tasks when I joined the FF was to 
> get a set of NDAs in place and a process by which any FreeBSD developer 
> could be given access to NDA'd information from various vendors.  We 
> have these types of relationships with several folks including Intel, 
> ARM, NVidia and others.

Thanks for clarifying this, coordinating an shepherding the NDA's with proper legal counsel sounds like a massive task!
The Foundation certainly does seem to be the place for this level of coordination to happen- (to keep the codebase in a known state with regard to various legalities).  Far better than over in Linux-land, where who knows who's got what NDA priviliged commit coming into what parts of the tree...

For example, in the Linux Kernel, it became quite obvious when Spectre/Meltdown was about to hit the street, that even Linus wasn't at all privvy to what was happening:
https://lkml.org/lkml/2018/1/3/797
(an interesting post on many levels.)

> 
> > I mean, a while back they were becomming involved with humanitarian 
> > projects- wonderful stuff, but not what I donate to the FreeBSD 
> > Foundation for- and certainly quite a deviation from strategically 
> > shepherding the legalities of NDA's with Intel.
> > These days, I see a lot of great press stuff on the Foundation site, 
> > but I don't see any clear "here is why we help sign NDA's with Intel." 
> >  I can't even find mention of "NDA" on the Foundation site.
> >
> 
> It's not something we'd put on the front page.  I and others talk about 
> this frequently at conferences and developer summits.  This kind of 
> stuff often takes time with our lawyer, and so we don't just do it for 
> everyone.   The typical process is that a developer comes to us with a 
> need and we work with them to get this in place so they get the 
> information they need.

I get that- but to clarify, I'm certainly not looking for a sign saying "get in line here for NDA privileged information".

As a user of FreeBSD, with my own concerns, I simply would prefer to understand more clearly what these NDA's impose on the codebase, and also what they provide, to the FreeBSD project.  Worthy of a blurb describing the scope of this critical effort under the "What We Do" tab, etc...

> 
> > Assuming FreeBSD devlopers/Foundation does have NDA with Intel 
> > (others) for chip documentation, as well as security vulnearabilities:
> 
> Chip docs mostly don't need NDAs, although some graphics stuff still 
> does, the security stuff from the FF side, is handled by Ed Maste and 
> one or two of our contractors, as well as by members of the community 
> who are contacted by Secteam.  Secteam and SO are the main project folks 
> handling this stuff.

I may be wrong here, but chip docs for stuff like these latest speculative execution and cache bugs, don't appear to be well documented  at all, (even with NDA)?  Is this a qbit- whereby the chip is simultaneously documented, and not documented, and also concurrently has a state which requires NDA?  I kid- I understand the scale and complexity of publishing relevant docs on fast moving hardware in our world- yet I don't understand where it's acceptable to step over a line into NDA territory.

Also, from Theo's BSDCan talk, the fun clarifying vocabulary of "Chicken Bits", (features on chips which are part of future features, baked into the die early- or perhaps never seeing the light of day- these require NDA- in recent years exposed only through security vulns, forcing vendor response documentation.)

Vulnerabilities that pop up from these "Chicken Bits", have had a profound effect on my professional life as a sysadmin/engineer in recent years.

To date, I don't see evidence that any NDA process here is mitigating anything for anybody, except perhaps mitigating dips in Intel's stock value.  I'd love to see the evidence which can change my mind!

Additionally, these issues are far from new or unique to the *BSD's, - as in the BlackHat/Defcon/CCC/ShmooCon/etc universe, Intel CPU vulnerabilities have been more frequent and more damaging every year for some years.  (And often, the technical details are absurdly inane- the issues are boneheaded- the secrecy and profound impact of the vluns is what's made them such a hot topic).

> 
> > For Meltdown/Spectre, well, I don't know if if NDA's were in place at 
> > the time- but if it was, it didn't seem to help FreeBSD get any 
> > mitigations out- (even for several weeks past public disclosure).
> 
> The biggest challenges here are not NDAs per se, but with hands and 
> communication.  Intel is big, and FreeBSD is not Linux so there are two 
> challenges there, and we're continuing to work with Intel to get the 
> right people to talk to at the right time.  Of course, in 6 months those 
> people will change jobs, so, wash rinse repeat.  The hands problem is a 
> serious one since capable volunteer security experts who can even 
> understand the problems of Spectre are few and far between.

With all due respect to everyone working toward fixes/mitigations, the esoteric nature of many important security vulnerabilities cannot ever be comprehensively handled by any single organization.  I do not see evidence here that maintaining NDA-backed secrecy is helping increase the number of hands- capable volunteers who are subject matter experts.

In my 20 years running internet-facing infrastructure, the most effective security responses in the world have often been mitigated/resolved by the parties which care most about them, (and understand them), not by teams dedicated to be solely responsible for solving everything.

And for things like the recent speculation bugs: it breaks my heart to see so many *great minds* with better things to do for the OS, get mired in bit-flipping on cache invalidation.  Even as an outsider who's never hacked on that subsystem, these problems look like an outright inane distraction for FreeBSD or any OS.  So again here, I fail to see why playing by Intel's rules (with NDA) helps us all transcend past these issues?

> 
> > Frankly, if the FreeBSD Foundation appears to have handled taking on 
> > the burden of NDA's which enforce embargos on critical security fixes 
> > to the FreeBSD codebase, why would various fixes be prevented from 
> > being applied to the codebase by developers not under NDA, (and not in 
> > the know)?  Excepting that this activity could naturally lead Intel 
> > legal or others to believe that NDA holders are leaking info, is there 
> > anything which actually prevents fixes from going into the FreeBSD 
> > codebase from non-NDA committers?
> >
> 
> Leaks have two negative side effects.  First and foremost they give some 
> number of potential attackers a window of opportunity, that's why the 
> industry has coordinated releases of vulnerabilities.  Secondly, leaks 
> lead to distrust and we'd like these folks to trust us, so that we can 
> have a good working relationship, and get information sooner rather than 
> later.

I can't agree more regarding leaks.
If one agrees to the rules, one must play by them - or face the consequences (loosing trust, here).

However, I'm well beyond that here.  I'm questioning the decision to engage the NDA's in the first place.

> 
> > Last question- stance on Intel::
> > Aside from the fact that I'm typing this on an Intel CPU, and this 
> > message has proably only touched Intel CPU's on it's way to your 
> > eye-balls reading it, why are we treating Intel so special not to hold 
> > them accountable or even to ask them to provide user manuals and 
> > documentation for the CPU's we buy from them, particularly in the case 
> > of security problems with undocumented features?
> >
> 
> I'm slightly confused here, the chip manuals are free and available on 
> their web site, including updated errata.  Of course reading them won't 
> help you in most cases.  Turns out a modern CPU is complicated :-)  

I understand a modern CPU is complicated- no doubt.  Yet, doing reasonable OS work for Intel CPU's requires NDA to get proper docs?

I mean, I'd much rather just start seeing more pressure on Intel to simply publish more transparently.
And if our relatively smaller *BSD voice isn't enough, I'm certain there are plenty of folks in the Linux multiverse who would agree:
(Again, Linus reaction to Meltdown/Spectre <https://lkml.org/lkml/2018/1/3/797>)

> It 
> would be hilarious to run an ML system against their manuals to try to 
> find security issues, but I've got other things to do...

I think you just threw a very good project out to someone in the world then :)
Any capable hands who have read this far: I'd be delighted to play support/sysadmin role for any required servers in the NYC*BUG cabinet for someone doing this sort of work!  :)  (call it d-messy-db, to match nycbug dmesgdb?)

Best,
.ike


> 
> Best,
> George
>