[talk] Embargo/Foundation questions, Intel CPU/speculation bugs

[George Neville-Neil]

The short answer is that without the NDAs we'd be shut out of a lot more 
things than we're already shut out of.  One of the reasons we have 
excellent support for high speed NICs (SolarFlare, Chelsio, Mellanox 
etc.) is due to the FF signing NDAs, getting things in place and then, 
when the time was right, getting developers at those companies to do the 
right things for us.  We can see this in other areas as well, graphics 
etc.  It also helps that companies like NetFlix, Hudson River (my former 
employer), Netgate and others have stepped up to demand access for 
FreeBSD, but access comes at a price, and, it's relatively impossible 
for a lone developer to get this stuff in place, at least for the 
hardware that matters.

I have yet to see a negative effect.  Generally there aren't that many 
people who can effectively do the work that an NDA is useful for.  It's 
not like there are 100 competent graphics driver writers on Earth, and 
if there were, I'd simply get them all under an NDA through the FF and 
off we'd go.  The fact of the matter is we're talking about < 10 people 
per area or per device provider (even Intel) who really need this kind 
of access.

In the end if we want to be able to be at the cutting edge of new 
hardware we're going to have to play ball with these folks, and that's 
what the FF is there for, to be the folks who have lawyers, and sign 
contracts and NDAs and help the community engage with the folks making 
the chips.

Best,
George

On 19 Jun 2018, at 2:36, Isaac (.ike) Levy wrote:

> Appreciate your time and thoughtful response GNN, 2 big questions 
> remain, specific to FreeBSD:
>
> - What benefit have the vendor NDA's had for the FreeBSD project 
> (particularly Intel)?
>
> - What negative effects have vendor NDA's had on the FreeBSD project?
>
>
> These are the only two questions that really matter, more commentary 
> for anyone with time to spend on it:
>
> Worth me stating ahead of my commentary (for folks on list who don't 
> know me or GNN), I have profound respect for GNN's epic work on 
> FreeBSD, from core Networking, to comprehensive OS internals, his code 
> touches us all in some way we cherish.  We're all better off because 
> of GNN's publishing and educational work, as well as the time he's 
> spent toiling for the Foundation- the list goes on and on.
>
> Just wanted to state this up front so it's clear in this discourse 
> that I'm aiming to keep this thread and any disagreement focused on 
> constructive understanding of the issues, and all our common aim to 
> keep working for *BSD systems to be as strong as ever.
> Even though I'm a mere mortal, I'll certainly say that GNN and myself 
> are every bit as invested in *BSD thriving, as so many of us are here.
>
> On Mon, Jun 18, 2018, at 5:24 AM, George Neville-Neil wrote:
>> Comments in line...
>>
>> On 18 Jun 2018, at 3:59, Isaac (.ike) Levy wrote:
>>
>>> Hi All,
>>>
>>> I'm sure folks have been tuned in to the fragmented security news of
>>> the week: Intel and more speculative execution security flaws.  (If
>>> not, check the URL footnotes below).
>>>
>>> With that, questions for list, genuinely looking for some logical
>>> answers in favor of the NDA/embargo on critical fixes we've seen in
>>> recent months:
>>>
>>> - What do folks think about these Intel NDA-covered embargo, (or any
>>> vendor for that matter)?
>>> The embargoed information really doesn't seem to be doing us all any
>>> good at all here as users.
>>> I'd really love to hear any constructive reasons why the NDA's are
>>> necessary or good for any project developement?!  How does this help
>>> the world?
>>>
>>> - Specific to FreeBSD, and the Foundation: I've been a huge fan of 
>>> the
>>> FreeBSD foundation since it's inception.  To separate assets from 
>>> the
>>> FreeBSD codebase was certainly born in the SCO era, with the the USL
>>> vs. BSDi era in mind, etc... this part of the mission of the
>>> Foundation has always made a lot of sense to me- separating legal
>>> assets from the codebase for the longevity of the codebase itself.
>>> The Foundation has done some excellent work, from handling 
>>> donations,
>>> coordinating conference attendance, project grants, developer 
>>> hardware
>>> and donations- and of course supporting the amazing RELENG work as a
>>> J.O.B. job for Glen Barber (et. al.).  This has produced amazing
>>> things for FreeBSD.
>>>
>>> Yet, what are the *boundaries* of the mission of the FreeBSD
>>> foundation?
>>> Does the foundation even facilitate vendor relationships and 
>>> shepherd
>>> NDA's for developers, or am I just totally mistaken here?
>>
>> Indeed we do.  In fact one of my first tasks when I joined the FF was 
>> to
>> get a set of NDAs in place and a process by which any FreeBSD 
>> developer
>> could be given access to NDA'd information from various vendors.  We
>> have these types of relationships with several folks including Intel,
>> ARM, NVidia and others.
>
> Thanks for clarifying this, coordinating an shepherding the NDA's with 
> proper legal counsel sounds like a massive task!
> The Foundation certainly does seem to be the place for this level of 
> coordination to happen- (to keep the codebase in a known state with 
> regard to various legalities).  Far better than over in Linux-land, 
> where who knows who's got what NDA priviliged commit coming into what 
> parts of the tree...
>
> For example, in the Linux Kernel, it became quite obvious when 
> Spectre/Meltdown was about to hit the street, that even Linus wasn't 
> at all privvy to what was happening:
> https://lkml.org/lkml/2018/1/3/797
> (an interesting post on many levels.)
>
>>
>>> I mean, a while back they were becomming involved with humanitarian
>>> projects- wonderful stuff, but not what I donate to the FreeBSD
>>> Foundation for- and certainly quite a deviation from strategically
>>> shepherding the legalities of NDA's with Intel.
>>> These days, I see a lot of great press stuff on the Foundation site,
>>> but I don't see any clear "here is why we help sign NDA's with 
>>> Intel."
>>>  I can't even find mention of "NDA" on the Foundation site.
>>>
>>
>> It's not something we'd put on the front page.  I and others talk 
>> about
>> this frequently at conferences and developer summits.  This kind of
>> stuff often takes time with our lawyer, and so we don't just do it 
>> for
>> everyone.   The typical process is that a developer comes to us with 
>> a
>> need and we work with them to get this in place so they get the
>> information they need.
>
> I get that- but to clarify, I'm certainly not looking for a sign 
> saying "get in line here for NDA privileged information".
>
> As a user of FreeBSD, with my own concerns, I simply would prefer to 
> understand more clearly what these NDA's impose on the codebase, and 
> also what they provide, to the FreeBSD project.  Worthy of a blurb 
> describing the scope of this critical effort under the "What We Do" 
> tab, etc...
>
>>
>>> Assuming FreeBSD devlopers/Foundation does have NDA with Intel
>>> (others) for chip documentation, as well as security 
>>> vulnearabilities:
>>
>> Chip docs mostly don't need NDAs, although some graphics stuff still
>> does, the security stuff from the FF side, is handled by Ed Maste and
>> one or two of our contractors, as well as by members of the community
>> who are contacted by Secteam.  Secteam and SO are the main project 
>> folks
>> handling this stuff.
>
> I may be wrong here, but chip docs for stuff like these latest 
> speculative execution and cache bugs, don't appear to be well 
> documented  at all, (even with NDA)?  Is this a qbit- whereby the chip 
> is simultaneously documented, and not documented, and also 
> concurrently has a state which requires NDA?  I kid- I understand the 
> scale and complexity of publishing relevant docs on fast moving 
> hardware in our world- yet I don't understand where it's acceptable to 
> step over a line into NDA territory.
>
> Also, from Theo's BSDCan talk, the fun clarifying vocabulary of 
> "Chicken Bits", (features on chips which are part of future features, 
> baked into the die early- or perhaps never seeing the light of day- 
> these require NDA- in recent years exposed only through security 
> vulns, forcing vendor response documentation.)
>
> Vulnerabilities that pop up from these "Chicken Bits", have had a 
> profound effect on my professional life as a sysadmin/engineer in 
> recent years.
>
> To date, I don't see evidence that any NDA process here is mitigating 
> anything for anybody, except perhaps mitigating dips in Intel's stock 
> value.  I'd love to see the evidence which can change my mind!
>
> Additionally, these issues are far from new or unique to the *BSD's, - 
> as in the BlackHat/Defcon/CCC/ShmooCon/etc universe, Intel CPU 
> vulnerabilities have been more frequent and more damaging every year 
> for some years.  (And often, the technical details are absurdly inane- 
> the issues are boneheaded- the secrecy and profound impact of the 
> vluns is what's made them such a hot topic).
>
>>
>>> For Meltdown/Spectre, well, I don't know if if NDA's were in place 
>>> at
>>> the time- but if it was, it didn't seem to help FreeBSD get any
>>> mitigations out- (even for several weeks past public disclosure).
>>
>> The biggest challenges here are not NDAs per se, but with hands and
>> communication.  Intel is big, and FreeBSD is not Linux so there are 
>> two
>> challenges there, and we're continuing to work with Intel to get the
>> right people to talk to at the right time.  Of course, in 6 months 
>> those
>> people will change jobs, so, wash rinse repeat.  The hands problem is 
>> a
>> serious one since capable volunteer security experts who can even
>> understand the problems of Spectre are few and far between.
>
> With all due respect to everyone working toward fixes/mitigations, the 
> esoteric nature of many important security vulnerabilities cannot ever 
> be comprehensively handled by any single organization.  I do not see 
> evidence here that maintaining NDA-backed secrecy is helping increase 
> the number of hands- capable volunteers who are subject matter 
> experts.
>
> In my 20 years running internet-facing infrastructure, the most 
> effective security responses in the world have often been 
> mitigated/resolved by the parties which care most about them, (and 
> understand them), not by teams dedicated to be solely responsible for 
> solving everything.
>
> And for things like the recent speculation bugs: it breaks my heart to 
> see so many *great minds* with better things to do for the OS, get 
> mired in bit-flipping on cache invalidation.  Even as an outsider 
> who's never hacked on that subsystem, these problems look like an 
> outright inane distraction for FreeBSD or any OS.  So again here, I 
> fail to see why playing by Intel's rules (with NDA) helps us all 
> transcend past these issues?
>
>>
>>> Frankly, if the FreeBSD Foundation appears to have handled taking on
>>> the burden of NDA's which enforce embargos on critical security 
>>> fixes
>>> to the FreeBSD codebase, why would various fixes be prevented from
>>> being applied to the codebase by developers not under NDA, (and not 
>>> in
>>> the know)?  Excepting that this activity could naturally lead Intel
>>> legal or others to believe that NDA holders are leaking info, is 
>>> there
>>> anything which actually prevents fixes from going into the FreeBSD
>>> codebase from non-NDA committers?
>>>
>>
>> Leaks have two negative side effects.  First and foremost they give 
>> some
>> number of potential attackers a window of opportunity, that's why the
>> industry has coordinated releases of vulnerabilities.  Secondly, 
>> leaks
>> lead to distrust and we'd like these folks to trust us, so that we 
>> can
>> have a good working relationship, and get information sooner rather 
>> than
>> later.
>
> I can't agree more regarding leaks.
> If one agrees to the rules, one must play by them - or face the 
> consequences (loosing trust, here).
>
> However, I'm well beyond that here.  I'm questioning the decision to 
> engage the NDA's in the first place.
>
>>
>>> Last question- stance on Intel::
>>> Aside from the fact that I'm typing this on an Intel CPU, and this
>>> message has proably only touched Intel CPU's on it's way to your
>>> eye-balls reading it, why are we treating Intel so special not to 
>>> hold
>>> them accountable or even to ask them to provide user manuals and
>>> documentation for the CPU's we buy from them, particularly in the 
>>> case
>>> of security problems with undocumented features?
>>>
>>
>> I'm slightly confused here, the chip manuals are free and available 
>> on
>> their web site, including updated errata.  Of course reading them 
>> won't
>> help you in most cases.  Turns out a modern CPU is complicated :-)
>
> I understand a modern CPU is complicated- no doubt.  Yet, doing 
> reasonable OS work for Intel CPU's requires NDA to get proper docs?
>
> I mean, I'd much rather just start seeing more pressure on Intel to 
> simply publish more transparently.
> And if our relatively smaller *BSD voice isn't enough, I'm certain 
> there are plenty of folks in the Linux multiverse who would agree:
> (Again, Linus reaction to Meltdown/Spectre 
> <https://lkml.org/lkml/2018/1/3/797>)
>
>> It
>> would be hilarious to run an ML system against their manuals to try 
>> to
>> find security issues, but I've got other things to do...
>
> I think you just threw a very good project out to someone in the world 
> then :)
> Any capable hands who have read this far: I'd be delighted to play 
> support/sysadmin role for any required servers in the NYC*BUG cabinet 
> for someone doing this sort of work!  :)  (call it d-messy-db, to 
> match nycbug dmesgdb?)
>
> Best,
> .ike
>
>
>>
>> Best,
>> George
>>