I spent some time working with the people at the advanced computation group there. It was a short time, and it was all in the formal systems/automatic analysis area, so this is a summary of what I picked up within my limited view...
The thing is that software's taken at a very different angle from there as in most parts.
First, the ones who know exactly what's supposed to happen aren't software people, they're engineers who don't write code. At least not real-time embedded control code.
Second, there are too many different people who's interests come together in the same codebase. I.E. lots of different engineering disciplines at once. Each one has their own work to do separate from the programming. You can get specifications & equations out of them, but the code's a different group.
Third, the need for high assurance is important. Note the word 'Assurance.' If something bad happens at Nasa, there's a lot of news and possibly a statement by The President. Being that this is a government agency building stuff, it's going to have similar problems to DoD agencies building stuff. That is, an incredible amount of the project will be spent to reduce the variability of the project. Even if the sum cost is many [, many]* times the likely variance.
The main space they have to work with is in mechanized assurance systems. Sadly, the formal systems out there aren't terribly evolved, as most people just test & debug their systems to get it right.
As for contributions to software, or lessons learned, there are some very good experiences & things learned for people who want to build better tools to make this stuff easier. E.g. higher-level languages that look more like Matlab than embedded C (don't get me started on Matlab-C converter). Stuff where you can be sure that no matter what instruction sequences you give to a robotic arm, that it never ends up in a position where one of joints is locked.
When I was there, the attempt was natural language to CSP traces, to be run through a theorem prover there later. We had a total of two grad students working on it over a summer, after an initial prototype was written earlier.
As other posts on HN have mentioned, and as one can surmise from the chatter about Nasa, it's costs are very closely looked at by a lot of people -- many who don't understand/know/care about the specific problems Nasa's got. That creates a whole new set of problems. Being observed under a microscope and then being hit for everything a nontechnical senator can get political points for hitting you for will make you start working in pretty strange ways.
Would you say the development process fit something like the waterfall model?
It sounds from the article like it's a very waterfally method, but they can get away with it because they don't have the same constraints and lack of clear specifications that most software does.
You can't show a program to bug free. You can't prove a negative.
"the last three versions of the program -- each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors."
Translation: in each of the last three versions, only one error was found, in the last 11 versions, only 17 errors were found.
"the shuttle group now finds 85% of its errors before formal testing begins, and 99.9% before the program is delivered"
In order to know you've found x% of the errors in a program, don't you have to have already found 100% of them? If I find 9 errors in a program, how can I know that I've found 90% of the errors unless I also know that there are 10 errors in it?
A lot of what the author says sounds impressive, but I'd argue a lot of it is either wrong or not even wrong (http://en.wikipedia.org/wiki/Not_even_wrong). Even granting the author some artistic license, he's going pretty overboard in a lot of his claims.
I'm sure the software produced by this group is less error-prone than your typical desktop or web app, but it doesn't sound like they're writing innovative software either. Most of what they do sounds like either mathematical calculations or driving hardware devices. Both areas which seem to me to be pretty well explored at this stage.
I would disagree with this. I'm not an expert in this field at all, but from what I've read, the software they write is tested to the point where every single possible input is tested for its assumed output on every subsystem. They write software that is mathematically proven to not fail.
My only point of reference for how they work is, admittedly, this article. I'm far from knowledgeable about formal methods but, in my understanding, if they're writing provably correct programs the code should, by definition, contain no 'errors'. (That is, it should always "do the thing right". There's no guarantee they've designed it to "do the right thing". Design errors may be exactly what the article is talking about.)
I'm not sure exactly what you mean when you say "every single possible input is tested". I would claim that this is not only impractical but impossible. The range of possible inputs to a system is infinite. This may sound pedantic, but in my experience as a software tester all sorts of failures often pop up when you look even slightly outside the range of values that seems sensible/likely/exhaustive for a given input.
I work on a project for a Japanese university which is of comparable complexity, at least in kloc size, to the Space Shuttle.
It most definitively does not have only one bug in it per release.
However, and this is a pretty key concept: we don't bill them half a billion a year for it. (Not everybody, after all, has access to a virtually unlimited slush fund whose primary purpose is to line the pockets of defense contractors in a PR-friendly way.)
Ugh, my bad. After a 16 hour day in the salt mines I was not doing math right.
I spent it, incidentally, supervising an outsourcing team which can't read the language the project is written in -- I kid you not, I actually had to say "Thank you for your inquiry about unknown red text under the login screen. The red text you have circled is not an error message. It is a weather report. The text is red because the forecast is potentially lethal. Closing as 'By Design'."
"John Munson, a software engineer and professor of computer science at the University of Idaho, is not quite so generous. 'Cave art,' he says. 'It's primitive. We supposedly teach computer science. There's no science here at all.'"
I'll give him his clever phrase, despite being technically false. What we create with our knowledge of computer science (our programming) is not the same thing as computer science itself.
This is an interesting way to think about programming, though. Compare it, for instance, to metalworking. The craft itself, clearly, is distinct from the science of metallurgy, but at the same time they have evolved alongside each other. I wonder if programming and computer science will go through a similar progression. As metalworkers just were learning to manipulate metal, we bit-twiddled and hand-assembled; as metalworkers evolved their tools and meticulously forged their creations, we created compilers but still hand-coded our programs. The next big step in metalworking was the automation of their craft via factories; is this way code will go as well? Will we simply guide computers as they produce their own code? And what advances in computer science would be necessary for this to happen?
Edit: Please forgive me if my history of metalworking is slightly incorrect, I just realized I mostly pulled it out of my ass.
Pseudocode-to-realcode converter, or something vaguely like it.
Honestly, we are already on our way there thanks to the popularity of 'expressive' languages like Python, Ruby, etc. I don't think the transition will happen overnight or anything.
By 2050 we might end up just coding everything in a very powerful dialect of Lisp.
Well, obviously we're progressing in terms of the ratio of human work to computer work, but I was thinking of a program that, rather than taking actual algorithms, takes specifications, and not exact ones like Prolog and the like require--some sort of natural language processing/artificial intelligence kind of thing that understands something like "I want a Firefox extension that will automatically do...". Of course, it would have to have knowledge about different protocols/methods/algorithms/etc., and we'd have to give it that knowledge manually at first, and sometimes you'd probably have to work with it, especially in the development of new algorithms. There would probably still require some significant work to put a non-trivial program together, but it would mean a lot less learning UI libraries, putting together basic systems (I'm thinking of games especially here), etc.
This isn't to say you couldn't put together programs manually still, of course--it'll always be fun. But you could get rid of most of the tedious bits. Also, Lisp (or something Lisp-y) seems like the most likely candidate for putting code together this way.
I remember discussing with a friend of mine who, at the time, was a recent hiree to Boeing. After a few months(!) of introduction and systems learning, he was tasked with writing a small portion of code. I don't remember the specifics, but it was a very specific task (e.g., "write code to display the fuel level in liters.)
After a day or so, he turned in his code (something like 30 lines) and was chastised for having done too much. "What are you trying to kill people?!?"
Turns out the regular output of their developers was something like 5-10 lines of code per day, and only certain days of the week. Finished products were peer reviewed before being QAd.
At the time, I was just learning early Ruby, and getting used to the concepts of rapid development, and it seemed so strange. Of course, if my webapp breaks, there aren't any lives on the line.
Also, for those who don't notice. This article is 12 years old. It doesn't take anything away from the article, but I first read it three years ago and got excited about the topic before I noticed that it was a bit aged.
I remember hearing stories in undergrad about code written on paper and mercilessly reviewed so that it would work perfectly the first time it was compiled. Can't remember the specifics and may have been an urban legend -- but this is the closest thing I've seen recently.
I wouldn't be surprised if this was done back when things were coded on punch cards and submitted for overnight processing. Iteration then was very time expensive so it would have made sense to check and re-check as much up front as possible.
My father (who graduated with a BS in EE in 1970), said he was only allowed one hour of computer time a week, so the students did everything by hand, including scouring for bugs, before they got near a keyboard.
pg: "It's natural for organizations to learn from mistakes. The problem is, people who propose new checks almost never consider that the check itself has a cost."
FTA: "The process is so pervasive, it gets the blame for any error -- if there is a flaw in the software, there must be something wrong with the way its being written, something that can be corrected. [...] Does a question need to be added to a checklist?"
Modern AAA computer games have at least double the code size, are produced in a fifth of the time, for the price of roughly 1 year of the NASA group's budget.
This is why we need NewSpace. It would actually be better to fail more often and more cheaply.
Of course, that's impossible for a Congressionally-funded organization.
I bet you are talking about the fast prototyping that danc (lostgarden.com) talks about. You probably don't want to use a buzzword that doesn't even give sensible results on a Google search.
Game programming is one field where "good enough" is not just a particular philosophy of coding, it is a law of the gaming universe. I would say that the "assurance" requirements of games are on the other end of the spectrum compared to NASA software.
On the other hand, NASA has recently proven with some of its probe projects that Soviet-style "shotgun" probes can be better than trying to make the perfect probe. Making 3 "good enough" probes and having only one or two actually accomplish the mission can cost less than making one perfect probe which even then may or may not be successful.
SpaceX, XCOR Aerospace, Scaled Composites, Armadillo Aerospace, etc.
My comment was actually based on lurking in on discussion with rocketeers who have worked both inside and outside the government.
As someone who's only worked in relatively small, private-sector software companies, it's been very eye-opening to get an insight into how and why government managers make their decisions.
The thing is that software's taken at a very different angle from there as in most parts.
First, the ones who know exactly what's supposed to happen aren't software people, they're engineers who don't write code. At least not real-time embedded control code.
Second, there are too many different people who's interests come together in the same codebase. I.E. lots of different engineering disciplines at once. Each one has their own work to do separate from the programming. You can get specifications & equations out of them, but the code's a different group.
Third, the need for high assurance is important. Note the word 'Assurance.' If something bad happens at Nasa, there's a lot of news and possibly a statement by The President. Being that this is a government agency building stuff, it's going to have similar problems to DoD agencies building stuff. That is, an incredible amount of the project will be spent to reduce the variability of the project. Even if the sum cost is many [, many]* times the likely variance.
The main space they have to work with is in mechanized assurance systems. Sadly, the formal systems out there aren't terribly evolved, as most people just test & debug their systems to get it right.
As for contributions to software, or lessons learned, there are some very good experiences & things learned for people who want to build better tools to make this stuff easier. E.g. higher-level languages that look more like Matlab than embedded C (don't get me started on Matlab-C converter). Stuff where you can be sure that no matter what instruction sequences you give to a robotic arm, that it never ends up in a position where one of joints is locked.
When I was there, the attempt was natural language to CSP traces, to be run through a theorem prover there later. We had a total of two grad students working on it over a summer, after an initial prototype was written earlier.
As other posts on HN have mentioned, and as one can surmise from the chatter about Nasa, it's costs are very closely looked at by a lot of people -- many who don't understand/know/care about the specific problems Nasa's got. That creates a whole new set of problems. Being observed under a microscope and then being hit for everything a nontechnical senator can get political points for hitting you for will make you start working in pretty strange ways.