I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome.
The concept of lines is a human one. It is how humans parse.
If humans could parse without needing the concept of a "line" then, e.g., there would be no problems with programming in C which has very shaky support for the concept of "lines".
But there are problems as we all know. It's proof that people do need to think in terms of "lines". Even though the computer does not need them. The only "problem" with this is that people are not computers.
I'm not sure anyone outside of the most unrealistic nerds would agree that this is a "problem".
1. As for the paper, I can think of at least one tool/language to work with text that does keep data in a binary format while one performs a series of transformations. It's not line-based. And I would guess that Mr. Pike does not know how to use it. It's fast and efficient. Probably faster than sam.
> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is?
My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern.
Let me ask you this; how would you do the equivalent of this hypothetical command?
$ ls | structured-grep 'file.size > 1M'
The answer is that you can't in today's world without writing a parser (or some code that calls readdir/stat manually). That is the problem with unstructured text.
Text formats like CSV that seem simple actually end up being hugely complicated once you push them to their limits. Nothing is worse than software that breaks once something unexpected happens, like a string that contains an embedded comma.
"ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly:
ls | find -size +1M
On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do:
For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.
>>Let me ask you this; how would you do the equivalent of this hypothetical command?
>> $ ls | structured-grep 'file.size > 1M'
# find . -maxdepth 1 -size +1M
really, flat text is fine. Maybe it's not perfect, but it's good enough that most want to not add any complexity to it that would make it incompatible.
And if you really need that complexity, it's usually worth whipping up a parser for.
"find" is a poor man's "structured-grep." It provides a bunch of functionality for filtering a result set, but is totally specific to lists of files. You can't use find with ps, netstat, iptables, ifconfig, or any other command-line program that produces a list of records.
> And if you really need that complexity, it's usually worth whipping up a parser for.
No work is worth doing if it could just as easily have been avoided.
The vision in my head has less complexity than the status quo, not more. How many flags does "find" have? One for every field name you can filter/sort by. Done right, a "structured-grep" that can grep on any field sent to it is much, much simpler.
I think I see your point, and it has merit ... but ... :)
in a way, the 'find' program is like what you envision, except it's just for files. That means that someone somewhere along the road, had the same idea/problem (but limited to files) as you and whipped up a parser to produce that meta-data. That particular parser proved to be so useful to so many people, it became it's own program.
There's more than 40 years of sofware-"evolution" contained in unix, and apparently retrieving structured on the command-line has only proven universally useful for files. Unix has outlived many at the time more modern operating systems, and I think it's partly because it lacked a "grand unifying vision".
Instead it has a "small, quick&dirty unifying vision" of which "flat text processing" on the commandline is a central part. It has turned out to be the greatest common denominator for being able to write programs, that might be quick and dirty oneliners, but ultimately they got the job done. And only those tiny little utilities that proved to be universally useful were developed into bigger more stuctured programs.
I'm not saying you idea is without merit, but it does apply the principle of "this concept A is useful for this particular problem-set. Let's apply it natively to all problem-sets so it can be useful there too!" ( in a way like Java did with the OO concept).
When simpler visions and concepts are actually implemented in the end it usually turns out one has been replacing witchcraft with voodoo.
The solution I often contemplate is to address the process by which the data is created. (Let's assume we're talking about data you would find on the web.) That is, why don't we impose rules on that process? Why can't we "mandate" that the output be structured data from the get go? Instead we allow vast quantities of unstructered data to be created and then we try to normalize it. I know this is a radical view, but it could make sense in some circumstances.
I disagree with your comment about CSV. But it's impossible to have a meaningful argument unless you provide an example: Give me a job to do, a CSV file and let me have a go at it. I'm serious. Post a link to a CSV file, define a task and let's see what we can do just using plain ole UNIX. Could be a fun exercise.
As for your hypothetical command, I do not understand what is so difficult about this. The stat command is what you want, not ls. No self-respecting UNIX user would parse ls when he can use stat (I recommend the BSD one over GNU.).
But here's what I would do:
1. If your UNIX filenames have spaces in them, rename them. There is no sensible reason to leave spaces in filenames in UNIX. Fix this first before it becomes a problem.
2. Write a one-liner and save it as a function, perhaps in your .profile, or maybe in RCS, or save it as a script. There's so many ways to manipulate output as a stream. Pick one that suits your tastes. That's the beauty of UNIX. Make your own solutions as you go. There is no right or wrong answer. It is a form of customization. My choice will no doubt make some people cringe. Assuming there's no user named "[0-9]M":
whatever(){ ls -lhS |tr '\011' '\040'|sed '/ [0-9]M /!d' ;}
or save what's between the brackets as a file named "whatever". Maybe you save it in a directory called "x" and add that to your PATH. Then you do
. whatever
Of course how long the list of files is going to be makes a difference. I might take a different approach if the output was going to be an enormous list.
There are so many ways to get what you're after. The point is that you should be able to tap out a one-liner that does the job. Maybe it takes a few iterations to get the right output. Tweak it until the output is what you want. Viewing command-line history is perfect for seeing the process of creating a one-liner to manipuate output. You can see the line grow incrementally as you build it, until you finally have the output you want. This sort of history allows you to go back to any stage in the process. If you're a vi fan, you can use vi-mode on the command line to move around the line quickly as you edit. Eventually you can hit "v" and edit the thing visually in your EDITOR, then save it. I've built over 700 useful functions this way and the number keeps growing.
I do understand there should be a way to "extract" the file size column the way Pike decribes in the article. To do this I think you have to free yourself from "line-oriented" thinking and imagine another type of structure. And I think using another language you can do it. But for something as simple as this -- manipulating ls command output (cf. manipulating large datasets) -- an "ugly" one-liner suits me fine. The more you use the boring old utilities the more you can get them to do.
Regular Expressions are indeed "crude". But, to me, that is just fine in a lot of cases.
Yes try working on the full OSI Stack you had to learn asn.1 just to beable to read what a concrete decode was doing.
Though my boss did impress me by watching an OSI transaction in flight on our network monitor stop it and point to a dword and say thats wong and its Sprints broken x.400 implimentation.
A company that relies on a bot that accesses others' resources to make money and at the same time relies on reCAPTCHA to frustrate other bots from accessing it own resources.
It may be easy to do today, but, going forward, how do we determine which bots are "good" and which ones are "bad"?
Clearly, simply being a "bot" does not imply "bad" intent. If it did then we should all be blocking search engine bots.
Yet this is what reCAPTCHA does: it blocks not based on intent, but based on the characteristic of being a "bot".
It's easy. One bot reads a text file and only accesses the resources outlined in the file. The other maliciously tries to eat up resources without restraint. Good - Bad.
Like I said, it's relatively easy today. But how about going forward?
There will be lots more bots and lots more usages. Things might not be so simple. For example, some sites might exclude all search engine bots except a chosen few despite the fact they all honor robots.txt and behave essentially the same.
(BTW, if by "text file" you mean robots.txt, isn't that an exclusion list? You seem to be saying it's an inclusion list.)
What would a malignant dictatorship be like?