Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Map-reduce enthusiasts were wrong: the two fundamental functions are not map and reduce, rather zip-with and group-by. If you want to do real work done.

I've heard Google learned this lesson and their data processing now rests upon these two.



`zip-with` and `group-by` just represent the most common uses of `map` and `reduce` respectively. Many data processing frameworks (cascading, spark, storm, etc.) build these abstractions right on top of map-reduce, and sometimes the layer is pretty thin.

Heck, zip-with in clojure is literally just map:

    (map + [1 1 2 3] [1 2 3 5]) ; => [2 3 5 8]


For the curious, here's a paper on the universality of fold (reduce) www.cs.nott.ac.uk/~gmh/fold.pdf


It's not zip-with that you wrote. You illustrate that given not enough tools, you lose the grip on the process.

zip-with is

  (zip-with #(+ % 1) [1 1 2 3]) ; => [[1 2] [1 2] [2 3] [3 4]]


This is not zip-with. zip-with is

  zip-with fun list1 list2…
  = [ fun list1_0 list1_0 list2_0 …,
      fun list1_1 list2_1 list2_1 …,
      … ]

In clojure zip-list ≡ map. Your example is just “map (λx → [x,x+1])”


Am I missing something? From my understanding ZIP should be over more than one sequence, otherwise it's just a normal map really.

Certainly the Haskell examples I have seen would strongly reinforce my understanding.


Well, it zips a sequence with another derived sequence.


Like

    (defn zip-with [f xs] (map (juxt identity f) xs))
or something else? Without showing what your zip-with (and all the other 'missing' things) implementation looks like, people need to guess what they are and why these might be useful.


Down voted? Really come on.


Lets race all the way down, seems like it's a meaningless metric anyway.


  (let [i1 [1 1 2 3]
        i2 (map #(inc %) i1)]
    (map vector i1 i2))

  => ([1 2] [1 2] [2 3] [3 4])
All standard tools. I don't know what your definition of zip-with is, but it certainly differs from the one I'm used to from Haskell, and it seems like other people disagree with you as well (but I suppose you'll just consider me 'constrainer' or whatever as well).


Ah, I see. I'm not sure I'd call it zip-with, but it's definitely familiar and useful. Scalding calls it map-to[1], and Trident's `each` can work this way as well.

[1] https://github.com/twitter/scalding/wiki/Fields-based-API-Re...


So:

  (map #(vector % (inc %)) [1 1 2 3]) ; => ([1 2] [1 2] [2 3] [3 4])
Is it really that bad the function is missing?


Yes. When you have no hammer you don't recognize nails.


Can you support your claims by evidence instead of trite idioms? A piece of code made drastically simpler using zip-with should be enough.

Don't get me wrong, I just want to learn, but your "hammer" just looks like a regular one to me without an explanation.


You keep repeating this, and I find it ill-thought. Clojure offers lots of hammers, saws, drills and other tools.

But you seem like you want millions of them -- a hardware store all your own.

More is not always better, and you don't need to have a tool for every little specialized case (nor it's good for your coding -- fewer but solid abstractions and a set of tools that can be combined to build the most elaborate structures are better than an explosion of tools for everything).


(for [i [1 1 2 3]] [i (inc i)])

is zip-with lazy like for?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: