I discussed the Clojure thrush operator once before. The way that I have been using it lately is to operate on a stream of data, perhaps condensing the stream at the end into one value. This suggested to me that I should be able to operate on the stream of data in parallel, since all of the functions that I am using are pure.

What I have done with the new package is to create a new macro called ||->>. This operates just like the ->> macro already in Clojure, except that it splits the data in the stream and runs it in parallel. Behind the scenes, it uses the Clojure pmap function, which advises that it should only be used when the operation being done is CPU-intensive enough to exceed the coordination cost. Since multiple functions are put together by my macro before they are passed to pmap, following this advice should become easier.

As an example:

(||->> data
       (map big-function)
       (filter odd?)
       :merge
       (reduce +))

is the same as

(->> data
     (map big-function)
     (filter odd?)
     (reduce +))

but everything before the :merge statement is executed in parallel.

Of course, the source is on my github. The jar can also be easily downloaded from clojars.

The project I've been working on has partly been a struggle to manage resources. Thus, I've needed a good way to keep some of the resources in memory and get rid of the ones that are no longer relevant. Using a hash-map data structure is always nice, so I've implemented a variant of a hash-map that has an upper limit on its size. When you try to add an element that would cause the map to have too many elements, it kicks out the element that was least recently added or modified.

(apply lru-map 4 (range 20))
;=> {12 13, 14 15, 16 17, 18 19}

Sometimes, you want to do something with an element as it gets kicked out. For that, there is lru-map-with. This takes two extra arguments – a function that operates on some "state" and the element that is getting kicked out and the initial "state". (Behind the scenes, this "state" isn't really state, but it is helpful to think of it as state.)

(apply lru-map-with 4 conj [] (range 20))
;=> {12 13, 14 15, 16 17, 18 19, :lru-state [[0 1] [2 3] [4 5] [6 7] [8 9] [10 11]]}

Again, this little package is on github and the jar is on clojars.