Showing posts with label Clojure. Show all posts
Showing posts with label Clojure. Show all posts

Wednesday, March 26, 2014

Clojure HashMap usage within a Java class

I attended the Hack (Make!) the Bank hackathon that took place on Level 39 (One Canada Square in London). My team decided to make a funds reallocating application that makes automatic payments once an account is topped up. Long story short, we got stuck at interoperability with the Java classes providing Paypal access.

We made the mistake of inserting a Clojure hash map directly to the Java class, which seemed to work but the moment we switched to paper account in the Java class (by modifying the incoming HashMap argument) the system threw an exeption. Ben from Erudine Financial realized that the structure we were passing was inmutable so we just created the proper Java HashMap instance.
(java.util.HashMap. {})
Remember that Clojure's {} is actually a clojure.core.PersistentHashMap.

This situation happens especially when clumsily using proof of concept code in production.

Saturday, February 8, 2014

Use Java JARs within Clojure projects

I am creating a clojure wrapper for the excellent library from Pekka Enberg Falcon, which will provide Denarius with FIX protocol communication capabilities. The new project's name is, obviously, falcon-clj.

Although the ideal packaging would be to include Falcon as a Git submodule, I decided to pre-compile Denarius into a JAR stored into the /lib directory to avoid Leiningen complications in the starting phase and be able to focus on the implementation.

I compiled the library and pushed it into Github and then I received my new laptop, so I installed everything from scratch, including the JDK, and cloned everything back into the new laptop. The next stage would be to use the REPL to quickly get used to Falcon.
=> (import [falcon.fix Session Versions])
UnsupportedClassVersionError falcon/fix/Session : Unsupported major.minor version 51.0  java.lang.ClassLoader.defineClass1 (:-2)
So I digged up a little bit and found some interesting information here, which led me to here and then to correctly configure Eclipse (Aptana) to use an external JRE to run things.

Basically, you need to explicitly tell Eclipse to use the external JDK you are using. You can find the setting under Window -> Preferences -> Java -> Installed JREs.



Saturday, January 25, 2014

Denarius Exchange organization created on Github

Dear all,

I've transferred the Denarius' repository to an organization focused on developing the exchange. Anybody is invidted to join and contribute.

Te new address is:
https://github.com/denarius-exchange/denarius

On the technical side, a matching is implemented with hash maps now, and we get slightly more speed, but we are introducing vectors and queue operations on them, which will yield about five times more speed.

Next ahead are the connection nodes and protocols implemented on them.

Monday, January 6, 2014

Denarius financial exchange announcement

I've been busy on Christmas working on Denarius, a financial exchange. This is the announcement that I've benn posting on several sites (see, for example, on bitcointalk).

Dear all,

I just started out a new open source project to implement a financial exchange project.

The implementation is in Clojure, which has already given the project a kick start with the matching engine concurrency. Interested Clojure programmers are invited to join.

Also, programmers working in finance, parallel systems and databases are kindly invited to join. Contributions to the core system design are highly welcome.

This project stems from unsatisfactory use of other existing projects regarding financial exchanges, and is of interest for the bitcoin community since cryptocoins are among the tradeable assets in the software.

This account is newbie's, so diffusion to other parts of bitcointalk is appreciated.

Subscribe just sending an email to: denarius@librelist.com
Source code: https://github.com/analyticbastard/denarius
Wiki: https://github.com/analyticbastard/denarius/wiki/Introduction

Sunday, December 22, 2013

Clojure concurrency and some niceties

I stumbled upon the barber problem at some webpage I don't remember; it was a loose and open approach to it. I then imagined my own version and implemented that in Clojure (I'll put the code as a Gist, even though the style just does not match this blog's). Basically, I want to adjust the rate of customers entering the barber shop to a little faster than the barber can dispatch one customer.

To do that, I create two functions, one that increases the queue of customers (with the shop supporting up to 3 customers) amd one that dispatces customers, decreasing the queue. I assume 4 hours for customers to be able to take a seat (the barber's will close after 4 hours and cut the remaining wating customers).


Notice here that the barber cuts the hair only if a random number meets a condition (here making him slower than his customers). I implement this with a watcher function, that gets called whenever the reference changes. In order to be called every time a customer enter the shop, we need to issue and identity change, that does not change the value of the queue but fires the watcher. This is a very nice feature of Clojure.

The important functional element is this

  (let [f (if (< @queue 3) inc identity)]
    (dosync (alter queue f)) ))
Notice the conditional assignment to the variable in the let block. This removes the boilerplate code needed in the expression section within the let block.

Tuesday, December 17, 2013

Damn, Clojure is fast!

Well I feel positive today, I struggled for the past two days to make my logistic regression in Clojure faster. I even made up to four different implementations of the logistic regression, with none of them giving satisfactory results. It all was even more disappointing when comparing to my MyML logistic regression implementation.

Well, I found out what the problem was: Actually I was making two fatal errors.
  • My input data in Clojure were lists instead of vectors
  • My input data in Python was 256 datapoints, instead of 1000 as in the Clojure version.
Both points stemmed from me being not so careful. In the first case, I knew already that one should use vectors instead of list when going after performance, but I was assuming that my data was in vector form. Staring at the variable X, I wondered if it was vector or list and voila, performance just got x5 better. Then I went to the Python interpreter, checked whether the number of iterations in the gradient descent object was the same as in Clojure, and then checked the data... well, my data was smaller (from a previous test with logistic regression). I re-generated my data and Python just lagged behind. In particular, I give you the figures (notice that I did not bother to put the wrong results I was getting because of my mistakes):

Python: 0.56 sec
Clojure: 0.19 sec (iterative implementation through loop-recur) 0.04 sec (concurrent implementation through agents).

Bear in mind that I am conduncting the tests on my girlfriend's borrowed machine and that I installed Cristoph Gohlke's Numpy distribution, which shippes with Intel's MKL statically-linked libraries, so it should be pretty fast in terms of algebraic computations. Perhaps the lack of performance comes from Python's interpreter itself (read-interpret-execute...). This is even more supporting of Clojure, since we are focusing on the infrastructure of both systems.

I will be putting everything in order, making my logistic regression more idiomatic and building some tests.

Wednesday, November 20, 2013

Google Highway 101 brainteaser

Some days ago I read about the famous Google application invitation in the form of an ad on Highway 101 passing through LA.


The ad shows a conceiled URL that we need to guess by computing the first 10-digit prime that appears in the series of consecutive digits of the irrational e. It is an oldie but goodie.

I thought I could solve this in Clojure. Here is my solution.

First we need to compute the number e with as many digits as necessary.
For that we can implementan unbounded spigot algorithm for the number e. I googled about this and found out a blog with superb material for this exercise, so I implemented the ideas in Clojure.

(defn digitse [N n] (if (= n N)  (.setScale  2.5M 200)
                      (if (= n 0)
                        (+ 2.0M (* (.divide
                                   (.setScale 1.0M 200 BigDecimal/ROUND_HALF_UP) 2.0M BigDecimal/ROUND_HALF_UP)
                                   (digitse N (inc n) )) )
                      (+ 1.0M (* (.divide
                                   (.setScale 1.0M 200 BigDecimal/ROUND_HALF_UP) (+ n 2.0M) BigDecimal/ROUND_HALF_UP)
                                 (digitse N (inc n)))) ) ))

We then create a subsequence with sufficient decimal places. It turs out that 150 digits is enough. We also convert it to string to improve the partition of our 10-digit (now characters) strings.
(def e (digitse 150 0))

(def se (.subSequence (.toString e) 0 150 ))
We need to define a function that tells us wheter a given number is a prime. As we know, we need only test up to its square root, but here a simple loop over the 10-digit numbers show us that none of the square roots exceeds 89000, so we take the first 8500 prime numbers, starting in 2, from a prime number generator, that I took from here.

(defn gen-primes "Generates an infinite, lazy sequence of prime numbers"
  []
  (let [reinsert (fn [table x prime]
                   (update-in table [(+ prime x)] conj prime))]
    (defn primes-step [table d]
                 (if-let [factors (get table d)]
                   (recur (reduce #(reinsert %1 d %2) (dissoc table d) factors)
                          (inc d))
                   (lazy-seq (cons d (primes-step (assoc table (* d d) (list d))
                                                 (inc d))))))
    (primes-step {} 2)))
Please note that this generator is pretty fast, compared to the answers that I have read on the internet. Now we take the primes we need from this lazy sequence.
(def first-primes (take 8500 (gen-primes) ) )
And define a function to test that the condition that some remainder of a given number between any of the prime denominators is zero does not happen, which means that our number is prime.

(defn prime-restricted? [n first-primes]
  (= nil (some #(= 0 (rem n %)) first-primes) ))

Finally, we recursivelly (with no stack overhead) until either a prime is found or we exceed the 150 digits capacity (in which case we would just increase it, but we don't need to).


(defn find-first-prime-in-e [se init first-primes]
  (if (<= (- (.length se) init) 10)
    nil
    (let [number (java.lang.Long/parseLong (.subSequence se init (+ init 10)) )]
      (println init number)
      (if (prime-restricted? number first-primes)
        number
        (recur se (inc init) first-primes)
        )))
  )



The last output lines and the returned value of the previous function are

96 6642742746
97 6427427466
98 4274274663
99 2742746639
100 7427466391
7427466391

So, it turns out that the first 10-digit prime number is in place 100, which makes the URL

7427466391.com

Upon connection, you would get another quizz, which having in mind the quantity you just computed is fairly straighforward.

Monday, October 21, 2013

Clojure structures, deconstruction and higher-order functions to make our lives better

Clojure provides an easy and more productive way of writing programs and it is available to us if we want to stay away from the common imperative, sequential, start-to-end way of thinking.

During my time as a developer, both (and especially) in the academia and in production software, I've come across the problem of getting the index of those elements in an array that meet a constrain. This is easy (although after some training, as usual) in Clojure
(defn index-filter [pred coll]
  (for [[i elem]
    (map-indexed (fn [a b] [a b]) coll) :when (pred elem)]
   i))
The way this works is: the for macro associates each i and elem  to the pairs in the sequence given by (fn [a b] [a b]) coll). This is filtered by executing the predicate on each element. The predicate, in turn, filters out the elements in which we are not interested. The for body then returns each of the index that passed the condition.

We can separate the functionality into two functions, the first to write the indexing of the original elements as an independent function:
(def make-index (partial map-indexed (fn [a b] [a b]) ))
We use (partial) to make a function that still needs an input argument and associate it into the make-index symbol. Placing it into the general code:
(defn index-filter [pred coll]
  (for [[i elem] (make-index coll) :when (pred elem)]
   i))
The way you call this function is with a predicate and a collection. For example, Now we have a very elegant solution that is valid for many data types.

Wednesday, August 28, 2013

Clojure for project development

In line with my last Clojure post, and with several comments I have come across the internet, I am going to post a guide to build a Clojure piece of software runnable from the beginning (i.e., outside of the REPL).

First of all, you need a JVM, obviously. If you haven't done that, install the latest version of Java SDK.

Now you want to install Leinigen. Leinigen is a building and dependency management tool favored by the Clojure community. Download Leinigen script here https://raw.github.com/technomancy/leiningen/stable/bin/lein.  Now you need to place it on a directory within your PAT (for example, /bin) and set it to be executable with chmod 755 ~/bin/lein.

Once we have done that, we need to tell the script to download the Leinigen system. You can easiy do that with lein self-install.

Now you can create a Clojure project, called hello:
lein new app hello
This uses the template app to create your new project. Now cd into the new directory, collect the dependencies and run the tests.
cd hello
lein deps
lein test
You'll see a single testcase which deliberately fails:
Testing hello.core-test FAIL in (replace-me) (core_test.clj:6) expected: false   actual: false Ran 1 tests containing 1 assertions. 1 failures, 0 errors.
Great! Clojure is installed in this project and working! To get a feel for Clojure, let's try out some basic stuff by starting a script console:
lein repl
You'll see something like:

nREPL server started on port 59654 on host 127.0.0.1
REPL-y 0.3.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=>
Type  
(println "Hello World!") 
and press return. You should get:
Hello World!
nil
user=>
Now let's define a function that does that:
(defn greet [] (println "Hello World!"))
The console will respond:
#'user/greet user=>
Run the function:  
(greet)
Hello World! nil user=>
Returning to the project, edit src/hello/core.clj (the basic source skeleton that Leiningen created for you above). Add our greet function to it and call it, so core.clj reads:
(ns hello.core) (defn greet [] (println "Hello World!"))

(defn -main[] (greet "Sean"))
The (ns hello.core) line declares the namespace (think Java package) in which the code lives. The -main function will be the regular Java main function and we tell Clojure by writing the - prefix.
We can run this via Leiningen:
lein run -m hello.core
The -m argument specifies the namespace in which -main is defined.

Now let's modify our script so we can compile it and run it via the JVM. First we need to update the namespace declaration to tell Clojure we want to generate a (Java) class file, we remove the spaces in the output by call str to construct a single string (so we need a space after Hello), and we change our main method to accept an argument:
(ns hello.core (:gen-class))
(defn greet[who] (println (str "Hello " who "!")))
(defn -main[who] (greet who))
We also need to tell Leiningen about our main class. Edit project.clj and add a :main declaration so it looks like this:
(defproject hello "1.0.0-SNAPSHOT" 
   :description "FIXME: write"
   :dependencies [[org.clojure/clojure "1.2.1"]]
     :main hello.core)
Don't worry about the rest of it, that's part of the Leiningen/Maven magic used to ensure the right libraries are available. Now tell Leiningen to compile your script and create a JAR that we can execute via Java:
lein uberjar
If you look in the current directory, you'll see hello-1.0.0-SNAPSHOT.jar and hello-1.0.0-SNAPSHOT-standalone.jar and it's the second one we'll use:
java -cp hello-1.0.0-SNAPSHOT-standalone.jar hello.core
 You have now a functional project and you are ready to write some code for production software.

Sunday, August 25, 2013

The perils of the REPL

Functional [programming] people are proud of their new toy called the REPL almost as if interactive development was a new concept. I guess that coming from Java and the generalist software development languages makes you think it is (although generalist but interpreted languages such as Python have always had an interactive interpreter). People who have worked with scientific modelling software such as Matlab or R (myself included) are used to this way of developing: rapidly modelling an idea into a few lines that could be recalled and modified according to one's needs.

This, however, becomes dangerous when developing software. When making a software product, one is one step ahead from bare modeling, in the sense that full working conditions are taken into account, one of them being program startup. I say this because I've read some books on Clojure and always found them to work with the REPL to describe de language, obviating the classical software bulding cicle of write a source code file which includes a main function or entry point, compile it and execute the result. The REPL gives you the advantage of quick modeling, but it is very different from writing Clojure source files and integrate them into a whose system intended for production.

Despite not being so difficult being Clojure a JVM system, almost none of them explain the entry point to the program, and they stick to explaining language sintax and basic libraries on the REPL, forgetting about entry points and other production software issues such as multiple file integration. Some of them don't even include a section to Leiningen or Maven, and jump to using advanced features such as databases (Redis, MySQL, HBase) or web toolkits. Even those that come with brief introduction don't even put the reader in a context of making a deployable piece of software. Therefore, readers must resort to blogs to find that kind of information.

The REPL is useful. It is as useful as it is in the scientific/modeling world, but as software developers with deployable product, programmers must deal with things further than testing live, more important to their business.

Saturday, June 29, 2013

Clojure vectors vs. PersistentQueue

Clojure has several pre-built data structures that can be used to implement our ideas. Here you have a comparison of the well-known vector and the poorly-documented clojure.lang.PersistentQueue. Edit: I changed the code in quotes to a Github Gist since I like it much better the way it looks. Sorry for the horrible contrast of the gists and the blogger template, I believe nothing will save me from modifiying the colors.

There was a fatal flaw in the implementation. It is now corrected. Thanks to the commentators.