Streaming access to provide high performance

This recipe will take a look at how to parse and write large amount of data. Since XML is probably the most verbose of data formats, we will look briefly at how to generate large amount of junk data and then how to process those in a memory efficient way.

Getting ready

The data.xml library is our library of choice here, so let's add it to the project.clj if it is not already:

[org.clojure/data.xml "0.1.0-beta1"] 

We will also make use of the core Java streaming and zipping functions that are available in the JVM. We know we can access host functions just by importing them in the current namespace, so nothing more to prepare here.

How to do it...

So that's it. Let's go for a large dump of data onto our file system. We will create an XML element.

Streaming out

We get ready by pulling the proper namespaces for XML and for writing to streams:

(use 'clojure.data.xml) 
(use '[clojure.java.io :only [writer output-stream]]) 

Now, we can generate...