Testing R/Rserve in the Cloud

I have to admit, this took me quite a while to figure out. But here we go.

Continuous Integration ?

My present idea of good software is, that good software MUST prove that it works. In order to show that, a continuous integration server is a developers best friend. Such a Continuous Integration server is JENKINS. But when I run such a server on my local laptop, I cannot show anyone out there in the world wide web, that my code seriously runs. So, searching the web for a CI service with low cost I happened to stumble upon the red hat openshift service. They offer free "gears" (modules), remote https login and configuration. Some security restriktions involve that I am not admin of the server, but nevertheless I can unpack ("install") R in my gear.

Running R/Rserve on openshift

After creating a standard Jenkins CI gear I need a bit of remote work to install Rserve and the required environment:
ssh ...7@ci-joris.rhcloud.com
#DOWNLOAD AND INSTALL R
cd $OPENSHIFT_DATA_DIR/tools
rm -rf .. (old version if installed)
wget ftp://ftp.stat.math.ethz.ch/pub/Software/R/R-devel.tar.gz
tar -xzf R-devel.tar.gz
cd R-devel
./configure --prefix=$OPENSHIFT_DATA_DIR/tools/R/ --exec-prefix=$OPENSHIFT_DATA_DIR/tools/R-tools --enable-R-shlib >r_configure.log 2>r_configure_error.log
make >r_make.log 2>r_make_error.log
#INSTALL Rserve
export PATH=$PATH:$OPENSHIFT_DATA_DIR/tools/R-devel/bin
wget http://www.rforge.net/Rserve/snapshot/Rserve_1.7-0.tar.gz
R CMD INSTALL Rserve_1.7-0.tar.gz >rserve_install.log 2>rserve_install_error.log
#Rserve requires a native domain socket:
wget http://junixsocket.googlecode.com/files/junixsocket-1.3-bin.tar.bz2
tar -xvjf junixsocket-1.3-bin.tar.bz2
mv junixsocket-1.3/lib-native/* .
rm *.bz2
rm -rf junixsocket-1.3/

Done so far. This installed R, Rserve and a Java Unixsocket library (junixsocket). The unix socket is required to allow connection to Rserve. "localhost" is not available as a server IP running on openshift. Nevertheless I can connect to Rserve through a socket.

Be aware that this continuos integration server will test R and Rserve with one installed version. I like to keep that combination fixed for now. A possible next extension is to install R and Rserve on the fly.

Nevertheless, you may take peek of what extensions are in development and see the health of j.o.r.i.s in the web: https://ci-joris.rhcloud.com

You may download the current source from SVN https://ralfseger.svn.cloudforge.com/repository/root and test yourselves.
To use unix-socket connection add some environment variables:

export DERBY_HOST="$OPENSHIFT_INTERNAL_IP"
export DERBY_PORT="3529"
export NATIVE_LIB_PATH="$OPENSHIFT_DATA_DIR"tools/native-libs
export RSERVE_SOCKET="$OPENSHIFT_DATA_DIR"socket/rserve
export JORIS_host="$OPENSHIFT_INTERNAL_IP"
export JORIS_port="9999"

the $OPENSHIFT_INTERNAL_IP is the internal IP when you run on openshift.
For your own computer you are free to use whatever fits your needs.

Big thanks to red hat for providing openshift and to cloudforge for their complimentary, great service.

The J.O.R.I.S Model View Presenter

The J.O.R.I.S Project does not only provide means to integrate R into Java using the Springframework.

Since you need means to create and test drive (see last post) your User Interface the Model View Presenter (see at Martin Fowlers page) was chosen. Model View Presenter separates View from Model, and since both can be represented as an interface the application can be tested easier and more effective than many other classic MVC approaches.

The Model and View are not directly coupled but are glued together in the Presenter implementation. This allows to test the application behavior as a whole, even when the UI Application grows.

Since I gave a small practical example last time how to test drive an application my next idea is to show how to combine some rich client frameworks with the pattern above.

You may check MainFrame.java from the subversion repository repository to get started.

The application services are maintained by a Spring context. 

context = new ClassPathXmlApplicationContext(new String[] { 

    "classpath*:spring/hibernate-service-impl.spring.xml",

    "classpath:/spring/mmi-jrConnectionContext.spring.xml", 

    "classpath*:spring/r-service.spring.xml",

    "classpath:/spring/mmi-databaseContext.spring.xml", 

    "classpath:/spring/mmi-applicationContext.spring.xml" });

rConnection = context.getBean("managedConnection", IRConnection.class);

The application frame is defined (and thus the layout separated) in XML, leveraging the swixml Framework.

<?xml version="1.0" encoding="UTF-8"?>
<frame size="800,600" defaultCloseOperation="JFrame.EXIT_ON_CLOSE">
 <menubar>
  <menu text="R">
   <menuitem text="Scan Workspace" accelerator="W" id="scanWorkspace"/>
   <menuitem text="Load Data Set" accelerator="L" id="loadDataSet"/>
   <separator />
   <menuitem text="Quit" id="quit" accelerator="shift meta Q"/>
  </menu>
  <menu text="Database">
   <menuitem text="Search" accelerator="meta F" id="searchData" />
   <menuitem text="Configure" accelerator="meta C" />
  </menu>
 </menubar>

 <panel layout="BorderLayout">
  <splitpane orientation="JSplitPane.VERTICAL_SPLIT"
   dividerSize="5">
   <scrollpane height="500" PreferredSize="800,500"
    HorizontalScrollBarPolicy="JScrollPane.HORIZONTAL_SCROLLBAR_NEVER"
    VerticalScrollBarPolicy="JScrollPane.VERTICAL_SCROLLBAR_AS_NEEDED">
    <editorpane id="protocol" Editable="false" MinimumSize="320,200" PreferredSize="800,500" ContentType="text/html">
    </editorpane>
   </scrollpane>
   <textarea id="input">
   </textarea>
  </splitpane>
 </panel>
</frame>

A custom UIProcessor binds the standard swing events easily to the MessageBus. This binding is acchieved 
by a naming convention for Buttons and MenuItems.

For the menuItem Quit in the above example see the QuitEventHandler.java.QuitEventHandler handling 
org.rosuda.ui.event.QuitEvents. 

These utilities leverage small and readable classes for easy and fast test driven UI design.

ui-less Test Driven User Interface Development

Whenever you start a project that deals with technical problems, you are forced to write a simple application to show the basic concepts. Since I am going to demonstrate a concept I will save most technical details for a later post.

My topic today is TDD (test driven development). Instead of writing lots of code and check if everything works by re-running the application you write a test for each feature. Unit testing is a state of art technique to ensure quality of code. But for most user interfaces, writing test is a really cumbersome work. Regularly a specific test framework, that starts your application and simulates the user interaction, is needed. This type of test is really useful for a thorough quality assurence. But to add new features quickly to your user interface thats a real costly in terms of labor.

Example

Lets get started by very simple UI component. You have one input field where you enter your commands and a large text area where you see what you type with additional information by some service.

The upper part, with the friendly "welcome" text is our protocol text area while the lower part is the input area.

Now the first step is, that we do some meaningful specification. Like, whenever I type a text into the input area and press the CRT Key, I want that some more text is in the protocol than before.

Let us create a unit test that does exactly that:

public class MainFrameTest {

    @Test
    public void pressingTheCrtKeySendsTheInputcontentToTheProtocol() {
        int initialLength = view.getProtocol().getValue().getLength();
        enterCommand("ein Text");
        assertThat(view.getProtocol().getValue().getLength(), greaterThan(initialLength));
    }

}

Of course this code, will not compile (yet). Since we test-drive there is no "view" yet nor a class for that view. The Method enterCommand is technical and yet unknown. Lets first write the enterCommand, and thus define what the view needs to provide:

    // -- helper
    private void enterCommand(String inputText) {
        setInputText(inputText);

        view.getInputEvent().sendEvent(HasKeyEvent.KeyEvent.Type.KEY_UP, 0, java.awt.event.KeyEvent.VK_ENTER);
    }

    private void setInputText(String inputText) {
        view.getInputValue().setValue(inputText);
    }

Currently the view requires these methods:

• getProtocol
• getInputValue
• getInputEvent

Knowing that we can define an interface for our user-interface to reflect upon that fact:

package org.rosuda.ui.main;

import javax.swing.text.html.HTMLDocument;
import org.rosuda.ui.core.mvc.HasKeyEvent;
import org.rosuda.ui.core.mvc.HasValue;
import org.rosuda.ui.core.mvc.MVP;

public interface MainView<C> extends MVP.View<C> {
    HasValue<String> getInputValue();
    HasValue<HTMLDocument> getProtocol();
    HasKeyEvent getInputEvent();
}

Defining a test has provided us with a java interface, instead of a bloated AWT, Swing or other user interface technique. You might ask, will this ever work ? I assure, stay with me, and I'll convince you.

To make this test go green and creating functional java code we will need some more java classes. Unless you are familiar with the MVP (Model View Presenter) pattern, which deserves an own post you might be confused how to write code for making your test go green.

Using j.o.r.i.s MVP and MVP Test support you can inherit from the base class MVPTest.

This MVPTest.java requires the MVP Components as generic arguments and an aditional Model-initialisation test helper class.

The signature looks overwhelming:

public abstract class MVPTest<MODEL, VIEW, PRESENTER extends MVP.Presenter<MODEL, VIEW>, MODELINITIALIZER extends ModelInitializer<MODEL>>

Since I want to cover MVP in an later post, let's write the code that is required to come back to green:

first we use the abstract class MVPTest for our unit Test:

public class MainFrameTest extends MVPTest<MainModel, MainView<Void>, MainPresenter<Void>, MainFrameTestModelData> {
@Override
 protected MainView<Void> createTestViewInstance() {
 return new MainFrameTestView();
 }
}

an implementation for the model initialization, currently we do not need special model values so this is an empty implementation

public class MainFrameTestModelData extends ModelInitializer{

    @Override
    protected void initModel(MainModel model) {
    }

}

we need a mock UI for test wiring:


public class MainFrameTestView extends DefaultTestView implements MainView {

    private HasValue input = new DefaultHasValue();
    private HasValue protocol = new DefaultHasValue();
    private HasKeyEvent inputEvent = new DefaultHasKeyEvent();

    @Override
    public HasValue getInputValue() {
 return input;
    }

    @Override
    public HasValue getProtocol() {
 return protocol;
    }

    @Override
    public HasKeyEvent getInputEvent() {
 return inputEvent;
    }

}




we have to create the MVP classes according to the j.o.r.i.s framework:

public class MainModel implements MVP.Model {

    private final HTMLDocument protocol;

    public MainModel() throws IOException {
 protocol = new HTMLDocument();
 protocol.setParser(new ParserDelegator());
 BufferedReader htmlStream = null;
 try {
     htmlStream = new BufferedReader(new InputStreamReader(MainFrame.class.getResourceAsStream("/gui/html/welcome.html")));
     EditorKit kit = getEditorKit();
     kit.read(htmlStream, protocol, 0);
 } catch (Exception e) {
     throw new RuntimeException(e);
 } finally {
     if (htmlStream != null) {
  htmlStream.close();
     }
 }
    }

    private EditorKit getEditorKit() {
 return new HTMLEditorKit();
    }

    HTMLDocument getProtocol() {
 return protocol;
    }
    
}



public class MainPresenter implements MVP.Presenter> {

    @Override
    public void bind(final MainModel model, final MainView<C> view, final MessageBus messageBus) {

    public void unbind(final MainModel model, final MainView<C> view, final MessageBus messageBus) {

    }
}




That's all to get the test running. We're not green yet, but this suffices to compile the test. 




Back to green


We've finally covered all required classes. A model class, holding the protocol data, a mock view object and a controller class, the MainPresenter.

Since the model and the view have no knowledge of each other yet we need to implement the bind(..) method in the presenter and we are done:





    @Override
    public void bind(final MainModel model, final MainView<C> view, final MessageBus messageBus) {
 view.getProtocol().setValue(model.getProtocol());
 view.getInputEvent().addKeyEventListener(new HasKeyEvent.KeyListener() {
     @Override
     public void onKeyEvent(HasKeyEvent.KeyEvent event) {
  if (HasKeyEvent.KeyEvent.Type.KEY_UP.equals(event.getType()) && KeyEvent.VK_ENTER == event.getKeyCode()) {
      final String currentValue = view.getInputValue().getValue();
      appendHTML(model, new StringBuilder("<div class=\"command\">&gt; ").append("<a href=\"").append(StringEscapeUtils.escapeHtml(currentValue))
       .append("\">").append(currentValue).append("</a>").append("</div>").toString());
      view.getInputValue().setValue("");
      messageBus.fireEvent(new CRTKeyEvent(currentValue));
  }
  
     }
 });
    }

    private void appendHTML(final MainModel model, final String htmlText) {
 final HTMLDocument targetDoc = model.getProtocol();
 final Element body = targetDoc.getElement("htmlbody");
 final Element lastChild = body.getElement(body.getElementCount() - 1);
 try {
     targetDoc.insertAfterEnd(lastChild, htmlText);
 } catch (final Exception e) {
     LOG.error(e);
 }
    }

Conclusions

Making use of the MVP Pattern we deal with a java interface for the UI. Thus we can test wiring code and logic without dealing with real UI - code. 

By following the test driven paradigma we could implement a view linked to the model. The features are defined in the test. This allows much faster test-driving without neglecting any feature test, or delay testing for later.

Apologies for the non-optimal code format. For better readability try any java GUI and download the sources.

Lots of statisticians, psychologists, physicists use R as their universal statistical tool. Others just employ the power of R within another programming language to leverage well-tested statistical tools with specialist tools.
In this article I want to show how you can combine JAVA with R.
Start with an installation of either R or JAVA first. After you have finished you need Rserve. You might want to follow the instructions from the Rserve doc. The simplest way is to launch R cli and use the command

install.packages("Rserve")

There are multiple ways of starting the Rserve server. Linux users are usually well aware of their paths and dependencies. But on Windows systems you might have several installations, like 32 and 64 bit and need to start the Rserve that fits your installation. Using the command line tool you should be able to start Rserve as follows

>C:\tools\R\R-2.14.2\bin\i386\R.exe CMD C:\Users\rse\Documents\R\win-library\2.14\Rserve\libs\i386\Rserve.exe --no-save --vanilla
Ok, ready to answer queries.

When you know, you can start Rserve and thus use R from JAVA.

Starting this Rserve server any time is again and again is annoying. So when you used JAVA for some time you certainly stumbled upen the spring framework. Wouldn't it be nice, instead of starting Rserve over and over again, but use some Spring magic and just have a code like:

@Autowired IRConnection managedConnection;

The project j.o.r.i.s provides that type of spring magic. Apart from auto-starting Rserve there is a handy wrapper class for Rconnections, database tools and much more I will write about in a later post.