Friday, May 24, 2013

Shiny + Concerto = YES !!!

So I have finally gotten beta access to the two most powerful R controlled web application makers in existence and produced very exciting experimental products

A few posts ago I posted a Visual Reasoning Test that I had made by hand and powered with Concerto.  I posted a bit of the Concerto R code in order for people to get a feel for how Concerto 4.0 beta works. Of those who saw the post, about 300 people chose to start the test. Of those those 300, 60 generous
Shiny- Item Response Evaluation App
volunteers completed the entire exam (which took between 20 and 50 minutes depending upon how hard the user was trying).

Since each test is 22 items long, I now have data on about 2000 responses to individual items.  This gives me on average about 20 responses per item though some items have substantially more and others less.  If you have not taken the opportunity to take this exam, this might be a good chance though I will come back to it with future revisions.

I understand that many a test taker or non-test taker may be curious about the results of this test so I have made an analysis tool using Shiny R.  This test tool is a dynamic test tool this allows users to look at the individual items and how the difficulty of those items compares with that of other items.

If you have not yet had the chance to take a look at Shiny R, I suggest you do so.  The application, developed by the RStudio team is awesome.   Through a handful of commands users are able to create a fully interactive web application leveraging R's power and flexibility.

Visual Reasoning Test v0.2b 
The dynamic interface that Shiny allows users to design makes high powered analysis easy to systematize and enjoyable.

Concerto takes a bit more work but it also is more flexible.  It allows for complex interfaces that users interact with sequentially.  Concerto can easily manage keeping track of a user identity system and has database management tools which make it a very attractive option.  Combining the two, as I have done, I believe gives developers some serious power to work with.

In this Concerto and Shiny paired application I give the test to the user via Concerto.  After the test is done, I make the results of the test in terms of how others did on the items available through Shiny.  How this is accomplished is that Concerto saves a copy of the responses in a file in on the concerto server every time a user completes the test.  Likewise, every time a user access the Shiny web application, Shiny pulls the latest item response data from the Concerto server.

It then does some basic transformations on that data and then provides the data in various forms through the Shiny generated user interface.
Shiny- Item Response Evaluation App
As no doubt is embarrassingly clear, I am very exciting about these new (free!) products entering the market.    I will continue to work with both of them.  Keep tuned on for future posts.


  1. Hey, I greatly enjoyed your test (I'm one of the 60 that complete) and the Shiny app is very slick.

    I did want to send you a bug report: when I took the test (I think last Friday?), for one of the questions the answer choices didn't load. On my test, I think it was the 16th question. Using the Shiny app I was able to identify it as Item 10. If others experienced the problem that might have contributed to it looking like a very hard question.

    When taking the test I tried reloading the page, viewing the instructions and coming back to the question, all to no avail.

    1. Hi Gregor,

      Thanks very much for writing and I am glad you enjoyed the test. Currently I am working on calibrating the items and making it an adaptive test so that the next release I can reduce the number of items to only 10 or so and probably get a better test.

      As for the bug. You are totally right about that and I have not decided how to deal with my data from that item. Someone else also reported the bug in a comment on the original post but I forgot to think about that when I was looking at the item responses.

      Well, I am glad you enjoyed the test. I hope to improve it and release other tests as well. If there is any additional suggests that you have on how to improve the test I am very receptive.

      Thanks again!