# Another promising use of the package is its implementation in conjunction with the deployment of a computer adaptive test on Concerto (an open source testing platform put forward by the Psychometrics Centre at the University of Cambridge).
# In this post I will explore the use of this package in generating data and simulating test administration and scoring.
require(catR)
# catR has the ability to very easily simulate an item bank with eaither the 1pl, 2pl, 3pl, or 4pl models available.
# Frequently one might already have an item bank that one is using. However, if you don't, then you can easily simulate an item bank with the following command. Though the command is used for more than simulating an item bank. It also generates information about your item bank.
# Will create an item bank of 400 items
mybank = createItemBank(model="2pl", items=400, thMin=-6, thMax=6)
# The thetas that the item information is generated on is:
theta = mybank$theta
# The information for the entire 200 items can be generated:
information = apply(mybank$infoTab,1,sum)
# The information function looks normal because information stacks highest around the b parameter which is most heavily distributed at the mean 0.
plot(theta, information, type="l", main="The b parameter default is normal(0,1)")
# We can see the parameters of the items we drew.
mybank$itemPar
# However, I am unsatisfied with the idea of an item bank that has more information at the mean and less at the tails since one of promises of computer adaptive tests is that they have the potential to be equally precise at all levels of the ability spectrum. I will instead generate an item bank that reflects this hope.
mybank2 = createItemBank(model="2pl", bPrior=c("unif",-4,4), items=400, thMin=-6, thMax=6)
information2 = apply(mybank2$infoTab,1,sum)
# We can see that information2 has much less information at the center of the distribution and more near the tails.
plot(theta, information, type="l", main="A wider spread of information means less at the center")
lines(theta, information2, pch=22, lty=2, lwd=2)
# We can check to see if both distributions has approximately the same amount of information over the tested range (-6,6). I chose -6 and 6 because it helps make the distribution information look more similar. If we restrict information to only the likely ranges of theta -4,4 then the first function has much more information.
sum(information)
sum(information2)
# Both item pools have the same amount of information (ignoring statistical noise) accross the entire possible continuum of thetas. However, the information function with a normally distributed b parameter has more information at the likely values of ability. That said we might not care to have an extremely precise estimate of student ability at the center of the ability distribution if we have little precision near the ends.
# Let's say we would like to require that the test any student takes provides a minimum infomration of 30 for any student taking the test with ability between -4 and 4.
abline(h = 30)
abline(v = c(-4,4))
# Using the uniform draw does not give us a perfect fit, but it is better than normal distribution.
# There is other more complex ways of ensuring that this criteria is met, dealt with in one of my prior posts. (http://www.econometricsbysimulation.com/2012/11/matching-test-information.html) I will not go into that here.
# Let's create a simple simulation of a cat test with our item bank.
# First let's draw the subjects true ability:
true.theta = rnorm(1)
# Let's select our item. Our initial estimate of theta is 0. nextItem has quite a number of interesting options available that make it flexible. Out tells the test not to select some items.
item.choice = nextItem(mybank2,theta=0, out=NA)
# our item is:
item.choice$item
# The parameters for that item are:
item.choice$par
# First I want to make sure the item parameters can be fed into the probability function Pi
pars = t(as.matrix(item.choice$par))
# This will give us the probability that the subject will get the answer right.
probCorrect = Pi(true.theta, pars)$Pi
# Now let's do a random draw to see if the subject answered correctly (by simulation of course)
correct = (probCorrect>runif(1))
# Finally, let's estimate subject's ability. I will use baysian estimator since the maximum likelihood estiamtor does not function with few observations.
theta.est = eapEst(pars, correct)
# Now we would start again with our theta estimate and plug it back into our item choice and repeat until we meet some test termination criterion.
# I will say for the sake of simplicity, I will loop through a lengthy 100 item test.
# Let's start from this point.
nitems = 100
user.response = matrix(NA, nrow=nitems, ncol=1)
user.response[1,] = correct
user.theta.est = theta.est
items.taken = item.choice$item
items.parm = matrix(NA, nrow=nitems, ncol=4)
items.parm[1,] = item.choice$par
for (i in 2:nitems) {
item.choice = nextItem(mybank2,theta=user.theta.est, out=items.taken)
items.parm[i,] = item.choice$par
user.response[i] = Pi(true.theta, t(as.matrix(items.parm[i,])))$Pi>runif(1)
user.theta.est[i] = eapEst(items.parm[!is.na(user.response),], user.response[!is.na(user.response)])
}
# Note, convergence is not guaranteed in any particular run nor can it be assuming an infinite number of items can be administered since there are only 400 items in our pool.
plot(1:nitems, user.theta.est, type="b", main="Estimates should converge on true ability"
, ylab = "Theta Estimates", xlab = "Number of items administered")
abline(h=true.theta, lwd=3, col="darkblue")
# catR even has a command called randomCAT which takes a particular item bank and user theta and generates a random simulated test and response pattern for that user just as we did previously but with many many options. This of course is unlikely to solve all of your needs. However, the creators of catR should be congradulated on their thoroughness!
# This post gives a simplified run through of some of catR's abilities. Overall, I am very impressed and look forward to using it in my work.
