Efficiently Saving and Sharing Data in R

After spending a day the other week struggling to make sense of a federal data set shared in an archaic format (ASCII fixed format dat file).

It is essential for the effective distribution and sharing of data that it use the minimum amount of disk space and be rapidly accessible for use by potential users.

In this post I test four different file formats available to R users. These formats are comma separated values csv (write.csv()), object representation format as a ASCII txt (dput()), a serialized R object (saveRDS()), and a Stata file (write.dta() from the foreign package). For reference, rds files seem to be identical to Rdata files except that they deal with only one object rather than potentially multiple.

In order to get an idea of how and where different formats outperformed each other I simulated a dataset composed of different common data formats. These formats were the following:

Numeric Formats

• Index 1 to N - ex. 1,2,3,4,...
• Whole Numbers - ex. 30, 81, 73, 5, ...
• Big Numbers - ex. 36374.989943, 15280.050850, 5.908210, 79.890601, 2.857904, ...
• Continous Numbers - ex. 1.1681155, 1.6963295 0.8964436, -0.5227753, ...

Text Formats

• String coded factor variables with 4 characters - ex. fdsg, jfkd, jfht, ejft, jfkd ...
• String coded factor variables with 16 characters coded as strings
• String coded factor variables with 64 characters coded as strings
• Factor coded variables with 4 characters - ex. fdsg, jfkd, jfht, ejft, jfkd - coded as 1,2,4,3,2, ...
• Factor coded variables with 16 characters
• Factor coded variables with 64 characters
• String variables with random 4 characters - ex. jdhd, jdjj, ienz, lsdk, ...
• String variables with random 16 characters
• String variables with random 64 characters

What type of format a variable is in is a predictive characteristic for how much space that variable takes up and therefore how time consuming that variable is to read or write. For variables that are easy to describe they tend to take up little space. An index variable in an extreme example and can take up almost no space as it can be expressed in an extremely compact format (1:N).

In contrast numbers which are very long or have a great degree of precision tend to have more information and therefore take more resources to access and store. String variables when filled with truly random or unique responses are some of the hardest data to compress as each value may be sampled from the full character spectrum. There is some significant potential for compression when strings are repeated in the variable. These repetitive entries can be either coded as a "factor" variable or a string variable in R.

As part of this exploration, I look at how string data is stored and saved when coded as either a string or as a factor within R.

Raw Files

Let's first look at space taken when saving uncompressed files.

Figure 1: File Size

Figure 1 shows the file size of each of the saved variables when 10,000 observations are generated. The dataframe object is the data.frame composed off all of the variables. From the height of the dataframe, we can see that rds is overall the winner. Looking at the other variable values we can see only that csv appear to consistently underperform for most file formats except for random strings.

Figure 2: File Size Log scaled

In Figure 2 we can see that rds is consistently outperforming all of the other formats with the one exception of index in which the txt encoding simply reads 1:10000. Apparently even serializing to bytes can't beat that.

Interestingly, there does not appear to be a effective size difference between repetitive strings encoded as factors accounting for the size of the strings (4, 16, or 64). We can see that the inability of csv to compress factor strings dramatically penalizes the efficiency of csv relative to the other formats.

File Compression

But data is rarely shared in uncompressed formats. How does compression change things?

Figure 3: Zipped File Sizes Logged

We can see from Figure 3 that if we zip our data after saving, the file size can do pretty much as well as rds. Comma delineated csv files are a bit of an exception with factor variables suffering under csv. Yet random strings perform slightly better under csv than other formats. Interesting rds files seem slightly larger than the other two file types. Overall though, it is pretty hard to see any significant difference in file size based on format after zipping.

So, should we stick with whatever format we prefer?

Not so fast. Sure, all of the files are similarly sized after zipping. This is useful for sharing files. But having to keep large file sizes on a hard drive is not ideal even if they can be compressed for distribution. There is finite space on any system and some files can be in the hundreds of MB to hundreds of GB range. Dealing with file formats and multiple file versions which are this large can easily drain the permanent storage capacity of most systems.

But an equally important concern, is how long it takes write and read different file formats.

Reading Speed

In order to test reading speeds, I loaded each of the different full dataframe files fifty times. I also tested how long it would take to unzip then load that file.

Figure 4: Reading and unzipping average speeds

From Figure 4, we can see that read speeds roughly correspond with the size of files. We can see that even a relatively small file (30 MB csv file) can take as long as 7 seconds to open. Working with large files saved in an inefficient format can be very frustrating.

In contrast, saving files in efficient formats can dramatically cut down on the time taken opening those files. Using the most efficient format (rds), files could be 100 times larger than those used in this simulation and still open in less than a minute.

Conclusions

Finding common file formats that any software can access is not easy. As a result many public data sets are provided in archaic formats which are poorly suited for end users.

This results in a wide pool of software sweets having the ability to access these datasets. However, with inefficient file formats comes a higher demand on the hardware of end users. I am unlikely to be the only person struggling with opening some of these large "public access" datasets.

Those maintaining these datasets will argue that sticking with the standard, inefficient format is the best of bad options. However, there is no reason they could not post datasets in rds formats in addition to the outdated formats they currently exist in.

And no we need not argue that selecting one software language to save data in will be biased toward those languages. Already many federal databases come with code supplements in Stata, SAS, or SPSS. To access these supplements, one is required to have paid access to that software.

Yet, R is free and its database format is public domain. Any user could download R, open a rds or Rdata file, then save that file in a format more suited to their purposes. None of these other proprietary database formats can boast the same.

ANOVAs and MANOVAs

# Anova's are frequently used in experimental setting when treatments is a categorical value and there are one or more response variables.  Rather than testing the statistical significance of each categorical value with respect to the response variable.  We instead test the joint significance.

# Let's see how this works.

# Imagine we are a pharmaceutical company and would like see is the effect on sleep of three different drug combinations at three different levels each.

# In order to test each level for all of the drug combinations we would have 27 combinations.  If we tested the individual significance of each combination then without adjusting the alpha value we would over-reject far too many times.

#
asbermatimitite=rbinom(1000,3,.5)*5
zugrimiitosoite=rbinom(1000,3,.5)*5
crelitotiserite=rbinom(1000,3,.5)*5

# Let's say each of our subjects on average sleeps a number of hours as a random draw from the poisson distribution.
base.sleep = rpois(1000,8)

sleep.hrs = base.sleep+asbermatimitite/5-zugrimiitosoite/2.5+crealitotiserite*0

# Let's say the most frequent side effect of these drugs is drowsiness.  We would also like to know the effect of the drugs on drowsiness levels.

base.drowsiness = runif(1000)

drowsiness = base.drowsiness + asbermatimitite*.01 + zugrimiitosoite*.02 - crealitotiserite*.01

# From this setup we can see that crelitotiserite does not actually increase sleep but does reduce drowsiness.

boxplot(sleep.hrs ~ asbermatimitite+zugrimiitosoite+crelitotiserite, horizontal = T, main="# of hours slept", ylab="Asbermatimitite.ZugrimiitosoiteCcrelitotiserite (Mg)")

# From the box plot it is clear there is movements in the means as the explanatory variables changes.  Other that that general statement I am no sure how else to read the grpah.

boxplot(drowsiness ~ asbermatimitite+zugrimiitosoite+crelitotiserite, horizontal = T, main="Self reported drowsiness", ylab="Asbermatimitite.Zugrimiitosoite.Crelitotiserite (Mg)")

# Likewise with drowsiness levels.  There is movement but it is hard to tell where and why.

man <- manova(cbind(sleep.hrs,drowsiness)~asbermatimitite+zugrimiitosoite+crelitotiserite)

# This will save the results of the manova in the list called man.

summary(man)

# We can see that all of the explanatory variables are significant.  crelitotiserite less so than the other two.  Unlike regression analysis we do not know what to do with this information now.  All we can say is that given random treatment there are differences in one or more of the dependent variables as a result of the different levels of treatment   I next logical approach would be to attempt to estimate the form of the effects.  If one of the variables was not significant then it might have been reasonably dropped from the analysis.

Introduction to Multi-Agent Simulations

Multi-agent simulations are simulations in which two or more agents take sequential actions frequently with respect to each other within a constructed environment.  This type of simulation is powerful in that it can demonstrate a hypothesis or generate results to situations which are frequently too complicated to solve by hand.  These types of simulations are frequently used to support hypotheses that researchers would otherwise have difficulty showing with real data.  They are also useful at exploring environments that are inherently difficult or impossible to run experiments in.

The simulations by Robert Axelrod and biologist W.D. Hamilton attempting to demonstrate the evolution of cooperation [1] use multi-agent simulations in an attempt to show that within an environment where agents can either cooperate or cheat with repeated outcomes the best strategy under many circumstances is to cooperate.  This type of simulation is uniquely powerful in that it evaluates many competing strategies ultimately identifying one that works under a well-defined scenario. 

Multi-agent simulations are appealing in that they allow researchers to generate data and evaluate scenarios that are rich and complex and potentially more similar to that of the “real world”.  Within a multi-agent simulation each agent acts with its own strategy.  That strategy can either be a common objective or an individual objective. 

These types of simulations might be effectively deployed to explain the behaviors of ants.  Each individual ant has a relatively simple set of commands.  By simulating many competing commands and evaluating the effectiveness of different types of ants at building colonies within a rich environment it could be informative.

Within applied economics multi-agent simulations often have a spatial component that is used to identify placement and relation of agents within the environment.  Other relational scales may be appropriate as well.  If looking at social network analysis it may be useful to have both a spatial as well as shared public spaces networks in which agents can interact.

One of the most challenging aspect of multi-agent simulations is the inherent complexity of the environment and the interactions that agents are making.  Often times, though you have defined the environment and the strategies of agents it is not clear exactly how those strategies interact with the environment to produce the results that they do.  

In a way this is the point.  Multi-agent simulations can demonstrate results that are otherwise difficult to demonstrate purely through math or argumentation.  But, as such it is often difficult to be sure what mechanisms are driving the results that are being produced: because of the agents’ actions, the environmental variables, or some other artifact not previously considered. 

As such, care in interpretation and rigorous error checking are extremely routines when designing and evaluating simulations.

1. http://en.wikipedia.org/wiki/The_Evolution_of_Cooperation

Why Simulate?

Simulations are a powerful tool available to researchers in all kinds of work from economics, to engineering, to psychological measurement.  They provide researchers with a unique tool that can be used to explore detailed thought experiments.  Researchers can simulate anything from the relatively straightforward testing of the inherently unobservable effectiveness of new or existing econometric/statistical methods to more complex simulations of human or non-human movement and behavior such as simulations of the transmission of malaria parasites or zombie hordes.

Simulations are frequently used to support theories developed a proiri to the simulation and advanced by authors who use simulations to complement their arguments.   The literature is replete with examples of new methods or theories that use simulations to complement their theoretical results.   Simulations can also be useful tools in exploring new problems and solutions.  An ideal exploratory simulation will have an environment rich in reasonable and representative environmental information. 

A perfect example of this is an engineering simulation I observed in which the automobiles with various numbers of wheels, wheel sizes, and wheel positioning were created in the simulation and the computer reran the model assigning different parameter values to each repetition searching the best design.  However, when simulating global environments it becomes all the more important that simulated agents are representational of real agents because nearly any result is possible if one controls the environment of the simulation.    

Of course there are other reasons for do simulations besides the ones listed above.  Sometimes simulations are seen as a mechanism for uncovering relationships that are too difficult or complex to identify outside of a constructed framework.  Typical examples of this are simulation that test violations of assumptions in statistical/econometric methods.  Looking at instrumental variables for instance, what happens when an instrumental variable does not have a very strong predictive power on the endogenous variable?  And what happens when it is also slightly endogenous?

Yet other simulations are used to dismiss or undermine criticisms.  Often times it is seen as the fatal flaw in an argument if the estimator is shown to be biased theoretically.  However, simulations can be used to look at the scope of reasonable bias expected and see how it would be expected to affect estimates.  It is not uncommon that biased, even inconsistent estimators turn out to be reasonable and effective even when the assumptions are not perfectly met.

As hopefully is clear, there are many good reasons for developing simulations.  This blog will hopefully guide you through of some of the most common types of simulations as well as through more obscure types as well.