package zoo

Display time series, spatial and space-time data with R

In previous posts, I have shown you how to rename, generate, and even recode variables.

For time‐series analysis, dates and times are critical. You need to have one variable which records the time index.

Annual Data
For annual data it is convenient if the time index is the year number (e.g. 2010). Suppose your first
observation is the year 1947. You can generate the time index by the commands:
generate t=1947+_n-1
tsset t, annual
The variable “_n” is the natural index of the observation, starting at 1 and running to the number of
observations n. The generate command creates a variable “t” which adds 1947 to “_n”, and then
subtracts 1, so it is a series with entries “1947”, “1948”, “1949”, etc. The tsset command declares the variable “t” to be the time index. The option “annual” is not necessary, but tells STATA that the time index is measured at the annual frequency.

Quarterly Data
STATA stores the time index as an integer series. It uses the convention that the first quarter of 1996 is 0. The second quarter of 1996 is 1, the first quarter of 1999 is 4, etc. Dates before 1996 are negative integers, so that the fourth quarter of 1995 is ‐1, the third is ‐2, etc..
When formatted as a date, STATA displays quarterly time periods as “1957q2”, meaning the second
quarter of 1957. (Even though STATA stores the number “‐11”, the eleventh quarter before 1996q1.)
STATA uses the formula “tq(1957q2)” to translate the formatted date “1957q2” to the numerical index “‐11”.
Suppose that your first observation is the third quarter of 1947. You can generate a time index for the data set by the commands;

generate t=tq(1947q3)+_n-1
format t %tq
tsset t
The generate command creates a variable “t” with integer entries, normalized so that 0 occurs in
1060q1. The format command formats the variable “t” using the time‐series quarterly format. The “tq” refers to “time‐series quarterly”. The tsset command declares that the variable “t” is the time index.
You could have alternatively typed
tsset t, quarterly
to tell STATA that it is a quarterly series, but it is not necessary as “t” has already been formatted as
quarterly. Now, when you look at the variable “t” you will see it displayed in year‐quarter format.
Monthly Data

Monthly data is similar, but with “m” replacing “q”. STATA stores the time index with the convention
that 1960m1 is 0. To generate a monthly index starting in the second month of 1962, use the commands

generate t=tm(1962m2)+_n-1
format t %tm
tsset t

Weekly Data
Weekly data is similar, with “w” instead of “q” and “m”, and the base period is 1960w1. For a series
starting in the 7th week of 1973, use the commands
generate t=tw(1973w7)+_n-1
format t %tw
tsset t

Daily Data
Daily data is stored by dates. For example, “01jan1960” is Jan 1, 1960, which is the base period. To
generate a daily time index staring on April 18, 1962, use the commands
generate t=td(18apr1962)+_n-1
format t %td

tsset t

Next session, we shall do regression on time series data and I will show you how to reshape data from one form to another.

Image via Wikipedia

In previous posts I discussed how to generate a single permutation from a fully-randomised or restricted permutation design using shuffle(). Here I want to briefly mention the shuffleSet() function and illustrate it’s usage.

Every time you call shuffle() it has to interpret the control list to identify the type of permutation required. Whilst the overhead of this interpretation is not too high, there is no reason that it need be incurred just to generate a set of permutations. This is where shuffleSet() comes in. It works exactly like shuffle() taking the number of observations and a control object but in addition it takes an extra argument nset which is the number of permutations required for the set.

> require(permute) ## load the package
> args(shuffleSet)
function (n, nset = 1, control = permControl())
NULL

To generate 10 random permutations of ten observations you would use

> set.seed(2)
> shuffleSet(10, 10)
      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
 [1,]    2    7    5   10    6    8    1    3    4     9
 [2,]    6    3    7    2    9    5    4    1   10     8
 [3,]    7    4   10    2    3    6    1    8    5     9
 [4,]    1    2    7    8    4    6    5   10    9     3
 [5,]   10    3    1    2    6    4    5    7    9     8
 [6,]    1   10    6    7    2    5    4    3    8     9
 [7,]    8   10    6    2    9    3    7    4    1     5
 [8,]    3   10    1    2    7    4    6    9    8     5
 [9,]    4    7    1    3    2    5   10    8    6     9
[10,]   10    4    9    8    3    1    2    5    6     7

If those 10 observations were collected as a time series and we wanted 10 restricted permutations you would use

> set.seed(2)
> shuffleSet(10, 10, control = permControl(within = Within(type = "series")))
      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
 [1,]    3    4    5    6    7    8    9   10    1     2
 [2,]    9   10    1    2    3    4    5    6    7     8
 [3,]    7    8    9   10    1    2    3    4    5     6
 [4,]    3    4    5    6    7    8    9   10    1     2
 [5,]    1    2    3    4    5    6    7    8    9    10
 [6,]    1    2    3    4    5    6    7    8    9    10
 [7,]    3    4    5    6    7    8    9   10    1     2
 [8,]   10    1    2    3    4    5    6    7    8     9
 [9,]    6    7    8    9   10    1    2    3    4     5
[10,]    7    8    9   10    1    2    3    4    5     6

From the above set of permutations, the cyclic shifts employed in the "series" permutation type are clear. One problem with the set we just produced is that the same permutation was returned more than once. In fact, there were only six unique permutations in the set requested. This is due to there only being 10 possible permutations of the numbers 1, 2, …, 10 if we allow cyclic shifts in a single direction

> numPerms(10, control = permControl(within = Within(type = "series")))
[1] 10

shuffle() and shuffleSet() know nothing of these limits, but there are functions in the permute package that can tell you the number of possible permutations (numPerms()) and generate the entire set of permutations for a stated design (allPerms()). I’ll take a look at allPerms() in a future posting.

I return now to the Golden Jackal mandible length example I used in an earlier post but update the example to make use of shuffleSet() instead of shuffle(). I will just show the code and output for the permutation test, refer to the previous post for details:

> data(jackal) ## load the data
> ## function to compute the difference of means
> meanDif <- function(x, grp) {
+     mean(x[grp == "Male"]) - mean(x[grp == "Female"])
+ }
> N <- nrow(jackal)
> ## make this reproducible and like previous post
> set.seed(42)
> ## generate the set of 4999 random permutations
> pSet  <- shuffleSet(N, 4999)
> ## iterate over the set
> Djackal <- apply(pSet, 1, function(i, data) with(data, meanDif(Length, Sex[i])),
+                  data = jackal)
> ## add the observed difference of mean mandible length
> Djackal <- c(Djackal, with(jackal, meanDif(Length, Sex)))
## No. obs equal to or greater than observed
> (Dbig <- sum(Djackal >= Djackal[5000]))
[1] 12
> ## permutational p-value
> Dbig/length(Djackal)
[1] 0.0024

The last two lines of R code compute the number of observations in the Null distribution with differences in mean mandible length as great or greater than the observed difference, and the resulting permutation p-value. These are the same as those computed in the previous post.

Generating entire sets of permutations is useful for several reasons. One recent example that we came across is with the new parallel processing capabilities in the forthcoming version of R. We are able to generate a set of permutations and then distribute the process of the permutation test over a number of CPUs or worker threads, each dealing with a subset of the permutations we generated. This can greatly reduce the compute time needed for the permutation test, especially where the objective function is computationally complex, but allows us to not worry about controlling the random number generator in each separate process — this is all done within the main function and only the relevant subset of permutations is passed to each worker process. An additional reason for generating a set of permutation to work with rather than individual permutations is that it is easy to switch between using a set of randomly generated permutations or the set of all possible permutations where that set is not overly large. allPerms() returns the set of permutations in the same way that shuffleSet() does, so we can simplify our code if we write the test to iterate over a set of permutations.

The full script for the Golden Jackal permutation test is shown below:

data(jackal) ## load the data
## function to compute the difference of means
meanDif <- function(x, grp) {
    mean(x[grp == "Male"]) - mean(x[grp == "Female"])
}
N <- nrow(jackal)
## make this reproducible and like previous post
set.seed(42)
## generate the set of 4999 random permutations
pSet  <- shuffleSet(N, 4999)
## iterate over the set
Djackal <- apply(pSet, 1, function(i, data) with(data, meanDif(Length, Sex[i])),
                 data = jackal)
## add the observed difference of mean mandible length
Djackal <- c(Djackal, with(jackal, meanDif(Length, Sex)))
## No. obs equal to or greater than observed
(Dbig <- sum(Djackal >= Djackal[5000]))
## permutational p-value
Dbig/length(Djackal)

shuffle() has two arguments: i) n, the number of observations in the data set to be permuted, and ii) control, a list that defines the permutation design describing how the samples are permuted.

R> args(shuffle)
function (n, control = permControl())
NULL

control is a list, and for complex permutation designs. As a result, several convenience functions are provided that make it easier to specify the design you want. The main convenience function is permControl() which if passed no arguments populates an appropriate control object with defaults that result in free permutation of observations.

> str(permControl())
List of 10
 $ strata     : NULL
 $ nperm      : num 199
 $ complete   : logi FALSE
 $ within     :List of 5
  ..$ type    : chr "free"
  ..$ constant: logi FALSE
  ..$ mirror  : logi FALSE
  ..$ ncol    : NULL
  ..$ nrow    : NULL
 $ blocks     :List of 4
  ..$ type  : chr "none"
  ..$ mirror: logi FALSE
  ..$ ncol  : NULL
  ..$ nrow  : NULL
 $ maxperm    : num 9999
 $ minperm    : num 99
 $ all.perms  : NULL
 $ observed   : logi FALSE
 $ name.strata: chr "NULL"
 - attr(*, "class")= chr "permControl"

Several types of permutation can be produced by functions in permute:

• Free permutation of objects, which we saw in the earlier post
• Time series or line transect designs, where the temporal or spatial ordering is preserved
• Spatial grid designs, where the spatial ordering is preserved in both coordinate directions
• Permutation of blocks or groups of samples

The first three of these can be nested within the levels of a factor or to the levels of that factor, or to both. Such flexibility allows the analysis of split-plot designs using permutation tests. permControl() is used to set up the design from which shuffle() will draw a permutation. permControl() has two main arguments that specify how samples are permuted within blocks of samples or at the block level itself. These are within and blocks. Two convenience functions, Within() and Blocks() can be used to set the various options for permutation. For example, to permute the observations 1:10 assuming a time series design for the entire set of observations, the following control object would be used

> set.seed(4)
> x <- 1:10
> CTRL <- permControl(within = Within(type = "series"))
> perm <- shuffle(10, control = CTRL)
> perm
 [1]  7  8  9 10  1  2  3  4  5  6
> x[perm]
 [1]  7  8  9 10  1  2  3  4  5  6

It is assumed that the observations are in temporal or transect order. We only specified the type of permutation within blocks, the remaining options are set to their defaults via Within().

A more complex design, with three blocks, and a 3 by 3 spatial grid arrangement within each block can be created as follows

> set.seed(4)
> block <- gl(3, 9)
> CTRL <- permControl(strata = block,
+                     within = Within(type = "grid", ncol = 3, nrow = 3))
> perm <- shuffle(length(block), control = CTRL)
> perm
 [1]  6  4  5  9  7  8  3  1  2 14 15 13 17 18 16 11 12 10 22 23
[21] 24 25 26 27 19 20 21

Visualising the permutation as the 3 matrices may help illustrate how the data have been shuffled

> ## Original
> lapply(split(1:27, block), matrix, ncol = 3)
$`1`
     [,1] [,2] [,3]
[1,]    1    4    7
[2,]    2    5    8
[3,]    3    6    9

$`2`
     [,1] [,2] [,3]
[1,]   10   13   16
[2,]   11   14   17
[3,]   12   15   18

$`3`
     [,1] [,2] [,3]
[1,]   19   22   25
[2,]   20   23   26
[3,]   21   24   27

> ## Shuffled
> lapply(split(perm, block), matrix, ncol = 3)
$`1`
     [,1] [,2] [,3]
[1,]    6    9    3
[2,]    4    7    1
[3,]    5    8    2

$`2`
     [,1] [,2] [,3]
[1,]   14   17   11
[2,]   15   18   12
[3,]   13   16   10

$`3`
     [,1] [,2] [,3]
[1,]   22   25   19
[2,]   23   26   20
[3,]   24   27   21

In the first grid, the lower-left corner of the grid was set to row 2 and column 2 of the original, to row 1 and column 2 in the second grid, and to row 3 column 2 in the third grid.

To have the same permutation within each level of block, use the constant argument of the Within() function, setting it to TRUE

> set.seed(4)
> CTRL <- permControl(strata = block,
+                     within = Within(type = "grid", ncol = 3, nrow = 3,
+                                     constant = TRUE))
> perm2 <- shuffle(length(block), control = CTRL)
> lapply(split(perm2, block), matrix, ncol = 3)
$`1`
     [,1] [,2] [,3]
[1,]    6    9    3
[2,]    4    7    1
[3,]    5    8    2

$`2`
     [,1] [,2] [,3]
[1,]   15   18   12
[2,]   13   16   10
[3,]   14   17   11

$`3`
     [,1] [,2] [,3]
[1,]   24   27   21
[2,]   22   25   19
[3,]   23   26   20

As you can see, at the moment, I make some assumptions about the ordering of samples within each spatial/temporal structure. The samples do not have the be arranged in strata order, but within the levels of the grouping variable the observations must be in the right order. For spatial grids, this means in column-major order—just as in the way R fills matrices by columns. In a future release, I hope to relax some of these assumptions to make it easier to apply permutations to the data to hand.

In the next post in this series, I’ll take a look at generating sets of permutations using the shuffleSet() function.