The Gibbs sampler is applicable for certain classes of problems, based on two main criterion. Given a target distribution , where , ),  The first criterion is 1) that it is necessary that we have an analytic (mathematical) expression for the conditional distribution of each variable in the joint distribution given all other variables in the joint. Formally, if the target distribution is -dimensional, we must have individual expressions for

.

Each of these expressions defines the probability of the -th dimension given that we have values for all other () dimensions. Having the conditional distribution for each variable means that we don’t need a proposal distribution or an accept/reject criterion, like in the Metropolis-Hastings algorithm. Therefore, we can simply sample from each conditional while keeping all other variables held fixed. This leads to the second criterion 2) that we must be able to sample from each conditional distribution. This caveat is obvious if we want an implementable algorithm.

The Gibbs sampler works in much the same way as the component-wise Metropolis-Hastings algorithms except that instead drawing from a proposal distribution for each dimension, then accepting or rejecting the proposed sample, we simply draw a value for that dimension according to the variable’s corresponding conditional distribution. We also accept all values that are drawn. Similar to the component-wise Metropolis-Hastings algorithm, we step through each variable sequentially, sampling it while keeping all other variables fixed. The Gibbs sampling procedure is outlined below

1. set
2. generate an initial state
3. repeat until

set

for each dimension

draw from

To get a better understanding of the Gibbs sampler at work, let’s implement the Gibbs sampler to solve the same multivariate sampling problem addressed in the previous post.

Example: Sampling from a bivariate a Normal distribution

This example parallels the examples in the previous post where we sampled from a 2-D Normal distribution using block-wise and component-wise Metropolis-Hastings algorithms. Here, we show how to implement a Gibbs sampler to draw samples from the same target distribution. As a reminder, the target distribution is a Normal form with following parameterization:

with mean

and covariance

In order to sample from this distribution using a Gibbs sampler, we need to have in hand the conditional distributions for variables/dimensions and :

(i.e. the conditional for the first dimension, )

and

(the conditional for the second dimension, )

Where is the previous state of the second dimension, and is the state of the first dimension after drawing from . The reason for the discrepancy between updating and using states and , can be is seen in step 3 of the algorithm outlined in the previous section. At iteration we first sample a new state for variable conditioned on the most recent state of variable , which is from iteration . We then sample a new state for the variable conditioned on the most recent state of variable , which is now from the current iteration, .

After some math (which which I will skip for some brevity, but see the following for some details), we find that the two conditional distributions for the target Normal distribution are:

and

,

which are both univariate Normal distributions, each with a mean that is dependent on the value of the most recent state of the conditioning variable, and a variance that is dependent on the target covariances between the two variables.

Using the above expressions for the conditional probabilities of variables and , we implement the Gibbs sampler using MATLAB below. The output of the sampler is shown here:

Gibbs sampler Markov chain and samples for bivariate Normal target distribution

Inspecting the figure above, note how at each iteration the Markov chain for the Gibbs sampler first takes a step only along the direction, then only along the direction.  This shows how the Gibbs sampler sequentially samples the value of each variable separately, in a component-wise fashion.

% EXAMPLE: GIBBS SAMPLER FOR BIVARIATE NORMAL
rand('seed' ,12345);
nSamples = 5000;

mu = [0 0]; % TARGET MEAN
rho(1) = 0.8; % rho_21
rho(2) = 0.8; % rho_12

% INITIALIZE THE GIBBS SAMPLER
propSigma = 1; % PROPOSAL VARIANCE
minn = [-3 -3];
maxx = [3 3];

% INITIALIZE SAMPLES
x = zeros(nSamples,2);
x(1,1) = unifrnd(minn(1), maxx(1));
x(1,2) = unifrnd(minn(2), maxx(2));

dims = 1:2; % INDEX INTO EACH DIMENSION

% RUN GIBBS SAMPLER
t = 1;
while t < nSamples
    t = t + 1;
    T = [t-1,t];
    for iD = 1:2 % LOOP OVER DIMENSIONS
        % UPDATE SAMPLES
        nIx = dims~=iD; % *NOT* THE CURRENT DIMENSION
        % CONDITIONAL MEAN
        muCond = mu(iD) + rho(iD)*(x(T(iD),nIx)-mu(nIx));
        % CONDITIONAL VARIANCE
        varCond = sqrt(1-rho(iD)^2);
        % DRAW FROM CONDITIONAL
        x(t,iD) = normrnd(muCond,varCond);
    end
end

% DISPLAY SAMPLING DYNAMICS
figure;
h1 = scatter(x(:,1),x(:,2),'r.');

% CONDITIONAL STEPS/SAMPLES
hold on;
for t = 1:50
    plot([x(t,1),x(t+1,1)],[x(t,2),x(t,2)],'k-');
    plot([x(t+1,1),x(t+1,1)],[x(t,2),x(t+1,2)],'k-');
    h2 = plot(x(t+1,1),x(t+1,2),'ko');
end

h3 = scatter(x(1,1),x(1,2),'go','Linewidth',3);
legend([h1,h2,h3],{'Samples','1st 50 Samples','x(t=0)'},'Location','Northwest')
hold off;
xlabel('x_1');
ylabel('x_2');
axis square

Wrapping Up

The Gibbs sampler is a popular MCMC method for sampling from complex, multivariate probability distributions. However, the Gibbs sampler cannot be used for general sampling problems. For many target distributions, it may difficult or impossible to obtain a closed-form expression for all the needed conditional distributions. In other scenarios, analytic expressions may exist for all conditionals but it may be difficult to sample from any or all of the conditional distributions (in these scenarios it is common to use univariate sampling methods such as rejection sampling and (surprise!) Metropolis-type MCMC techniques to approximate samples from each conditional). Gibbs samplers are very popular for Bayesian methods where models are often devised in such a way that conditional expressions for all model variables are easily obtained and take well-known forms that can be sampled from efficiently.

Gibbs sampling, like many MCMC techniques suffer from what is often called “slow mixing.” Slow mixing occurs when the underlying Markov chain takes a long time to sufficiently explore the values of in order to give a good characterization of . Slow mixing is due to a number of factors including the “random walk” nature of the Markov chain, as well as the tendency of the Markov chain to get “stuck,” only sampling a single region of having high-probability under . Such behaviors are bad for sampling distributions with multiple modes or heavy tails. More advanced techniques, such as Hybrid Monte Carlo have been developed to incorporate additional dynamics that increase the efficiency of the Markov chain path. We will discuss Hybrid Monte Carlo in a future post.

A journey of a thousand miles begins with a single step. Lao-tzu. Photo credit: Larry Becker

In older posts (here, here, here and here), we have already analyzed upper bounds for the MSE or MISE which have the form

where is going to zero (for the density estimation case ) and . We would like to know if these rates are tight, in the sense that there is no other estimator that is better. To make this, we define a lower bound like

We will assume the following three element:

• A class of functions , containing the “true” function . For example, could be a Hölder class or a Sobolev class .
• A probability measure family , indexed by on a measurable space associated with the data. For example, in the density model, is the probability measure associated with a sample when the ‘s density is .
• A distance which compares the performance between the estimate and the real value . For example , for fix.
  In fact, it is enough use a semi-distance which satisfies the triangle inequality.

Define, for a stochastic model and a distance , the minimax riskas

where the is taken over all estimators. The main goal is to get a result of the form

for and .

Suppose that we have proved that

and if for a particular

It implies directly that

If inequalities (1) and (2) are satisfied simultaneously, we say that is the optimal rate of convergence for this problem and that attains that rate.

Remark 1 Two rates of convergence and are equivalent (we write ) if

The big question here is: How can we prove the equation (1) if it is necessary bound for all the estimators of ?

At a first glance, it will be an impossible task, just imagine the massive quantity of possible estimators for . Fortunately, we can apply a “reduction procedure” in order to simplify the problem and find the bound required. We will study it the next post.

As always any thoughts, suggestions or improvements are welcome in the commentaries.

Related articles

• Rates of convergence for the MISE in Sobolev classes of densities
• A first minimax lower bound in the two hypothesis scenario
• A general reduction scheme for minimax lower bounds
• Introduction to Minimax Lower Bounds
• A global measure of risk for kernel estimators in Nikol’ski classes

Now I was wondering if this approach can be extended to situations where the pixel cover fractions are changing over time? In analogy to change detection for qualitative classes, where we for example want to find the areas that have changed from Forest to Agriculture to Grassland, we could use multi-temporal unmixing to answer this question even more precise.

When dealing with change detection there are to common ways to deal with multi-temporal data: a) you classify each date individually and calculate the change classes afterword, b) you use the stack of all images and directly deal with the change classes. Both approaches do have some disadvantages: for a), only single-date information is available, and for b) the number of change classes quickly rises with the number of classes and the number of dates. To overcome this, one could use the whole image stack to calculate each single date classification (let us call this c)). Some tests show that this approach works well (to my knowledge, it is not yet published).

To come back to the multi-temporal unmixing problem with changing pixel cover fractios: approach a) and b) can directly be transfered for unmixing problems. Unfortunately, approach c) is a bit tricky, because we want to use the stack of all images to unmix a single date image. But unfortunately, the single date endmembers do not match the image stack feature space. Have to thing about a solution for this!

By writing all this, it came to my mind, that even for the regression case, their might be an analogy to the multi-temporal setting. To be even more general, all of the function learning concepts like classification, regression and density estimation might be put into a multi-temporal framework like described above.

Short summary:

• New change detection approach: use the multi-temporal image stack to classify single-date images; calculate the change classes afterward; advantage: use all multi-temporal information without exploding class combinations; disadvantages: no (?)
• Transfer this approach to multi-temporal unmixing; unresolved problem: single-date endmembers do not match multi-temporal stack
• general multi-temporal function learning framework might be worth thinking of

A Markov chain is a stochastic process that operates sequentially (e.g. temporally), transitioning from one state to another within an allowed set of states.^†

A Markov chain is defined by three elements:

1. A state space , which is a set of values that the chain is allowed to take
2. A transition operator that defines  the probability of moving from state to  .
3. An initial condition distribution which defines the probability of being in any one of the possible states at the initial iteration .

The Markov chain starts at some initial state, which is sampled from , then transitions from one state to another according to the transition operator .

A Markov chain is called memoryless if the next state only depends on the current state and not on any of the states previous to the current:

(This memoryless property is formally know as the Markov property).

If the transition operator for a Markov chain does not change across transitions, the Markov chain is called time homogenous.  A nice  property of time homogenous Markov chains is that as the chain runs for a long time and , the chain will reach an equilibrium that is called the chain’s stationary distribution:

We’ll see later how the stationary distribution of a Markov chain is important for sampling from probability distributions, a technique that is at the heart of Markov Chain Monte Carlo (MCMC) methods.

Finite state-space (time homogenous) Markov chain

If the state space of a Markov chain takes on a finite number of distinct values, and it is time homogenous, then the transition operator can be defined by a matrix , where the entries of are:

This means that if the chain is currently in the -th state, the transition operator assigns the probability of moving to the  -th state by the entries of -th row of (i.e.  each row of defines a conditional probability distribution on the state space). Let’s take a look at a finite state-space Markov chain in action with a simple example.

Example: Predicting the weather with a finite state-space Markov chain

In Berkeley, CA, there are (literally) only 3 types of weather: sunny, foggy, or rainy (this is analogous to a state-space that takes on three discrete values). The weather patterns are very stable there, so a Berkeley weatherman can easily predict the weather next week based on the weather today with the following transition rules:

If it is sunny today, then

• it is highly likely that it will be sunny next week
  • ,
• it is very unlikely that it will be raining next week
  • 
• and somewhat likely that it will foggy next week
  • 

If it is foggy today then

• it is somewhat likely that it will be sunny next week
  • 
• but slightly less likely that it will be foggy next week
  • ,
• and fairly unlikely that it will be raining next week.
  • ,

If it is rainy today then

• it is unlikely that it will be sunny next week
  • ,
• it is somewhat likely that it will be foggy next week
  • ,
• and it is fairly likely that it will be rainy next week
  • ,

All of these transition rules can be instantiated in a single 3 x 3 transition operator matrix:

Where each row of corresponds to the weather at iteration (today), and each column corresponds to the weather the next week. Let’s say that it is rainy today, what is the probability it will be sunny next week, in two weeks, or in 6 months? We can answer these questions by running a Markov chain from the initial state of “rainy,” transitioning according to . The following chunk of MATLAB code runs the Markov chain.

% FINITE STATE-SPACE MARKOV CHAIN EXAMPLE

% TRANSITION OPERATOR
%     S  F  R
%     U  O  A
%     N  G  I
%     N  G  N
%     Y  Y  Y
P = [.8 .15 .05;  % SUNNY
     .4 .5  .1;   % FOGGY
     .1 .3  .6];  % RAINY
nWeeks = 25

% INITIAL STATE IS RAINY
X(1,:) = [0 0 1];

% RUN MARKOV CHAIN
for iB = 2:nWeeks
    X(iB,:) = X(iB-1,:)*P; % TRANSITION
end

% DISPLAY
figure; hold on
h(1) = plot(1:nWeeks,X(:,1),'r','Linewidth',2);
h(2) = plot(1:nWeeks,X(:,2),'k','Linewidth',2);
h(3) = plot(1:nWeeks,X(:,3),'b','Linewidth',2);
h(4) = plot([15 15],[0 1],'g--','Linewidth',2);
hold off
legend(h, {'Sunny','Foggy','Rainy','Burn In'});
xlabel('Week')
ylabel('p(Weather)')
xlim([1,nWeeks]);
ylim([0 1]);

% PREDICTIONS
fprintf('\np(weather) in 1 week -->'), disp(X(2,:))
fprintf('\np(weather) in 2 weeks -->'), disp(X(3,:))
fprintf('\np(weather) in 6 months -->'), disp(X(25,:))

Finite state-space Markov chain for predicting the weather

Here we see that at week 1 the probability of sunny weather is 0.1. The next week, the probability of sunny weather is 0.26, and in 6 months, there is a 60% chance that it will be sunny. Also note that after approximately 15 weeks the Markov chain has reached the equilibrium/stationary distribution and, chances are, the weather will be sunny. This 15-week period is what is known as the burn in period for the Markov chain, and is the number of transitions it takes the chain to move from the initial conditions to the stationary distribution.

A cool thing about finite state-space time-homogeneous Markov chain is that it is not necessary  to run the chain sequentially through all iterations in order to predict a state in the future. Instead we can predict by first raising the transition operator to the -th power, where is the iteration at which we want to predict, then multiplying the result by the distribution over the initial state, . For instance, to predict the probability of the weather in 2 weeks, knowing that it is rainy today (i.e. ):

and in six months:

These are the same results we get by running the Markov chain sequentially through each number of transitions. Therefore we can calculate an approximation to the stationary distribution from by setting to a large number. It turns out that it is also possible to analytically derive the stationary distribution from (hint: think about the properties of eigenvectors).

Continuous state-space Markov chains

A Markov chain can also have a continuous state space that exists in the real numbers . In this case the transition operator cannot be instantiated simply as a matrix, but is instead some continuous function on the real numbers. Note that the continuous state-space Markov chain also has a burn in period and a stationary distribution. However, the stationary distribution will also be over a continuous set of variables. To get a better understanding of the workings of a continuous state-space Markov chain, let’s look at a simple example.

Example: Sampling from a continuous distribution using continuous state-space Markov chains

We can use the stationary distribution of a continuous state-space Markov chain in order to sample from a continuous probability distribution: we  run a Markov chain for a sufficient amount of time so that it has reached its stationary distribution, then keep the states that the chain visits as samples from that stationary distribution.

In the following example we define a continuous state-space Markov chain. The transition operator is a Normal distribution with unit variance and a mean that is half the distance between zero and the previous state, and the distribution over initial conditions is a Normal distribution with zero mean and unit variance.

To ensure that the chain has moved sufficiently far from the initial conditions and that we are sampling  from the chain’s stationary distribution,  we will choose to throw away the first 50 burn in states of the chain. We can also run multiple chains simultaneously in order to sample the stationary distribution more densely. Here we choose to run 5 chains simultaneously.

% EXAMPLE OF CONTINUOUS STATE-SPACE MARKOV CHAIN

% INITIALIZE
randn('seed',12345)
nBurnin = 50; % # BURNIN
nChains = 5;  % # MARKOV CHAINS

% DEFINE TRANSITION OPERATOR
P = inline('normrnd(.5*x,1,1,nChains)','x','nChains');
nTransitions = 1000;
x = zeros(nTransitions,nChains);
x(1,:) = randn(1,nChains);

% RUN THE CHAINS
for iT = 2:nTransitions
    x(iT,:) = P(x(iT-1),nChains);
end

% DISPLAY BURNIN
figure
subplot(221); plot(x(1:100,:)); hold on;
minn = min(x(:));
maxx = max(x(:));
l = line([nBurnin nBurnin],[minn maxx],'color','k','Linewidth',2);
ylim([minn maxx])
legend(l,'~Burn-in','Location','SouthEast')
title('First 100 Samples'); hold off

% DISPLAY ENTIRE MARKOV CHAIN
subplot(223); plot(x);hold on;
l = line([nBurnin nBurnin],[minn maxx],'color','k','Linewidth',2);
legend(l,'~Burn-in','Location','SouthEast')
title('Entire Chain');

% DISPLAY SAMPLES FROM STATIONARY DISTRIBUTION
samples = x(nBurnin+1:end,:);
subplot(122);
[counts,bins] = hist(samples(:),100); colormap hot
b = bar(bins,counts);
legend(b,sprintf('Markov Chain\nSamples'));
title(['\mu=',num2str(mean(samples(:))),' \sigma=',num2str(var(samples(:)))])

Sampling from the stationary distribution of a continuous state-space Markov chain

In the upper left panel of the code output we see a close up of the first 100 of the 1000 transitions made by the 5 simultaneous Markov chains; the burn in cutoff is marked by the black line. In the lower left panel we see the entire sequence of transitions for the Markov chains. In the right panel, we can tell from the sampled states that the stationary distribution for this chain is a Normal distribution, with mean equal to zero, and a variance equal to 1.3.

Wrapping Up

In the previous example we were able to deduce the stationary distribution of the Markov chain by looking at the samples generated from the chain after the burn in period. However, in order to use Markov chains to sample from a specific target distribution, we have to design the transition operator such that the resulting chain reaches a stationary distribution that matches the target distribution. This is where MCMC methods like the Metropolis sampler, the Metropolis-Hastings sampler, and the Gibbs sampler come to rescue. We will discuss each of these Markov-chain-based sampling methods separately in later posts.

^† On notation:

• Here we use the shorthand notation to correspond to , for some random variable
• A superscript in parentheses is an index into an iteration or point in time, not a power

Envelope distribution and rejection criterion

In order to be able to reject samples from such that they are sampled from , must “cover” or envelop the distribution . This is generally done by choosing a constant such that   for all . For this reason is often called the envelope distribution. A common criterion for accepting samples from is based on the ratio of the target distribution to that of the envelope distribution. The samples are accepted if

where , and rejected otherwise. If the ratio is close to one, then must have a large amount of probability mass around and that sample should  be more likely accepted. If the ratio is small, then it means that has low probability mass around and we should be less likely to accept the sample. This criterion is demonstrated in the chunk of MATLAB code and the resulting figure below:

rand('seed',12345);
x = -10:.1:10;
% CREATE A "COMPLEX DISTRIBUTION" f(x) AS A MIXTURE OF TWO NORMAL
% DISTRIBUTIONS
f = inline('normpdf(x,3,2) + normpdf(x,-5,1)','x');
t = plot(x,f(x),'b','linewidth',2); hold on;

% PROPOSAL IS A CENTERED NORMAL DISTRIBUTION
q = inline('normpdf(x,0,4)','x');

% DETERMINE SCALING CONSTANT
c = max(f(x)./q(x))

%PLOT SCALED PROPOSAL/ENVELOP DISTRIBUTION
p = plot(x,c*q(x),'k--');

% DRAW A SAMPLE FROM q(x);
qx = normrnd(0,4);
fx = f(qx);

% PLOT THE RATIO OF f(q(x)) to cq(x)
a = plot([qx,qx],[0 fx],'g','Linewidth',2);
r = plot([qx,qx],[fx,c*q(qx)],'r','Linewidth',2);
legend([t,p,a,r],{'Target','Proposal','Accept','Reject'});
xlabel('x');

Rejection Sampling with a Normal proposal distribution

Here a zero-mean Normal distribution is used as the proposal distribution. This distribution is scaled by a factor , determined from and to ensure that the proposal distribution covers . We then sample from , and compare the proportion of occupied by . If we compare this proportion to a random number sampled  from (i.e. the criterion outlined above), then we would accept this sample with probability proportional to the length of the green line segment and reject the sample with probability proportional to the length of the red line segment.

Rejection sampling of a random discrete distribution

This next example shows how rejection sampling can be used to sample from any arbitrary distribution, continuous or not, and with or without an analytic probability density function.

Random Discrete Target Distribution and Proposal that Bounds It.

The figure above shows a random discrete probability density function generated on the interval (0,15). We will use rejection sampling as described above to sample from . Our proposal/envelope distribution is the uniform discrete distribution on the same interval (i.e. any of the integers from 1-15 are equally probable) multiplied by a constant that is determined such that the maximum value of lies under (or equal to) .

Rejection Samples For Discrete Distribution on interval [1 15]

rand('seed',12345)
randn('seed',12345)

fLength = 15;
% CREATE A RANDOM DISTRIBUTION ON THE INTERVAL [1 fLength]
f = rand(1,fLength); f = f/sum(f);

figure; h = plot(f,'r','Linewidth',2);
hold on;
l = plot([1 fLength],[max(f) max(f)],'k','Linewidth',2);

legend([h,l],{'f(x)','q(x)'},'Location','Southwest');
xlim([0 fLength + 1])
xlabel('x');
ylabel('p(x)');
title('Target (f(x)) and Proposal (q(x)) Distributions');

% OUR PROPOSAL IS THE DISCRETE UNIFORM ON THE INTERVAL [1 fLength]
% SO OUR CONSTANT IS
c = max(f/(1/fLength));

nSamples = 10000;
i = 1;
while i < nSamples
   proposal = unidrnd(fLength);
   q = c*/fLenth; % ENVELOPE DISTRIBUTION
   if rand < f(proposal)/q
      samps(i) = proposal;
      i = i + 1;
   end
end

% DISPLAY THE SAMPLES AND COMPARE TO THE TARGET DISTRIBUTION
bins = 1:fLength;
counts = histc(samps,bins);
figure
b = bar(1:fLength,counts/sum(counts),'FaceColor',[.8 .8 .8])
hold on;
h = plot(f,'r','Linewidth',2)
legend([h,b],{'f(x)','samples'});
xlabel('x'); ylabel('p(x)');
xlim([0 fLength + 1]);

Rejection sampling from the unit circle to estimate

Though the ratio-based acceptance-rejection criterion introduced above is a common choice for drawing samples from complex distributions, it is not the only criterion we could use. For instance we could use a different set of criteria to generate some geometrically-bounded distribution. If we wanted to generate points uniformly within the unit circle (i.e. a circle centered at and with radius ), we could do so by sampling Cartesian spatial coordinates and uniformly from the interval (-1,1)–which samples form a square centered at (0,0)–and reject those points that lie outside of the radius

Unit Circle Inscribed in Square

Something clever that we can do with such a set of samples is to approximate the value : Because a square that inscribes the unit circle has area:

and the unit circle has the area:

We can use the ratio of their areas to approximate :

The figure below shows the rejection sampling process and the resulting estimate of from the samples. One-hundred thousand 2D points are sampled uniformly from the interval (-1,1).  Those points that lie within the unit circle are plotted as blue dots. Those points that lie outside of the unit circle are plotted as red x’s. If we take four times the ratio of the area in blue to the entire area, we get a very close approximation to 3.14 for .

Rejection Criterion

The MATLAB code used to generate the example figures is below:

% DISPLAY A CIRCLE INSCRIBED IN A SQUARE

figure;
a = 0:.01:2*pi;
x = cos(a); y = sin(a);
hold on
plot(x,y,'k','Linewidth',2)

t = text(0.5, 0.05,'r');
l = line([0 1],[0 0],'Linewidth',2);
axis equal
box on
xlim([-1 1])
ylim([-1 1])
title('Unit Circle Inscribed in a Square')

pause;
rand('seed',12345)
randn('seed',12345)
delete(l); delete(t);

% DRAW SAMPLES FROM PROPOSAL DISTRIBUTION
samples = 2*rand(2,100000) - 1;

% REJECTION
reject = sum(samples.^2) > 1;

% DISPLAY REJECTION CRITERION
scatter(samples(1,~reject),samples(2,~reject),'b.')
scatter(samples(1,reject),samples(2,reject),'rx')
hold off
xlim([-1 1])
ylim([-1 1])

piHat = mean(sum(samples.*samples)<1)*4;

title(['Estimate of \pi = ',num2str(piHat)]);

Wrapping Up

Rejection sampling is a simple way to generate samples from complex distributions. However, Rejection sampling also has a number of weaknesses:

• Finding a proposal distribution that can cover the support of the target distribution is a non-trivial task.
• Additionally, as the dimensionality of the target distribution increases, the proportion of points that are rejected also increases. This curse of dimensionality makes rejection sampling an inefficient technique for sampling multi-dimensional distributions, as the majority of the points proposed are not accepted as valid samples.
• Some of these problems are solved by changing the form of the proposal distribution to “hug” the target distribution as we gain knowledge of the target from observing accepted samples. Such a process is called Adaptive Rejection Sampling, which will be covered in another post.

The idea behind IT Sampling is that the probability mass for a random variable distributed according to the probability density function  integrates to one and therefore the cumulative distribution function can be used to map from values in the interval (i.e. probabilities) to the domain of . Because it is easy to sample values uniformly from the interval , we can use the inverse of the CDF to transform these sampled probabilities into samples . The code below demonstrates this process at work in order to sample from a student’s t distribution with 10 degrees of freedom.

rand('seed',12345)

% DEGREES OF FREEDOM
dF = 10;
x = -3:.1:3;
Cx = cdf('t',x,dF)
z = rand;

% COMPARE VALUES OF
zIdx = min(find(Cx>z));

% DRAW SAMPLE
sample = x(zIdx);

% DISPLAY
figure; hold on
plot(x,Cx,'k','Linewidth',2);
plot([x(1),x(zIdx)],[Cx(zIdx),Cx(zIdx)],'r','LineWidth',2);
plot([x(zIdx),x(zIdx)],[Cx(zIdx),0],'b','LineWidth',2);
plot(x(zIdx),z,'ko','LineWidth',2);
text(x(1)+.1,z + .05,'z','Color','r')
text(x(zIdx)+.05,.05,'x_{sampled}','Color','b')
ylabel('C(x)')
xlabel('x')
hold off

IT Sampling from student’s-t(10)

However, the scheme used to create to plot above is inefficient in that one must compare current values of with the for all values of . A much more efficient method is to evaluate directly:

1. Derive (or a good approximation) from
2. for

• - draw from
• -
• - end for

The IT sampling process is demonstrated in the next chunk of code to sample from the Beta distribution, a distribution for which  is easy to approximate using Netwon’s method (which we let MATLAB do for us within the function icdf.m)

rand('seed',12345)
nSamples = 1000;

% BETA PARAMETERS
alpha = 2; beta = 10;

% DRAW PROPOSAL SAMPLES
z = rand(1,nSamples);

% EVALUATE PROPOSAL SAMPLES AT INVERSE CDF
samples = icdf('beta',z,alpha,beta);
bins = linspace(0,1,50);
counts = histc(samples,bins);
probSampled = counts/sum(counts)
probTheory = betapdf(bins,alpha,beta);

% DISPLAY
b = bar(bins,probSampled,'FaceColor',[.9 .9 .9]);
hold on;
t = plot(bins,probTheory/sum(probTheory),'r','LineWidth',2);
xlim([0 1])
xlabel('x')
ylabel('p(x)')
legend([t,b],{'Theory','IT Samples'})
hold off

Inverse Transform Sampling of Beta(2,10)

Wrapping Up

The IT sampling method is generally only used for univariate distributions where can be computed in closed form, or approximated. However, it is a nice example of how uniform random variables can be used to sample from much more complicated distributions.

Suppose that the data is d-dimensional so that . We will use the product kernel

The risk is given by

Multivariate density estimation.  (Photo credit: Wikipedia)

where is the second partial derivative of . The optimal bandwidth satisfies leading to a risk of order (for further details see Hardle (2004)).

The interesting effect of here is that the risk increase exponentially as the dimension grows. We call to this behavior the curse of dimensionality. This phenomena says that the data is more sparse as we increase the dimensionality. This table from Silverman (1986) shows the sample size required to ensure a relative mean squared error less than 0.1 at 0 when the density is multivariate normal and the optimal bandwidth is selected.

For this reason it is important to search methods for dimension reduction. One of these methods was proposed by Li (1991) in its article Sliced Inverse Regression for Dimension Reduction. I used this method to find another efficient estimator based in a Taylor approximation (see Solís Chacón, M et al. (2012) ). In a next post I going to talk a little about the details of those articles.

Sources:

• Hardle, W. (2004). Nonparametric and Semiparametric Models. Springer Series in Statistics. Springer.
• Li, K.-C. (1991). Sliced Inverse Regression for Dimension Reduction. Journal of the American Statistical Association, 86(414), 316-327. Retrieved from http://www.jstor.org/stable/2290563
• Tsybakov, A. (2009). Introduction to nonparametric estimation. Springer.
• Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis, volume 26. Chapman & Hall/CRC.
• Solís Chacón, M., Loubes, J.-M., Clement, M. & Da Veiga, S. (2012). Efficient estimation of conditional covariance matrices for dimension reduction. Arxiv preprint arXiv: Retrieved from http://arxiv.org/abs/1110.3238

The normal reference rule

This method works well only if the true density is very smooth. Assume that is normal distributed. Then we have

Usually is estimated by where is the sample standard deviation and is the interquartile range. Recall that the interquartile range is the percentile minus the percentile. Here, gives a consistent estimate of if the data comes from a .

We can summarize this method in the following way

Cross-validation

Define the integrated squared error as

Notice that the MISE is indeed the expected value of ISE. Our goal is minimize the ISE as small as possible. Remark that the last term in (0) does not depends on , so minimize this risk the is equivalent to minimizing the expected value of

If we look closer the term we notice that is the expected value of . The straight estimate for this expected value is

The problem with it is that the observations to estimate the expectation are dependent of the observations to estimate . The solution to solve this, it is remove the observation for . Then, we define the leave-one-out cross-validation estimator of as

where

The following figure illustrates the idea behind the leave-one cross validation. The idea is to take one data point as your test data and the rest as your training data for each iteration.

Leave-one-out cross-validation

Following with the term we have

where means the convolution of with itself.

Finally it is possible define a reasonable criterion to choose the bandwidth,

Note: An alternative way to implement the leave-one-out cross validation is,

and then calculate numerically the integral.

Sources:

• Hardle, W. (2004). Nonparametric and Semiparametric Models. Springer Series in Statistics. Springer.
• Tsybakov, A. (2009). Introduction to nonparametric estimation. Springer.
• Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis, volume 26. Chapman & Hall/CRC.

Related articles

• Check your missing-data imputations using cross-validation (andrewgelman.com)

A 1D Kernel density estimation

I will make a summary of ideas about nonparametric estimation, including some basics results to develop more advanced theory later.

In the first post  we talk something about the density estimation and the nonparametric regression. Later, in posts about histogram (I,II,III,IV) , we saw how the histogram is a nonparametric estimator and we studied its properties.

Now, we are ready to go one step further and study the kernel density estimators in general.

1. Kernel density estimation

Let i.i.d random variables with common probability in . The distribution function of is . Consider the empirical distribution function

By the strong law of large numbers we have for all in almost surely as . Therefore, is a consistent estimator for all in . The natural question is: How can we estimate

The following argument was one of the first solutions. For small we have

Replacing by its estimator , define

where is the Rosenblatt estimator. We can rewrite it in the form

where , which is in fact the histogram estimator.

In general,

where , and . The function is the kernel and is the bandwidth which depends on . The function is the kernel (Parzen-Ronsenblatt) estimator.

Some classical examples of kernels are:

• Rectangular: .
• Triangular: .
• Epanechnikov: .
• Gaussian: .

2. Mean Squared Error (or Risk) of Kernel Estimators

At an arbitrary point we have

A simple calculation gives us

where is the bias and is the variance of the estimator and we define as

We shall analyze both terms separately.

2.1. Variance of the estimator

Proposition 1 Suppose that the density satisfies for all . Let be a function such that

Then for any , , and we have

where

Proof: We have

Finally if as then the variance goes to 0 as .

2.2. Bias of the estimator

The bias has the form

To analyze the behavior of as function of , we need some regularity conditions on the density and the kernel . Denote as the integer part of the real number .

Definition 2 Let be an interval in and let and be two positives numbers. The Holder class on is the set of times differential functions whose derivative satisfies

Definition 3 Let be an integer. We say that is a kernel of order if the functions , , are integrable and satisfy

Often in the literature exists an alternative definition of an order for a kernel. A kernel is of order if Definition 2 holds and which is more restrictive. Define the class of densities as follows:

Proposition 2 Assume that and let be a kernel of order satisfying

Then for all , and we have

where

Proof: We have

Next, since is a kernel of order and making a Taylor’s development of , for , we obtain,

and

2.3. Upper bound on the mean squared risk

We see from before that the bias and variance behave in opposite direction. If we chose a small we get a large variance and for consequence an undersmoothed estimator. By the other side, if is large the bias cannot be controlled which lead to an oversmoothed estimator.

If the assumptions of Propositions 2 and 2 hold, we get

The minimum with respect to is attained at

and for

uniformly in . For and , those observations lead us to get

where is a constant depending only on , , and the kernel .

The next time, we will formalize the MISE (Mean Integrated Squared Error) and study its properties.

If you have any comments/suggestions/improvements please let me know below.

Source: Tsybakov, A. (2009). Introduction to nonparametric estimation. Springer.

What is exactly the if we assume that

How is the density of standard normal distribution. It is easy to see that , so we have

We recognized the integral as the variance of a random variable with mean equal to 0 and variance equal to . Then we get,

Finally, using the results of Part III,

This is not useful in many cases, but it could work as a rule-of-thumb bindwidth.

For the example, we will use a sample of 1000 normal distributed numbers and we will try to use all the theory seen before. I did this little script in R to compute the MSE, MISE and plot the bias-variance trade-offs  between them (Note: this is my very first script in R so I will appreciate any comment ).

<pre># Function to get the breaks
breaks <- function(x,h){
  b = floor(min(x)/h) : ceiling(max(x)/h)
  b = b*h
  return(b)
}
#Generate 1000 numbers with normal standard distribution
x=rnorm(1000)
n=length(x);
# Point to evaluate the MSE
x0 = 0
# Real value of f(x0)
f_x0 = dnorm(x0);
# || f' ||_2^2 for a normal standard distribution
norm_f_prime = 1/(4*sqrt(pi));
#Sequence of bins
hvec = seq(0.1,0.7,by=0.0005)
Bias = numeric();
Var = numeric();
MSE = numeric();
Bias_MISE = numeric();
Var_MISE = numeric();
MISE = numeric();

par(mfrow=c(1,2))
hist(x,breaks=breaks(x,0.001),freq=F,xlab ="Bandwidth with h=0.001")
hist(x,breaks=breaks(x,2),freq=F,xlab ="Bandwidth with h=2")
par(mfrow=c(1,1))

for(h in hvec){
  xhist = hist(x,breaks=breaks(x,h),plot=FALSE)
  # Average of bins near x0
  bins_near_x0 = xhist$breaks=x0-h;
  p = mean(xhist$counts[bins_near_x0])/n;
  # Expectation of \hat{f} in x0
  E_fhat_x0 = p/h;
  #Compute of Bias, Var, MSE and MISE
  Bias = c(Bias,E_fhat_x0 - f_x0);
  Var = c(Var,(p*(1-p))/(n*h^2))
  MSE = c(MSE,tail(Var,1) + tail(Bias,1)^2)
  Var_MISE = c(Var_MISE,1/(n*h))
  Bias_MISE = c(Bias_MISE,h^2 * norm_f_prime /12)
  MISE = c(MISE,tail(Var_MISE,1) + tail(Bias_MISE,1))
}

#Tradeoff MSE in x0
max_range = range(Bias^2,Var,MSE)
plot(hvec,MSE,ylim=max_range,type="l",col="blue",lwd=3, xlab="Bandwidth h")
lines(hvec,Bias^2,type="l",lty=2,col="red",lwd=3)
lines(hvec,Var,type="S",lty=6,col="black",lwd=3)
legend(x="topleft",max_range,legend=c("MSE","Bias^2","Var"),col=c("blue","red","black"),lwd=3,lty=c(1,2,6))

# Tradeoff MISE
max_range = range(Bias_MISE,Var_MISE,MISE)
plot(hvec,MISE,ylim=max_range,type="l",col="blue",lwd=3,xlab="Bandwidth h")
lines(hvec,Bias_MISE,type="l",lty=5,col="red",lwd=3)
lines(hvec,Var_MISE,type="S",lty=6,col="black",lwd=3)
legend(x="topleft",max_range,legend=c("MISE","Bias^2_MISE","Var_MISE"),col=c("blue","red","black"),lwd=3,lty=c(1,2,6))

# h optimal for the point x0
h_x0=hvec[MSE==min(MSE)]

# h optimal for any point using the minimal MISE
h_opt_MISE=hvec[MISE==min(MISE)]

# h optimal for any point using the rule-of-thumb
h_opt = (6/(n*norm_f_prime))^(1/3)# histogram with h_opt
breaks = floor((min(x))/h_opt):ceiling((max(x))/h_opt)
breaks = breaks*h_opt#Plot the histogram
h=hist(x,breaks=breaks,freq=FALSE,ylim=c(0,0.5))
lines(sort(x),dnorm(sort(x)),type="l")

To start notice that if the binwidth is too small or too large we will get a very bad approximation. As we can see, in the next plot, we will get a terrible fitting of the histogram for and :

Comparison of Histograms with different Binwidths

The plots of the MSE and MISE trade-offs are respectively

Trade-off Bias-Variance of Histogram in x0=0

Trade-off Bias-Variance for MISE

For the MSE in the optimal value is . In the case of MISE the optimal value by minimization is and the real value, using the formula seen before, is . Finally, we get the best fitted histogram:

Histogram of 1000 numbers normal standard distributed with h=0.349083021225025

The next week, we will start with density estimation using kernels more generals and we will see that the histogram is, in fact, a particular case of one of them.

Related articles

• Density Estimation by Histograms (Part II) (maikolsolis.wordpress.com)
• Density Estimation by Histograms (Part III) (maikolsolis.wordpress.com)

Example of linear regression

I would like to start with some basic ideas about density estimation and nonparametric regression.

The study of the probability density function (pdf) is called nonparametric estimation. This kind of estimation can serve as a block building in nonparametric regression.

Let us the typical linear regression problem. Assume that we have a set of explanatory variables and an explained variable related in the following way:

where is independent of and . We can also see the model 1 as,

Here is the conditional expectation of given . It is clear  that the model 2 is not enough for many complex problems. To tackle this situation we will transform 2 into the model:

Here is the true, unknown regression function.

Just to get the things in perspective. Let and assume that the model is

Now given a data sample, suppose that you have to estimate as accurately as possible in one trial. That means, you can not change the model if the data does not fit well.

Of course you can simply use 4, but in general you do not know any information about the function (except that it is a smooth function). We call this type of regression like nonparametric and we will return to it later.

In the next post we will study the nonparametric density estimation and its importance.

We know that for any p.d. kernel there exists a deterministic map that has the aforementioned property but it may be infinite dimensional. The paper presents results indicating that with the randomized map we  can get away with only a “small” number of features (at least for a classification setting).

Before applying the method to density estimation let us review the relevant section of the paper briefly.

Bochner’s Theorem and Random Fourier Features

Assume that we have data in and a continuous p.d. kernel defined for every pair of points . Assume further that the kernel is shift-invariant, i.e., and that the kernel is scaled so that .

The theorem by Bochner states that under the above conditions must be the Fourier transform of a non-negative measure on . In other words, there exists a probability density function for such that .

where (1) is because is real. Equation (2) says that if we draw a random vector according to and form two vectors and , then the expected value of is .

Therefore, for , if we choose the transformation

with drawn according to , linear inner products in this transformed space will approximate .

Gaussian RBF Kernel

The Gaussian radial basis function kernel satisfies all the above conditions and we know that the Fourier transform of the Gaussian is another Gaussian (with the reciprocal variance). Therefore for “linearizing” the Gaussian r.b.f. kernel, we draw samples from a Gaussian distribution for the transformation.

Parzen Window Density Estimation

Given a data  set  , the the so-called Parzen window probability density estimator is defined as follows

where is often a positive, symmetric, shift-invariant kernel and is the bandwidth parameter that controls the scale of influence of the data points.

A common kernel that is used for Parzen window density estimation is the Gaussian density. If we make the same choice we can apply our feature transformation to linearize the procedure. We have

where has been absorbed into the kernel variance.

Therefore all we need to do is take the mean of the transformed data points and estimate the pdf at a new point to be (proportional to) the inner product its transformed feature vector with the mean.

Of course since the kernel value is only approximated by the inner product of the random Fourier features we expect that the estimate pdf will differ from a plain unadorned Parzen window estimate.  But different how?

Experiments

Below are some pictures showing how the method performs on some synthetic data. I generated a few dozen points from a mixture of Gaussians and plotted contours of the estimated pdf for the region around the points. I did this for several choices of and (the scale parameter for the Gaussian kernel).

First let us check that the method performs as expected for large values of because the kernel value is well approximated by the inner product of the Fourier features. The first 3 pictures are for for various values of .

D = 10000 and gamma = 2.0

D = 10000 and gamma = 1.0

D = 10000 and gamma = 0.5

—————————————————————————

—————————————————————————

Now let us see what happens when we decrease . We expect the error in approximating the kernel would lead to obviously erroneous pdf.  This is clearly evident for the case of .

D=1000 and gamma = 1.0

D=100 and gamma = 1.0

—————————————————————————

—————————————————————————

The following picture for   and is even stranger.

D = 1000 and gamma = 2.0

—————————————————————————

—————————————————————————

Discussion

It seems that even for a simple 2D example, we seem to need to compute a very large number of random Fourier features to make the estimated pdf accurate. (For this small example this is very wasteful, since a plain Parzen window estimate would require less memory and computation.)

However, the pictures do indicate that if the approach is to be used for outlier detection (aka novelty detection) from a given data set, we might be able get away with much smaller . That is, even if the estimated pdf has a big error on the entire space, on the points from the data it seems to be reasonably accurate.

As a part of introducing nonparametric statistics, I wanted to write about applications of computation geometry from the nonparametric 2/3 dimensional density estimation perspective. Also, the following article came along when I just began to collect statistical applications in astronomy (my [ArXiv] series). This [arXiv] paper, in fact, initiated me to investigate Voronoi Tessellations in astronomy in general.

[arxiv/astro-ph:0707.2877]
Voronoi Tessellations and the Cosmic Web: Spatial Patterns and Clustering across the Universe
by Rien van de Weygaert

Since then, quite time has passed. In the mean time, I found more publications in astronomy specifically using tessellation as a main tool of nonparametric density estimation and for data analysis. Nonetheless, in general, topics in spatial statistics tend to be unrecognized or almost ignored in analyzing astronomical spatial data (I mean data points with coordinate information). Many seem only utilizing statistics partially or not at all. Some might want to know how often Voronoi tessellation is applied in astronomy. Here, I listed results from my ADS search by limiting tessellation in title key words. :

• [arxiv/astro-ph:0110259]
  Detecting Clusters of Galaxies in the Sloan Digital Sky Survey I : Monte Carlo Comparison of Cluster Detection Algorithms
  by Kim, R.S.J. et al. (2002) AJ, 123, pp.20-36.
• [arxiv/astro-ph:0906.1905]
  The VOISE Algorithm: a Versatile Toll for Automatic Segmentation of Astronomical Images
  by Guio, P. and Achilleos, N. (2009)
• Using Voronoi Techniques to determine the shapes of photon sources
  by Wilkinson and Meurs Irish Astronomical Journal, 1998, 25(1), 37
• High-order 3D Voronoi tessellation for identifying isolated galaxies, pairs and triplets
  by Elyiv, A.; Melnyk, O.; Vavilova, I. 2009..MNRAS..394..1409E
• 3-D Voronoi’s Tessellation as a Tool for Identifying Galaxy Groups
  by Melnyk, Olga V.; Elyiv, Andrii A.; Vavilova, Iryna B. 2007..IAUS..235..223M
• Adaptive binning of X-ray data with weighted Voronoi tessellations
  by Diehl, Steven; Statler, Thomas S. 2006..MNRAS..368..497D
• Adaptive spatial binning of integral-field spectroscopic data using Voronoi tessellations
  by Cappellari, M. and Copin, Y. 2003..MNRAS..342..345C
• Adaptive Spatial Binning of 2D Spectra and Images Using Voronoi Tessellations
  by Cappellari, M.; Copin, Y. 2002..ASPC..282..515CA
• Finding galaxy clusters using Voronoi tessellations
  by Ramella, M.; Boschin, W.; Fadda, D.; Nonino, M. 2001..A&A…368..776R
• The Forest Method as a New Parallel Tree Method with the Sectional Voronoi Tessellation
  by Yahagi, Hideki; Mori, Masao; Yoshii, Yuzuru 1999..ApJS..124..1
• Cluster Identification via Voronoi Tesselation ..1999..ASPC..176..108
• The accuracy of parameters determined with the core-sampling method: Application to Voronoi tessellations 1997..A&AS..123..495
• Dynamical Voronoi tessellation. V. Thickness and incompleteness.
  by Zaninetti, L 1995..A&AS..109..71
• Fragmenting the Universe. 3: The constructions and statistics of 3-D Voronoi tessellations
  by van de Weygaert, Rien 1994..A&A..283..361
• Dynamical Voronoi tessellation. IV. The distribution of the asteroids
  by Zaninetti, L 1993..A&A..276..255
• Quasi-periodic structures in the large-scale galaxy distribution and three-dimensional Voronoi tessellation
  1991..MNRAS..250..519
• Dynamical Voronoi tessellation. III – The distribution of galaxies
  by Zaninetti, L 1991..A&A..246..291
• Dynamical Voronoi tessellation. II – The three-dimensional case
  by Zaninetti, L 1990..A&A..233..293
• Dynamical Voronoi tessellation. I – The two-dimensional case
  by Zaninetti, L 1989..A&A..224..345

Then, the topic has been forgotten for a while until this recent [arXiv] paper, which reminded me my old intention for introducing tessellation for density estimation and for understanding large scale structures or clusters (astronomers’ jargon, not the term in machine or statistical learning).

[arxiv:stat.ME:0910.1473] Moment Analysis of the Delaunay Tessellation Field Estimator
by M.N.M van Lieshout

Looking into plots of the papers by van de Weygaert or van Lieshout, without mathematical jargon and abstraction, one can immediately understand what Voronoi and Delaunay Tessellation is (Delaunay Tessellation is also called as Delaunay Triangulation (wiki). Perhaps, you want to check out wiki:Delaunay Tessellation Field Estimator as well). Voronoi tessellations have been adopted in many scientific/engineering fields to describe the spatial distribution. Astronomy is not an exception. Voronoi Tessellation has been used for field interpolation.

van de Weygaert described Voronoi tessellations as follows:

1. the asymptotic frame for the ultimate matter distribution,
2. the skeleton of the cosmic matter distribution,
3. a versatile and flexible mathematical model for weblike spatial pattern, and
4. a natural asymptotic result of an evolution in which low-density expanding void regions dictate the spatial organization of the Megaparsec universe, while matter assembles in high-density filamentary and wall-like interstices between the voids.

van Lieshout derived explicit expressions for the mean and variance of Delaunay Tessellatoin Field Estimator (DTFE) and showed that for stationary Poisson processes, the DTFE is asymptotically unbiased with a variance that is proportional to the square intensity.

We’ve observed voids and filaments of cosmic matters with patterns of which theory hasn’t been discovered. In general, those patterns are manifested via observed galaxies, both directly and indirectly. Individual observed objects, I believe, can be matched to points that construct Voronoi polygons. They represent each polygon and investigating its distributional properly helps to understand the formation rules and theories of those patterns. For that matter, probably, various topics in stochastic geometry, not just Voronoi tessellation, can be adopted.

There are plethora information available on Voronoi Tessellation such as the website of International Symposium on Voronoi Diagrams in Science and Engineering. Two recent meeting websites are ISVD09 and ISVD08. Also, the following review paper is interesting.

Centroidal Voronoi Tessellations: Applications and Algorithms (1999) Du, Faber, and Gunzburger in SIAM Review, vol. 41(4), pp. 637-676

By the way, you may have noticed my preference for Voronoi Tessellation over Delaunay owing to the characteristics of this centroidal Voronoi that each observation is the center of each Voronoi cell as opposed to the property of Delaunay triangulation that multiple simplices are associated one observation/point. However, from the perspective of understanding the distribution of observations as a whole, both approaches offer summaries and insights in a nonparametric fashion, which I put the most value on.

The sixth data release of the Sloan Digital Sky Survey (SDSS DR6) is available at http://www.sdss.org/dr6. Additionally, Catalog Archive Service (CAS) and
SQL interface to access the catalog would be useful to data searching statisticians. Simple SQL commends, which are well documented, could narrow down the size of data and the spatial coverage.

Part of my dissertation was about creating nonparametric multivariate analysis tools with convex hull peeling and I used SDSS DR4 to apply those convex hull peeling tools to explore celestial objects in the multidimensional color space without projections (dimension reduction). SDSS CAS might fulfill the needs of those who are looking for data sets to conduct

• massive multivariate data analysis,
• streaming data analysis (strictly, SDSS is not streaming but the data base is updated yearly by adding new observations and depending on memory, streaming data analysis can be easily simulated) and
• application of his/her new machine learning and statistical multivariate analysis tools for new discoveries.

Particularly, thanks to whole northern hemisphere survey, interesting spatial statistics can be developed such as voronoi tessellation for spatial density estimation. It also provides a vast image reservoir as well as the catalog of massive multivariate spatial data.

Oh, by the way, the paper discusses changes and improvement in the recent data release. The SDSS DR6 includes the complete imaging of the Northern Galactic Cap and contains images and parameters of 287 million objects over 9583 deg^2, and 1.27 million spectra over 7425 deg^2. The photometric calibration has improved with uncertainties of 1% in g,r,i and 2% in u, significantly better than previous data releases. The method of spectrophotometric calibration has changed and resulted 0.35 mags brighter in the spectrophotometric scale. Two independent codes for spectral classifications and redshifts are available as well.