http://theanalysisofdata.com/probability/viewer1.html (single page viewer)

http://theanalysisofdata.com/probability/viewer2.html (two page viewer)

http://theanalysisofdata.com/probability/0/0-2.pdf (table of contents)

http://theanalysisofdata.com/computing/R.pdf (R programming pdf version)

http://theanalysisofdata.com/computing/graphics.pdf (R graphics pdf version)

The book represents my best effort at capturing the part of probability that is necessary and sufficient for an in-depth understanding of machine learning and statistics. It does not cover statistics, which will be described in a later volume. The book is part of the Analysis of Data Project (TAOD), which provides educational material on the area of data analysis.

• The project features comprehensive coverage of all relevant disciplines including probability, statistics, computing, and machine learning.
• The content is almost self-contained and includes mathematical prerequisites and basic computing concepts.
• The R programming language is used to demonstrate the contents. Full code is available, facilitating reproducibility of experiments and letting readers experiment with variations of the code.
• The presentation is mathematically rigorous, and includes derivations and proofs in most cases.
• HTML versions are freely available on the website http://theanalysisofdata.com. Hardcopies are available at affordable prices.

Example of syntax highlighting in R:

<pre lang=”rsplus”>
a=4 # assigning value of 4 to a
a # print value of a
</pre>

Example of equation using LaTeX code:

Example of syntax highlighting in R:

a=4 # assigning value of 4 to a
a # print value of a

EM is a special case of coordinate descent algorithms known as majorization-minimization procedures (MMP). Majorization-minimization procedures operate by iteratively (nearly-) minimizing an upper bound of the objective which is tight at the previous iteration’s (near-) minimizer. In other words, to guarantee the procedure minimizes our objective , we require the following three properties:

That these conditions guarantee convergence, i.e., , follows from the so-called descent property, i.e.,

From this view we see that EM is a special case, i.e.,

This result (and that the MM conditions are met) follows from Gibbs’ Inequality, i.e., for all distributions with equality iff is identically .

For more information and a handful of alternative majorization recipes, see the excellent 2004 tutorial by David Hunter and Kenneth Lange, A Tutorial on MM Algorithms. For a clever application of MM, check-out this paper by Andrew Ng and pals.

Most (all?) of the majorizers listed by Lange exploit convexity of the objective. When the objective is nonlinear, it is always possible to rewrite it as a difference of convex functions and then apply the MMP to the subtracted convex function. However it is typically quite challenging to formulate as a difference of convex functions.

Miscellanea:
Alternative (equivalent) ways to derive EM:

Alt1: (KL-Div)

where the bound follows from Gibbs’ inequality.

Alt2: (Jensen)

where the bound follows from Jensen’s inequality.

Note that all of these derivations essentially just use and the definition of convexity. For a great (but dated) overview on different incarnations of EM, see this survey.

(this does not reflect the views of the anyone in the smlv group – post is intended as a comic relief)

Two classes that are relevant to my area are AI and Machine Learning. Both will for sure be outstanding. The first is taught by Sebastian Thrun and Peter Norvig – both leading researchers in AI. The second is taught by Andrew Ng who is also an outstanding researcher. I also believe all three are excellent teachers.

58,000 people already registered for the AI class. What a nice opportunity for grad students, but also for undergrads, highschool students, professionals, and retired people.

SEMAPHORE Interfaces with POSIX semaphore.

This mex file provides an interface with the POSIX semaphore functionality. For more information, see [1,2].

SEMAPHORE(‘create’,KEY,VAL)
Initializes a semaphore which can later by accessed by KEY. The argument VAL specifies the initial value for the semaphore.

SEMAPHORE(‘destroy’,KEY)
Destroys the semaphore indexed by KEY. Destroying a semaphore that other processes or threads are currently blocked on (in ‘wait’) produces undefined behavior. Using a semaphore that has been destroyed produces undefined results, until the semaphore has been reinitialized using ‘init’.

SEMAPHORE(‘wait’,KEY)
Decrements (locks) the semaphore indexed by KEY. If the semaphore’s value is greater than zero, then the decrement proceeds, and the function returns, immediately. If the semaphore currently has the value zero, then the call blocks until either it becomes possible to perform the decrement (i.e., the semaphore value rises above zero), or a signal handler interrupts the call.

SEMAPHORE(‘post’,KEY)
Increments (unlocks) the semaphore indexed by KEY. If the semaphore’s value consequently becomes greater than zero, then another process or thread blocked in a ‘wait’ call will be woken up and proceed to lock the semaphore.

[1] – `man 7posix semaphore.h`
[2] – http://en.wikipedia.org/wiki/Semaphore_(programming)

>> S = rand(2000);
>> S = S + S’;
>> t= tic();[v,e] = eig(S);toc(t);
Elapsed time is 20.249647 seconds.
>> t= tic();S(1,2) = S(1,2) + 0.0001;[v,e] = eig(S);toc(t);
Elapsed time is 16.336485 seconds.

Stupid enough. I really need eigenvectors faster so I wrote a simple wrapper with Intel MKL for eigenvector computation on symmetric matrices.

http://www.cc.gatech.edu/~fli/mex_dsyev.mexa64

http://www.cc.gatech.edu/~fli/mex_dsyev.cpp

>> t = tic();[v,e] = mex_dsyev(S);toc(t);
Elapsed time is 2.971105 seconds.

Experiments are done on dice with 8 cores. You can see that MKL implicitly use all the 8 cores to compute, that’s the simplest way to parallelize to me.

I’ll look to compile it in Windows, but I don’t have MKL in Windows for now.

In Matlab this is particularly annoying when one wishes to compute the dot product p’*log(p) where p is a pmf shaped as a column vector since the result is NaN whenever one or more elements of p are zero.

One workaround is to use the statistics toolbox, i.e., nansum(p.*log(p)). However this method is substantially slower, uses more intermediate memory, and doesn’t always work well for more general multiplication operators (tensor product).

A neat trick is to exploit how Matlab sparse vectors work. Assuming length(p)>1, the desired result can be achieved by: sparse(p)’*log(p) since Matlab assumes the product of any zero element and some other element is zero.

Interestingly enough, although favorable for statistical computing, it does violate the implications of floating-point conventions and could lead to unexpected results.

I’m new to the group and blog but somehow think that I should at least post something. But I’d never figured out what to write here. Well I’ve got some small stuff today. Because we discussed about Hadoop at some point and we were talking about doing parallel optimization and potential out-of-sync from different processors doing a part of the task and possibilities to make them communicate at stages, I’m sorta interested in seeing how bad will the syncronization problem be.

Today I’m trying to clog the server by running multiple optimizations in parallel. And then I realized that would a small showcase of the sync problem. My setting is like I run 4 different pretty slow optimizations (same dataset one-against-all classification, therefore with different labels) on 4 CPUs on the server simulatenously using MATLAB’s parfor. And one can see the output of the 4 optimizations coming back in parallel. I’ve drawn this small figure to show the synchronization behaviors of these 4 processes.

So with this pretty basic setting (all processes on the same machine, using the same processor and running almost the same optimization), they are out-of-sync starting from around 14 iterations. However the sync problem is not huge in this case. One expects to get around +-2 iterations deviation between the processes.

Conclusion: 1) Out-of-sync exists even with very constrained setting. 2) The amount of it may be controlled when the setting is well-constrained.

Question: What happens if we have both global sync problem (sync between different computers on a grid) and local ones (sync within one computer), is there something to do to take advantage of the structure that local sync is more controllable than the global? Suppose each computer in the grid has 100 cores, is there something to do to exploit this architecture so we perform some better-synced parallel optimization within the computer and less well-synced ones outside on the less reliable network?

So despite swearing I’ll never do it I decided to take the plunge and get a Mac – specifically Macbook Air 13.3”. The verdict is that it’s a great laptop but I still like many things in Linux that Apple annoyingly decided to do differently. I include below some tips on making a Mac behave more like Linux. The steps below are listed in chronological order – execute them one by one right when you get your Mac.

1. When the computer asks for your full name and the user name choose two identical short names (think username for both). This ensures that you won’t get your full name in some places where linux expects the username.
2. Follow the steps in http://support.apple.com/kb/HT1528 to enable root account (that means you can sometimes use the su command rather than sudo). This is necessary for some of the steps.
3. To move your home directory from /User/username to /home/username. Start by “unlocking” the /home directory by commenting out the line in /etc/auto_master that starts with /home. Then remove and re-create the directory /home/username (use sudo for both actions). Next, move your home directory by CTRL-click the user name in the accounts part of the system preferences (first unlock it by entering your password). If you are doing that after the initial installation you may have files in your old home directory that should be moved (while making sure to change their ownership to the new username if needed).
4. Mac doesn’t let you maximize windows to full screen. It thinks it is smarter than you and decides by itself how to react when you press the + button to maximize a window. This is particularly annoying for laptops with small screens. Installing the application Right Zoom (http://www.macupdate.com/app/mac/30591/right-zoom) let’s you change this behavior. I recommend selecting the options apply for all applications and full screen maximization when the option button is pressed. This means you can still use Apple’s window maximization (click on window max button) or full screen maximization (click on window max button while pressing the Option key).
5. The default hostname in Mac is your computer name appended by a network id. This is bad since whenever you connect to the internet from different spots your hostname will be different which prevents you for example from using ssh with automatic authentication (no need to enter password every time). I am sure there are other complications caused by this. The hostname should be changed in two places to one short name: once in system_preferences->sharing and once with

   sudo scutil --set HostName newName

6. You probably want to customize your bash shell- either write it from scratch or import it from a previous computer. First, note that Mac uses .bash_profile rather than .bashrc so simply take your .bashrc file and rename it to .bash_profile. Second, remove any –color mention in your ls aliases since this will not work and cause Mac to complain. To show colors in ls put the following or something similar in your .bash_profile. This works nicely for black background shell but you may want to change the colors if you prefer white background

   export CLICOLOR=1 export LSCOLORS=Gxfxcxdxbxegedabagacad

7. Create an Apple account, install X-code, and then install Macports. You are now free to install a lot of linux tools including many excellent gnome programs:

   sudo port install program_name

8. Proceed with the standard linux setup depending on your preferences: create ssh keys for automatic authentication, install emacs (my favorite is http://vgoulet.act.ulaval.ca/en/emacs/mac/ which includes ESS), latex (type sudo port install texlive), R, git, libreoffice, etc. Many of these programs may be installed either using macports (see above) or by installing the Mac application (search google, download dmg file and double click it). I chose the latter which works nicely.