TSNE test

We’ve recently been using the R tsne package to implement the TSNE plotting in SeqMonk, but with somewhat unexpectedly poor results, even from quite different datasets. I’ve now been made aware that there is also an Rtsne package, and that several people have reported problems with the tsne package which weren’t present in Rtnse so I wanted to test both performance and output from the two.

I have a smallish dataset (available from here) with big differneces in it so it should be really easy to separate.

setwd("C:/Users/andrewss/Desktop/Tsne_test/")
read.delim("tsne_data.txt") -> tsne.data
rownames(tsne.data) <- tsne.data$Probe
tsne.data[14:31] -> tsne.data
tsne.data

dim(tsne.data)

[1] 1510   18

PCA

As a point of comparison we’ll start with a simple PCA to see how well that does.

start.time <- proc.time()
prcomp(t(tsne.data)) -> pca.result
proc.time()-start.time

   user  system elapsed 
   0.01    0.00    0.03 

We can plot this to see how well it separates.

plot(
  pca.result$x[,1], 
  pca.result$x[,2], 
  pch=19, xlab="PCA PC1", 
  ylab="PCA PC2", 
  main="Separation with PCA",
  col=as.factor(gsub("_.*","",colnames(tsne.data)))
  )
text(pca.result$x[,1], pca.result$x[,2],gsub("_.*","",colnames(tsne.data)),pos=1, cex=0.7)

So this data separates ridiculously well (it was selected to), and works very quickly through PCA.

tsne

Now let’s try the same thing with TSNE.

We’ll try the tsne package first

library(tsne)
start.time <- proc.time()
tsne(t(tsne.data), perplexity = 5) -> tsne.result
proc.time()-start.time

   user  system elapsed 
   9.67    0.00    9.69 

So for a pretty small dataset (18 samples and 1510 measures), it’s pretty slow (~300x slower than PCA).

Let’s plot the results to see if it looks sensible

plot(
  tsne.result, 
  pch=19,
  col=as.factor(gsub("_.*","",colnames(tsne.data))),
  xlab="TSNE dim1", 
  ylab="TSNE dim2", 
  main="Separation with tsne package")
text(tsne.result[,1],tsne.result[,2],gsub("_.*","",colnames(tsne.data)),pos=1, cex=0.7)

Not exactly earth shattering separation.

Rtsne

Now lets try the Rtsne package

library(Rtsne)

start.time <- proc.time()
Rtsne(t(tsne.data), perplexity = 5) -> rtsne.result
proc.time()-start.time

Well that’s a whole lot faster (about 3X slower than PCA) - which seems a lot more reasonable. Let’s look at the separation

plot(
  rtsne.result$Y,
  col=as.factor(gsub("_.*","",colnames(tsne.data))),
  pch=19, 
  xlab="TSNE dim1", 
  ylab="TSNE dim2", 
  main="Separation with the Rtsne package"
  )
text(rtsne.result$Y[,1],rtsne.result$Y[,2],gsub("_.*","",colnames(tsne.data)),pos=1, cex=0.5)

Well that looks much more reasonable, so the slowness and poor separation in the tnse package isn’t inherent to the procedure, but are something to do with the implementation.

We’re not the only ones to spot this - there’s even a patch which fixed a very similar looking error which was submitted back in 2016 and is supposedly in v1.3 (the one I used here), but doesn’t seem to have actually fixed it (or wasn’t included).

Regardless of any bugs, the speed difference alone would mean that you should go for Rtsne over tsne, but it would be good if the tsne package could at least be fixed.

sessionInfo()

R version 3.3.1 (2016-06-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RColorBrewer_1.1-2 Rtsne_0.13         tsne_0.1-3        

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.10    digest_0.6.12   rprojroot_1.2   jsonlite_1.3    backports_1.0.5 magrittr_1.5    evaluate_0.10  
 [8] stringi_1.1.3   rmarkdown_1.4   tools_3.3.1     stringr_1.2.0   yaml_2.1.14     base64enc_0.1-3 htmltools_0.3.5
[15] knitr_1.16