New landing page for CRMDA guides

Thanks to the hard work of our staff, we have a new landing page for the CRMDA guide documents.

Please feast your eyes on this page

https://crmda.ku.edu/guides

Which has been extensively vetted and inspected by a highly motivated series of focus groups. Or people that walked at random through the office. You can guess which.

Also, there's a new guide document about estimating rates with Poisson regression.

"Poisson Rates" https://crmda.ku.edu/node/622

Posted in Data Analysis | Leave a comment

R 3.5.1 and packages on cluster

In the KU HPC cluster, we now have a module for R-3.5.1 and there is a CRMDA
module, Rstats, that will link together R with the collection of packages I maintain.

The list of packages that were installed today is below. We have a few that won't compile
with the available resources (tkrplot), but the big happy news is we have success with
Rstan for the first time in quite a while. Special thanks to help from rstan-dev forums.

How to use this:

In your ~/.bashrc, add these lines. Or log in on a compute node and run these lines
interactively:

$ export PATH=/panfs/pfs.local/work/crmda/tools/bin:$PATH
$ source crmda_env-3.5.sh

After that, you should see:

$ module list

Currently Loaded Modules:
  1) legacy       3) compiler/gcc/6.3   5) java/1.8.0_131   7) icu/59.1      9) Rstats/3.5.1
  2) emacs/24.5   4) openmpi/1.10       6) xz/5.2.3         8) tcltk/8.6.6  10) mplus/7.3

Packages installed today

/panfs/pfs.local/software/install/MRO/3.5.1/ropen/3.5.1/lib64/R/library
base    3.5.1
boot    1.3-20
checkpoint  0.4.4
class   7.3-14
cluster 2.0.7-1
codetools   0.2-15
compiler    3.5.1
curl    3.2
datasets    3.5.1
deployrRserve   9.0.0
doParallel  1.0.13
foreach 1.5.0
foreign 0.8-71
graphics    3.5.1
grDevices   3.5.1
grid    3.5.1
iterators   1.0.10
jsonlite    1.5
KernSmooth  2.23-15
lattice 0.20-35
MASS    7.3-50
Matrix  1.2-14
methods 3.5.1
mgcv    1.8-24
MicrosoftR  3.5.1.3
nlme    3.1-137
nnet    7.3-12
parallel    3.5.1
png 0.1-7
R6  2.2.2
RevoIOQ 10.0.0
RevoMods    11.0.1
RevoUtils   11.0.1
RevoUtilsMath   11.0.0
rpart   4.1-13
RUnit   0.4.32
sp  1.3-1
spatial 7.3-11
splines 3.5.1
stats   3.5.1
stats4  3.5.1
survival    2.42-6
tcltk   3.5.1
tools   3.5.1
utils   3.5.1
/panfs/pfs.local/work/crmda/tools/mro/3.5/site-library
abind   1.4-5
acepack 1.4.1
actuar  2.3-1
ada 2.0-5
ade4    1.7-13
ADGofTest   0.3
AER 1.2-5
akima   0.6-2
alr3    2.0.8
amap    0.8-16
Amelia  1.7.5
aod 1.3
ape 5.2
aplpack 1.3.2
arm 1.10-1
arules  1.6-1
assertthat  0.2.0
backports   1.1.2
base64enc   0.1-3
bayesm  3.1-0.1
bcp 4.0.3
bdsmatrix   1.3-3
bestglm 0.37
betareg 3.1-1
BH  1.66.0-1
bibtex  0.4.2
biglm   0.9-1
bindr   0.1.1
bindrcpp    0.2.2
bit 1.1-14
bit64   0.9-7
bitops  1.0-6
BMA 3.18.9
bnlearn 4.4
BradleyTerry2   1.0-8
brew    1.0-6
brglm   0.6.1
broom   0.5.0
Cairo   1.5-9
cairoDevice 2.25
callr   3.0.0
car 3.0-2
carData 3.0-2
caret   6.0-80
caTools 1.17.1.1
cellranger  1.1.0
censReg 0.5-26
checkmate   1.8.5
chron   2.3-53
cli 1.0.1
clipr   0.4.1
clue    0.3-56
clv 0.3-2.1
cocorresp   0.3-0
coda    0.19-2
coin    1.2-2
colorspace  1.3-2
combinat    0.0-8
commonmark  1.6
copula  0.999-18
corpcor 1.6.9
crayon  1.3.4
crosstalk   1.0.0
cubature    1.4-1
Cubist  0.2.2
CVST    0.2-2
data.table  1.11.8
DBI 1.0.0
dbplyr  1.2.2
DCluster    0.2-7
ddalpha 1.3.4
deldir  0.1-15
DEoptimR    1.0-8
desc    1.2.0
descr   1.1.4
Devore7 0.7.6
DiagrammeR  1.0.0
DiceKriging 1.5.6
dichromat   2.0-0
digest  0.6.18
dimRed  0.1.0
diptest 0.75-7
distr   2.7.0
dlm 1.1-5
doBy    4.6-2
doMC    1.3.5
doMPI   0.2.2
doSNOW  1.0.16
dotCall64   1.0-0
downloader  0.4
dplyr   0.7.7
DRR 0.0.3
dse 2015.12-1
e1071   1.7-0
earth   4.6.3
Ecdat   0.3-1
Ecfun   0.1-7
ecodist 2.0.1
effects 4.0-3
egcm    1.0.12
eha 2.6.0
eiPack  0.1-7
emplik  1.0-4.3
ENmisc  1.2-7
estimability    1.3
evaluate    0.12
expint  0.1-5
expm    0.999-3
fansi   0.4.0
faraway 1.0.7
fastICA 1.2-1
fastmatch   1.1-0
fBasics 3042.89
fda 2.4.8
ffmanova    0.2-2
fields  9.6
flexmix 2.3-14
forcats 0.3.0
formatR 1.5
Formula 1.2-3
forward 1.0.4
fs  1.2.6
fUnitRoots  3042.79
futile.logger   1.4.3
futile.options  1.0.1
gam 1.16
gamlss  5.1-2
gamlss.data 5.1-0
gamlss.dist 5.1-0
gamm4   0.2-5
gbm 2.1.4
gclus   1.3.1
gdata   2.18.0
gee 4.13-19
geepack 1.2-1
geometry    0.3-6
geoR    1.7-5.2.1
geoRglm 0.9-11
ggm 2.3
ggplot2 3.1.0
glmc    0.2-4
glmmBUGS    2.4.2
glmmML  1.0.3
glmnet  2.0-16
glmpath 0.98
glue    1.3.0
gmodels 2.18.1
gmp 0.5-13.2
goftest 1.1-1
gower   0.1.2
GPArotation 2014.11-1
gpclib  1.5-5
grf 0.10.1
gridBase    0.4-7
gridExtra   2.3
grpreg  3.2-0
gsl 1.9-10.3
gss 2.1-9
gsubfn  0.7
gtable  0.2.0
gtools  3.8.1
haven   1.1.2
hexbin  1.27.2
highr   0.7
HistData    0.8-4
Hmisc   4.1-1
hms 0.4.2
htmlTable   1.12
htmltools   0.3.6
htmlwidgets 1.3
httpuv  1.4.5
httr    1.3.1
hunspell    2.9
HyperbolicDist  0.6-2
igraph  1.2.2
ineq    0.2-13
influence.ME    0.9-9
influenceR  0.1.0
inline  0.3.15
inum    1.0-0
iplots  1.1-7.1
ipred   0.9-8
irlba   2.3.2
Iso 0.0-17
ISOcodes    2018.06.29
ISwR    2.0-7
itertools   0.1-3
janeaustenr 0.1.5
JavaGD  0.6-1
JGR 1.8-6
JM  1.4-8
JMdesign    1.1
jomo    2.6-4
jpeg    0.1-8
Kendall 2.2
kernlab 0.9-27
knitr   1.20
kutils  1.51
labeling    0.3
laeken  0.4.6
lambda.r    1.2.3
languageR   1.4.1
lars    1.2
later   0.7.5
latticeExtra    0.6-28
lava    1.6.3
lavaan  0.6-3
lavaan.survey   1.1.3.1
lazyeval    0.2.1
leaps   3.0
LearnBayes  2.15.1
libcoin 1.0-1
lme4    1.1-18-1
lmec    1.0
lmerTest    3.0-1
lmeSplines  1.1-10
lmm 1.2
lmtest  0.9-36
locfit  1.5-9.1
logspline   2.1.11
longitudinal    1.1.12
longitudinalData    2.4.1
loo 2.0.0
lpSolve 5.6.13
ltm 1.1-1
lubridate   1.7.4
magic   1.5-9
magrittr    1.5
manipulate  1.0.1
manipulateWidget    0.10.0
maps    3.3.0
maptools    0.9-4
markdown    0.8
Matching    4.9-3
MatchIt 3.0.2
matrixcalc  1.0-3
MatrixModels    0.4-1
matrixStats 0.54.0
maxLik  1.3-4
mboost  2.9-1
mcgibbsit   1.1.0
mclust  5.4.1
mcmc    0.9-5
MCMCglmm    2.26
MCMCpack    1.4-4
MCPAN   1.1-21
mda 0.4-10
mediation   4.4.6
memisc  0.99.14.12
memoise 1.1.0
MEMSS   0.9-2
mi  1.0
mice    3.3.0
micEcon 0.6-14
microbenchmark  1.4-6
mime    0.6
miniUI  0.1.1.1
minqa   1.2.4
misc3d  0.8-4
miscTools   0.6-22
mitml   0.3-6
mitools 2.3
mix 1.0-10
mixtools    1.1.0
mlbench 2.1-1
mnormt  1.5-5
MNP 3.1-0
ModelMetrics    1.2.2
modelr  0.1.2
modeltools  0.2-22
MplusAutomation 0.7-2
MPV 1.53
msm 1.6.6
multcomp    1.4-8
munsell 0.5.0
mvbutils    2.7.4.1
mvProbit    0.1-8
mvtnorm 1.0-8
network 1.13.0.1
nloptr  1.2.1
NLP 0.2-0
NMF 0.21.0
nnls    1.4
nor1mix 1.2-3
norm    1.0-9.5
nortest 1.0-4
np  0.60-9
numDeriv    2016.8-1
nws 1.7.0.1
openssl 1.0.2
openxlsx    4.1.0
ordinal 2018.8-25
orthopolynom    1.0-5
pan 1.6
pander  0.6.3
partDSA 0.9.14
party   1.3-1
partykit    1.2-2
PASWR   1.1
pbivnorm    0.6.0
pbkrtest    0.4-7
PBSmapping  2.70.5
pcaPP   1.9-73
permute 0.9-4
pillar  1.3.0
pixmap  0.4-11
pkgbuild    1.0.2
pkgconfig   2.0.2
pkgKitten   0.1.4
pkgload 1.0.2
pkgmaker    0.27
plm 1.6-6
plogr   0.2.0
plotmo  3.5.0
plotrix 3.7-4
pls 2.7-0
plyr    1.8.4
pmml    1.5.6
pmmlTransformations 1.3.2
polspline   1.1.13
polyclip    1.9-1
polycor 0.7-9
polynom 1.3-9
PolynomF    1.0-2
portableParallelSeeds   0.97
ppcor   1.1
pracma  2.1.8
praise  1.0.0
prettyunits 1.0.2
processx    3.2.0
prodlim 2018.04.18
profileModel    0.5-9
promises    1.0.1
proto   1.0.0
proxy   0.4-22
ps  1.2.1
pscl    1.5.2
psidR   1.9
pspline 1.0-18
psych   1.8.10
purrr   0.2.5
quadprog    1.5-5
quantmod    0.4-13
quantreg    5.36
qvcalc  0.9-1
R2HTML  2.3.2
R2OpenBUGS  3.2-3.2
R6  2.3.0
RandomFields    3.1.50
RandomFieldsUtils   0.3.25
randomForest    4.6-14
randomForestSRC 2.7.0
rattle  5.2.0
rbenchmark  1.0.0
rbugs   0.5-9
Rcmdr   2.5-1
RcmdrMisc   2.5-1
RColorBrewer    1.1-2
Rcpp    0.12.19
RcppArmadillo   0.9.100.5.0
RcppEigen   0.3.3.4.0
RcppRoll    0.3.0
RCurl   1.95-4.11
Rd2roxygen  1.7
rda 1.0.2-2.1
readr   1.1.1
readstata13 0.9.2
readxl  1.1.0
recipes 0.1.3
registry    0.5
relimp  1.0-5
rematch 1.0.1
repr    0.15.0
reprex  0.2.1
reshape 0.8.8
reshape2    1.4.3
rgexf   0.15.3
rgl 0.99.16
RGtk2   2.20.35
rio 0.5.10
rJava   0.9-10
rlang   0.3.0.1
rlecuyer    0.3-4
rmarkdown   1.10
Rmpi    0.6-7
rms 5.1-2
rngtools    1.3.1
robustbase  0.93-3
rockchalk   1.8.126
Rook    1.1-1
roxygen2    6.1.1
rpart.plot  3.0.5
rpf 0.59
rprojroot   1.3-2
rrcov   1.4-4
rstan   2.18.1
rstudioapi  0.8
RSvgDevice  0.6.4.4
rvest   0.3.2
sandwich    2.5-0
SAScii  1.0
SASmixed    1.0-4
scales  1.0.0
scatterplot3d   0.3-41
segmented   0.5-3.0
selectr 0.4-1
sem 3.1-9
SemiPar 1.0-4.2
semTools    0.5-1
setRNG  2013.9-1
sets    1.0-18
sfsmisc 1.1-2
shapefiles  0.7
shiny   1.2.0
simsem  0.5-14
slam    0.1-43
sm  2.2-5.6
smoothSurv  2.0.1
sna 2.4
snow    0.4-3
SnowballC   0.5.1
snowFT  1.6-0
SoDA    1.0-6
sourcetools 0.1.7
spam    2.2-0
SparseM 1.77
spatialCovariance   0.6-9
spatialkernel   0.4-23
spatstat    1.57-1
spatstat.data   1.4-0
spatstat.utils  1.13-0
spData  0.2.9.4
spdep   0.7-9
splancs 2.01-40
SQUAREM 2017.10-1
stabledist  0.7-1
stabs   0.6-3
StanHeaders 2.18.0
startupmsg  0.9.5
StatDataML  1.0-26
statmod 1.4.30
statnet.common  4.1.4
stepwise    0.3
stopwords   0.9.0
stringi 1.2.4
stringr 1.3.1
strucchange 1.5-1
subselect   0.14
survey  3.34
systemfit   1.1-22
tables  0.8.4
tcltk2  1.2-11
TeachingDemos   2.10
tensor  1.5
tensorA 0.36.1
testthat    2.0.1
texreg  1.36.23
tfplot  2015.12-1
tframe  2015.12-1
TH.data 1.0-9
tibble  1.4.2
tidyr   0.8.2
tidyselect  0.2.5
tidytext    0.2.0
tidyverse   1.2.1
timeDate    3043.102
timeSeries  3042.102
tinytex 0.9
tis 1.34
tm  0.7-5
tokenizers  0.2.1
topicmodels 0.2-7
tree    1.0-39
triangle    0.11
trimcluster 0.1-2.1
trust   0.1-7
tseries 0.10-45
TTR 0.23-4
ucminf  1.1-4
urca    1.3-0
UsingR  2.0-6
utf8    1.1.4
vcd 1.4-4
vegan   2.5-3
VGAM    1.0-6
VIM 4.7.0
viridis 0.5.1
viridisLite 0.3.0
visNetwork  2.0.4
waveslim    1.7.5
webshot 0.5.1
whisker 0.3-2
withr   2.1.2
wnominate   1.2.5
xfun    0.4
xgboost 0.71.2
XML 3.98-1.16
xml2    1.2.0
xtable  1.8-3
xts 0.11-2
yaml    2.2.0
Zelig   5.1.6
zip 1.0.0
zipfR   0.6-10
zoo 1.8-4
Posted in Data Analysis | Leave a comment

ACF Cluster changing to SLURM queue manager November 7

In case you did not hear yet, the current scheduler, which was implemented February 2017, is being replaced by SLURM, on November 8. There will be a cluster shutdown Nov. 6- Nov 7 and after that the submission scripts that used "MSUB" notation will no longer work.

Here is how we are preparing in CRMDA.

  1. We have a test cluster for people who want to practice SLURM code. We reallocated 5 notes out of the CRMDA share into this temporary test system:

    slurmsubmit.hpc.crc.ku.edu

    Users who currently are in the CRMDA cluster group should be able to log in there and test revisions. Because we ONLY have 5 nodes there, one should not submit MASSIVE jobs in slurmsubmit. It is intended primarily for testing and learning SLURM to prepare for the transition.

  2. Our set of working examples for high performance computing, hpcexample, is being re-worked. The webpage for the Gitlab repository is https://gitlab.crmda.ku.edu/crmda/hpcexample The master branch on the repository is the currently used MSUB notation. We have started a new branch slurm and on that branch we are putting in revised submission scripts. The slurm branch You can tell if an example has been updated because the notation #MSUB will be changed to #SBATCH.

The Center for Research Computing has a help page for the transition (https://crc.ku.edu/hpc/slurm/how-to) and there have been training sessions (https://crc.ku.edu/hpc/meetings).

To give you a rough idea of what is needed, we compare an old MSUB-style submission script (in this case, the submission for Example 65 (https://gitlab.crmda.ku.edu/crmda/hpcexample/tree/master/Ex65-R-parallel)

#!/bin/sh
#
#
#MSUB -M your-name_here@ku.edu
#MSUB -N RParallelHelloWorld 
#MSUB -q sixhour 
#MSUB -l nodes=1:ppn=11:ib
#MSUB -l walltime=00:05:00
#MSUB -m bea

cd $PBS_O_WORKDIR

## Please check your ~/.bash_profile to make sure
## the correct modules will be loaded with new shells.
## See discussion:
## http://www.crmda.dept.ku.edu/timeline/archives/184
## In a terminal on a compute node, you can test this
## by running
## $ module list
## It is not helpful or necessary to insert module
## commands in this script (contrary to previous advice).
   mpiexec -n 1 R --vanilla -f parallel-hello.R

To the SLURM style, like so:

#!/bin/sh
#
#
#SBATCH --mail-user=your_name_here@ku.edu
#SBATCH --job-name=RParallelHelloWorld 
#SBATCH --partition=sixhour
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=11
#SBATCH --time=00:05:00
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --error="%x.e%j"
#SBATCH --output="%x.o%j"

cd $SLURM_SUBMIT_DIR

## Please check your ~/.bash_profile to make sure
## the correct modules will be loaded with new shells.
## See discussion:
## http://www.crmda.dept.ku.edu/timeline/archives/184
## In a terminal on a compute node, you can test this
## by running
## $ module list
## It is not helpful or necessary to insert module
## commands in this script (contrary to previous advice).

mpiexec -n 1 R --vanilla -f parallel-hello.R

Then, to actually launch the job into the cluster, we will no longer use "msub", instead use "sbatch".

After November 7, the slurmsubmit.hpc.crc.ku.edu test login will no longer function, users will just need to go to the address that we currently use hpc.crc.ku.edu.

The slurm branch in hpcexample has just a few fully converted directories, please check Ex65 for a fully working example if you need it today. However, the other examples are rapidly approaching a presentable state.

Other chores we need to do:

  1. See about creating "shortcuts" to create user sessions. In SLURM system, to request an interactive session (as explained in the CRC slurm/how-to document) can be obtained thus:

    $ srun --time=4:00:00 --ntasks=1 --nodes=1 --partition=sixhour --x11 --pty /bin/bash

    I would have an easier-to-remember way to launch that, like our old qlogin script. But I don't yet know how we can make that available to the users of CRMDA-based accounts. We'll figure it out.

  2. Did you notice that our old MSUB script asked for Infiniband (ib) nodes, but the new one does not. Its like that because I don't know the right approach. Find out if it is smart to ask for or avoid the infiniband nodes. Several years ago, we were avoiding IB because it made jobs hang. More recently, we've been asking for IB nodes because it was a way to make sure that a job's calculations were not sent to a very old compute node (only newer had IB). I don't know what is the best advice now.

  3. If we have a job that requires 11 cores, are we smarter to ask for 1 node with 11 tasks, or are we smarter to allow cluster to send the 11 cores-worth of work out to 11 different nodes. Over time, the best advice has changed. In our old cluster, the latter was faster. In the community cluster, we noticed that there were errors caused by slightly different types of compute nodes and we were smarter to try to keep all computations on the same system (or a small number of systems). I don't know what the best advice will be in the new SLURM framework. We'll figure it out.

  4. A wholesale overhaul of our training materials in http://crmda.ku.edu/computing will need to occur. That's the most daunting part in all of this.

Posted in Data Analysis | Leave a comment

R Package Updates: Celebrating Independence Day

We had a couple of new package requests. I'm taking the occasion, as we approach the holiday celebrating American independence, to update the other R packages. There are still complications building newer rstan, I log in every month or so and try for a few hours. Otherwise I think we are working:

Remember, that packages you have in your HOME folder, such as R/x86/3.3 or R/x86/3.4 will PRECEDE the ones I provide systemwide. So if you have an old one, then that's what R will use for you. Because your packages are in version-specific folder (note 3.3 and 3.4 on end of path), it WILL be necessary for you to install packages for R 3.4 if you update the environment file (described next).

It is important to set your R selection in the environment (fully described here: R Modules: Super Exciting New Updates) because, if you don't, then you might launch a parallel job that will use different/inconsistent packages and versions of R on different nodes. This was a very hard-to-understand problem in the new cluster. So configure one of the environment scripts "crmda_env.sh" (for R 3.3) and "crmda_env-test.sh" (for R 3.4). I am thinking hard on what new pleasant names we could use for those scripts.

For R-3.4, the package set that comes along with the R environment we provide (via crmda_env-test.sh) are in the second block here, while the ones provided by central sysadmin are in the first block (with indicated directory). The ones in the first set, which I do not administer, are out of date. There's a philosophical problem here. Some people think the packages should never be updated for replication purposes, but generally I tend to want the newest. If users want older packages, they can install in $HOME/R, of course. The out-of-date-packages problem is more serious in R-3.3 that 3.4, and maybe users ought to adopt that anyway.

Right now, here is the situation:

/panfs/pfs.local/software/install/MRO/3.4.0/microsoft-r/3.4/lib64/R/library
KernSmooth  2.23-15
MASS    7.3-47
Matrix  1.2-9
MicrosoftR  3.4.0.0068
R6  2.2.0
RUnit   0.4.26
RevoIOQ 8.0.7
RevoMods    11.0.0
RevoUtils   10.0.4
RevoUtilsMath   10.0.0
base    3.4.0
boot    1.3-19
checkpoint  0.4.0
class   7.3-14
cluster 2.0.6
codetools   0.2-15
compiler    3.4.0
curl    2.6
datasets    3.4.0
deployrRserve   9.0.0
doParallel  1.0.10
foreach 1.4.4
foreign 0.8-67
grDevices   3.4.0
graphics    3.4.0
grid    3.4.0
iterators   1.0.8
jsonlite    1.4
lattice 0.20-35
methods 3.4.0
mgcv    1.8-17
nlme    3.1-131
nnet    7.3-12
parallel    3.4.0
png 0.1-7
rpart   4.1-11
spatial 7.3-11
splines 3.4.0
stats   3.4.0
stats4  3.4.0
survival    2.41-3
tcltk   3.4.0
tools   3.4.0
utils   3.4.0
/panfs/pfs.local/work/crmda/tools/mro/3.4/site-library
ADGofTest   0.3
AER 1.2-5
Amelia  1.7.5
BH  1.66.0-1
BMA 3.18.8
BradleyTerry2   1.0-8
CVST    0.2-2
Cairo   1.5-9
Cubist  0.2.2
DBI 1.0.0
DCluster    0.2-7
DEoptimR    1.0-8
DRR 0.0.3
Devore7 0.7.6
DiagrammeR  1.0.0
DiceKriging 1.5.5
ENmisc  1.2-7
Ecdat   0.3-1
Ecfun   0.1-7
Formula 1.2-3
GPArotation 2014.11-1
HistData    0.8-4
Hmisc   4.1-1
HyperbolicDist  0.6-2
ISOcodes    2018.06.29
ISwR    2.0-7
Iso 0.0-17
JM  1.4-8
JMdesign    1.1
JavaGD  0.6-1
Kendall 2.2
Lahman  6.0-0
LearnBayes  2.15.1
MCMCpack    1.4-3
MCPAN   1.1-21
MEMSS   0.9-2
MNP 3.1-0
MPV 1.53
MatchIt 3.0.2
Matching    4.9-3
MatrixModels    0.4-1
ModelMetrics    1.1.0
MplusAutomation 0.7-2
NLP 0.1-11
NMF 0.21.0
PASWR   1.1
PolynomF    1.0-2
R2HTML  2.3.2
R2OpenBUGS  3.2-3.2
R6  2.2.2
RColorBrewer    1.1-2
RCurl   1.95-4.10
RGtk2   2.20.35
RMySQL  0.10.15
RPostgreSQL 0.6-2
RSQLite 2.1.1
RSvgDevice  0.6.4.4
RUnit   0.4.32
RandomFields    3.1.50
RandomFieldsUtils   0.3.25
Rcmdr   2.4-4
RcmdrMisc   1.0-10
Rcpp    0.12.17
RcppArmadillo   0.8.600.0.0
RcppEigen   0.3.3.4.0
RcppRoll    0.3.0
Rd2roxygen  1.6.2
Rmpi    0.6-7
Rook    1.1-1
SAScii  1.0
SASmixed    1.0-4
SQUAREM 2017.10-1
SemiPar 1.0-4.2
SnowballC   0.5.1
SoDA    1.0-6
SparseM 1.77
StanHeaders 2.17.2
StatDataML  1.0-26
SweaveListingUtils  0.7.7
TH.data 1.0-8
TTR 0.23-3
TeachingDemos   2.10
UsingR  2.0-6
VGAM    1.0-5
VIM 4.7.0
XML 3.98-1.11
Zelig   5.1.6
abind   1.4-5
acepack 1.4.1
actuar  2.3-1
ada 2.0-5
ade4    1.7-11
akima   0.6-2
alr3    2.0.8
amap    0.8-16
aod 1.3
ape 5.1
aplpack 1.3.2
arm 1.10-1
arules  1.6-1
assertthat  0.2.0
backports   1.1.2
base64enc   0.1-3
bayesm  3.1-0.1
bcp 4.0.2
bdsmatrix   1.3-3
bestglm 0.37
betareg 3.1-0
bibtex  0.4.2
biglm   0.9-1
bindr   0.1.1
bindrcpp    0.2.2
bit 1.1-14
bit64   0.9-7
bitops  1.0-6
blob    1.1.1
bnlearn 4.3
brew    1.0-6
brglm   0.6.1
broom   0.4.4
caTools 1.17.1
cairoDevice 2.25
callr   2.0.4
car 3.0-0
carData 3.0-1
caret   6.0-80
cellranger  1.1.0
censReg 0.5-26
checkmate   1.8.5
chron   2.3-52
cli 1.0.0
clipr   0.4.1
clue    0.3-55
clv 0.3-2.1
cocorresp   0.3-0
coda    0.19-1
coin    1.2-2
colorspace  1.3-2
combinat    0.0-8
copula  0.999-18
corpcor 1.6.9
covr    3.1.0
crayon  1.3.4
crosstalk   1.0.0
cubature    1.4
data.table  1.11.4
dbplyr  1.2.1
ddalpha 1.3.4
debugme 1.1.0
deldir  0.1-15
descr   1.1.4
dichromat   2.0-0
digest  0.6.15
dimRed  0.1.0
diptest 0.75-7
distr   2.6.2
dlm 1.1-5
doBy    4.6-1
doMC    1.3.5
doMPI   0.2.2
doParallel  1.0.11
doSNOW  1.0.16
dotCall64   0.9-5.2
downloader  0.4
dplyr   0.7.6
dse 2015.12-1
e1071   1.6-8
earth   4.6.3
ecodist 2.0.1
effects 4.0-2
egcm    1.0.12
eha 2.5.1
eiPack  0.1-7
emplik  1.0-3
estimability    1.3
evaluate    0.10.1
expint  0.1-5
expm    0.999-2
fBasics 3042.89
fUnitRoots  3042.79
faraway 1.0.7
fastICA 1.2-1
fastmatch   1.1-0
fda 2.4.7
ffmanova    0.2-2
fields  9.6
flexmix 2.3-14
forcats 0.3.0
foreach 1.4.4
formatR 1.5
forward 1.0.3
gam 1.15
gamlss  5.1-0
gamlss.data 5.0-1
gamlss.dist 5.0-6
gamm4   0.2-5
gbm 2.1.3
gclus   1.3.1
gdata   2.18.0
gee 4.13-19
geepack 1.2-1
geoR    1.7-5.2
geoRglm 0.9-11
geometry    0.3-6
ggm 2.3
ggplot2 2.2.1
glmc    0.2-4
glmmBUGS    2.4.2
glmmML  1.0.3
glmnet  2.0-16
glmpath 0.98
glue    1.2.0
gmodels 2.18.1
gmp 0.5-13.1
gower   0.1.2
gpclib  1.5-5
grf 0.10.0
gridBase    0.4-7
gridExtra   2.3
grpreg  3.1-4
gsl 1.9-10.3
gss 2.1-9
gsubfn  0.7
gtable  0.2.0
gtools  3.8.1
haven   1.1.2
hexbin  1.27.2
highr   0.7
hms 0.4.2
htmlTable   1.12
htmltools   0.3.6
htmlwidgets 1.2
httpuv  1.4.4.2
httr    1.3.1
hunspell    2.9
igraph  1.2.1
ineq    0.2-13
influence.ME    0.9-9
influenceR  0.1.0
inline  0.3.15
inum    1.0-0
iplots  1.1-7.1
ipred   0.9-6
irlba   2.3.2
iterators   1.0.9
itertools   0.1-3
janeaustenr 0.1.5
jomo    2.6-2
jpeg    0.1-8
jsonlite    1.5
kernlab 0.9-26
knitr   1.20
kutils  1.46
labeling    0.3
laeken  0.4.6
languageR   1.4.1
lars    1.2
later   0.7.3
latticeExtra    0.6-28
lava    1.6.2
lavaan  0.6-1
lavaan.survey   1.1.3.1
lazyeval    0.2.1
leaps   3.0
libcoin 1.0-1
lme4    1.1-17
lmeSplines  1.1-10
lmec    1.0
lmm 1.2
lmtest  0.9-36
locfit  1.5-9.1
logspline   2.1.11
longitudinal    1.1.12
longitudinalData    2.4.1
lpSolve 5.6.13
ltm 1.1-1
lubridate   1.7.4
magic   1.5-8
magrittr    1.5
manipulate  1.0.1
manipulateWidget    0.10.0
maps    3.3.0
maptools    0.9-2
markdown    0.8
matrixcalc  1.0-3
maxLik  1.3-4
mboost  2.9-0
mcgibbsit   1.1.0
mclust  5.4.1
mcmc    0.9-5
mda 0.4-10
memisc  0.99.14.9
memoise 1.1.0
mi  1.0
micEcon 0.6-14
mice    3.1.0
microbenchmark  1.4-4
mime    0.5
miniUI  0.1.1.1
minqa   1.2.4
misc3d  0.8-4
miscTools   0.6-22
mitml   0.3-5
mitools 2.3
mix 1.0-10
mixtools    1.1.0
mlbench 2.1-1
mnormt  1.5-5
modelr  0.1.2
modeltools  0.2-21
msm 1.6.6
multcomp    1.4-8
munsell 0.5.0
mvProbit    0.1-8
mvbutils    2.7.4.1
mvtnorm 1.0-8
network 1.13.0.1
nloptr  1.0.4
nnls    1.4
nor1mix 1.2-3
norm    1.0-9.5
nortest 1.0-4
np  0.60-8
numDeriv    2016.8-1
nws 1.7.0.1
nycflights13    1.0.0
openssl 1.0.1
openxlsx    4.1.0
ordinal 2018.4-19
orthopolynom    1.0-5
pan 1.6
pander  0.6.1
partDSA 0.9.14
party   1.3-0
partykit    1.2-2
pbivnorm    0.6.0
pbkrtest    0.4-7
pcaPP   1.9-73
permute 0.9-4
pillar  1.2.3
pixmap  0.4-11
pkgKitten   0.1.4
pkgconfig   2.0.1
pkgmaker    0.27
plm 1.6-6
plogr   0.2.0
plotmo  3.4.1
plotrix 3.7-2
pls 2.6-0
plyr    1.8.4
pmml    1.5.4
pmmlTransformations 1.3.1
png 0.1-7
polspline   1.1.13
polycor 0.7-9
polynom 1.3-9
portableParallelSeeds   0.97
ppcor   1.1
pracma  2.1.4
praise  1.0.0
prettyunits 1.0.2
processx    3.1.0
prodlim 2018.04.18
profileModel    0.5-9
promises    1.0.1
proto   1.0.0
proxy   0.4-22
pscl    1.5.2
psidR   1.5
pspline 1.0-18
psych   1.8.4
purrr   0.2.5
quadprog    1.5-5
quantmod    0.4-13
quantreg    5.36
qvcalc  0.9-1
rJava   0.9-10
randomForest    4.6-14
randomForestSRC 2.6.1
rattle  5.1.0
rbenchmark  1.0.0
rbugs   0.5-9
rda 1.0.2-2
readr   1.1.1
readstata13 0.9.2
readxl  1.1.0
recipes 0.1.3
registry    0.5
relimp  1.0-5
rematch 1.0.1
repr    0.15.0
reprex  0.2.0
reshape 0.8.7
reshape2    1.4.3
rex 1.1.2
rgenoud 5.8-2.0
rgexf   0.15.3
rgl 0.99.16
rio 0.5.10
rlang   0.2.1
rlecuyer    0.3-4
rmarkdown   1.10
rms 5.1-2
rngtools    1.3.1
robustbase  0.93-1
rockchalk   1.8.111
roxygen2    6.0.1
rpart.plot  2.2.0
rpf 0.59
rprojroot   1.3-2
rrcov   1.4-4
rstan   2.17.3
rstudioapi  0.7
rvest   0.3.2
sandwich    2.4-0
scales  0.5.0
scatterplot3d   0.3-41
segmented   0.5-3.0
selectr 0.4-1
sem 3.1-9
semTools    0.5-0
setRNG  2013.9-1
sets    1.0-18
sfsmisc 1.1-2
shapefiles  0.7
shiny   1.1.0
simsem  0.5-14
slam    0.1-43
sm  2.2-5.5
smoothSurv  2.0
sna 2.4
snow    0.4-2
snowFT  1.6-0
sourcetools 0.1.7
sp  1.3-1
spData  0.2.9.0
spam    2.2-0
spatialCovariance   0.6-9
spdep   0.7-7
splancs 2.01-40
stabledist  0.7-1
stabs   0.6-3
startupmsg  0.9.4
statmod 1.4.30
statnet.common  4.1.4
stepwise    0.3
stopwords   0.9.0
stringi 1.2.3
stringr 1.3.1
strucchange 1.5-1
subselect   0.14
survey  3.33-2
systemfit   1.1-22
tables  0.8.4
tcltk2  1.2-11
tensorA 0.36
testthat    2.0.0
texreg  1.36.23
tfplot  2015.12-1
tframe  2015.12-1
tibble  1.4.2
tidyr   0.8.1
tidyselect  0.2.4
tidytext    0.1.9
tidyverse   1.2.1
timeDate    3043.102
timeSeries  3042.102
tinytex 0.5
tis 1.34
tm  0.7-4
tokenizers  0.2.1
topicmodels 0.2-7
tree    1.0-39
triangle    0.11
trimcluster 0.1-2
trust   0.1-7
tseries 0.10-45
ucminf  1.1-4
urca    1.3-0
utf8    1.1.4
vcd 1.4-4
vegan   2.5-2
viridis 0.5.1
viridisLite 0.3.0
visNetwork  2.0.4
waveslim    1.7.5
webshot 0.5.0
whisker 0.3-2
withr   2.1.2
wnominate   1.2.5
xml2    1.2.0
xtable  1.8-2
xts 0.10-2
yaml    2.1.19
zip 1.0.0
zipfR   0.6-10
zoo 1.8-2

And the session info was:

> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server release 6.4 (Santiago)

Matrix products: default
BLAS: /panfs/pfs.local/software/install/MRO/3.4.0/microsoft-r/3.4/lib64/R/lib/libRblas.so
LAPACK: /panfs/pfs.local/software/install/MRO/3.4.0/microsoft-r/3.4/lib64/R/lib/libRlapack.so

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RevoUtilsMath_10.0.0

loaded via a namespace (and not attached):
[1] compiler_3.4.0   RevoUtils_10.0.4 tools_3.4.0    

For R-3.3, the packages are provided via an environment file (crmda_env.sh). The package list will be mostly the same, however the effort to compile the latest versions of these packages fails: c("inum", "Rmpi", "partykit", "OpenMx", "mboost"). Rmpi is a foundational piece an I'll try to figure out what is wrong. Anyway, here is the firecracker 3.3 package list.

/panfs/pfs.local/software/install/MRO/3.3/microsoft-r/3.3/lib64/R/library
KernSmooth  2.23-15
MASS    7.3-45
Matrix  1.2-7.1
MicrosoftR  3.3.2.0000
R6  2.2.0
RUnit   0.4.26
RevoIOQ 8.0.6
RevoMods    10.0.0
RevoUtils   10.0.2
RevoUtilsMath   10.0.0
base    3.3.2
boot    1.3-18
checkpoint  0.3.18
class   7.3-14
cluster 2.0.5
codetools   0.2-15
compiler    3.3.2
curl    2.2
datasets    3.3.2
deployrRserve   9.0.0
doParallel  1.0.10
foreach 1.4.3
foreign 0.8-67
grDevices   3.3.2
graphics    3.3.2
grid    3.3.2
iterators   1.0.8
jsonlite    1.1
lattice 0.20-34
methods 3.3.2
mgcv    1.8-15
nlme    3.1-128
nnet    7.3-12
parallel    3.3.2
png 0.1-7
rpart   4.1-10
spatial 7.3-11
splines 3.3.2
stats   3.3.2
stats4  3.3.2
survival    2.39-5
tcltk   3.3.2
tools   3.3.2
utils   3.3.2
/panfs/pfs.local/work/crmda/tools/mro/3.3/site-library
ADGofTest   0.3
AER 1.2-5
Amelia  1.7.5
BH  1.66.0-1
BMA 3.18.8
BradleyTerry2   1.0-8
CVST    0.2-2
Cairo   1.5-9
Cubist  0.2.2
DBI 1.0.0
DCluster    0.2-7
DEoptimR    1.0-8
DRR 0.0.3
Devore7 0.7.6
DiagrammeR  1.0.0
DiceKriging 1.5.5
ENmisc  1.2-7
Ecdat   0.3-1
Ecfun   0.1-7
Formula 1.2-3
GPArotation 2014.11-1
HistData    0.8-4
Hmisc   4.1-1
HyperbolicDist  0.6-2
ISOcodes    2018.06.29
ISwR    2.0-7
Iso 0.0-17
JGR 1.8-4
JM  1.4-8
JMdesign    1.1
JavaGD  0.6-1
Kendall 2.2
Lahman  6.0-0
LearnBayes  2.15.1
MCMCglmm    2.25
MCMCpack    1.4-3
MCPAN   1.1-21
MEMSS   0.9-2
MNP 3.1-0
MPV 1.53
MatchIt 3.0.2
Matching    4.9-3
MatrixModels    0.4-1
ModelMetrics    1.1.0
MplusAutomation 0.7-2
NLP 0.1-11
NMF 0.21.0
OpenMx  2.8.3
PASWR   1.1
PBSmapping  2.70.5
PolynomF    1.0-2
R2HTML  2.3.2
R2OpenBUGS  3.2-3.2
R6  2.2.2
RColorBrewer    1.1-2
RCurl   1.95-4.10
RGtk2   2.20.31
RMySQL  0.10.15
RPostgreSQL 0.6-2
RSQLite 2.1.1
RSvgDevice  0.6.4.4
RUnit   0.4.32
RandomFields    3.1.50
RandomFieldsUtils   0.3.25
Rcmdr   2.4-1
RcmdrMisc   1.0-7
Rcpp    0.12.17
RcppArmadillo   0.8.600.0.0
RcppEigen   0.3.3.4.0
RcppRoll    0.3.0
Rd2roxygen  1.6.2
Rmpi    0.6-6
Rook    1.1-1
SAScii  1.0
SASmixed    1.0-4
SQUAREM 2017.10-1
SemiPar 1.0-4.2
SnowballC   0.5.1
SoDA    1.0-6
SparseM 1.77
StanHeaders 2.17.2
StatDataML  1.0-26
SweaveListingUtils  0.7.7
TH.data 1.0-8
TTR 0.23-3
TeachingDemos   2.10
UsingR  2.0-6
VGAM    1.0-3
VIM 4.7.0
XML 3.98-1.11
Zelig   5.1.6
abind   1.4-5
acepack 1.4.1
actuar  2.3-1
ada 2.0-5
ade4    1.7-11
adehabitat  1.8.20
akima   0.6-2
alr3    2.0.8
amap    0.8-16
aod 1.3
ape 5.1
aplpack 1.3.2
arm 1.10-1
arules  1.5-4
assertthat  0.2.0
backports   1.1.2
base64enc   0.1-3
bayesm  3.1-0.1
bcp 4.0.2
bdsmatrix   1.3-3
bestglm 0.37
betareg 3.1-0
bibtex  0.4.2
biglm   0.9-1
bindr   0.1.1
bindrcpp    0.2.2
bit 1.1-14
bit64   0.9-7
bitops  1.0-6
blob    1.1.1
bnlearn 4.3
brew    1.0-6
brglm   0.6.1
broom   0.4.4
caTools 1.17.1
cairoDevice 2.25
callr   2.0.4
car 3.0-0
carData 3.0-1
caret   6.0-80
cellranger  1.1.0
censReg 0.5-26
checkmate   1.8.5
chron   2.3-52
cli 1.0.0
clipr   0.4.1
clue    0.3-55
clv 0.3-2.1
cocorresp   0.3-0
coda    0.19-1
coin    1.2-2
colorspace  1.3-2
combinat    0.0-8
commonmark  1.5
copula  0.999-18
corpcor 1.6.9
covr    3.1.0
crayon  1.3.4
crosstalk   1.0.0
cslogistic  0.1-3
cubature    1.4
data.table  1.11.4
dbplyr  1.2.1
ddalpha 1.3.4
debugme 1.1.0
deldir  0.1-15
desc    1.2.0
descr   1.1.4
dichromat   2.0-0
digest  0.6.15
dimRed  0.1.0
diptest 0.75-7
distr   2.6.2
dlm 1.1-5
doBy    4.6-1
doMC    1.3.5
doMPI   0.2.2
doParallel  1.0.11
doSNOW  1.0.16
dotCall64   0.9-5.2
downloader  0.4
dplyr   0.7.6
dse 2015.12-1
e1071   1.6-8
earth   4.6.3
ecodist 2.0.1
effects 4.0-2
egcm    1.0.12
eha 2.5.1
eiPack  0.1-7
emplik  1.0-3
estimability    1.3
evaluate    0.10.1
expint  0.1-5
expm    0.999-2
fBasics 3042.89
fUnitRoots  3042.79
faraway 1.0.7
fastICA 1.2-1
fastmatch   1.1-0
fda 2.4.7
ffmanova    0.2-2
fields  9.6
flexmix 2.3-14
forcats 0.3.0
foreach 1.4.4
formatR 1.5
forward 1.0.3
gam 1.15
gamlss  5.1-0
gamlss.data 5.0-1
gamlss.dist 5.0-6
gamm4   0.2-5
gbm 2.1.3
gclus   1.3.1
gdata   2.18.0
gee 4.13-19
geepack 1.2-1
geoR    1.7-5.2
geoRglm 0.9-11
geometry    0.3-6
ggm 2.3
ggplot2 2.2.1
glmc    0.2-4
glmmBUGS    2.4.2
glmmML  1.0.3
glmnet  2.0-16
glmpath 0.98
glue    1.2.0
gmodels 2.18.1
gmp 0.5-13.1
goftest 1.1-1
gower   0.1.2
gpclib  1.5-5
grf 0.10.0
gridBase    0.4-7
gridExtra   2.3
grpreg  3.1-4
gsl 1.9-10.3
gss 2.1-9
gsubfn  0.7
gtable  0.2.0
gtools  3.8.1
haven   1.1.2
hexbin  1.27.2
highr   0.7
hms 0.4.2
htmlTable   1.12
htmltools   0.3.6
htmlwidgets 1.2
httpuv  1.4.4.2
httr    1.3.1
hunspell    2.9
igraph  1.2.1
ineq    0.2-13
influence.ME    0.9-9
influenceR  0.1.0
inline  0.3.15
iplots  1.1-7.1
ipred   0.9-6
irlba   2.3.2
iterators   1.0.9
itertools   0.1-3
janeaustenr 0.1.5
jomo    2.6-2
jpeg    0.1-8
jsonlite    1.5
kernlab 0.9-26
knitr   1.20
kutils  1.46
labeling    0.3
laeken  0.4.6
languageR   1.4.1
lars    1.2
later   0.7.3
latticeExtra    0.6-28
lava    1.6.2
lavaan  0.6-1
lavaan.survey   1.1.3.1
lazyeval    0.2.1
leaps   3.0
lme4    1.1-17
lmeSplines  1.1-10
lmec    1.0
lmerTest    3.0-1
lmm 1.2
lmtest  0.9-36
locfit  1.5-9.1
logspline   2.1.11
longitudinal    1.1.12
longitudinalData    2.4.1
lpSolve 5.6.13
ltm 1.1-1
lubridate   1.7.4
magic   1.5-8
magrittr    1.5
manipulate  1.0.1
manipulateWidget    0.10.0
maps    3.3.0
maptools    0.9-2
markdown    0.8
matrixStats 0.53.1
matrixcalc  1.0-3
maxLik  1.3-4
mcgibbsit   1.1.0
mclust  5.4.1
mcmc    0.9-5
mda 0.4-10
mediation   4.4.6
memisc  0.99.14.9
memoise 1.1.0
mi  1.0
micEcon 0.6-14
mice    3.1.0
microbenchmark  1.4-4
mime    0.5
miniUI  0.1.1.1
minqa   1.2.4
misc3d  0.8-4
miscTools   0.6-22
mitml   0.3-5
mitools 2.3
mix 1.0-10
mixtools    1.1.0
mlbench 2.1-1
mnormt  1.5-5
modelr  0.1.2
modeltools  0.2-21
msm 1.6.6
multcomp    1.4-8
munsell 0.5.0
mvProbit    0.1-8
mvbutils    2.7.4.1
mvtnorm 1.0-8
neighbr 1.0
network 1.13.0.1
nloptr  1.0.4
nnls    1.4
nor1mix 1.2-3
norm    1.0-9.5
nortest 1.0-4
np  0.60-8
numDeriv    2016.8-1
nws 1.7.0.1
nycflights13    1.0.0
openssl 1.0.1
openxlsx    4.1.0
ordinal 2018.4-19
orthopolynom    1.0-5
pan 1.6
pander  0.6.1
partDSA 0.9.14
party   1.3-0
pbivnorm    0.6.0
pbkrtest    0.4-7
pcaPP   1.9-73
permute 0.9-4
pillar  1.2.3
pixmap  0.4-11
pkgKitten   0.1.4
pkgconfig   2.0.1
pkgmaker    0.27
plm 1.6-6
plogr   0.2.0
plotmo  3.4.1
plotrix 3.7-2
pls 2.6-0
plyr    1.8.4
pmml    1.5.4
pmmlTransformations 1.3.1
png 0.1-7
polspline   1.1.13
polyclip    1.9-0
polycor 0.7-9
polynom 1.3-9
portableParallelSeeds   0.97
ppcor   1.1
pracma  2.1.4
praise  1.0.0
prettyunits 1.0.2
processx    3.1.0
prodlim 2018.04.18
profileModel    0.5-9
promises    1.0.1
proto   1.0.0
proxy   0.4-19
pscl    1.5.2
psidR   1.5
pspline 1.0-18
psych   1.8.4
purrr   0.2.5
quadprog    1.5-5
quantmod    0.4-13
quantreg    5.36
qvcalc  0.9-1
rJava   0.9-10
randomForest    4.6-14
randomForestSRC 2.6.1
rattle  5.1.0
rbenchmark  1.0.0
rbugs   0.5-9
rda 1.0.2-2
readr   1.1.1
readstata13 0.9.2
readxl  1.1.0
recipes 0.1.3
registry    0.5
relimp  1.0-5
rematch 1.0.1
repr    0.15.0
reprex  0.2.0
reshape 0.8.7
reshape2    1.4.3
rex 1.1.2
rgenoud 5.8-2.0
rgexf   0.15.3
rgl 0.99.16
rio 0.5.10
rlang   0.2.1
rlecuyer    0.3-4
rmarkdown   1.10
rms 5.1-2
rngtools    1.3.1
robustbase  0.93-1
rockchalk   1.8.111
roxygen2    6.0.1
rpart   4.1-13
rpart.plot  2.2.0
rpf 0.59
rprojroot   1.3-2
rrcov   1.4-4
rstan   2.17.3
rstudio 0.98.978
rstudioapi  0.7
rvest   0.3.2
sandwich    2.4-0
scales  0.5.0
scatterplot3d   0.3-41
segmented   0.5-3.0
selectr 0.4-1
sem 3.1-9
semTools    0.5-0
setRNG  2013.9-1
sets    1.0-18
sfsmisc 1.1-2
shapefiles  0.7
shiny   1.1.0
simsem  0.5-14
slam    0.1-40
sm  2.2-5.5
smoothSurv  2.0
sna 2.4
snow    0.4-2
snowFT  1.6-0
sourcetools 0.1.7
sp  1.3-1
spData  0.2.9.0
spam    2.2-0
spatialCovariance   0.6-9
spatialkernel   0.4-23
spatstat    1.56-0
spatstat.data   1.3-1
spatstat.utils  1.8-2
spdep   0.7-7
splancs 2.01-40
stabledist  0.7-1
stabs   0.6-3
startupmsg  0.9.4
statmod 1.4.30
statnet.common  4.1.4
stepwise    0.3
stopwords   0.9.0
stringi 1.2.3
stringr 1.3.1
strucchange 1.5-1
subselect   0.14
survey  3.33-2
survival    2.42-4
systemfit   1.1-22
tables  0.8.4
tcltk2  1.2-11
tensor  1.5
tensorA 0.36
testthat    2.0.0
texreg  1.36.23
tfplot  2015.12-1
tframe  2015.12-1
tibble  1.4.2
tidyr   0.8.1
tidyselect  0.2.4
tidytext    0.1.9
tidyverse   1.2.1
timeDate    3043.102
timeSeries  3042.102
tinytex 0.5
tis 1.34
tkrplot 0.0-23
tm  0.7-4
tokenizers  0.2.1
topicmodels 0.2-7
tree    1.0-39
triangle    0.11
trimcluster 0.1-2
truncnorm   1.0-7
trust   0.1-7
tseries 0.10-45
ucminf  1.1-4
urca    1.3-0
utf8    1.1.4
vcd 1.4-4
vegan   2.5-2
viridis 0.5.1
viridisLite 0.3.0
visNetwork  2.0.4
waveslim    1.7.5
webshot 0.5.0
whisker 0.3-2
withr   2.1.2
wnominate   1.2.5
xgboost 0.71.2
xml2    1.2.0
xtable  1.8-2
xts 0.10-2
yaml    2.1.19
zip 1.0.0
zipfR   0.6-10
zoo 1.8-2
Posted in Data Analysis | Leave a comment

GRA positions open beginning Summer, 2018

We have 2 GRA positions for 2018-2019 available now. Please consider applying:

http://crmda.ku.edu/gra

Posted in Data Analysis | Leave a comment

stationery package for R in testing

The R package "stationery" offers themes and templates for 8 document types. It offers working examples of documents that use R markdown and LaTeX/Sweave and LaTeX/knitr to prepare guides, slides, and reports.

The 8 document types are

  1. HTML webpages prepared with Rmarkdown
  2. PDF guide documents prepared with Rmarkdown
  3. PDF report documents prepared with Rmarkdown
  4. PDF guide documents prepared with LaTeX/NoWeb using the knitr code chunk engine
  5. PDF guide documents prepared with LaTeX/NoWeb using the Sweave code chunk engine
  6. PDF report documents prepared with LaTeX/NoWeb using the knitr code chunk engine
  7. PDF report documents prepared with LaTeX/NoWeb using the Sweave code chunk engine
  8. PDF slides prepared with a KU customized Beamer slide template and the Sweave code chunk engine.

At the current time, the customization and integration of the Sweave code chunk engine with our document types is considerably further along. The knitr chunk engine documents are much more difficult to customize and we are currently in a default state with them.

Today we have uploaded the first version for general testing, numbered 0.71. To install in R, run the following.

CRAN <-"http://rweb.crmda.ku.edu/cran"
KRAN <-"http://rweb.crmda.ku.edu/kran"
options(repos =c(KRAN,CRAN)) 
install.packages("stationery", dep=TRUE, type="source")

After the signature, see the R session that proves this really will work.

Right now, all 8 of the document types can be initiated by the function we provide called "initWriteup". To create a new template folder, it is as simple as running

initWriteup("rnw2pdf-slides-sweave")

That will create a directory "writeup/rnw2pdf-slides-sweave". The first time you compile the document, it will create a folder named "theme" and copy some template files and logos into that folder. After that, if you want to replace the logos with your own pictures, or customize the templates, it will be quite obvious how to do so.

The vignettes directory right now has several instruction sheets that are nearing a presentable state. The vignettes "Rmarkdown", "code_chunks", and "HTML_special_features" are readable for humans, if not prize winning. The document-type-specific vignettes are going to undergo clarification this weekend.

Among those document specific guides listed here, the asterixed writeups are most complete:

instructions-rmd2pdf-guide.pdf
instructions-rmd2pdf-report.pdf
instructions-rnw2pdf-guide-knit.pdf (*)
instructions-rnw2pdf-guide-sweave.pdf
instructions-rnw2pdf-report-knit.pdf (*)
instructions-rnw2pdf-report-sweave.pdf
instructions-rnw2pdf-slides-sweave.pdf

pj

> CRAN <-"http://rweb.crmda.ku.edu/cran"
> KRAN <-"http://rweb.crmda.ku.edu/kran"
> options(repos =c(KRAN,CRAN))
> install.packages("stationery", dep=TRUE, type="source")
Installing package into ‘/home/pauljohn/R/x86_64-pc-linux-gnu-library/3.4’
(as ‘lib’ is unspecified)
trying URL 'http://rweb.crmda.ku.edu/kran/src/contrib/stationery_0.71.tar.gz'
Content type 'application/x-gzip' length 6786069 bytes (6.5 MB)
==================================================
downloaded 6.5 MB

* installing *source* package ‘stationery’ ...
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
   ‘HTML_special_features.Rmd’ using ‘UTF-8’
   ‘Rmarkdown.Rmd’ using ‘UTF-8’
   ‘code_chunks.Rmd’ using ‘UTF-8’
   ‘instructions-rmd2pdf-guide.Rmd’ using ‘UTF-8’
   ‘instructions-rmd2pdf-report.Rmd’ using ‘UTF-8’
   ‘instructions-rnw2pdf-guide-knit.Rnw’ using ‘UTF-8’
   ‘instructions-rnw2pdf-report-knit.Rnw’ using ‘UTF-8’
   ‘instructions-rnw2pdf-guide-sweave.Rnw’ using ‘UTF-8’
   ‘instructions-rnw2pdf-report-sweave.Rnw’ using ‘UTF-8’
   ‘instructions-rnw2pdf-slides-sweave.Rnw’ using ‘UTF-8’
   ‘stationery.Rnw’ using ‘UTF-8’
** testing if installed package can be loaded
* DONE (stationery)

The downloaded source packages are in
        ‘/tmp/Rtmpc2b9sK/downloaded_packages’
Posted in Data Analysis | Leave a comment

Using too many files in your cluster account? What to do!

After the cluster restart yesterday, I got this message indicating I'm over the limit on the number of files allowed in a home account. Previously I was allowed 100,000, now it appears that is lowered to 85,000. Oh, well.

The email looks like this:

PanActive Manager Warning: User Quota Violation Soft (files)

Date:        Wed Feb 07 01:19:29 CST 2018           
System Name: pfs.local                              
System IP:   10.71.31.203 10.71.31.100 10.71.31.102 
Version      6.3.1.a-1371318.1                      
Customer ID: 1021288                                

User Quota Violation Soft (files):  Limit reached on volume /home for Unix User:   (Id:  uid:142424) Limit = 85.00 K.

The above message applies to the following component:
    Volume: /home 

If I wanted to find the directories in my account where I'm using a lot of storage, I would do this

$ du -h --max-depth=1

to see the total disk size used by the files in /home. However, that does not help me isolate the number of files in use.

I checked with Riley at KU CRC and he gave me this magic recipe.

$ for i in $(find . -maxdepth 1 ! -path . -type d); do echo -n $i": "; (find $i -type f | wc -l); done | sort -k 2 -n

I'd seen that before, but forgotten. I can use same trick to get a proper sorting of the du output, so I can also see which folders are holding the most knowledge (er, files).

$ du --max-depth=1 | sort -n

Note I removed the human-readable flag ("-h") and I instruct the sort function to treat the first return value as a number, rather than text.

Posted in Data Analysis | Leave a comment

New update of kutils package for R available

I have uploaded kutils-1.31 on KRAN. To try it, fix your repositories like so:

CRAN <- "http://rweb.crmda.ku.edu/cran"
KRAN <- "http://rweb.crmda.ku.edu/kran"
options(repos = c(KRAN, CRAN))

If you have kutils already, then fire away with an update:

update.packages(ask = FALSE, checkBuilt = TRUE)

In case you don't have kutils before now (how could that possibly be?), run

install.packages("kutils", dep = TRUE)

Todays updates include new functions

importQualtrics : Imports Qualtrics files (csv or xlsx), gets column names correct and avoids corruption caused by lines 2 and 3 in most Qualtrics files. I need an example file for this one, if you have a Qualtrics file for which you have legal rights, lets see about it.

mergeCheck : reports on bad and unmatched values of the "by" (key) variable. Has examples!

anonomize : convert ID to conceal identity of survey participants

keyTemplateSPSS : creates a key describing the value_old, value_new changes implied by SPSS value labels

Please test mergeCheck, it is the function I've been promising you for some time. (example pasted below). I need to hear what it does to mangle your own key variables and such.

Also this includes the miscellaneous little updates for variable key that we've been implementing for the past 2 months.

keysPoolCheck : for several keys from related data sets, checks for mis-matches in class

keysPool: tries to blend keys. We have had this since summer, but did not export it into the workspace because it was not well tested. So please test it.

Why now?

Soon we will have a separate package for the SEM table writer functions, semTable. I mean, we have that now, soon to be on CRAN. That other package relies on some general purpose functions in kutils and it is necessary to upload kutils on CRAN and have it approved before we can upload semTable.

mergeCheck output to tantalyze you.

> library(kutils)
> example(mergeCheck)

mrgChc> df1 <- data.frame(id = 1:7, x = rnorm(7))

mrgChc> df2 <- data.frame(id = c(2:6, 9:10), x = rnorm(7))

mrgChc> mc1 <- mergeCheck(df1, df2, by = "id")
Merge difficulties detected

Unmatched cases from df1 and df2 :
df1
  id          x
1  1 -0.9292122
7  7  0.3819611
df2
  id          x
6  9 -1.3481932
7 10  0.6325223

mrgChc> ## Use mc1 objects mc1$keysBad, mc1$keysDuped, mc1$unmatched
mrgChc> df1 <- data.frame(id = c(1:3, NA, NaN, "", " "), x = rnorm(7))

mrgChc> df2 <- data.frame(id = c(2:6, 5:6), x = rnorm(7))

mrgChc> mergeCheck(df1, df2, by = "id")
Merge difficulties detected

Unacceptable key values
df1
    id          x
4 <NA> -0.5295721
5  NaN -0.1538291
6       0.3073913
Duplicated key values
df2
  id           x
4  5  0.57832104
5  6 -0.01897907
6  5  1.08232745
7  6  1.88448038
Unmatched cases from df1 and df2 :
df1
    id          x
1    1  0.9593052
4 <NA> -0.5295721
5  NaN -0.1538291
6       0.3073913
7      -0.3304785
df2
  id           x
3  4 -0.47677841
4  5  0.57832104
5  6 -0.01897907
6  5  1.08232745
7  6  1.88448038

mrgChc> df1 <- data.frame(idx = c(1:5, NA, NaN), x = rnorm(7))

mrgChc> df2 <- data.frame(idy = c(2:6, 9:10), x = rnorm(7))

mrgChc> mergeCheck(df1, df2, by.x = "idx", by.y = "idy")
Merge difficulties detected

Unacceptable key values
df1
  idx         x
6  NA -1.457980
7 NaN -0.707062
Unmatched cases from df1 and df2 :
df1
  idx          x
1   1  0.8150989
6  NA -1.4579796
7 NaN -0.7070620
df2
  idy         x
5   6 1.2318111
6   9 1.1306955
7  10 0.1466441
Posted in Data Analysis | Leave a comment

kutils package update: more powerful SEM table maker

I have uploaded kutils-1.25, a beta testing version on KRAN. If your computer is not set to automatically pull updates from KRAN already, run this:

CRAN <- "http://rweb.crmda.ku.edu/cran"
KRAN <- "http://rweb.crmda.ku.edu/kran"
options(repos = c(KRAN, CRAN))
update.packages(ask = FALSE, checkBuilt = TRUE)

In case you don't have kutils before now (how could that possibly be?), run

install.packages("kutils", dep = TRUE)

The special feature we are testing now is the enhanced semTable function. In a nutshell, here is what this can do. Provide one or more structural equation models fitted with lavaan. Then summary tables can be presented that display the models side by side. Using the arguments columns and paramSets, the user is able to control which columns are displayed for which model, and which model sections are included.

The examples section of the help page for semTable includes 20 examples. It also demonstrates a new function called testtable, which can compile and display the PDF output from LaTeX tables.

To whet your appetite, this code fits a multi-group CFA:

 require(lavaan)
 tempdir <- tempdir()
 HS.model <- ' visual  =~ x1 + x2 + x3
                   textual =~ x4 + x5 + x6
                   speed   =~ x7 + x8 + x9'
 fit1.g <- cfa(HS.model, data = HolzingerSwineford1939, std.lv = TRUE, group = "school")
 fit1.gt1 <- semTable(fit1.g, columns = c("estsestars", "p"),
                    columnLabels = c(estsestars = "Est w/stars", p = "p-value"),
                    file = file.path(tempdir, "fit1.gt1"))
if (interactive()) testtable("fit1.gt1", tempdir)

Creates a PDF file that shows the 2 groups side by side:

The default settings will display all of the groups, side by side. That might become crowded, so we allow the user to select which columns are to be displayed for the groups. It is possible also to use the groups argument to name one or more groups to be displayed in the presentation (helps when there are too many groups to fit in one table). In the multi group model, we do not allow different columns to be displayed for the various groups.

If the user has several SEM, they can be displayed side by side and the user is allowed to customize the list of displayed columns for each separate model. As we demonstrate here, we can fit a non-standardized SEM and a standardized model and display them side by side.

## Fit same model with standardization
fit1 <- cfa(HS.model, data = HolzingerSwineford1939,
             std.lv = TRUE, meanstructure = TRUE)
 fit1.std <- update(fit1, std.lv = TRUE, std.ov = TRUE, meanstructure = TRUE)
 ## include 2 models in table request
 fit1.t2 <- semTable(list("Ordinary" = fit1, "Standardized" = fit1.std),
                     file = file.path(tempdir, "fit1.2.1"))
 semTable(list("Ordinary" = fit1, "Standardized" = fit1.std),
     columns = list("Ordinary" = c("est", "se"), "Standardized" = c("est")),
     columnLabels = c(est = "Est", se = "SE"), file = file.path(tempdir, "fit1.2.2"))
 if (interactive()) testtable("fit1.2.2", tempdir)

These examples demonstrate the ability to create LaTeX tables, because that's what I use. However, the function
is designed to also create tables in HTML and CSV formats, The magic recipe for that it to insert, say, type = c("latex", "html", "csv") into the semTable funciton call. If you also add file="whatever" then three files will be created, "whatever.tex", "whatever.html" and "whatever.csv". The HTML file can be viewed within the R session by running

browseURL("whatever.html")

The appeal of the HTML output is that an HTML file can be opened by a word processor, say LibreOffice or MS Word, and it will generally be turned into a table object which can be edited. The CSV file may be used in the same way, in conjunction with a spread sheet program.

This semTable edition is intended to allow one to fit different models that can be combined into one table. The standardized example above is perhaps not the most persuasive demonstration. Consider the following instead.

## Model 5 - Mediation model with equality constraints              
model5 <-                                                           
    '                                                               
    # latent variable definitions                                   
    ind60 =~ x1 + x2 + x3                                           
    dem60 =~ y1 + e*y2 + d*y3 + y4                                  
    dem65 =~ y5 + e*y6 + d*y7 + y8                                  
    # regressions                                                   
    dem60 ~ a*ind60                                                 
    dem65 ~ c*ind60 + b*dem60                                       
    # residual correlations                                         
    y1 ~~ y5                                                        
    y2 ~~ y4 + y6                                                   
    y3 ~~ y7                                                        
    y4 ~~ y8                                                        
    y6 ~~ y8                                                        

    # indirect effect (a*b)                                         
    ## := operator defines new parameters                           
    ab := a*b                                                       

    ## total effect                                                 
    total := c + (a*b)                                              
    '                                                               
 fit5 <- sem(model5, data=PoliticalDemocracy)                        
 fit5boot <- sem(model5, data=PoliticalDemocracy, se = "bootstrap", boot = 100)                                               
 ## Model 5b - Revision of Model 5s                                  
 model5b <-                                                          
    '                                                               
    # latent variable definitions                                   
    ind60 =~ x1 + x2                                                
    dem60 =~ y1 + e*y2 + d*y3 + y4                                  
    dem65 =~ y5 + e*y6 + d*y7 + y8                                  
    # regressions                                                   
    dem60 ~ a*ind60                                                 
    dem65 ~ c*ind60 + b*dem60                                       
    # no residual correlations                                      
    # indirect effect (a*b)                                         
    ## := operator defines new parameters                           
    ab := a*b                                                       

    ## total effect                                                 
    total := c + (a*b)                                              
    '                                                               

fit5b <- sem(model5b, data=PoliticalDemocracy, se = "bootstrap",    
bootstrap = 100)                                                    
semTable(list("Model 5" = fit5boot, "Model 5b" = fit5b),            
         columns = c("estsestars", "rsquare"),                      
         file = file.path(tempdir, "fit5.5"),                       
          type = c("latex", "html", "csv"),                         
         longtable = TRUE)                                          
testtable("fit5.5", tempdir)                                        

It seems to me this result is not exactly perfect. I wish it did not print "NA" on the omitted loadings. But I'm headed in the right direction. This lines up the parts of the models, makes it plain which pieces are included in each model.

Posted in Data Analysis, R | Leave a comment

R modules: Super Exciting New Updates

This is revised Monday, July 24, 2017.

Some of you have reported segmentation faults during the past week. We learned they come from 3 different problems. First, some people have R packages compiled in their user accounts. These fall out-of-date with the R packages we provide, causing incompatability. Second, some new compute nodes came on line during the past 2 weeks and some are missing support libraries. When these are missing, the R packages that rely on them (such as our beloved kutils or rockchalk) would fail to load. This was a tricky problem because it only happened on some nodes, which only became recently available. Third, I did not understand the gravity and drama involved with the user account setup and the Rmpi package.

Lets skip to the chase. What should users do now.

Step 1. remove module statements from submission scripts.

Those statements are not having the anticipated effect, and they will destroy the benefits of the changes I suggest next.

I'm told this problem does not affect all MPI jobs, just ones that use R and the style of parallelization that we understand.

Step 2. Configure your individual user account to load appropriate modules.

Some module should be available for every session launched for your account, in every node. These have to be THE SAME in all nodes and cores launched by the job. There are 2 ways to get this done.

Option 1. The easy way: Use my R package module stanza, crmda_env.sh

In the cluster file system, I have a file /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh with contents like this:

#!/bin/bash

module purge
module load legacy
module load emacs
module use /panfs/pfs.local/work/crmda/tools/modules
module load Rstats/3.3
OMPI_MCA_btl=^openib
export OMPI_MCA_btl

I say "like this" because I may insert new material there. The last 2 lines were inserted July 22, 2017. The goal is to conceal all of the details from users by putting them in a module that's loaded, such as Rstats/3.3. When we are ready to transition to R-3.4, I'll change that line accordingly.

In your user accounts, there are 2 files where you can incorporate this information, they are ~/.bashrc and ~/.bash_profile. Add a last line in those files like this:

source /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh

I'll show you my ~/.bashrc file so you can see the larger context:

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi

# User specific aliases and functions
export LS_COLORS=$LS_COLORS:'di=0;33:'
# alert for rm, cp, mv
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# color and with classification
alias ls='ls -F --color=auto'
alias ll='ls -alF'

source /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh

I strongly urge all of our cluster users to include the "alert for rm, cp, mv" piece. This causes the system to ask for confirmation before deleting or replacing files. But that's up to you. I also have some an adjustment to the colors of the directory listing.

I insert the same "source" line at the end of ~/.bash_profile as well.

On 2017-07-23, I made a minor edit in my .bashrc and .bash_profile files:

export PATH=/panfs/pfs.local/work/crmda/tools/bin:$PATH
source crmda_env.sh

This is equivalent, but gives me a side benefit. Instead of adding the source function with the full path, I inserted that bin folder into my path. That means I can use any script in that folder without typing out the full path. When I find very handy shell scripts that I use often, and I think the other users should have access to them as well, then I will put them in that folder. For example, if you look there today, you should see "crmda_env-test.sh", which is the new one I'm working on. When that's ready, it will become "crmda_env.sh" and the old one will get renamed as "crmda_env-2017xxxx.sh", where xxxx is the date on which it becomes the old one.

Option 2. Add your own module statements in ~/.bashrc and ~/.bash_profile

Make sure you put the same modules in both ~./bashrc and ~./bash_profile. Look at the file /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh to get ideas of what you need. For example, run

$ cat /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh

You might consider creating a file similar to /panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh in your account. Then source that at the end of your ~/.bashrc and ~/.bash_profile. If you do that, they will always stay consistent.

Frequently Asked Questions that I anticipate

Can you explain why the segmentation fault happens?

Answer: Yes, I have some answers.

Here is the basic issue. Suppose you have a submission script that looks like this:

#!/bin/sh
#
#
#MSUB -N RParallelHelloWorld 
#MSUB -q crmda
#MSUB -l nodes=1:ppn=11:ib
#MSUB -l walltime=00:50:00
#MSUB -M your-name-here@ku.edu
#MSUB -m bea

cd $PBS_O_WORKDIR

module purge 
module load legacy 
module load emacs 
module use /panfs/pfs.local/work/crmda/tools/modules 
module load Rstats/3.3

mpiexec -n 1 R --vanilla -f parallel-hello.R 

I though we were supposed to do that, until last week. Here's what is wrong with it.

The environment specifies Rstats/3.3, but that ONLY applies to the "master" node in the R session. It does not apply to the "child" nodes that are spawned by Rmpi. Those nodes are spawned, they are completely separate shell sessions and they are launched by settings in ~/.bash_profile. If your ~/.bash_profile does not have the required modules, then the new nodes are going to have the system default R session, and guess what you get with that? The wrong shared libraries for just about everything. Possibly you get a different version of Rmpi or Rcpp loaded, and when the separate nodes start taking to each other, they notice the difference and sometimes crash.

As a result, the submission scripts, for example, in hpcexample/Ex65-R-parallel, will now look like this:

#!/bin/sh
#
#
#MSUB -N RParallelHelloWorld
#MSUB -q crmda
#MSUB -l nodes=1:ppn=11:ib
#MSUB -l walltime=00:50:00
#MSUB -M pauljohn@ku.edu
#MSUB -m bea

cd $PBS_O_WORKDIR

## Please check your ~/.bash_profile to make sure
## the correct modules will be loaded with new shells.
## See discussion:
## http://www.crmda.dept.ku.edu/timeline/archives/184

mpiexec -n 1 R --vanilla -f parallel-hello.R

Why is this needed for both ~/.bashrc and ~/.bash_profile?

Answer: You ask a lot of questions.

The short answer is "there's some computer nerd detail". The long answer is, "when you log in on a system, the settings in ~/.bash_profile are used. That is a 'login shell'. If you are in already, and you run a command that launches a new shell inside your session, for example by running "bash", then your new shell is not a 'login shell'. It will be created with settings in ~./bashrc.

If you will never run an interactive session, never interact with R via Emacs or Rstudio, then it might be enough to change ~/.bash_profile. If you think you might ever want to log in and run a small test case, then you should have same in both ~/.bashrc and ~/.bash_profile.

What are the benefits of Option 1?

Answer: Over time, the CRMDA R setup may evolve. Right now, I've already built a setup Rstats/3.4. After we do some bug-testing, then I can easily update the shell file (/panfs/pfs.local/work/crmda/tools/bin/crmda_env.sh) and use that. If you maintain your own modules, then you have to do that yourself.

What are the dangers of Option 1?

Answer: If I get it wrong, then you get it wrong.

Does this mean you need to revise all of the code examples in the hpcexample (https://gitlab.crmda.ku.edu/crmda/hpcexample​) set?

Answer: Yes. It has not been a good week. And it looks like it won't be a good week again.

Why didn't we hear about this in the old community cluster, or in CRMDA's HPC cluster

Answer: Because "we" were in control of the cluster settings and user accounts, the cluster administrators would work all of this out for us and they inserted the settings in the shell for us. Some of you may open your ~/.bashrc or ~/.bash_profile and see the old cluster settings. When I opened mine on 2017-07-07, I noticed that I had modules loaded from the old cluster. I also noticed I'd made an error of editing ~/.bashrc and not ~/.bash_profile.

Why didn't we see these problems before?

Answer: Dumb luck.

In the new CRC-supervised cluster, some modules are loaded automatically. As those modules were more-or-less consistent with what we need to do, then the different environments were not causing segmentation faults. However, when we update the R packages like Rstan, Rcpp, and, well, anything with a lot of shared libraries, then we hit the crash.

I notice you don't have oreterun in your submission example. Do you mean mpiexec really?

Answer: The documentation says that orterun, mpiexec, and mpirun are all interchangeable. I rather enjoyed orterun, it sounds fancy. However, it appears mpiexec is more widely used. There are more advanced tools (such as mpiexec.hydra, which we might start using).

In your submission script, why don't you specify the $PBS_NODEFILE any more.

Answer: The program mpiexec is compiled in a way that makes this no longer necessary. It is not harmful to specify $PBS_NODEFILE, but it is not needed either. The hpcexamples will get cleaned up. The CRMDA cluster documentation will need to be corrected.

Posted in Computing, R | Leave a comment