使用多个大栅格计算PCA

Question

在计算皮尔逊相关性时，下面的脚本对我来说也适用于相同的数据。我最近对其进行了调整，创建了一个协方差矩阵，以输入到pca中。我在论坛上读到，输入预先创建的协方差矩阵可能会避免记忆问题，但我的情况并非如此。运行协方差矩阵时，我会出现以下错误：

Error: cannot allocate vector of size 1.1 Gb
In addition: Warning messages:
1: In na.omit.default(cbind(x, y)) :
  Reached total allocation of 6141Mb: see help(memory.size)
2: In na.omit.default(cbind(x, y)) :
  Reached total allocation of 6141Mb: see help(memory.size)
3: In na.omit.default(cbind(x, y)) :
  Reached total allocation of 6141Mb: see help(memory.size)
4: In na.omit.default(cbind(x, y)) :
  Reached total allocation of 6141Mb: see help(memory.size)

有人能提出一个更有效的方法来做到这一点，这样我就不会遇到内存问题了吗？如果我在计算协方差方面完全偏离了基础，那很好。PCA是我最终唯一需要的东西。我的数据是12,1波段光栅，采用的是ArcGIS的光栅格式，每个都很大，为581.15 mb。非常感谢任何帮助。

library(rgdal)
library(raster)


setwd("K:/Documents/SDSU/Thesis/GIS Data All/GIS Layers/Generated_Layers/GridsForCor")


# List the full path to each raster:
raster_files = c('aspectclp',
                 'lakedistclp',
                 'ocdistclp',
                 'popdenclp',
                 'roaddistclp',
                 'scurveclp',
                 'sdemclp',
                 'solarradclp',
                 'sslopeclp',
                 'vegcatclp',
                 'canopcvrclp',
                 'canophtclp')

cov_matrix <- matrix(NA, length(raster_files), length(raster_files))
for (outer_n in 1:length(raster_files)) {
  outer_raster <- raster(raster_files[outer_n])
  # Start this loop at outer_n rather than 1 so that we don't compute the 
  # same covariance twice. At the end of the loops cov_matrix will be upper 
  # triangular, with the lower triangle all NA, and the diagonal all NA 
  # (since the diagonal would all be 1 anyway).
  for (inner_n in (outer_n):length(raster_files)) {
    # Don't compute correlation of a raster with itself:
    if (inner_n == outer_n) {next}
    inner_raster <- raster(raster_files[inner_n])
    cov_matrix[outer_n, inner_n] <- cov(outer_raster[], inner_raster[], 
                                    use='complete.obs', method = "spearman")
  }
}

pca_matrix <- princomp(raster_files, cor = FALSE, covmat = cov_matrix))

# Writing to a txt file & csv file
write.table(pca_matrix, "PCA.txt", sep="\t", row.names = FALSE)
write.csv(pca_matrix, "PCA.csv") enter code here

Answer 1

我在ffdf对象上进行pca时也有类似的困难。尝试在（内部）循环中插入一个gc（），如下所示：

for (inner_n in (outer_n):length(raster_files)) {
  # Don't compute correlation of a raster with itself:
  if (inner_n == outer_n) {next}
  inner_raster <- raster(raster_files[inner_n])
  cov_matrix[outer_n, inner_n] <- cov(outer_raster[], inner_raster[], 
                                use='complete.obs', method = "spearman")
  gc()
}

这会强制立即进行垃圾收集，为循环的继续释放足够的内存-至少对我来说，这足以让它工作。