sub-.hdd.RdThis function extract data from HDD files, in a similar fashion as data.table but with more arguments.
# S3 method for class 'hdd'
`[`(
x,
index,
...,
file,
destDir = NULL,
newfile = NULL,
replace = TRUE,
all.vars = FALSE
)An hdd data set.
An index, you can use .N and variable names, like in data.table.
Other components of the extraction to be passed to
data.table.
Integer vector. Which file to extract from?
Remember hdd data is split in several files.
You can use the special variable .N to refer to the last file.
A destination directory, default is NULL.
If destDir=NULL, the default, then the result of the query is loaded into memory.
To save the result of the query into a new directory, use destDir.
(Deprecated, please use destDir now.)
A destination directory, default is NULL.
Logical scalar, default is TRUE.
Only used if argument destDir is not missing. If replace=TRUE, the default,
the destination directory is cleaned before running the query.
If replace=FALSE and the destination directory already contains an HDD data set,
an error is thrown.
Logical scaalr, default is FALSE.
By default, if the first argument of ... is provided (i.e. argument j)
then only variables appearing in all ... plus the variable names found
in index are extracted. If TRUE all variables are extracted
before any selection is done. (This can be useful when the algorithm getting
the variable names gets confused in case of complex queries.)
Returns a data.table extracted from a HDD file (except if newwfile is not missing).
The extraction of variables look like a regular data.table extraction
but in fact all operations are made chunk-by-chunk behind the scene.
The extra arguments file, destDir and replace are added to
a regular data.table call.
Argument file is used to select the chunks, you can use the special variable .N
to identify the last chunk.
By default, the operation loads the data in memory. But if the expected size is still too large,
you can use the argument destDir to create a new HDD data set without size restriction.
See hdd, sub-.hdd and cash-.hdd
for the extraction and manipulation of out of memory data. For importation of
HDD data sets from text files: see txt2hdd.
See hdd_slice to apply functions to chunks of data (and create
HDD objects) and hdd_merge to merge large files.
To create/reshape HDD objects from memory or from other HDD objects, see
write_hdd.
To display general information from HDD objects: origin,
summary.hdd, print.hdd,
dim.hdd and names.hdd.
# Toy example with iris data
# First we create a hdd data set to run the example
hdd_path = tempfile()
write_hdd(iris, hdd_path, rowsPerChunk = 40)
# your data set is in the hard drive, in hdd format already.
data_hdd = hdd(hdd_path)
# summary information on the whole file:
summary(data_hdd)
#> Hard drive data of 7.36 KB. Made of 4 files.
#> Location: C:/Users/berge028/AppData/Local/Temp/RtmpCKHydE/file19c858c41f15/
#> 150 lines, 5 variables.
# You can use the argument 'file' to subselect slices.
# Let's have some descriptive statistics of the first slice of HDD
summary(data_hdd[, file = 1])
#> Sepal.Length Sepal.Width Petal.Length Petal.Width
#> Min. :4.300 Min. :2.900 Min. :1.000 Min. :0.1000
#> 1st Qu.:4.800 1st Qu.:3.200 1st Qu.:1.400 1st Qu.:0.2000
#> Median :5.000 Median :3.400 Median :1.500 Median :0.2000
#> Mean :5.057 Mean :3.462 Mean :1.465 Mean :0.2405
#> 3rd Qu.:5.400 3rd Qu.:3.700 3rd Qu.:1.600 3rd Qu.:0.3000
#> Max. :5.800 Max. :4.400 Max. :1.900 Max. :0.5000
#> Species
#> setosa :37
#> versicolor: 0
#> virginica : 0
#>
#>
#>
# It extract the data from the first HDD slice and
# returns a data.table in memory, we then apply summary to it
# You can use the special argument .N, as in data.table.
# the following query shows the first and last lines of
# each slice of the HDD data set:
data_hdd[c(1, .N), file = 1:.N]
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <num> <num> <num> <num> <fctr>
#> 1: 5.1 3.5 1.4 0.2 setosa
#> 2: 5.5 3.5 1.3 0.2 setosa
#> 3: 4.9 3.6 1.4 0.1 setosa
#> 4: 6.4 2.9 4.3 1.3 versicolor
#> 5: 6.6 3.0 4.4 1.4 versicolor
#> 6: 6.4 2.7 5.3 1.9 virginica
#> 7: 6.8 3.0 5.5 2.1 virginica
#> 8: 5.9 3.0 5.1 1.8 virginica
# Extraction of observations for which the variable
# Petal.Width is lower than 0.1
data_hdd[Petal.Width < 0.2, ]
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <num> <num> <num> <num> <fctr>
#> 1: 4.9 3.1 1.5 0.1 setosa
#> 2: 4.8 3.0 1.4 0.1 setosa
#> 3: 4.3 3.0 1.1 0.1 setosa
#> 4: 5.2 4.1 1.5 0.1 setosa
#> 5: 4.9 3.6 1.4 0.1 setosa
# You can apply data.table syntax:
data_hdd[, .(pl = Petal.Length)]
#> pl
#> <num>
#> 1: 1.4
#> 2: 1.4
#> 3: 1.3
#> 4: 1.5
#> 5: 1.4
#> ---
#> 146: 5.2
#> 147: 5.0
#> 148: 5.2
#> 149: 5.4
#> 150: 5.1
# and create variables
data_hdd[, pl2 := Petal.Length**2]
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species pl2
#> <num> <num> <num> <num> <fctr> <num>
#> 1: 5.1 3.5 1.4 0.2 setosa 1.96
#> 2: 4.9 3.0 1.4 0.2 setosa 1.96
#> 3: 4.7 3.2 1.3 0.2 setosa 1.69
#> 4: 4.6 3.1 1.5 0.2 setosa 2.25
#> 5: 5.0 3.6 1.4 0.2 setosa 1.96
#> ---
#> 146: 6.7 3.0 5.2 2.3 virginica 27.04
#> 147: 6.3 2.5 5.0 1.9 virginica 25.00
#> 148: 6.5 3.0 5.2 2.0 virginica 27.04
#> 149: 6.2 3.4 5.4 2.3 virginica 29.16
#> 150: 5.9 3.0 5.1 1.8 virginica 26.01
# You can use the by clause, but then
# the by is applied slice by slice, NOT on the full data set:
data_hdd[, .(mean_pl = mean(Petal.Length)), by = Species]
#> Species mean_pl
#> <fctr> <num>
#> 1: setosa 1.464865
#> 2: setosa 1.453846
#> 3: versicolor 4.312000
#> 4: versicolor 4.208000
#> 5: virginica 5.675000
#> 6: virginica 5.513158
# If the data you extract does not fit into memory,
# you can create a new HDD file with the argument 'destDir':
hdd_path_new = tempfile()
data_hdd[, pl2 := Petal.Length**2, destDir = hdd_path_new]
#> NULL
# check the result:
data_hdd_bis = hdd(hdd_path_new)
summary(data_hdd_bis)
#> Hard drive data of 8.77 KB. Made of 4 files.
#> Location: C:/Users/berge028/AppData/Local/Temp/RtmpCKHydE/file19c8454752ed/
#> 150 lines, 6 variables.
print(data_hdd_bis)
#> Sepal.Length Sepal.Width Petal.Length Petal.Width Species pl2
#> 1 5.1 3.5 1.4 0.2 setosa 1.96
#> 2 4.9 3 1.4 0.2 setosa 1.96
#> 3 4.7 3.2 1.3 0.2 setosa 1.69
#> ----
#> 148 6.5 3 5.2 2 virginica 27
#> 149 6.2 3.4 5.4 2.3 virginica 29.2
#> 150 5.9 3 5.1 1.8 virginica 26