nestedscales provides a multi-level discrete representation of a data dimension and the operations that move data between its levels. This vignette introduces the representation and the main functions on a small industry classification. The dimension packages timescales and geoscales apply the same functions to calendars and region hierarchies. The package website includes a roadmap of the package’s scope.
A nested scale is a table with one row per atom, the finest unit of the dimension, and one column per frame, a coarser level that groups the atoms. The frames are given coarsest first. Numeric columns that are not frames are weights.
industries <- data.frame(
section = c("C", "C", "C", "D", "D"),
division = c("C10", "C10", "C11", "D35", "D35"),
class = c("C101", "C102", "C110", "D351", "D352"),
gva = c(120, 80, 45, 300, 150)
)
ind <- scale_from_leaftable(
industries,
frames = c("section", "division", "class"),
name = "nace"
)
ind
#> NestedScale: nace
#> Frames (3, coarsest first):
#> - section (2)
#> - division (3)
#> - class (5)
#> Atoms: 5
#> Weights: gva (default: gva)The scale records the frames, the units at each frame, the atom key and the weights:
scale_frames(ind)
#> [1] "section" "division" "class"
scale_units(ind, "division")
#> [1] "C10" "C11" "D35"
scale_key(ind)
#> [1] "class"
scale_weights(ind)
#> [1] "gva"Every frame partitions the atoms. Frames need not nest in each other:
a unit at one frame may belong to several units at a coarser frame, and
two frames may be independent, in which case the atoms are their
combinations. scale_nests(), scale_crosses()
and scale_is_uniform() report the relation a given pair of
frames has; no relation is assumed.
scale_nests(ind, "section", "division")
#> [1] TRUE
scale_family(ind, "section", "division")
#> parent_frame parent child_frame child
#> 1 section C division C10
#> 2 section C division C11
#> 3 section D division D35Data keyed at one frame is converted to another frame with
recast_scale(). Each value column follows a rule from
SCALE_RULES:
| rule | aggregation (fine to coarse) | disaggregation (coarse to fine) |
|---|---|---|
sum |
sum | split in proportion to the weight |
weighted_mean |
weighted mean | copy |
mean |
mean over atoms | copy |
copy |
the common value | copy |
sd |
standard deviation over atoms | not defined |
share |
value as a share of its parent’s total | not defined |
The rule is given per value column, so extensive and intensive quantities convert in one call:
x <- data.frame(
class = c("C101", "C102", "C110", "D351", "D352"),
jobs = c(12, 8, 5, 30, 20),
wage = c(31, 29, 27, 45, 40)
)
recast_scale(x, ind,
source = "class", target = "section",
rule = c(jobs = "sum", wage = "weighted_mean")
)
#> section jobs wage
#> 1 C 25 29.61224
#> 2 D 50 43.33333The same function converts in the other direction. A total at the section level is split over the classes in proportion to the weight:
y <- data.frame(section = c("C", "D"), investment = c(100, 400))
recast_scale(y, ind, source = "section", target = "class", rule = "sum")
#> class investment
#> 1 C101 48.97959
#> 2 C102 32.65306
#> 3 C110 18.36735
#> 4 D351 266.66667
#> 5 D352 133.33333Columns that are neither the key nor a value column are identifiers
and are preserved as groups, so a panel of several years or scenarios
converts in one call. A rule may also be registered once for a named
value column with register_scale_rule(), after which
rule = can be omitted for that column.
join_scale() attaches the scale’s labels to a dataset:
the unit at the keyed frame, optionally the coarser frames
(attach =) and the unit’s share and weight
(meta =):
join_scale(x, ind, frame = "class", attach = TRUE)
#> class jobs wage nace.section nace.division nace
#> 1 C101 12 31 C C10 C101
#> 2 C102 8 29 C C10 C102
#> 3 C110 5 27 C C11 C110
#> 4 D351 30 45 D D35 D351
#> 5 D352 20 40 D D35 D352filter_scale() keeps the atoms under given units; the
result records what fraction of each weight it retains.
prune_scale() collapses a scale to a coarser atom
level.
manufacturing <- filter_scale(ind, "section", "C")
scale_units(manufacturing)
#> [1] "C101" "C102" "C110"
scale_coverage(manufacturing)
#> gva
#> 0.352518
scale_frames(prune_scale(ind, "division"))
#> [1] "section" "division"A scale may carry values that belong to its members but are not
weights: a calorific value per fuel, a density, an emission factor. They
live in an attribute table keyed by (frame, unit), at
whatever level they are published, and scale_attributes()
resolves one value per member of a frame, walking to the nearest coarser
level when a member has none of its own. convert_scale()
applies the resolved value row by row. Alternative labels live in an
alias table, and scale_match() resolves labels against
member names and aliases exactly, reporting ambiguity rather than
guessing. The package ships one such scale, the energypal fuel taxonomy
with the IPCC 2006 defaults:
carriers <- read_scale("carriers")
lhv <- scale_attributes(carriers, "lhv", frame = "fuel")
head(lhv[!is.na(lhv$value), c("unit", "value", "unit_of_measure")])
#> unit value unit_of_measure
#> 1 Anthracite 26.7 GJ/t
#> 2 Bituminous 25.8 GJ/t
#> 3 SubBituminous 18.9 GJ/t
#> 4 Lignite 11.9 GJ/t
#> 5 Coke 28.2 GJ/t
#> 7 Crude 42.3 GJ/t
fuel_use <- data.frame(fuel = c("Lignite", "NaturalGas"), t = c(1000, 1000))
convert_scale(fuel_use, carriers, "lhv", from_unit = "t", to_unit = "GJ")
#> fuel t lhv lhv.origin_frame
#> 1 Lignite 11900 11.9 fuel
#> 2 NaturalGas 48000 48.0 fuel
scale_match(carriers, c("GAS", "lignite", "Hydrogen"))
#> label frame unit method candidates
#> 1 GAS carrier FossilGas alias <NA>
#> 2 lignite fuel Lignite exact <NA>
#> 3 Hydrogen <NA> <NA> ambiguous carrier/Hydrogen; fuel/Hydrogenvignette("attributes") has the full account.
A scale is stored as a definition: a JSON header with its tables
inline, or a bundle directory with Arrow or CSV tables, which the Python
binding reads as well. write_scale() and
read_scale() exchange them, and
scale_catalog() lists the definitions installed packages
ship:
path <- file.path(tempdir(), "nace.json")
write_scale(ind, path)
scale_frames(read_scale(path))
#> [1] "section" "division" "class"
scale_catalog()[c("id", "name", "frames", "n_atoms", "package")]
#> id name frames n_atoms package
#> 1 carriers carriers group/carrier/fuel 71 nestedscalesscale_product() combines scales into one
multi-dimensional index without materialising the product of their
atoms. recast_product(), join_product() and
filter_product() apply the single-scale operations axis by
axis.
p <- scale_product(a = scale_example(), b = scale_example2())
product_size(p)
#> $axes
#> a b
#> 7 4
#>
#> $total
#> [1] 28cluster_scale() groups units by their data and returns
the scale with the clusters as a new frame.reconcile_scale() compares aggregated data with an
independent total.as_scale_array() converts long data to a dense array
over a scale, and write_scale_dataset() stores data as an
Arrow dataset together with its scales.All operations accept a data.frame, tibble,
data.table, dtplyr table or arrow dataset and return the
same class; lazy inputs return a query unless
collect = TRUE.