diff --git a/R/dig.R b/R/dig.R index 000c0f6..c0b8af6 100644 --- a/R/dig.R +++ b/R/dig.R @@ -70,6 +70,15 @@ #' - `support`: a numeric scalar value of relative frequency of rows satisfying \eqn{C}, #' \eqn{supp = sum / |R|}. #' +#' - `indices`: an integer vector of row indices of rows satisfying \eqn{C} for +#' logical data, or the indices of rows with non-zero truth degrees for fuzzy +#' data, \eqn{indices = \{r \in R : \mu_C(r) > 0\}}. +#' +#' - `weights`: a numeric vector of truth degrees of \eqn{C} for each row in +#' \eqn{R}, \eqn{weights[r] = \mu C(r)}. Logical data is treated as a special +#' case of fuzzy data, where \eqn{\mu_C(r)} is 1 for rows satisfying \eqn{C} +#' and 0 otherwise. +#' #' - `pp`, `pn`, `np`, `nn`: a numeric vector of entries of a contingency table #' for \eqn{C} and \eqn{F}, satisfying the Ruspini condition #' \eqn{pp + pn + np + nn = |R|}. diff --git a/_pkgdown.yml b/_pkgdown.yml index 896ebef..3fd3cf0 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -15,6 +15,7 @@ articles: - nuggets - data-preparation - association-rules + - custom-patterns reference: - title: "Data Preparation" diff --git a/man/dig.Rd b/man/dig.Rd index 7d8b499..14d6529 100644 --- a/man/dig.Rd +++ b/man/dig.Rd @@ -215,6 +215,13 @@ logical data, or the sum of truth degrees for fuzzy data, \eqn{sum = \sum_{r \in R} \mu_C(r)}. \item \code{support}: a numeric scalar value of relative frequency of rows satisfying \eqn{C}, \eqn{supp = sum / |R|}. +\item \code{indices}: an integer vector of row indices of rows satisfying \eqn{C} for +logical data, or the indices of rows with non-zero truth degrees for fuzzy +data, \eqn{indices = \{r \in R : \mu_C(r) > 0\}}. +\item \code{weights}: a numeric vector of truth degrees of \eqn{C} for each row in +\eqn{R}, \eqn{weights[r] = \mu C(r)}. Logical data is treated as a special +case of fuzzy data, where \eqn{\mu_C(r)} is 1 for rows satisfying \eqn{C} +and 0 otherwise. \item \code{pp}, \code{pn}, \code{np}, \code{nn}: a numeric vector of entries of a contingency table for \eqn{C} and \eqn{F}, satisfying the Ruspini condition \eqn{pp + pn + np + nn = |R|}. diff --git a/vignettes/association-rules.Rmd b/vignettes/association-rules.Rmd index ba41606..59efc80 100644 --- a/vignettes/association-rules.Rmd +++ b/vignettes/association-rules.Rmd @@ -147,7 +147,7 @@ to_logical <- function(x) { In many applications, you want to constrain which predicates can appear on each side of the rule. This is done with the `antecedent` and `consequent` arguments, which accept -[tidyselect](https://tidyselect.r-lib.org/articles/syntax.html) expressions. +[tidyselect expression](https://tidyselect.r-lib.org/reference/language.html) expressions. For example, to find rules that predict the `uptake` rate from all other variables: @@ -453,3 +453,6 @@ This vignette demonstrated how to search for association rules using the For further details, consult the function documentation: `dig_associations()`, `add_interest()`, `dig_tautologies()`, `parse_condition()`, `partition()`, `var_names()`. + +For more advanced usage with custom pattern types, see +`vignette("custom-patterns")`. diff --git a/vignettes/custom-patterns.Rmd b/vignettes/custom-patterns.Rmd new file mode 100644 index 0000000..3ed11e7 --- /dev/null +++ b/vignettes/custom-patterns.Rmd @@ -0,0 +1,501 @@ +--- +title: "Custom Pattern Search with dig()" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{Custom Pattern Search with dig()} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + +```{r, include = FALSE} +knitr::opts_chunk$set( + collapse = TRUE, + comment = "#>" +) +``` + +```{r, include = FALSE} +options(tibble.width = Inf) +``` + + + +# Introduction + +`dig()` is the general function behind custom pattern search in the `nuggets` +package. It searches for patterns of a custom type by generating conditions as +elementary conjunctions of predicates and by executing a user-defined callback +function on each generated condition. + +This makes `dig()` the low-level building block behind more specialized +functions such as `dig_associations()` and `dig_correlations()`. Use it when +you want to keep the search over conditions, but define your own evaluation +logic. + +This vignette focuses on how to use the `dig()` function: how to write the +callback function and how to control the search. For preparation of crisp and +fuzzy predicates, see `vignette("data-preparation")`. + +Examples in this vignette require loading the following packages: + +```{r, message = FALSE} +library(nuggets) +library(dplyr) # for data manipulation +``` + + + +# A Small Working Dataset + +`dig()` expects a matrix or data frame whose columns are logical predicates or +numeric fuzzy predicates. We will use `iris` and prepare a small crisp predicate +dataset that is rich enough for the first examples below: + +```{r} +crisp_iris <- iris |> + partition(Species) |> + partition(Sepal.Length:Petal.Width, .method = "crisp", .breaks = 3) + +head(crisp_iris, n = 3) +``` + +The commands above create crisp predicates for the four numeric columns of +`iris`, plus the three species predicates. The preparation step is intentionally +brief here; the dedicated `vignette("data-preparation")` explains `partition()`, +fuzzy predicates, and breakpoints in detail. + + + +# A Simple `dig()` Call + +The `dig()` function generates conditions from the selected predicates in +a recursive manner. It starts with the empty condition and adds one predicate at +a time, up to the specified `max_length`. Meanwhile, it evaluates the generated +condition and tests whether it meets the minimum support requirement. By support +we mean the relative frequency of rows satisfying the condition. If the condition +is frequent enough, `dig()` calls the user-defined callback function with the +generated condition and other information. The callback can then compute any +output you want. The `dig()` function collects those outputs and returns them as +a list. + +The simplest callback function can handle just the generated condition. +In the following example, the callback generates some debug output and +returns the formatted condition: + +```{r} +simple_callback <- function(condition) { + str(condition) + cat("------\n") + + list(condition = format_condition(names(condition))) +} + +simple_result <- dig(x = crisp_iris, + f = simple_callback, + condition = starts_with("Sepal"), + min_length = 0, + max_length = 2, + min_support = 0.2) +``` + +As you can see from the debug output issued by the `str()` call within the +callback, `dig()` enumerates all conditions that can be formed from the selected +predicates (in this case, all predicates starting with "Sepal") and that meet +the minimum support requirement. The callback receives each condition in the +form of a named integer vector, where the names are the predicate names and the +values are the column indices in the original data frame. The callback then +returns a named list with the formatted condition. All callback results are +collected into a list and returned by `dig()`: + +```{r} +str(simple_result) +``` + +As you can see, the result is a list of named lists, one for each condition that +was generated and passed to the callback. You can flatten the result into +a tibble with `dplyr`'s `bind_rows()`: + +```{r} +bind_rows(simple_result) +``` + +Note also the attributes of the result list. They contain information about the +search, such as the search statistics and the arguments that were passed to +`dig()`. You can use this information for debugging or for reproducing the search +later. For example, you can obtain the vector of predicate names that were used +to generate conditions with: + +```{r} +attributes(simple_result)$call_args$condition +``` + +This simple example illustrates the basic workflow: + +1. choose which predicates may form conditions; +2. set the search parameters (length, support, etc.); +3. define a callback function that computes the desired output for each condition; +4. let `dig()` enumerate conditions and collect callback results. + + + +# Condition and Focus + +The simple example above only used the predicates for generating conditions. +In many cases, you will also want to evaluate other predicates within each +generated condition. For example, you may want to know how often each species +occurs within a condition. That's where the *foci* (plural of *focus*) come into +play. + +Condition and focus predicates are selected separately with the `condition` and ` +focus` arguments of `dig()`. These arguments accept +[tidyselect expressions](https://tidyselect.r-lib.org/reference/language.html) +for selecting columns of the input data frame `x`. + + +## Using Foci + +Foci are predicates that are not used to generate conditions, but are evaluated +within each generated condition. You can select foci with the `focus` argument +of `dig()`. The callback function can then receive information about how often +each focus occurs within the generated condition. This is useful, e.g., for +finding conditions that are strongly associated with certain foci. + +For instance, let us define a callback that provides the number of occurences +of each species within each generated condition: + +```{r} +focus_callback <- function(condition, sum, pp) { + str(list(condition = condition, + sum = sum, + species = pp)) + cat("------\n") + + NULL +} + +focus_result <- dig(x = crisp_iris, + f = focus_callback, + condition = starts_with("Sepal"), + focus = starts_with("Species"), + min_length = 2, + max_length = 2, + max_results = 1) +``` + +We have defined a callback that, besides `condition`, also receives `sum` and +`pp`. The `sum` argument provides the number of rows satisfying the generated +condition, and the `pp` argument contains the number of rows that satisfy both +the generated condition and each focus predicate. In this case, the focus +predicates are the species predicates, so `pp` tells us how many rows of each +species satisfy the generated condition. + +Also note that we are using a neat trick that is useful during development of +the callback: we set `max_results = 1` to stop the search after the first +condition that meets the criteria. This allows us to see the output of the +callback without waiting for the entire search to complete, which can be +time-consuming for large datasets or complex conditions. + +So far, our callback only prints the information to the console and returns +`NULL`. Once we understand the structure of the data we receive, we can modify +the callback to return a list of patterns, where each pattern contains the +formatted condition, a single species, the count of data rows satisfying the +condition, and the count of the species within that condition: + +```{r} +focus_callback <- function(condition, sum, pp) { + species_names <- names(pp) + species_counts <- as.integer(pp) + + lapply(seq_along(species_names), function(i) { + list(condition = format_condition(names(condition)), + species = species_names[i], + condition_count = sum, + species_count = species_counts[i]) + }) +} + +focus_result <- dig(x = crisp_iris, + f = focus_callback, + condition = starts_with("Sepal"), + focus = starts_with("Species"), + min_length = 0, + max_length = 2) +``` + +The result of `dig()` is a list of lists, where each inner list corresponds to +a species within a generated condition. We use `unlist(recursive = FALSE)` to +flatten the list of lists into a single list of patterns, and then `bind_rows()` +to convert it into a tibble for easier viewing: + +```{r} +focus_result |> + unlist(recursive = FALSE) |> + bind_rows() |> + head(n = 6) +``` + + +## Filtering Foci + +As discussed in the previous section, `dig()` evaluates the focus predicates +within each generated condition. You may or may not want to keep all foci for +each condition. Some patterns may require all foci to be evaluated every time, +while others may only require a subset of foci to be considered that are +sufficiently frequent within the condition. Therefore, `dig()` provides several +arguments to filter foci based on their support: + +- `min_focus_support`: minimum support of a focus within a condition. I.e., the + relative frequency of rows satisfying both the condition and the focus must be + at least this value for the focus to be kept. Foci with support below this + threshold are filtered out. +- `min_conditional_focus_support`: minimum conditional support of a focus within + a condition. I.e., the relative frequency of rows satisfying both the condition + and the focus, divided by the number of rows satisfying the condition, must be + at least this value for the focus to be kept. Foci with conditional support + below this threshold are filtered out. + +The focus filtering may result in some conditions having no remaining foci. +If you want to skip the callback for such conditions, set `filter_empty_foci = TRUE`. +Otherwise, the callback will be called with empty focus information. Filtering +empty foci also improves performance, because it also stops early the evaluation +of longer conditions that would not have any remaining foci anyway. + +The following example shows how this can be used to emulate association-rule +search with a custom callback. Association rules are implications of the form +"if condition then focus". Condition is named the *antecedent* and focus is named +the *consequent*. The callback computes the confidence of each +antecedent-consequent pair, filters the pairs based on minimum support and +confidence, and returns a list of rules: + +```{r} +min_support <- 0.1 +min_confidence <- 0.8 + +rule_callback <- function(condition, pp, support) { + conf <- pp / support / nrow(crisp_iris) + sel <- !is.na(conf) & conf >= min_confidence & !is.na(pp) & pp >= min_support + conf <- conf[sel] + supp <- pp[sel] / nrow(crisp_iris) + + lapply(seq_along(conf), function(i) { + list(antecedent = format_condition(names(condition)), + consequent = names(conf)[[i]], + antecedent_support = support, + rule_support = supp[[i]], + confidence = conf[[i]] + ) + }) +} + +rule_result <- dig(x = crisp_iris, + f = rule_callback, + condition = !starts_with("Species"), + focus = starts_with("Species"), + min_length = 1, + min_support = min_support, + min_focus_support = min_support, + min_conditional_focus_support = min_confidence, + filter_empty_foci = TRUE) |> + unlist(recursive = FALSE) |> + bind_rows() |> + arrange(desc(confidence)) + +head(rule_result, n = 6) +``` + +This is a useful illustration of focus filtering, but association rules already +have a dedicated implementation: `dig_associations()` searches for them more +efficiently. For that purpose, prefer `dig_associations()` and see +`vignette("association-rules")`. + + + + +# What the Callback Function Can Receive + +As seen in the previous section, the callback function `f` may obtain not only +the generated condition, but also other information. The amount of received +information is controlled by declaring the arguments of the callback function. `dig()` +inspects the callback function argument names and computes only the requested +values. This is important for performance, because some values are expensive to +compute and may not be needed for every search. + +The callback function may declare any subset of the following arguments: + +- `condition`: named integer vector of column indices representing the generated + condition. +- `sum`: number of rows satisfying the condition for logical data, or the sum + of truth degrees for fuzzy data. +- `support`: relative frequency of the condition, i.e., `sum / nrow(x)`. +- `indices`: row indices of the original dataset `x` satisfying the condition + in crisp searches or the indices of rows with non-zero truth degrees in fuzzy + searches. +- `weights`: per-row truth degrees of the condition in dataset `x`; logical + (crisp) data is treated as 0/1 weights. +- `pp`, `pn`, `np`, `nn`: contingency-table entries for foci. The *i*-th* entry + of each vector corresponds to the *i*-th focus predicate. The + entries are defined as follows: + - `pp`: sum of truth degrees of rows satisfying both the condition and the focus + (**p**ositive condition, **p**ositive focus), + - `pn`: sum of truth degrees of rows satisfying the condition but not the focus, + (**p**ositive condition, **n**egative focus), + - `np`: sum of truth degrees of rows satisfying the focus but not the condition, + (**n**egative condition, **p**ositive focus), + - `nn`: sum of truth degrees of rows satisfying neither (**n**egative condition, + **n**egative focus). + +In practice: + +- use `condition` when you need condition predicate names, +- use `support` or `sum` for condition-level filtering or ranking, +- use `indices` when you want to compute something on the original rows, +- use `weights` for custom fuzzy summaries, +- use `pp`, `pn`, `np`, and `nn` when your pattern depends on foci and their + frequency within the condition. + +Note: only declare the arguments you need. For example, if you don't need foci, +don't declare `pp`, `pn`, `np`, or `nn`. This will save computation time, +especially for large datasets or complex conditions. The most expensive values +to compute are `indices` and `weights`. They require scanning the entire dataset +for each generated condition. So avoid them if not needed. + + + +# Advanced Examples + + +## Example: Fixed-Variable Correlations + +`dig_correlations()` searches over both generated conditions and combinations of +numeric variables. A simpler custom variant can be built directly with `dig()` +when the two variables are fixed in advance and only the condition should vary. + +Here we search for conditions under which `Sepal.Length` and `Petal.Length` +correlate strongly. The callback receives `indices`, uses them to select the +corresponding rows from the original `iris` data, and runs `cor.test()` on that +sub-data: + +```{r} +correlation_callback <- function(condition, support, indices) { + if (length(indices) < 10) { + return(NULL) + } + fit <- cor.test(iris$Sepal.Length[indices], + iris$Petal.Length[indices], + method = "pearson") + + list(condition = format_condition(names(condition)), + support = support, + correlation = unname(fit$estimate), + p_value = fit$p.value, + n = length(indices)) +} + +correlation_result <- dig(x = crisp_iris, + f = correlation_callback, + condition = everything(), + min_length = 1, + max_length = 2, + min_support = 0.1) |> + bind_rows() |> + arrange(desc(abs(correlation))) + +head(correlation_result, n = 6) +``` + +This example follows the same idea as `dig_correlations()`: + +- generate conditions, +- evaluate a statistic on the sub-data induced by each condition, +- return one row per successful evaluation. + +The difference is that `dig()` leaves the statistic entirely in your hands. +That is useful when you want to fix the variables, apply a custom test, return +additional diagnostics, or combine several criteria in one callback. + + + +## Example: Handling Fuzzy Data + +For fuzzy searches, conditions are no longer simply satisfied or not satisfied. +Instead, each row has a truth degree in the interval $[0,1]$. In that setting, +`indices` and `weights` play different roles: + +- `indices` tell you which rows have a non-zero truth degree for the condition, +- `weights` tell you on scale $[0, 1]$ how strongly each row satisfies the condition. + +The following example prepares fuzzy predicates from `iris` and then compares +an unweighted summary based on `indices` with a weighted summary based on +`weights`: + +```{r} +fuzzy_iris <- iris |> + partition(Species) |> + partition(Sepal.Length:Petal.Width, .method = "triangle", .breaks = 3) + +head(fuzzy_iris, n = 3) + +fuzzy_callback <- function(condition, indices, weights) { + if (length(indices) < 20) { + return(NULL) + } + + list(condition = format_condition(names(condition)), + nonzero_rows = sum(indices), + weighted_support = sum(weights) / nrow(fuzzy_iris), + mean_petal_length_by_indices = mean(iris$Petal.Length[indices]), + mean_petal_length_by_weights = weighted.mean(iris$Petal.Length, weights)) +} + +fuzzy_result <- dig(x = fuzzy_iris, + f = fuzzy_callback, + condition = starts_with("Sepal"), + min_length = 1, + max_length = 1, + min_support = 0.2) |> + bind_rows() + +fuzzy_result +``` + +The unweighted mean based on `indices` treats all rows with non-zero membership +equally. The weighted mean based on `weights` respects the fuzzy truth degrees, +so rows that satisfy the condition more strongly contribute more. This is the +main practical difference: `indices` identify the relevant rows, while +`weights` quantify the strength of their membership. + + + +# Practical Notes + +- Condition predicates are used to generate conditions, while focus predicates + are evaluated within each generated condition. Use `condition` and `focus` + arguments to select them separately. +- The result of `dig()` is a list. When each callback returns one named list, + `bind_rows()` is a convenient way to flatten it into a tibble. +- If the callback returns multiple patterns per condition, return a list of + named lists and flatten the result afterwards. +- Use only the arguments you need in the callback function. This saves + computation time, especially for large datasets or complex conditions. +- For fuzzy searches, use `weights` when truth degrees matter, and `indices` + when you only need the rows with non-zero membership. + + + +# Summary + +`dig()` is the most flexible search interface in `nuggets`. It lets you: + +1. generate conditions from selected predicates, +2. optionally evaluate focus predicates within each condition, +3. receive only the callback inputs you need, +4. define your own pattern logic, statistics, and output format. + +Use `dig()` when the built-in search functions are close to what you need, but +not exact. For related material, see: + +- `vignette("data-preparation")` for creating crisp and fuzzy predicates from + raw data, +- `vignette("association-rules")` for searching for association rules with + `dig_associations()`, +- `vignette("nuggets")` for an overview of the package and its main workflows. diff --git a/vignettes/data-preparation.Rmd b/vignettes/data-preparation.Rmd index ca06914..0414659 100644 --- a/vignettes/data-preparation.Rmd +++ b/vignettes/data-preparation.Rmd @@ -84,7 +84,7 @@ is `TRUE` and `x=F` for rows where `x` is `FALSE`. Missing values are excluded from both predicates. The `partition()` function requires the dataset as its first argument and a -*tidyselect* selection expression to select the columns to be transformed. +[tidyselect expression](https://tidyselect.r-lib.org/reference/language.html) to select the columns to be transformed. For example, the `vs` column in `mtcars_example` is a logical column indicating the engine type (V-shaped or straight): @@ -708,7 +708,7 @@ remove_almost_constant(d, .threshold = 0.5, .na_rm = FALSE) remove_almost_constant(d, .threshold = 0.5, .na_rm = TRUE) ``` -You can also restrict the check to a subset of columns using tidyselect syntax: +You can also restrict the check to a subset of columns using [tidyselect expression](https://tidyselect.r-lib.org/reference/language.html) syntax: ```{r} # Only check columns a1 through b2 @@ -897,5 +897,9 @@ package: With these tools, you can effectively prepare your data for pattern discovery using the various `dig_*()` functions provided by the `nuggets` package. For -information on pattern discovery itself, see the main "Getting Started" vignette -and the function documentation. +information on pattern discovery itself, see: + +- `vignette("nuggets")` for an overview of the package and its main workflows, +- `vignette("association-rules")` for a specialized pattern family based on the + `dig_associations()` function, +- `vignette("custom-patterns")` for defining custom pattern types with `dig()`. diff --git a/vignettes/nuggets.Rmd b/vignettes/nuggets.Rmd index c08d4df..bbb729f 100644 --- a/vignettes/nuggets.Rmd +++ b/vignettes/nuggets.Rmd @@ -276,7 +276,7 @@ The `dig_associations()` function searches for association rules. Its main arguments are: - `x`: the data matrix or data frame (logical or numeric); -- `antecedent`, `consequent`: tidyselect expressions selecting columns for each +- `antecedent`, `consequent`: [tidyselect expressions](https://tidyselect.r-lib.org/reference/language.html) selecting columns for each side of the rule; - `disjoint`: a vector defining mutually exclusive predicates; - rule filtering thresholds such as `min_support`, `min_confidence`, @@ -694,6 +694,8 @@ This vignette has introduced the core functionality of the `nuggets` package for ## Next Steps - Explore the [Data Preparation vignette](data-preparation.html) for advanced preprocessing techniques +- Explore the [Association Rules vignette](association-rules.html) for a specialized pattern family +- Explore the [Custom Patterns vignette](custom-patterns.html) for defining your own pattern types with `dig()` - Review function documentation (e.g., `?dig_associations`) for detailed parameter descriptions - Experiment with your own datasets to discover meaningful patterns - Use interactive exploration (`explore()`) to gain insights into discovered patterns