102
Chapter 6
Screening or Removing Fields
Toscreen out fiel ds with too many missing values, you have several options:
Youcan use a Data Audit node to filter fields based on qu ality.
Youcan use a Feature Selection node to screen out fields with more than a s pecified percentage
of missing values and to rank fields based on importance relative to a specified target.
Instead of removing the fields, you can use a Type node to set the field role to None. This will
keep the fields in the data set but exclude them from the modeling processe s.
Imputing or Filling Missing Values
In cases where there are only a few missing values, it may be useful to insert values to replace
the blanks. You can do this from the Data Audit report, which allows you to specify options for
specificfields as appropriate and then generate a SuperNode that imputes v alues using a number
of methods. This is the most flexible method, and it al so allows you to specify handling for large
numbers of fields in a single node.
The following methods are available for imputing missing values:
Fixed. Substitutes a fixed value (either the field mean, midpoint of the range, or a constant that
you specify).
Random. Substitutes a random value based on a normal or uniform distributi on.
Expression. Allows you to specify a custom expression. For example, you coul dr eplace values
with a global variable created by the Set Globals node.
Algorithm. Substitutes a value predicted by a model based on the C&RT algorithm. For each field
imputed using this method, there will be a separate C&RT model, along with a Filler node that
replaces blanks and nulls with the value predicted by the model. A Filter node is then used to
remove the prediction fields generated by the model.
Alternatively, to coerce values for specificfields, you can use a Type node to ensure that the
field types cover only legal values and then set the Check column to Coerce f ort he fields whose
blank values need replacing.
CLEM Functions for Missing Values
There are several functions used to handle missing values. The following function sa re often used
in Select and Filler nodes to discard or fill missing values:
count_nulls(LIST)
@BLANK(FIELD)
@NULL(FIELD)
undef