standard_scale

StandardScaleEstimator

StandardScaleEstimator(
    inputCol=None,
    outputCol=None,
    inputDtype=None,
    outputDtype=None,
    layerName=None,
    maskValue=None,
    sampleFraction=None,
)

Bases: BaseEstimator, SampleFractionParams, SingleInputSingleOutputParams, MaskValueParams

Standard scaler estimator for use in Spark pipelines. This estimator is used to calculate the mean and standard deviation of the input feature column. When fit is called it returns a StandardScaleTransformer which can be used to standardize/transform additional features.

WARNING: If the input is an array, we assume that the array has a constant shape across all rows.

Initializes a StandardScaleEstimator estimator. Sets all parameters to given inputs.

Parameters:

Name	Type	Description	Default
inputCol	Optional[str]	Input column name to standardize.	None
outputCol	Optional[str]	Output column name.	None
inputDtype	Optional[str]	Input data type to cast input column to before transforming.	None
outputDtype	Optional[str]	Output data type to cast the output column to after transforming.	None
layerName	Optional[str]	Name of the layer. Used as the name of the tensorflow layer in the keras model. If not set, we use the uid of the Spark transformer.	None
sampleFraction	Optional[float]	Fraction of data to sample for statistics estimation (exclusive 0.0-1.0). Default None (no sampling).	None

Returns:

Type	Description
None	None - class instantiated.

Source code in src/kamae/spark/estimators/standard_scale.py

@keyword_only
def __init__(
    self,
    inputCol: Optional[str] = None,
    outputCol: Optional[str] = None,
    inputDtype: Optional[str] = None,
    outputDtype: Optional[str] = None,
    layerName: Optional[str] = None,
    maskValue: Optional[float] = None,
    sampleFraction: Optional[float] = None,
) -> None:
    """
    Initializes a StandardScaleEstimator estimator.
    Sets all parameters to given inputs.

    :param inputCol: Input column name to standardize.
    :param outputCol: Output column name.
    :param inputDtype: Input data type to cast input column to before
    transforming.
    :param outputDtype: Output data type to cast the output column to after
    transforming.
    :param layerName: Name of the layer. Used as the name of the tensorflow layer
     in the keras model. If not set, we use the uid of the Spark transformer.
    :param sampleFraction: Fraction of data to sample for statistics
     estimation (exclusive 0.0-1.0). Default None (no sampling).
    :returns: None - class instantiated.
    """
    super().__init__()
    self._setDefault(maskValue=None, sampleFraction=None)
    kwargs = self._input_kwargs
    self.setParams(**kwargs)

compatible_dtypes property

compatible_dtypes

List of compatible data types for the layer. If the computation can be performed on any data type, return None.

Returns:

Type	Description
Optional[List[DataType]]	List of compatible data types for the layer.