Package

com.salesforce.op.stages.impl

feature

Permalink

package feature

Visibility
  1. Public
  2. All

Type Members

  1. class AbsoluteValueTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Absolute value transformer

    Absolute value transformer

    I

    input feature type

  2. class AddTransformer[I1 <: OPNumeric[_], I2 <: OPNumeric[_]] extends BinaryTransformer[I1, I2, Real]

    Permalink

    Plus function truth table (Real as example):

    Plus function truth table (Real as example):

    Real.empty + Real.empty = Real.empty Real.empty + Real(x) = Real(x) Real(x) + Real.empty = Real(x) Real(x) + Real(y) = Real(x + y)

  3. class AliasTransformer[I <: FeatureType] extends UnaryTransformer[I, I]

    Permalink

    No-op (identity) alias feature transformer allowing renaming features without applying a transformation on values.

    No-op (identity) alias feature transformer allowing renaming features without applying a transformation on values.

    I

    feature type

  4. class BinaryMapVectorizer[T <: OPMap[Boolean]] extends OPMapVectorizer[Boolean, T]

    Permalink

    Class for vectorizing BinaryMap features.

    Class for vectorizing BinaryMap features. Fills missing keys with args.defaultValue, which does not depend on the key, so getFillByKey returns an empty sequence.

    T

    input feature type to vectorize into an OPVector

  5. final class BinaryMapVectorizerModel[T <: OPMap[Boolean]] extends OPMapVectorizerModel[Boolean, T]

    Permalink
  6. class BinaryVectorizer extends SequenceTransformer[Binary, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Vectorizes Binary inputs where each input is transformed into 2 vector elements where the first element is [1 -> true] or [0 -> false] and the second element is [1 -> filled value] or [0 -> original value].

    Vectorizes Binary inputs where each input is transformed into 2 vector elements where the first element is [1 -> true] or [0 -> false] and the second element is [1 -> filled value] or [0 -> original value]. The vector representation for each input is concatenated into a final vector representation.

    Example:

    Data: Seq[(Binary, Binary)] = ((Some(false), None)) => f1, f2 new BinaryVectorizer().setInput(f1, f2).setFillValue(10)

    will produce Array(0.0, 0.0, 10.0, 1.0)

  7. class CeilTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Integral]

    Permalink

    Ceil transformer

    Ceil transformer

    I

    input feature type

  8. trait CleanTextFun extends AnyRef

    Permalink
  9. trait CleanTextMapFun extends CleanTextFun

    Permalink
  10. sealed trait DateListPivot extends EnumEntry with Serializable

    Permalink
  11. class DateListVectorizer[T <: OPList[Long]] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of DateLists features into a vector feature.

    Converts a sequence of DateLists features into a vector feature. Can choose how to pivot the features

  12. class DateMapToUnitCircleVectorizer[T <: DateMap] extends SequenceEstimator[T, OPVector] with DateToUnitCircleParams with MapVectorizerFuns[Long, T]

    Permalink

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    parameter timePeriod The time period to extract from the timestamp enum from: DayOfMonth, DayOfWeek, DayOfYear, HourOfDay, MonthOfYear, WeekOfMonth, WeekOfYear

    We extract the timePeriod from the timestamp and map this onto the unit circle containing the number of time periods equally spaced. For example, when timePeriod = HourOfDay, the timestamp 01/01/2018 6:37 maps to the point on the circle with angle radians = 2*math.Pi*6/24 We return the cartesian coordinates of this point: (math.cos(radians), math.sin(radians))

    The first time period always has angle 0.

    Note: We use the ISO week date format https://en.wikipedia.org/wiki/ISO_week_date#First_week Monday is the first day of the week & the first week of the year is the week wit the first Monday after Jan 1.

  13. final class DateMapToUnitCircleVectorizerModel[T <: DateMap] extends SequenceModel[T, OPVector] with CleanTextMapFun

    Permalink

    Model for DateMapToUnitCircleVectorizer

    Model for DateMapToUnitCircleVectorizer

    T

    DateMap type

  14. class DateMapVectorizer[T <: OPMap[Long]] extends OPMapVectorizer[Long, T]

    Permalink

    Class for vectorizing DateMap features.

    Class for vectorizing DateMap features. Fills missing keys with args.defaultValue, which does not depend on the key, so getFillByKey returns an empty sequence.

    T

    input feature type to vectorize into an OPVector

  15. final class DateMapVectorizerModel[T <: OPMap[Long]] extends OPMapVectorizerModel[Long, T]

    Permalink
  16. trait DateToUnitCircleParams extends Params

    Permalink
  17. class DateToUnitCircleTransformer[T <: Date] extends SequenceTransformer[T, OPVector] with DateToUnitCircleParams

    Permalink

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    parameter timePeriod The time period to extract from the timestamp enum from: DayOfMonth, DayOfWeek, DayOfYear, HourOfDay, MonthOfYear, WeekOfMonth, WeekOfYear

    We extract the timePeriod from the timestamp and map this onto the unit circle containing the number of time periods equally spaced. For example, when timePeriod = HourOfDay, the timestamp 01/01/2018 6:37 maps to the point on the circle with angle radians = 2*math.Pi*6/24 We return the cartesian coordinates of this point: (math.cos(radians), math.sin(radians))

    The first time period always has angle 0.

    Note: We use the ISO week date format https://en.wikipedia.org/wiki/ISO_week_date#First_week Monday is the first day of the week & the first week of the year is the week wit the first Monday after Jan 1.

  18. class DecisionTreeNumericBucketizer[N, I2 <: OPNumeric[N]] extends BinaryEstimator[RealNN, I2, OPVector] with DecisionTreeNumericBucketizerParams with VectorizerDefaults with TrackInvalidParam with TrackNullsParam with NumericBucketizerMetadata with AllowLabelAsInput[OPVector]

    Permalink

    Smart bucketizer for numeric values based on a Decision Tree classifier.

    Smart bucketizer for numeric values based on a Decision Tree classifier.

    N

    numeric feature type value

    I2

    numeric feature type

  19. final class DecisionTreeNumericBucketizerModel[I2 <: OPNumeric[_]] extends BinaryModel[RealNN, I2, OPVector] with AllowLabelAsInput[OPVector]

    Permalink
  20. trait DecisionTreeNumericBucketizerParams extends AnyRef

    Permalink
  21. class DecisionTreeNumericMapBucketizer[N, I2 <: OPMap[N]] extends BinaryEstimator[RealNN, I2, OPVector] with DecisionTreeNumericBucketizerParams with VectorizerDefaults with TrackInvalidParam with TrackNullsParam with NumericBucketizerMetadata with MapPivotParams with CleanTextMapFun with AllowLabelAsInput[OPVector]

    Permalink

    Smart bucketizer for numeric map values based on a Decision Tree classifier.

    Smart bucketizer for numeric map values based on a Decision Tree classifier.

    N

    numeric feature type value

    I2

    numeric map feature type

  22. final class DecisionTreeNumericMapBucketizerModel[I2 <: OPMap[_]] extends BinaryModel[RealNN, I2, OPVector] with CleanTextMapFun with AllowLabelAsInput[OPVector]

    Permalink
  23. final class DescalerTransformer[I1 <: Real, I2 <: Real, O <: Real] extends BinaryTransformer[I1, I2, O]

    Permalink

    A transformer that takes as inputs a feature to descale and (potentially different) scaled feature which contains the metadata for reconstructing the inverse scaling function.

    A transformer that takes as inputs a feature to descale and (potentially different) scaled feature which contains the metadata for reconstructing the inverse scaling function. Transforms the 2nd input feature by applying the inverse of the scaling function found in the metadata - 1st input feature the feature to descale - 2nd input feature the scaled feature containing metadata for constructing the scaling used to make this column

    I1

    feature type for first input

    I2

    feature type for the second input

    O

    output feature type

  24. class DivideTransformer[I1 <: OPNumeric[_], I2 <: OPNumeric[_]] extends BinaryTransformer[I1, I2, Real]

    Permalink

    Divide function truth table (Real as example):

    Divide function truth table (Real as example):

    Real.empty / Real.empty = Real.empty Real.empty / Real(x) = Real.empty Real(x) / Real.empty = Real.empty Real(x) / Real(y) = Real(x * y) filter ("is not NaN or Infinity")

  25. class DropIndicesByTransformer extends UnaryTransformer[OPVector, OPVector]

    Permalink

    Allows columns to be dropped from a feature vector based on properties of the metadata about what is contained in each column (will work only on vectors) created with OpVectorMetadata

  26. case class EmptyArgs() extends ScalingArgs with Product with Serializable

    Permalink

    Case class for Scaling families that take no parameters

  27. class ExpTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Exp transformer: returns Euler's number e raised to the power of feature value

    Exp transformer: returns Euler's number e raised to the power of feature value

    I

    input feature type

  28. class FillMissingWithMean[N, I <: OPNumeric[N]] extends UnaryEstimator[I, RealNN]

    Permalink

    Fill missing values with mean for any numeric feature

  29. final class FillMissingWithMeanModel[I <: OPNumeric[_]] extends UnaryModel[I, RealNN]

    Permalink
  30. class FilterMap[I <: OPMap[_]] extends UnaryTransformer[I, I] with MapPivotParams with TextParams with CleanTextMapFun

    Permalink

    Filters maps by keys provided in a whiteList or blackList

    Filters maps by keys provided in a whiteList or blackList

    I

    input feature type

  31. class FloorTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Integral]

    Permalink

    Floor transformer

    Floor transformer

    I

    input feature type

  32. class GeolocationMapVectorizer extends SequenceEstimator[GeolocationMap, OPVector] with MapVectorizerFuns[Seq[Double], GeolocationMap] with TrackNullsParam

    Permalink
  33. final class GeolocationMapVectorizerModel extends SequenceModel[GeolocationMap, OPVector] with CleanTextMapFun

    Permalink
  34. class GeolocationVectorizer extends SequenceEstimator[Geolocation, OPVector] with VectorizerDefaults with TrackNullsParam with GeolocationFunctions

    Permalink

    Converts a sequence of Geolocation features into a vector feature.

    Converts a sequence of Geolocation features into a vector feature. Can choose to fill null values with the mean or a constant

  35. final class GeolocationVectorizerModel extends SequenceModel[Geolocation, OPVector] with VectorizerDefaults

    Permalink
  36. sealed trait HashAlgorithm extends EnumEntry with Serializable

    Permalink

    Hashing Algorithms

  37. sealed trait HashSpaceStrategy extends EnumEntry with Serializable

    Permalink

    Hash space strategy

  38. case class HashingFunctionParams(hashWithIndex: Boolean, prependFeatureName: Boolean, numFeatures: Int, numInputs: Int, maxNumOfFeatures: Int, binaryFreq: Boolean, hashAlgorithm: HashAlgorithm, hashSpaceStrategy: HashSpaceStrategy) extends Product with Serializable

    Permalink

    Hashing Parameters

    Hashing Parameters

    hashWithIndex

    if true, include indices when hashing a feature that has them (OPLists or OPVectors)

    prependFeatureName

    if true, prepends a input feature name to each token of that feature

    numFeatures

    number of features (hashes) to generate

    numInputs

    number of inputs

    maxNumOfFeatures

    max number of features (hashes)

    binaryFreq

    if true, term frequency vector will be binary such that non-zero term counts will be set to 1.0

    hashAlgorithm

    hash algorithm to use

    hashSpaceStrategy

    strategy to determine whether to use shared hash space for all included features

  39. sealed abstract class Inclusion extends EnumEntry with Serializable

    Permalink
  40. sealed trait IndexToStringHandleInvalid extends EnumEntry with Serializable

    Permalink
  41. class IntegralMapVectorizer[T <: OPMap[Long]] extends OPMapVectorizer[Long, T]

    Permalink

    Class for vectorizing IntegralMap features.

    Class for vectorizing IntegralMap features. Fills missing keys with the mode for that key.

    T

    input feature type to vectorize into an OPVector

  42. final class IntegralMapVectorizerModel[T <: OPMap[Long]] extends OPMapVectorizerModel[Long, T]

    Permalink
  43. class IntegralVectorizer[T <: Integral] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of Integral features into a vector feature.

    Converts a sequence of Integral features into a vector feature. Can choose to fill null values with the mean or a constant

  44. final class IntegralVectorizerModel[T <: Integral] extends SequenceModel[T, OPVector] with VectorizerDefaults

    Permalink
  45. class IsValidPhoneDefaultCountry extends UnaryTransformer[Phone, Binary] with PhoneParams

    Permalink

    Transformer to determine if a phone numbers is valid when no country code is available.

    Transformer to determine if a phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary feature true if phone is valid false if invalid and none if phone number is none

  46. class IsValidPhoneMapDefaultCountry extends UnaryTransformer[PhoneMap, BinaryMap] with PhoneParams

    Permalink

    Transformer to determine if a map of phone numbers is valid when no country code is available.

    Transformer to determine if a map of phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary map feature true if phone is valid false if invalid and none if phone number is none

  47. class IsValidPhoneNumber extends BinaryTransformer[Phone, Text, Binary] with PhoneCountryParams

    Permalink

    Determine whether a phone number is valid given the country's regional code.

    Determine whether a phone number is valid given the country's regional code. By default the regional code will be checked against those provided in Google's PhoneNumber library. If the input regional code is not found, the default locale will be used for validation.

    If the User provided a Country name to code mapping, the phone number can only be validated against the input mapping. This transformer will first match on regional code, failing that, it will select the country with the closest Q-Distance.

    All phone numbers with less than 2 characters will be categorized as invalid

    All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary feature true if phone is valid false if invalid and none if phone number is none

  48. class JaccardSimilarity extends BinaryTransformer[MultiPickList, MultiPickList, RealNN]

    Permalink

    Calculates the Jaccard Similarity between two sets.

    Calculates the Jaccard Similarity between two sets. If both inputs are empty, Jaccard Similarity is defined as 1.0

  49. class LangDetector[T <: Text] extends UnaryTransformer[T, RealMap]

    Permalink

    Transformer that detects the language of the text

  50. trait LanguageDetectionParams extends Params

    Permalink
  51. case class LinearScaler(args: LinearScalerArgs) extends Scaler with Product with Serializable

    Permalink

    A case class representing a linear scaling function

    A case class representing a linear scaling function

    args

    case class containing the slope and intercept of the scaling function

  52. case class LinearScalerArgs(slope: Double, intercept: Double) extends ScalingArgs with Product with Serializable

    Permalink

    Parameters need to uniquely define a linear scaling function

    Parameters need to uniquely define a linear scaling function

    slope

    the slope of the linear scaler

    intercept

    the x axis intercept of the linear scaler

  53. case class LogScaler() extends Scaler with Product with Serializable

    Permalink

    A case class representing a logarithmic scaling function

  54. class LogTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Log base N transformer

    Log base N transformer

    I

    input feature type

  55. trait MapPivotParams extends Params

    Permalink
  56. trait MapStringPivotHelper extends SaveOthersParams

    Permalink
  57. trait MapVectorizerFuns[A, T <: OPMap[A]] extends VectorizerDefaults with MapPivotParams with CleanTextMapFun

    Permalink
  58. trait MaxCardinalityParams extends Params

    Permalink
  59. trait MaxPctCardinalityParams extends Params

    Permalink
  60. class MimeTypeDetector extends UnaryTransformer[Base64, Text] with MimeTypeDetectorParams

    Permalink

    Detects MIME type for Base64 encoded binary data.

  61. class MimeTypeMapDetector extends UnaryTransformer[Base64Map, PickListMap] with MimeTypeDetectorParams

    Permalink

    Detects MIME type for Base64Map encoded binary data.

  62. trait MinSupportParam extends Params

    Permalink
  63. class MultiPickListMapVectorizer[T <: OPMap[Set[String]]] extends SequenceEstimator[T, OPVector] with PivotParams with MapPivotParams with TextParams with MapStringPivotHelper with CleanTextMapFun with MinSupportParam with TrackNullsParam with MaxPctCardinalityParams with MaxPctCardinalityFun

    Permalink

    Converts a sequence of KeyMultiPickList features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features).

    Converts a sequence of KeyMultiPickList features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features). Each key found will also generate an other column which will capture values that do not make the cut or where not seen in training. Note that any keys not seen in training will be ignored.

  64. final class MultiPickListMapVectorizerModel[T <: OPMap[Set[String]]] extends SequenceModel[T, OPVector] with CleanTextMapFun

    Permalink
  65. class MultiplyTransformer[I1 <: OPNumeric[_], I2 <: OPNumeric[_]] extends BinaryTransformer[I1, I2, Real]

    Permalink

    Multiply function truth table (Real as example):

    Multiply function truth table (Real as example):

    Real.empty * Real.empty = Real.empty Real.empty * Real(x) = Real.empty Real(x) * Real.empty = Real.empty Real(x) * Real(y) = Real(x * y) filter ("is not NaN or Infinity")

  66. class NameEntityRecognizer[T <: Text] extends UnaryTransformer[T, MultiPickListMap] with LanguageDetectionParams

    Permalink

    Name Entity NameEntityType text recognizer.

    Name Entity NameEntityType text recognizer.

    Note: when providing your own the analyzer/splitter/tagger make sure they can work together, for instance OpenNLP models require their own analyzers to be provided when tokenizing. The returned feature type is a MultiPickListMap which contains sets of entities for all the tokens

    T

    text feature type

  67. class NumericBucketizer[I1 <: OPNumeric[_]] extends UnaryTransformer[I1, OPVector] with VectorizerDefaults with NumericBucketizerParams with NumericBucketizerMetadata

    Permalink

    Numeric Bucketizer

    Numeric Bucketizer

    I1

    numeric feature type

  68. trait NumericBucketizerParams extends TrackInvalidParam with TrackNullsParam

    Permalink
  69. sealed trait NumericMapDefaultParam extends Params

    Permalink
  70. class OPCollectionHashingVectorizer[T <: OPCollection] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with PivotParams with CleanTextFun with HashingFun with HashingVectorizerParams

    Permalink

    Generic hashing vectorizer to convert features of type OPCollection into Vectors

    Generic hashing vectorizer to convert features of type OPCollection into Vectors

    In more details: It tries to hash entries in the collection using the specified hashing algorithm to build a single vector. If the desired number of features (= hash space size) for all features combined is larger than Integer.Max (the maximal index for a vector), then all the features use the same hash space. There are also options for the user to hash indices with collections where that makes sense (OPLists and OPVectors), and to force a shared hash space, even if the number of feature is not high enough to require it.

  71. sealed abstract class OPCollectionTransformer[I <: FeatureType, O <: FeatureType, ICol <: OPCollection, OCol <: OPCollection] extends UnaryTransformer[ICol, OCol]

    Permalink

    Abstract base class for a set of transformer wrappers that allow unary transformers between non-collection types to be used on collection types.

    Abstract base class for a set of transformer wrappers that allow unary transformers between non-collection types to be used on collection types. For example, we can use a UnaryLambdaTransformer[Email, Integer] on a map's values, creating a UnaryLambdaTransformer[EmailMap, IntegralMap]. This base class will be inherited by concrete classes for OPMaps, OPList, and OPSets (in order to enforce not allowing these collection types to be transformed into each other, eg. no MultiPickList to RealMap transformations).

    The OP type hierarchy does not allow direct type checking of such transformer wrappers (eg. Real#Value is Option[Double] and RealMap#Value is Map[String, Double], so there's no way to enforce that a RealMap can only hold what is contained in a Real) since the types themselves are not created with typetags for performance reasons. However, we can still enforce that operations like building a UnaryLambdaTransformer[RealMap, StringMap] from a UnaryLambdaTransformer[Real, Integer] is not possible by using the Spark types in validateTypes.

    I

    input feature type for supplied non-collection transformer

    O

    output feature type for supplied non-collection transformer

    ICol

    input feature type for desired collection transformer

    OCol

    output feature type for desired collection transformer

  72. abstract class OPMapVectorizer[A, T <: OPMap[A]] extends SequenceEstimator[T, OPVector] with MapVectorizerFuns[Double, RealMap] with NumericMapDefaultParam with TrackNullsParam

    Permalink

    Base class for vectorizing OPMap[A] features.

    Base class for vectorizing OPMap[A] features. Individual vectorizers for different feature types need to implement the getFillByKey function (which calculates any fill values that differ by key - means, modes, etc.) and the makeModel function (which specifies which type of model will be returned).

    A

    value type for underlying map

    T

    input feature type to vectorize into an OPVector

  73. sealed abstract class OPMapVectorizerModel[A, I <: OPMap[A]] extends SequenceModel[I, OPVector] with CleanTextMapFun

    Permalink
  74. sealed case class OPMapVectorizerModelArgs(allKeys: Seq[Seq[String]], fillByKey: Seq[Map[String, Double]], shouldCleanKeys: Boolean, shouldCleanValues: Boolean, defaultValue: Double, trackNulls: Boolean = TransmogrifierDefaults.TrackNulls) extends Product with Serializable

    Permalink

    OPMap vectorizer model arguments

    OPMap vectorizer model arguments

    allKeys

    all keys per feature

    fillByKey

    fill values for features

    shouldCleanKeys

    should clean map keys

    shouldCleanValues

    should clean map values

    defaultValue

    default value to replace with

    trackNulls

    add column to track null values for each map key

  75. class OpCountVectorizer extends OpEstimatorWrapper[TextList, OPVector, CountVectorizer, CountVectorizerModel]

    Permalink

    Wrapper around spark ml CountVectorizer for use with OP pipelines

  76. class OpIndexToString extends OpTransformerWrapper[RealNN, Text, IndexToString]

    Permalink

    OP wrapper for org.apache.spark.ml.feature.IndexToString

    OP wrapper for org.apache.spark.ml.feature.IndexToString

    NOTE THAT THIS CLASS EITHER FILTERS OUT OR THROWS AN ERROR IF PREVIOUSLY UNSEEN VALUES APPEAR

    A transformer that maps a feature of indices back to a new feature of corresponding text values. The index-string mapping is either from the ML attributes of the input feature, or from user-supplied labels (which take precedence over ML attributes).

    See also

    OpStringIndexer for converting text into indices

  77. class OpIndexToStringNoFilter extends UnaryTransformer[RealNN, Text] with SaveOthersParams

    Permalink

    A transformer that maps a feature of indices back to a new feature of corresponding text values.

    A transformer that maps a feature of indices back to a new feature of corresponding text values. The index-string mapping is either from the ML attributes of the input feature, or from user-supplied labels (which take precedence over ML attributes).

    See also

    OpStringIndexerNoFilter for converting text into indices

  78. class OpLDA extends OpEstimatorWrapper[OPVector, OPVector, LDA, LDAModel]

    Permalink

    Wrapper around spark ml LDA (Latent Dirichlet Allocation) for use with OP pipelines

  79. abstract class OpOneHotVectorizer[T <: FeatureType] extends SequenceEstimator[T, OPVector] with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with OneHotFun with MaxPctCardinalityParams

    Permalink

    Converts a sequence of features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  80. abstract class OpOneHotVectorizerModel[T <: FeatureType] extends SequenceModel[T, OPVector] with CleanTextFun with OneHotModelFun[T]

    Permalink
  81. class OpScalarStandardScaler extends UnaryEstimator[RealNN, RealNN]

    Permalink

    Wraps Spark's native StandardScaler, which operates on vectors, to enable it to operate directly on scalars.

  82. final class OpScalarStandardScalerModel extends UnaryModel[RealNN, RealNN]

    Permalink
  83. class OpSetVectorizer[T <: OPSet[_]] extends OpOneHotVectorizer[T]

    Permalink

    Converts a sequence of OpSet features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of OpSet features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  84. final class OpSetVectorizerModel[T <: OPSet[_]] extends OpOneHotVectorizerModel[T]

    Permalink
  85. class OpStringIndexer[T <: Text] extends OpEstimatorWrapper[T, RealNN, StringIndexer, StringIndexerModel]

    Permalink

    OP wrapper for org.apache.spark.ml.feature.StringIndexer

    OP wrapper for org.apache.spark.ml.feature.StringIndexer

    NOTE THAT THIS CLASS EITHER FILTERS OUT OR THROWS AN ERROR IF PREVIOUSLY UNSEEN VALUES APPEAR

    A label indexer that maps a text column of labels to an ML feature of label indices. The indices are in [0, numLabels), ordered by label frequencies. So the most frequent label gets index 0.

    See also

    OpIndexToString for the inverse transformation

  86. class OpStringIndexerNoFilter[I <: Text] extends UnaryEstimator[I, RealNN] with SaveOthersParams

    Permalink

    A label indexer that maps a text column of labels to an ML feature of label indices.

    A label indexer that maps a text column of labels to an ML feature of label indices. The indices are in [0, numLabels), ordered by label frequencies. So the most frequent label gets index 0.

    See also

    OpIndexToStringNoFilter for the inverse transformation

  87. final class OpStringIndexerNoFilterModel[I <: Text] extends UnaryModel[I, RealNN]

    Permalink
  88. class OpTextPivotVectorizer[T <: Text] extends OpOneHotVectorizer[T]

    Permalink

    Converts a sequence of Text features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of Text features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  89. final class OpTextPivotVectorizerModel[T <: Text] extends OpOneHotVectorizerModel[T]

    Permalink
  90. class OpWord2Vec extends OpEstimatorWrapper[TextList, OPVector, Word2Vec, Word2VecModel]

    Permalink

    Wrapper around spark ml word2vec for use with OP pipelines

  91. class ParsePhoneDefaultCountry extends UnaryTransformer[Phone, Phone] with PhoneParams

    Permalink

    Transformer to determine if a phone numbers is valid when no country code is available.

    Transformer to determine if a phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluate with international formatting

    Returns stripped number if number is valid. And None other wise.

  92. class ParsePhoneNumber extends BinaryTransformer[Phone, Text, Phone] with PhoneCountryParams

    Permalink

    Determine whether a phone number is valid given the country's regional code.

    Determine whether a phone number is valid given the country's regional code. By default the regional code will be checked against those provided in Google's PhoneNumber library. If the input regional code is not found, the default locale will be used for validation.

    If the User provided a Country name to code mapping, the phone number can only be validated against the input mapping. This transformer will first match on regional code, failing that, it will select the country with the closest Q-Distance.

    All phone numbers with less than 2 characters will be categorized as invalid

    All phone numbers that starts with "+" will be evaluated with international formatting

    Returns stripped number if number is valid. And None other wise.

  93. class PercentileCalibrator extends UnaryEstimator[RealNN, RealNN]

    Permalink

    Wraps around org.apache.spark.ml.feature.QuantileDiscretizer

  94. final class PercentileCalibratorModel extends UnaryModel[RealNN, RealNN]

    Permalink
  95. trait PhoneCountryParams extends PhoneParams

    Permalink
  96. trait PhoneParams extends Params

    Permalink
  97. trait PivotParams extends TextParams

    Permalink
  98. class PowerTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Power transformer

    Power transformer

    I

    input feature type

  99. final class PredictionDescaler[I <: Real, O <: Real] extends BinaryTransformer[Prediction, I, O]

    Permalink

    Applies to the input column the inverse of the scaling function defined in the Prediction feature metadata.

    Applies to the input column the inverse of the scaling function defined in the Prediction feature metadata. - 1st input feature is the Prediction feature to descale - 2nd input feature is scaled Prediction feature containing the metadata for constructing the scaling used to make this column

    I

    input feature type

    O

    output feature type

  100. class RealMapVectorizer[T <: OPMap[Double]] extends OPMapVectorizer[Double, T]

    Permalink

    Class for vectorizing RealMap features.

    Class for vectorizing RealMap features. Fills missing keys with the mean for that key.

    T

    input feature type to vectorize into an OPVector

  101. final class RealMapVectorizerModel[T <: OPMap[Double]] extends OPMapVectorizerModel[Double, T]

    Permalink
  102. class RealNNVectorizer extends SequenceTransformer[RealNN, OPVector] with VectorizerDefaults

    Permalink

    Converts a sequence of real non nullable features into a vector feature

  103. class RealVectorizer[T <: Real] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of Nullable Numeric features into a vector feature.

    Converts a sequence of Nullable Numeric features into a vector feature. Can choose to fill null values with the mean or a constant

  104. final class RealVectorizerModel[T <: Real] extends SequenceModel[T, OPVector] with VectorizerDefaults

    Permalink
  105. class RoundDigitsTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Round digits transformer

    Round digits transformer

    I

    input feature type

  106. class RoundTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Integral]

    Permalink

    Round transformer

    Round transformer

    I

    input feature type

  107. trait SaveOthersParams extends Params

    Permalink
  108. class ScalarAddTransformer[I <: OPNumeric[_], N] extends UnaryTransformer[I, Real]

    Permalink

    Scalar addition transformer

    Scalar addition transformer

    I

    input feature type

    N

    value type

  109. class ScalarDivideTransformer[I <: OPNumeric[_], N] extends UnaryTransformer[I, Real]

    Permalink

    Scalar divide transformer

    Scalar divide transformer

    I

    input feature type

    N

    value type

  110. class ScalarMultiplyTransformer[I <: OPNumeric[_], N] extends UnaryTransformer[I, Real]

    Permalink

    Scalar multiply transformer

    Scalar multiply transformer

    I

    input feature type

    N

    value type

  111. class ScalarSubtractTransformer[I <: OPNumeric[_], N] extends UnaryTransformer[I, Real]

    Permalink

    Scalar subtract transformer

    Scalar subtract transformer

    I

    input feature type

    N

    value type

  112. trait Scaler extends Serializable

    Permalink

    A trait for defining a new family of scaling functions scalingType: a ScalingType Enum for the scaling name args: A case class containing the args needed to define scaling and inverse scaling functions scale: The scaling function descale: The inverse scaling function

    A trait for defining a new family of scaling functions scalingType: a ScalingType Enum for the scaling name args: A case class containing the args needed to define scaling and inverse scaling functions scale: The scaling function descale: The inverse scaling function

    To add a new family of scaling functions: Add an entry to the scalingType enum, define a Case class extending Scaler, and add a case statement to both the Scaler and ScalerMetaData case classes

  113. case class ScalerMetadata(scalingType: ScalingType, scalingArgs: ScalingArgs) extends Product with Serializable

    Permalink

    Metadata containing the info needed to reconstruct a Scaler instance

    Metadata containing the info needed to reconstruct a Scaler instance

    scalingType

    the family of functions containing the scaler

    scalingArgs

    the args uniquely defining a function in the scaling family

  114. final class ScalerTransformer[I <: Real, O <: Real] extends UnaryTransformer[I, O]

    Permalink

    Scaling transformer that applies a scaling function to a numerical feature

    Scaling transformer that applies a scaling function to a numerical feature

    I

    input feature type

    O

    output feature type

  115. trait ScalingArgs extends JsonLike

    Permalink

    A trait to be extended by a case class containing the args needed to define a family of scaling & descaling functions

  116. sealed trait ScalingType extends EnumEntry with Serializable

    Permalink
  117. class SetNGramSimilarity extends NGramSimilarity[MultiPickList]

    Permalink

    Compute char ngram distance for MultiPickList features.

  118. case class SmartTextFeatureInfo(key: String, isCategorical: Boolean, topValues: Array[String]) extends JsonLike with Product with Serializable

    Permalink

    Info about each feature within a text map

    Info about each feature within a text map

    key

    name of a feature

    isCategorical

    indicate whether a feature is categorical or not

    topValues

    most common values of a feature (only for categoricals)

  119. class SmartTextMapVectorizer[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with TextTokenizerParams with TrackTextLenParam with HashingVectorizerParams with MapHashingFun with OneHotFun with MapStringPivotHelper with MapVectorizerFuns[String, OPMap[String]] with MaxCardinalityParams

    Permalink

    Convert a sequence of text map features into a vector by detecting categoricals that are disguised as text.

    Convert a sequence of text map features into a vector by detecting categoricals that are disguised as text. A categorical will be represented as a vector consisting of occurrences of top K most common values of that feature plus occurrences of non top k values and a null indicator (if enabled). Non-categoricals will be converted into a vector using the hashing trick. In addition, a null indicator is created for each non-categorical (if enabled).

  120. final class SmartTextMapVectorizerModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with TextTokenizerParams with TrackTextLenParam with MapHashingFun with TextMapPivotVectorizerModelFun[OPMap[String]]

    Permalink
  121. case class SmartTextMapVectorizerModelArgs(allFeatureInfo: Seq[Seq[SmartTextFeatureInfo]], shouldCleanKeys: Boolean, shouldCleanValues: Boolean, shouldTrackNulls: Boolean, hashingParams: HashingFunctionParams) extends JsonLike with Product with Serializable

    Permalink

    Arguments for SmartTextMapVectorizerModel

    allFeatureInfo

    info about each feature with each text map

    shouldCleanKeys

    should clean feature keys

    shouldCleanValues

    should clean feature values

    shouldTrackNulls

    should track nulls

    hashingParams

    hashing function params

  122. class SmartTextVectorizer[T <: Text] extends SequenceEstimator[T, OPVector] with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with TextTokenizerParams with TrackTextLenParam with HashingVectorizerParams with HashingFun with OneHotFun with MaxCardinalityParams

    Permalink

    Convert a sequence of text features into a vector by detecting categoricals that are disguised as text.

    Convert a sequence of text features into a vector by detecting categoricals that are disguised as text. A categorical will be represented as a vector consisting of occurrences of top K most common values of that feature plus occurrences of non top k values and a null indicator (if enabled). Non-categoricals will be converted into a vector using the hashing trick. In addition, a null indicator is created for each non-categorical (if enabled).

  123. final class SmartTextVectorizerModel[T <: Text] extends SequenceModel[T, OPVector] with TextTokenizerParams with TrackTextLenParam with HashingFun with OneHotModelFun[Text]

    Permalink
  124. case class SmartTextVectorizerModelArgs(isCategorical: Array[Boolean], topValues: Array[Seq[String]], shouldCleanText: Boolean, shouldTrackNulls: Boolean, hashingParams: HashingFunctionParams) extends JsonLike with Product with Serializable

    Permalink

    Arguments for SmartTextVectorizerModel

    isCategorical

    is feature a categorical or not

    topValues

    top values to each feature

    shouldCleanText

    should clean text value

    shouldTrackNulls

    should track nulls

    hashingParams

    hashing function params

  125. class SqrtTransformer[I <: OPNumeric[_]] extends UnaryTransformer[I, Real]

    Permalink

    Square root transformer

    Square root transformer

    I

    input feature type

  126. sealed trait StringIndexerHandleInvalid extends EnumEntry with Serializable

    Permalink
  127. class SubstringTransformer[I1 <: Text, I2 <: Text] extends BinaryTransformer[I1, I2, Binary] with TextMatchingParams

    Permalink

    Checks if the first input is a substring of the second input

    Checks if the first input is a substring of the second input

    I1

    first input feature type

    I2

    second input feature type

  128. class SubtractTransformer[I1 <: OPNumeric[_], I2 <: OPNumeric[_]] extends BinaryTransformer[I1, I2, Real]

    Permalink

    Minus function truth table (Real as example):

    Minus function truth table (Real as example):

    Real.empty - Real.empty = Real.empty Real.empty - Real(x) = Real(-x) Real(x) - Real.empty = Real(x) Real(x) - Real(y) = Real(x - y)

  129. class TextLenTransformer[T <: TextList] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with TextTokenizerParams with TextParams

    Permalink

    Sequence transformer for generating a sequence of text lengths from a sequence of TextList values (eg.

    Sequence transformer for generating a sequence of text lengths from a sequence of TextList values (eg. tokenized raw text)

  130. class TextListNullTransformer[T <: TextList] extends SequenceTransformer[T, OPVector] with VectorizerDefaults

    Permalink

    Creates null indicator columns for a sequence of input TextList features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs.

    Creates null indicator columns for a sequence of input TextList features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs. separate hash spaces.

  131. class TextMapHashingVectorizer[T <: OPMap[String]] extends OPMapVectorizer[String, T] with TextParams

    Permalink
  132. final class TextMapHashingVectorizerModel[T <: OPMap[String]] extends OPMapVectorizerModel[String, T] with TextTokenizerParams with HashingFun

    Permalink
  133. class TextMapLenEstimator[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with MapVectorizerFuns[String, T]

    Permalink

    Estimator for computing text lengths on fields stored in text maps.

    Estimator for computing text lengths on fields stored in text maps. Note that because there are no maps from String to TextList, we need to do the tokenization here (unlike the TextLenTransformer).

  134. final class TextMapLenModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with CleanTextMapFun with TextTokenizerParams

    Permalink
  135. class TextMapNullEstimator[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with MapVectorizerFuns[String, T]

    Permalink

    Creates null indicator columns for a sequence of input TextMap features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs.

    Creates null indicator columns for a sequence of input TextMap features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs. separate hash spaces.

  136. final class TextMapNullModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with CleanTextMapFun with TextTokenizerParams

    Permalink
  137. class TextMapPivotVectorizer[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with PivotParams with MapPivotParams with TextParams with MapStringPivotHelper with CleanTextMapFun with MinSupportParam with TrackNullsParam with MaxPctCardinalityParams with MaxPctCardinalityFun

    Permalink

    Converts a sequence of KeyString features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features).

    Converts a sequence of KeyString features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features). Each key found will also generate an other column which will capture values that do not make the cut or where not seen in training. Note that any keys not seen in training will be ignored.

  138. final class TextMapPivotVectorizerModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with TextMapPivotVectorizerModelFun[T]

    Permalink
  139. trait TextMatchingParams extends Params

    Permalink
  140. class TextNGramSimilarity[T <: Text] extends NGramSimilarity[T]

    Permalink

    Compute char ngram distance for Text features.

  141. trait TextParams extends Params

    Permalink
  142. class TextTokenizer[T <: Text] extends UnaryTransformer[T, TextList] with TextTokenizerParams

    Permalink

    Transformer that takes anything of type Text or lower and returns a TextList of tokens extracted from that text

  143. trait TextTokenizerParams extends LanguageDetectionParams with TextMatchingParams

    Permalink
  144. sealed abstract class TimePeriod extends EnumEntry with Serializable

    Permalink
  145. class ToOccurTransformer[I <: FeatureType] extends UnaryTransformer[I, RealNN]

    Permalink

    Transformer that converts input feature of type I into doolean feature using a user specified function that maps object type I to a Boolean

    Transformer that converts input feature of type I into doolean feature using a user specified function that maps object type I to a Boolean

    I

    Object type to be mapped to a double (doolean).

  146. trait TrackInvalidParam extends Params

    Permalink

    Param that decides whether or not the values that are considered invalid are tracked

  147. trait TrackNullsParam extends Params

    Permalink

    Param that decides whether or not the values that were missing are tracked

  148. trait TrackTextLenParam extends Params

    Permalink

    Param that decides whether or not lengths of text are tracked during vectorization

  149. class ValidEmailTransformer extends UnaryTransformer[Email, Binary]

    Permalink

    Checks if an email is valid

  150. trait VectorizerDefaults extends OpPipelineStageBase

    Permalink
  151. class VectorsCombiner extends SequenceEstimator[OPVector, OPVector]

    Permalink

    Takes in a sequence of vectors and combines them into a single vector

  152. final class VectorsCombinerModel extends SequenceModel[OPVector, OPVector]

    Permalink

Value Members

  1. object DateListPivot extends Enum[DateListPivot] with Serializable

    Permalink

    Enumeration object that contains the option to pivot the DateList feature

    Enumeration object that contains the option to pivot the DateList feature

    1) SinceFirst - replace the feature by the number of days between the first event and reference date

    2) SinceLast - replace the feature by the number of days between the last event and reference date

    3) ModeDay - replace the feature by a pivot that indicates the mode of the day of the week Example : If the mode is Monday then it will return (1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)

    4) ModeMonth - replace the feature by a pivot that indicates the mode of the month

    5) ModeHour - replace the feature by a pivot that indicates the mode of the hour of the day.

  2. object DecisionTreeNumericBucketizer extends Product with Serializable

    Permalink
  3. object HashAlgorithm extends Enum[HashAlgorithm] with Serializable

    Permalink
  4. object HashSpaceStrategy extends Enum[HashSpaceStrategy] with Serializable

    Permalink
  5. object Inclusion extends Enum[Inclusion] with Serializable

    Permalink
  6. object IndexToStringHandleInvalid extends Enum[IndexToStringHandleInvalid] with Serializable

    Permalink
  7. object LangDetector extends Serializable

    Permalink
  8. object NGramSimilarity extends Serializable

    Permalink
  9. object NameEntityRecognizer extends Serializable

    Permalink
  10. object NumericBucketizer extends Serializable

    Permalink
  11. object OpIndexToStringNoFilter extends Serializable

    Permalink
  12. object OpOneHotVectorizer extends Serializable

    Permalink
  13. object OpStringIndexerNoFilter extends Serializable

    Permalink
  14. object PercentileCalibrator extends Product with Serializable

    Permalink
  15. object PhoneNumberParser extends Product with Serializable

    Permalink
  16. object Scaler extends Serializable

    Permalink

    Scaler instance factory

  17. object ScalerMetadata extends Serializable

    Permalink
  18. object ScalingType extends Enum[ScalingType] with Serializable

    Permalink
  19. object SmartTextVectorizer extends Serializable

    Permalink
  20. object StringIndexerHandleInvalid extends Enum[StringIndexerHandleInvalid] with Serializable

    Permalink
  21. object TextMapHashingVectorizerNames

    Permalink
  22. object TextTokenizer extends Serializable

    Permalink
  23. object TikaHelper

    Permalink

    Tika helper

  24. object TimePeriod extends Enum[TimePeriod] with Serializable

    Permalink
  25. object ToOccurTransformer extends Serializable

    Permalink
  26. object VectorizerUtils extends Product with Serializable

    Permalink

Ungrouped