Package

com.salesforce.op.stages.impl

feature

Permalink

package feature

Visibility
  1. Public
  2. All

Type Members

  1. class AliasTransformer[I <: FeatureType] extends UnaryTransformer[I, I]

    Permalink

    No-op (identity) alias feature transformer allowing renaming features without applying a transformation on values.

    No-op (identity) alias feature transformer allowing renaming features without applying a transformation on values.

    I

    feature type

  2. class BinaryMapVectorizer[T <: OPMap[Boolean]] extends OPMapVectorizer[Boolean, T]

    Permalink

    Class for vectorizing BinaryMap features.

    Class for vectorizing BinaryMap features. Fills missing keys with args.defaultValue, which does not depend on the key, so getFillByKey returns an empty sequence.

    T

    input feature type to vectorize into an OPVector

  3. final class BinaryMapVectorizerModel[T <: OPMap[Boolean]] extends OPMapVectorizerModel[Boolean, T]

    Permalink
  4. class BinaryVectorizer extends SequenceTransformer[Binary, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Vectorizes Binary inputs where each input is transformed into 2 vector elements where the first element is [1 -> true] or [0 -> false] and the second element is [1 -> filled value] or [0 -> original value].

    Vectorizes Binary inputs where each input is transformed into 2 vector elements where the first element is [1 -> true] or [0 -> false] and the second element is [1 -> filled value] or [0 -> original value]. The vector representation for each input is concatenated into a final vector representation.

    Example:

    Data: Seq[(Binary, Binary)] = ((Some(false), None)) => f1, f2 new BinaryVectorizer().setInput(f1, f2).setFillValue(10)

    will produce Array(0.0, 0.0, 10.0, 1.0)

  5. trait CleanTextFun extends AnyRef

    Permalink
  6. trait CleanTextMapFun extends CleanTextFun

    Permalink
  7. sealed trait DateListPivot extends EnumEntry with Serializable

    Permalink
  8. class DateListVectorizer[T <: OPList[Long]] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of DateLists features into a vector feature.

    Converts a sequence of DateLists features into a vector feature. Can choose how to pivot the features

  9. class DateMapToUnitCircleVectorizer[T <: DateMap] extends SequenceEstimator[T, OPVector] with DateToUnitCircleParams with MapVectorizerFuns[Long, T]

    Permalink

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    parameter timePeriod The time period to extract from the timestamp enum from: DayOfMonth, DayOfWeek, DayOfYear, HourOfDay, MonthOfYear, WeekOfMonth, WeekOfYear

    We extract the timePeriod from the timestamp and map this onto the unit circle containing the number of time periods equally spaced. For example, when timePeriod = HourOfDay, the timestamp 01/01/2018 6:37 maps to the point on the circle with angle radians = 2*math.Pi*6/24 We return the cartesian coordinates of this point: (math.cos(radians), math.sin(radians))

    The first time period always has angle 0.

    Note: We use the ISO week date format https://en.wikipedia.org/wiki/ISO_week_date#First_week Monday is the first day of the week & the first week of the year is the week wit the first Monday after Jan 1.

  10. final class DateMapToUnitCircleVectorizerModel[T <: DateMap] extends SequenceModel[T, OPVector] with CleanTextMapFun

    Permalink

    Model for DateMapToUnitCircleVectorizer

    Model for DateMapToUnitCircleVectorizer

    T

    DateMap type

  11. class DateMapVectorizer[T <: OPMap[Long]] extends OPMapVectorizer[Long, T]

    Permalink

    Class for vectorizing DateMap features.

    Class for vectorizing DateMap features. Fills missing keys with args.defaultValue, which does not depend on the key, so getFillByKey returns an empty sequence.

    T

    input feature type to vectorize into an OPVector

  12. final class DateMapVectorizerModel[T <: OPMap[Long]] extends OPMapVectorizerModel[Long, T]

    Permalink
  13. trait DateToUnitCircleParams extends Params

    Permalink
  14. class DateToUnitCircleTransformer[T <: Date] extends SequenceTransformer[T, OPVector] with DateToUnitCircleParams

    Permalink

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    Following: http://webspace.ship.edu/pgmarr/geo441/lectures/lec%2016%20-%20directional%20statistics.pdf Transforms a Date or DateTime field into a cartesian coordinate representation of an extracted time period on the unit circle

    parameter timePeriod The time period to extract from the timestamp enum from: DayOfMonth, DayOfWeek, DayOfYear, HourOfDay, MonthOfYear, WeekOfMonth, WeekOfYear

    We extract the timePeriod from the timestamp and map this onto the unit circle containing the number of time periods equally spaced. For example, when timePeriod = HourOfDay, the timestamp 01/01/2018 6:37 maps to the point on the circle with angle radians = 2*math.Pi*6/24 We return the cartesian coordinates of this point: (math.cos(radians), math.sin(radians))

    The first time period always has angle 0.

    Note: We use the ISO week date format https://en.wikipedia.org/wiki/ISO_week_date#First_week Monday is the first day of the week & the first week of the year is the week wit the first Monday after Jan 1.

  15. class DecisionTreeNumericBucketizer[N, I2 <: OPNumeric[N]] extends BinaryEstimator[RealNN, I2, OPVector] with DecisionTreeNumericBucketizerParams with VectorizerDefaults with TrackInvalidParam with TrackNullsParam with NumericBucketizerMetadata with AllowLabelAsInput[OPVector]

    Permalink

    Smart bucketizer for numeric values based on a Decision Tree classifier.

    Smart bucketizer for numeric values based on a Decision Tree classifier.

    N

    numeric feature type value

    I2

    numeric feature type

  16. final class DecisionTreeNumericBucketizerModel[I2 <: OPNumeric[_]] extends BinaryModel[RealNN, I2, OPVector] with AllowLabelAsInput[OPVector]

    Permalink
  17. trait DecisionTreeNumericBucketizerParams extends AnyRef

    Permalink
  18. class DecisionTreeNumericMapBucketizer[N, I2 <: OPMap[N]] extends BinaryEstimator[RealNN, I2, OPVector] with DecisionTreeNumericBucketizerParams with VectorizerDefaults with TrackInvalidParam with TrackNullsParam with NumericBucketizerMetadata with MapPivotParams with CleanTextMapFun with AllowLabelAsInput[OPVector]

    Permalink

    Smart bucketizer for numeric map values based on a Decision Tree classifier.

    Smart bucketizer for numeric map values based on a Decision Tree classifier.

    N

    numeric feature type value

    I2

    numeric map feature type

  19. final class DecisionTreeNumericMapBucketizerModel[I2 <: OPMap[_]] extends BinaryModel[RealNN, I2, OPVector] with CleanTextMapFun with AllowLabelAsInput[OPVector]

    Permalink
  20. class DropIndicesByTransformer extends UnaryTransformer[OPVector, OPVector]

    Permalink

    Allows columns to be dropped from a feature vector based on properties of the metadata about what is contained in each column (will work only on vectors) created with OpVectorMetadata

  21. class FillMissingWithMean[N, I <: OPNumeric[N]] extends UnaryEstimator[I, RealNN]

    Permalink

    Fill missing values with mean for any numeric feature

  22. final class FillMissingWithMeanModel[I <: OPNumeric[_]] extends UnaryModel[I, RealNN]

    Permalink
  23. class FilterMap[I <: OPMap[_]] extends UnaryTransformer[I, I] with MapPivotParams with TextParams with CleanTextMapFun

    Permalink

    Filters maps by keys provided in a whiteList or blackList

    Filters maps by keys provided in a whiteList or blackList

    I

    input feature type

  24. class GeolocationMapVectorizer extends SequenceEstimator[GeolocationMap, OPVector] with MapVectorizerFuns[Seq[Double], GeolocationMap] with TrackNullsParam

    Permalink
  25. final class GeolocationMapVectorizerModel extends SequenceModel[GeolocationMap, OPVector] with CleanTextMapFun

    Permalink
  26. class GeolocationVectorizer extends SequenceEstimator[Geolocation, OPVector] with VectorizerDefaults with TrackNullsParam with GeolocationFunctions

    Permalink

    Converts a sequence of Geolocation features into a vector feature.

    Converts a sequence of Geolocation features into a vector feature. Can choose to fill null values with the mean or a constant

  27. final class GeolocationVectorizerModel extends SequenceModel[Geolocation, OPVector] with VectorizerDefaults

    Permalink
  28. sealed trait HashAlgorithm extends EnumEntry with Serializable

    Permalink

    Hashing Algorithms

  29. sealed trait HashSpaceStrategy extends EnumEntry with Serializable

    Permalink

    Hash space strategy

  30. case class HashingFunctionParams(hashWithIndex: Boolean, prependFeatureName: Boolean, numFeatures: Int, numInputs: Int, maxNumOfFeatures: Int, binaryFreq: Boolean, hashAlgorithm: HashAlgorithm, hashSpaceStrategy: HashSpaceStrategy) extends Product with Serializable

    Permalink

    Hashing Parameters

    Hashing Parameters

    hashWithIndex

    if true, include indices when hashing a feature that has them (OPLists or OPVectors)

    prependFeatureName

    if true, prepends a input feature name to each token of that feature

    numFeatures

    number of features (hashes) to generate

    numInputs

    number of inputs

    maxNumOfFeatures

    max number of features (hashes)

    binaryFreq

    if true, term frequency vector will be binary such that non-zero term counts will be set to 1.0

    hashAlgorithm

    hash algorithm to use

    hashSpaceStrategy

    strategy to determine whether to use shared hash space for all included features

  31. sealed abstract class Inclusion extends EnumEntry with Serializable

    Permalink
  32. sealed trait IndexToStringHandleInvalid extends EnumEntry with Serializable

    Permalink
  33. class IntegralMapVectorizer[T <: OPMap[Long]] extends OPMapVectorizer[Long, T]

    Permalink

    Class for vectorizing IntegralMap features.

    Class for vectorizing IntegralMap features. Fills missing keys with the mode for that key.

    T

    input feature type to vectorize into an OPVector

  34. final class IntegralMapVectorizerModel[T <: OPMap[Long]] extends OPMapVectorizerModel[Long, T]

    Permalink
  35. class IntegralVectorizer[T <: Integral] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of Integral features into a vector feature.

    Converts a sequence of Integral features into a vector feature. Can choose to fill null values with the mean or a constant

  36. final class IntegralVectorizerModel[T <: Integral] extends SequenceModel[T, OPVector] with VectorizerDefaults

    Permalink
  37. class IsValidPhoneDefaultCountry extends UnaryTransformer[Phone, Binary] with PhoneParams

    Permalink

    Transformer to determine if a phone numbers is valid when no country code is available.

    Transformer to determine if a phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary feature true if phone is valid false if invalid and none if phone number is none

  38. class IsValidPhoneMapDefaultCountry extends UnaryTransformer[PhoneMap, BinaryMap] with PhoneParams

    Permalink

    Transformer to determine if a map of phone numbers is valid when no country code is available.

    Transformer to determine if a map of phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary map feature true if phone is valid false if invalid and none if phone number is none

  39. class IsValidPhoneNumber extends BinaryTransformer[Phone, Text, Binary] with PhoneCountryParams

    Permalink

    Determine whether a phone number is valid given the country's regional code.

    Determine whether a phone number is valid given the country's regional code. By default the regional code will be checked against those provided in Google's PhoneNumber library. If the input regional code is not found, the default locale will be used for validation.

    If the User provided a Country name to code mapping, the phone number can only be validated against the input mapping. This transformer will first match on regional code, failing that, it will select the country with the closest Q-Distance.

    All phone numbers with less than 2 characters will be categorized as invalid

    All phone numbers that starts with "+" will be evaluated with international formatting

    Returns binary feature true if phone is valid false if invalid and none if phone number is none

  40. class JaccardSimilarity extends BinaryTransformer[MultiPickList, MultiPickList, RealNN]

    Permalink

    Calculates the Jaccard Similarity between two sets.

    Calculates the Jaccard Similarity between two sets. If both inputs are empty, Jaccard Similarity is defined as 1.0

  41. class LangDetector[T <: Text] extends UnaryTransformer[T, RealMap]

    Permalink

    Transformer that detects the language of the text

  42. trait LanguageDetectionParams extends Params

    Permalink
  43. trait MapPivotParams extends Params

    Permalink
  44. trait MapStringPivotHelper extends SaveOthersParams

    Permalink
  45. trait MapVectorizerFuns[A, T <: OPMap[A]] extends VectorizerDefaults with MapPivotParams with CleanTextMapFun

    Permalink
  46. trait MaxCardinalityParams extends Params

    Permalink
  47. class MimeTypeDetector extends UnaryTransformer[Base64, Text] with MimeTypeDetectorParams

    Permalink

    Detects MIME type for Base64 encoded binary data.

  48. class MimeTypeMapDetector extends UnaryTransformer[Base64Map, PickListMap] with MimeTypeDetectorParams

    Permalink

    Detects MIME type for Base64Map encoded binary data.

  49. trait MinSupportParam extends Params

    Permalink
  50. class MultiPickListMapVectorizer[T <: OPMap[Set[String]]] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with PivotParams with MapPivotParams with TextParams with MapStringPivotHelper with CleanTextMapFun with MinSupportParam with TrackNullsParam

    Permalink

    Converts a sequence of KeyMultiPickList features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features).

    Converts a sequence of KeyMultiPickList features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features). Each key found will also generate an other column which will capture values that do not make the cut or where not seen in training. Note that any keys not seen in training will be ignored.

  51. final class MultiPickListMapVectorizerModel[T <: OPMap[Set[String]]] extends SequenceModel[T, OPVector] with VectorizerDefaults with CleanTextMapFun

    Permalink
  52. class NameEntityRecognizer[T <: Text] extends UnaryTransformer[T, MultiPickListMap] with LanguageDetectionParams

    Permalink

    Name Entity NameEntityType text recognizer.

    Name Entity NameEntityType text recognizer.

    Note: when providing your own the analyzer/splitter/tagger make sure they can work together, for instance OpenNLP models require their own analyzers to be provided when tokenizing. The returned feature type is a MultiPickListMap which contains sets of entities for all the tokens

    T

    text feature type

  53. class NumericBucketizer[I1 <: OPNumeric[_]] extends UnaryTransformer[I1, OPVector] with VectorizerDefaults with NumericBucketizerParams with NumericBucketizerMetadata

    Permalink

    Numeric Bucketizer

    Numeric Bucketizer

    I1

    numeric feature type

  54. trait NumericBucketizerParams extends TrackInvalidParam with TrackNullsParam

    Permalink
  55. sealed trait NumericMapDefaultParam extends Params

    Permalink
  56. class OPCollectionHashingVectorizer[T <: OPCollection] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with PivotParams with CleanTextFun with HashingFun with HashingVectorizerParams

    Permalink

    Generic hashing vectorizer to convert features of type OPCollection into Vectors

    Generic hashing vectorizer to convert features of type OPCollection into Vectors

    In more details: It tries to hash entries in the collection using the specified hashing algorithm to build a single vector. If the desired number of features (= hash space size) for all features combined is larger than Integer.Max (the maximal index for a vector), then all the features use the same hash space. There are also options for the user to hash indices with collections where that makes sense (OPLists and OPVectors), and to force a shared hash space, even if the number of feature is not high enough to require it.

  57. sealed abstract class OPCollectionTransformer[I <: FeatureType, O <: FeatureType, ICol <: OPCollection, OCol <: OPCollection] extends UnaryTransformer[ICol, OCol]

    Permalink

    Abstract base class for a set of transformer wrappers that allow unary transformers between non-collection types to be used on collection types.

    Abstract base class for a set of transformer wrappers that allow unary transformers between non-collection types to be used on collection types. For example, we can use a UnaryLambdaTransformer[Email, Integer] on a map's values, creating a UnaryLambdaTransformer[EmailMap, IntegralMap]. This base class will be inherited by concrete classes for OPMaps, OPList, and OPSets (in order to enforce not allowing these collection types to be transformed into each other, eg. no MultiPickList to RealMap transformations).

    The OP type hierarchy does not allow direct type checking of such transformer wrappers (eg. Real#Value is Option[Double] and RealMap#Value is Map[String, Double], so there's no way to enforce that a RealMap can only hold what is contained in a Real) since the types themselves are not created with typetags for performance reasons. However, we can still enforce that operations like building a UnaryLambdaTransformer[RealMap, StringMap] from a UnaryLambdaTransformer[Real, Integer] is not possible by using the Spark types in validateTypes.

    I

    input feature type for supplied non-collection transformer

    O

    output feature type for supplied non-collection transformer

    ICol

    input feature type for desired collection transformer

    OCol

    output feature type for desired collection transformer

  58. abstract class OPMapVectorizer[A, T <: OPMap[A]] extends SequenceEstimator[T, OPVector] with MapVectorizerFuns[Double, RealMap] with NumericMapDefaultParam with TrackNullsParam

    Permalink

    Base class for vectorizing OPMap[A] features.

    Base class for vectorizing OPMap[A] features. Individual vectorizers for different feature types need to implement the getFillByKey function (which calculates any fill values that differ by key - means, modes, etc.) and the makeModel function (which specifies which type of model will be returned).

    A

    value type for underlying map

    T

    input feature type to vectorize into an OPVector

  59. sealed abstract class OPMapVectorizerModel[A, I <: OPMap[A]] extends SequenceModel[I, OPVector] with CleanTextMapFun

    Permalink
  60. sealed case class OPMapVectorizerModelArgs(allKeys: Seq[Seq[String]], fillByKey: Seq[Map[String, Double]], shouldCleanKeys: Boolean, shouldCleanValues: Boolean, defaultValue: Double, trackNulls: Boolean = TransmogrifierDefaults.TrackNulls) extends Product with Serializable

    Permalink

    OPMap vectorizer model arguments

    OPMap vectorizer model arguments

    allKeys

    all keys per feature

    fillByKey

    fill values for features

    shouldCleanKeys

    should clean map keys

    shouldCleanValues

    should clean map values

    defaultValue

    default value to replace with

    trackNulls

    add column to track null values for each map key

  61. class OpCountVectorizer extends OpEstimatorWrapper[TextList, OPVector, CountVectorizer, CountVectorizerModel]

    Permalink

    Wrapper around spark ml CountVectorizer for use with OP pipelines

  62. class OpIndexToString extends OpTransformerWrapper[RealNN, Text, IndexToString]

    Permalink

    OP wrapper for org.apache.spark.ml.feature.IndexToString

    OP wrapper for org.apache.spark.ml.feature.IndexToString

    NOTE THAT THIS CLASS EITHER FILTERS OUT OR THROWS AN ERROR IF PREVIOUSLY UNSEEN VALUES APPEAR

    A transformer that maps a feature of indices back to a new feature of corresponding text values. The index-string mapping is either from the ML attributes of the input feature, or from user-supplied labels (which take precedence over ML attributes).

    See also

    OpStringIndexer for converting text into indices

  63. class OpIndexToStringNoFilter extends UnaryTransformer[RealNN, Text] with SaveOthersParams

    Permalink

    A transformer that maps a feature of indices back to a new feature of corresponding text values.

    A transformer that maps a feature of indices back to a new feature of corresponding text values. The index-string mapping is either from the ML attributes of the input feature, or from user-supplied labels (which take precedence over ML attributes).

    See also

    OpStringIndexerNoFilter for converting text into indices

  64. class OpLDA extends OpEstimatorWrapper[OPVector, OPVector, LDA, LDAModel]

    Permalink

    Wrapper around spark ml LDA (Latent Dirichlet Allocation) for use with OP pipelines

  65. abstract class OpOneHotVectorizer[T <: FeatureType] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with OneHotFun

    Permalink

    Converts a sequence of features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  66. abstract class OpOneHotVectorizerModel[T <: FeatureType] extends SequenceModel[T, OPVector] with VectorizerDefaults with CleanTextFun with OneHotModelFun[T]

    Permalink
  67. class OpScalarStandardScaler extends UnaryEstimator[RealNN, RealNN]

    Permalink

    Wraps Spark's native StandardScaler, which operates on vectors, to enable it to operate directly on scalars.

  68. final class OpScalarStandardScalerModel extends UnaryModel[RealNN, RealNN]

    Permalink
  69. class OpSetVectorizer[T <: OPSet[_]] extends OpOneHotVectorizer[T]

    Permalink

    Converts a sequence of OpSet features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of OpSet features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  70. final class OpSetVectorizerModel[T <: OPSet[_]] extends OpOneHotVectorizerModel[T]

    Permalink
  71. class OpStringIndexer[T <: Text] extends OpEstimatorWrapper[T, RealNN, StringIndexer, StringIndexerModel]

    Permalink

    OP wrapper for org.apache.spark.ml.feature.StringIndexer

    OP wrapper for org.apache.spark.ml.feature.StringIndexer

    NOTE THAT THIS CLASS EITHER FILTERS OUT OR THROWS AN ERROR IF PREVIOUSLY UNSEEN VALUES APPEAR

    A label indexer that maps a text column of labels to an ML feature of label indices. The indices are in [0, numLabels), ordered by label frequencies. So the most frequent label gets index 0.

    See also

    OpIndexToString for the inverse transformation

  72. class OpStringIndexerNoFilter[I <: Text] extends UnaryEstimator[I, RealNN] with SaveOthersParams

    Permalink

    A label indexer that maps a text column of labels to an ML feature of label indices.

    A label indexer that maps a text column of labels to an ML feature of label indices. The indices are in [0, numLabels), ordered by label frequencies. So the most frequent label gets index 0.

    See also

    OpIndexToStringNoFilter for the inverse transformation

  73. final class OpStringIndexerNoFilterModel[I <: Text] extends UnaryModel[I, RealNN]

    Permalink
  74. class OpTextPivotVectorizer[T <: Text] extends OpOneHotVectorizer[T]

    Permalink

    Converts a sequence of Text features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs).

    Converts a sequence of Text features into a vector keeping the top K most common occurrences of each feature (ie the final vector has length K * number of inputs). Plus an additional column for "other" values - which will capture values that do not make the cut or values not seen in training, and an additional column for empty values unless null tracking is disabled.

  75. final class OpTextPivotVectorizerModel[T <: Text] extends OpOneHotVectorizerModel[T]

    Permalink
  76. class OpWord2Vec extends OpEstimatorWrapper[TextList, OPVector, Word2Vec, Word2VecModel]

    Permalink

    Wrapper around spark ml word2vec for use with OP pipelines

  77. class ParsePhoneDefaultCountry extends UnaryTransformer[Phone, Phone] with PhoneParams

    Permalink

    Transformer to determine if a phone numbers is valid when no country code is available.

    Transformer to determine if a phone numbers is valid when no country code is available. The default locale will be used for validation. All phone numbers with less than 2 characters will be categorized as invalid All phone numbers that starts with "+" will be evaluate with international formatting

    Returns stripped number if number is valid. And None other wise.

  78. class ParsePhoneNumber extends BinaryTransformer[Phone, Text, Phone] with PhoneCountryParams

    Permalink

    Determine whether a phone number is valid given the country's regional code.

    Determine whether a phone number is valid given the country's regional code. By default the regional code will be checked against those provided in Google's PhoneNumber library. If the input regional code is not found, the default locale will be used for validation.

    If the User provided a Country name to code mapping, the phone number can only be validated against the input mapping. This transformer will first match on regional code, failing that, it will select the country with the closest Q-Distance.

    All phone numbers with less than 2 characters will be categorized as invalid

    All phone numbers that starts with "+" will be evaluated with international formatting

    Returns stripped number if number is valid. And None other wise.

  79. class PercentileCalibrator extends UnaryEstimator[RealNN, RealNN]

    Permalink

    Wraps around org.apache.spark.ml.feature.QuantileDiscretizer

  80. final class PercentileCalibratorModel extends UnaryModel[RealNN, RealNN]

    Permalink
  81. trait PhoneCountryParams extends PhoneParams

    Permalink
  82. trait PhoneParams extends Params

    Permalink
  83. trait PivotParams extends TextParams

    Permalink
  84. class RealMapVectorizer[T <: OPMap[Double]] extends OPMapVectorizer[Double, T]

    Permalink

    Class for vectorizing RealMap features.

    Class for vectorizing RealMap features. Fills missing keys with the mean for that key.

    T

    input feature type to vectorize into an OPVector

  85. final class RealMapVectorizerModel[T <: OPMap[Double]] extends OPMapVectorizerModel[Double, T]

    Permalink
  86. class RealNNVectorizer extends SequenceTransformer[RealNN, OPVector] with VectorizerDefaults

    Permalink

    Converts a sequence of real non nullable features into a vector feature

  87. class RealVectorizer[T <: Real] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with TrackNullsParam

    Permalink

    Converts a sequence of Nullable Numeric features into a vector feature.

    Converts a sequence of Nullable Numeric features into a vector feature. Can choose to fill null values with the mean or a constant

  88. final class RealVectorizerModel[T <: Real] extends SequenceModel[T, OPVector] with VectorizerDefaults

    Permalink
  89. trait SaveOthersParams extends Params

    Permalink
  90. class SetNGramSimilarity extends NGramSimilarity[MultiPickList]

    Permalink

    Compute char ngram distance for MultiPickList features.

  91. case class SmartTextFeatureInfo(key: String, isCategorical: Boolean, topValues: Array[String]) extends JsonLike with Product with Serializable

    Permalink

    Info about each feature within a text map

    Info about each feature within a text map

    key

    name of a feature

    isCategorical

    indicate whether a feature is categorical or not

    topValues

    most common values of a feature (only for categoricals)

  92. class SmartTextMapVectorizer[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with TextTokenizerParams with HashingVectorizerParams with MapHashingFun with OneHotFun with MapStringPivotHelper with MapVectorizerFuns[String, OPMap[String]] with MaxCardinalityParams

    Permalink

    Convert a sequence of text map features into a vector by detecting categoricals that are disguised as text.

    Convert a sequence of text map features into a vector by detecting categoricals that are disguised as text. A categorical will be represented as a vector consisting of occurrences of top K most common values of that feature plus occurrences of non top k values and a null indicator (if enabled). Non-categoricals will be converted into a vector using the hashing trick. In addition, a null indicator is created for each non-categorical (if enabled).

  93. final class SmartTextMapVectorizerModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with TextTokenizerParams with MapHashingFun with TextMapPivotVectorizerModelFun[OPMap[String]]

    Permalink
  94. case class SmartTextMapVectorizerModelArgs(allFeatureInfo: Seq[Seq[SmartTextFeatureInfo]], shouldCleanKeys: Boolean, shouldCleanValues: Boolean, shouldTrackNulls: Boolean, hashingParams: HashingFunctionParams) extends JsonLike with Product with Serializable

    Permalink

    Arguments for SmartTextMapVectorizerModel

    allFeatureInfo

    info about each feature with each text map

    shouldCleanKeys

    should clean feature keys

    shouldCleanValues

    should clean feature values

    shouldTrackNulls

    should track nulls

    hashingParams

    hashing function params

  95. class SmartTextVectorizer[T <: Text] extends SequenceEstimator[T, OPVector] with PivotParams with CleanTextFun with SaveOthersParams with TrackNullsParam with MinSupportParam with TextTokenizerParams with HashingVectorizerParams with HashingFun with OneHotFun with MaxCardinalityParams

    Permalink

    Convert a sequence of text features into a vector by detecting categoricals that are disguised as text.

    Convert a sequence of text features into a vector by detecting categoricals that are disguised as text. A categorical will be represented as a vector consisting of occurrences of top K most common values of that feature plus occurrences of non top k values and a null indicator (if enabled). Non-categoricals will be converted into a vector using the hashing trick. In addition, a null indicator is created for each non-categorical (if enabled).

  96. final class SmartTextVectorizerModel[T <: Text] extends SequenceModel[T, OPVector] with TextTokenizerParams with HashingFun with OneHotModelFun[Text]

    Permalink
  97. case class SmartTextVectorizerModelArgs(isCategorical: Array[Boolean], topValues: Array[Seq[String]], shouldCleanText: Boolean, shouldTrackNulls: Boolean, hashingParams: HashingFunctionParams) extends JsonLike with Product with Serializable

    Permalink

    Arguments for SmartTextVectorizerModel

    isCategorical

    is feature a categorical or not

    topValues

    top values to each feature

    shouldCleanText

    should clean text value

    shouldTrackNulls

    should track nulls

    hashingParams

    hashing function params

  98. sealed trait StringIndexerHandleInvalid extends EnumEntry with Serializable

    Permalink
  99. class TextLenTransformer[T <: TextList] extends SequenceTransformer[T, OPVector] with VectorizerDefaults with TextTokenizerParams with TextParams

    Permalink

    Sequence transformer for generating a sequence of text lengths from a sequence of TextList values (eg.

    Sequence transformer for generating a sequence of text lengths from a sequence of TextList values (eg. tokenized raw text)

  100. class TextListNullTransformer[T <: TextList] extends SequenceTransformer[T, OPVector] with VectorizerDefaults

    Permalink

    Creates null indicator columns for a sequence of input TextList features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs.

    Creates null indicator columns for a sequence of input TextList features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs. separate hash spaces.

  101. class TextMapHashingVectorizer[T <: OPMap[String]] extends OPMapVectorizer[String, T] with TextParams

    Permalink
  102. final class TextMapHashingVectorizerModel[T <: OPMap[String]] extends OPMapVectorizerModel[String, T] with TextTokenizerParams with HashingFun

    Permalink
  103. class TextMapLenEstimator[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with MapVectorizerFuns[String, T]

    Permalink

    Estimator for computing text lengths on fields stored in text maps.

    Estimator for computing text lengths on fields stored in text maps. Note that because there are no maps from String to TextList, we need to do the tokenization here (unlike the TextLenTransformer).

  104. final class TextMapLenModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with VectorizerDefaults with CleanTextMapFun with TextTokenizerParams

    Permalink
  105. class TextMapNullEstimator[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with MapVectorizerFuns[String, T]

    Permalink

    Creates null indicator columns for a sequence of input TextMap features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs.

    Creates null indicator columns for a sequence of input TextMap features, originally for use as a separate stage in null tracking for hashed text features (easier to do outside the hashing vectorizer since we can make a null indicator column for each input feature without having to add lots of complex logic in the hashing vectorizer to deal with metadata for shared vs. separate hash spaces.

  106. final class TextMapNullModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with VectorizerDefaults with CleanTextMapFun with TextTokenizerParams

    Permalink
  107. class TextMapPivotVectorizer[T <: OPMap[String]] extends SequenceEstimator[T, OPVector] with VectorizerDefaults with PivotParams with MapPivotParams with TextParams with MapStringPivotHelper with CleanTextMapFun with MinSupportParam with TrackNullsParam

    Permalink

    Converts a sequence of KeyString features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features).

    Converts a sequence of KeyString features into a vector keeping the top K most common occurrences of each key in the maps for that feature (ie the final vector has length k * number of keys * number of features). Each key found will also generate an other column which will capture values that do not make the cut or where not seen in training. Note that any keys not seen in training will be ignored.

  108. final class TextMapPivotVectorizerModel[T <: OPMap[String]] extends SequenceModel[T, OPVector] with VectorizerDefaults with TextMapPivotVectorizerModelFun[T]

    Permalink
  109. class TextNGramSimilarity[T <: Text] extends NGramSimilarity[T]

    Permalink

    Compute char ngram distance for Text features.

  110. trait TextParams extends Params

    Permalink
  111. class TextTokenizer[T <: Text] extends UnaryTransformer[T, TextList] with TextTokenizerParams

    Permalink

    Transformer that takes anything of type Text or lower and returns a TextList of tokens extracted from that text

  112. trait TextTokenizerParams extends LanguageDetectionParams

    Permalink
  113. sealed abstract class TimePeriod extends EnumEntry with Serializable

    Permalink
  114. class ToOccurTransformer[I <: FeatureType] extends UnaryTransformer[I, RealNN]

    Permalink

    Transformer that converts input feature of type I into doolean feature using a user specified function that maps object type I to a Boolean

    Transformer that converts input feature of type I into doolean feature using a user specified function that maps object type I to a Boolean

    I

    Object type to be mapped to a double (doolean).

  115. trait TrackInvalidParam extends Params

    Permalink

    Param that decides whether or not the values that are considered invalid are tracked

  116. trait TrackNullsParam extends Params

    Permalink

    Param that decides whether or not the values that were missing are tracked

  117. trait VectorizerDefaults extends OpPipelineStageBase

    Permalink
  118. class VectorsCombiner extends SequenceEstimator[OPVector, OPVector]

    Permalink

    Takes in a sequence of vectors and combines them into a single vector

  119. final class VectorsCombinerModel extends SequenceModel[OPVector, OPVector]

    Permalink

Value Members

  1. object DateListPivot extends Enum[DateListPivot] with Serializable

    Permalink

    Enumeration object that contains the option to pivot the DateList feature

    Enumeration object that contains the option to pivot the DateList feature

    1) SinceFirst - replace the feature by the number of days between the first event and reference date

    2) SinceLast - replace the feature by the number of days between the last event and reference date

    3) ModeDay - replace the feature by a pivot that indicates the mode of the day of the week Example : If the mode is Monday then it will return (1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)

    4) ModeMonth - replace the feature by a pivot that indicates the mode of the month

    5) ModeHour - replace the feature by a pivot that indicates the mode of the hour of the day.

  2. object DecisionTreeNumericBucketizer extends Product with Serializable

    Permalink
  3. object HashAlgorithm extends Enum[HashAlgorithm] with Serializable

    Permalink
  4. object HashSpaceStrategy extends Enum[HashSpaceStrategy] with Serializable

    Permalink
  5. object Inclusion extends Enum[Inclusion] with Serializable

    Permalink
  6. object IndexToStringHandleInvalid extends Enum[IndexToStringHandleInvalid] with Serializable

    Permalink
  7. object LangDetector extends Serializable

    Permalink
  8. object NGramSimilarity extends Serializable

    Permalink
  9. object NameEntityRecognizer extends Serializable

    Permalink
  10. object NumericBucketizer extends Serializable

    Permalink
  11. object OpIndexToStringNoFilter extends Serializable

    Permalink
  12. object OpStringIndexerNoFilter extends Serializable

    Permalink
  13. object PercentileCalibrator extends Product with Serializable

    Permalink
  14. object PhoneNumberParser extends Product with Serializable

    Permalink
  15. object SmartTextVectorizer extends Serializable

    Permalink
  16. object StringIndexerHandleInvalid extends Enum[StringIndexerHandleInvalid] with Serializable

    Permalink
  17. object TextMapHashingVectorizerNames

    Permalink
  18. object TextTokenizer extends Serializable

    Permalink
  19. object TikaHelper

    Permalink

    Tika helper

  20. object TimePeriod extends Enum[TimePeriod] with Serializable

    Permalink
  21. object ToOccurTransformer extends Serializable

    Permalink
  22. object VectorizerUtils extends Product with Serializable

    Permalink

Ungrouped