OpVectorColumnMetadata

Represents the metadata a column in a vector.

Because we expect every vector column to have been produced by some vectorization process, we provide the name of the feature that led to this column.

Also note that each column's indicator value should be unique, meaning that they represent mutually exclusive values. The output of a hashing vectorizer, for instance, does not produce mutually exclusive values.

parentFeatureName: The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)
parentFeatureType: The type of the parent feature(s) for the column
grouping: The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None
indicatorValue: A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted
descriptorValue: A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None
index: Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata

Linear Supertypes

Product, Equals, JsonLike, Serializable, Serializable, AnyRef, Any

Instance Constructors

new OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0)

parentFeatureName

The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)

parentFeatureType

The type of the parent feature(s) for the column

grouping

The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None

indicatorValue

A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted

descriptorValue

A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None

index

Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata

Value Members

final def !=(arg0: Any): Boolean

Definition Classes

AnyRef → Any
final def ##(): Int

Definition Classes

AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes

AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes

Any
def clone(): AnyRef

Attributes

protected[java.lang]

Definition Classes

AnyRef

Annotations

@throws( ... )
val descriptorValue: Option[String]

A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None
final def eq(arg0: AnyRef): Boolean

Definition Classes

AnyRef
def featureGroup(): Option[String]

Get the feature grouping qualified by the parent feature name

Get the feature grouping qualified by the parent feature name

returns

Optional string of feature grouping
def finalize(): Unit

Attributes

protected[java.lang]

Definition Classes

AnyRef

Annotations

@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes

AnyRef → Any
val grouping: Option[String]

The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys).

The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None
def hasParentOfSubType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

Does column have parent features of which are subtypes of feature type O
def hasParentOfType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

Does column have parent features of specified feature type O
val index: Int

Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata
val indicatorValue: Option[String]

A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted
final def isInstanceOf[T0]: Boolean

Definition Classes

Any
def isNullIndicator: Boolean

Is this column corresponds to a null-encoded value

Is this column corresponds to a null-encoded value

returns

true if this column corresponds to a null-encoded value
def isOtherIndicator: Boolean

Is this column corresponds the other category of a one hot encoded categorical

Is this column corresponds the other category of a one hot encoded categorical

returns

true if this column corresponds to the other category of a one hot encoded categorical
def makeColName(): String

Make unique name for this column

Make unique name for this column

returns

String name for this column
final def ne(arg0: AnyRef): Boolean

Definition Classes

AnyRef
final def notify(): Unit

Definition Classes

AnyRef
final def notifyAll(): Unit

Definition Classes

AnyRef
val parentFeatureName: Seq[String]

The name of the parent feature(s) for the column.

The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)
val parentFeatureType: Seq[String]

The type of the parent feature(s) for the column
def parentNamesWithMapKeys(): Seq[String]

Return parent features names with the key (grouping) from any map parents included in name

Return parent features names with the key (grouping) from any map parents included in name

returns

Sequence of parent feature names, simple names when features are not maps, names plus keys for columns with map parent features
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes

AnyRef
def toJson(pretty: Boolean = true): String

Write this instance to json string

Write this instance to json string

pretty

should pretty print

returns

json string of the instance

Definition Classes

JsonLike
def toMetadata(): Metadata

Convert this column into Spark metadata.

Convert this column into Spark metadata.

returns

column Spark metadata
def toMetadata(ind: Array[Int]): Metadata

Convert this column into Spark metadata.

Convert this column into Spark metadata.

ind

Indexes of vector columns that match this OpVectorColumnMetadata

returns

column Spark metadata
def toString(): String

This instance json string

This instance json string

returns

json string of the instance

Definition Classes

JsonLike → AnyRef → Any
final def wait(): Unit

Definition Classes

AnyRef

Annotations

@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes

AnyRef

Annotations

@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes

AnyRef

Annotations

@throws( ... )

Related Docs: object OpVectorColumnMetadata | package spark

case class OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0) extends JsonLike with Product with Serializable

Instance Constructors

new OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val descriptorValue: Option[String]

final def eq(arg0: AnyRef): Boolean

def featureGroup(): Option[String]

def finalize(): Unit

final def getClass(): Class[_]

val grouping: Option[String]

def hasParentOfSubType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

def hasParentOfType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

val index: Int

val indicatorValue: Option[String]

final def isInstanceOf[T0]: Boolean

def isNullIndicator: Boolean

def isOtherIndicator: Boolean

def makeColName(): String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val parentFeatureName: Seq[String]

val parentFeatureType: Seq[String]

def parentNamesWithMapKeys(): Seq[String]

final def synchronized[T0](arg0: ⇒ T0): T0

def toJson(pretty: Boolean = true): String

def toMetadata(): Metadata

def toMetadata(ind: Array[Int]): Metadata

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Product

Inherited from Equals

Inherited from JsonLike

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped