test

Type Members

trait FeatureAsserts extends Matchers

Asserts for Feature instances on a given input/output
trait FeatureTestBase extends TestSparkContext with PropertyChecks

Common functionality for testing Feature operations (transformers / estimators)
trait FeatureTypeEquality[O <: FeatureType] extends AnyRef

Feature Type equality instances mixin.

Feature Type equality instances mixin. Allowing users to customize equality in tests, for example to allow numerical tolerance.

O

feature type
final class Gender extends Enum[Gender]
class JoinTestData extends SpecificRecordBase with SpecificRecord
abstract class OpEstimatorSpec[O <: FeatureType, ModelType <: Model[ModelType] with OpPipelineStage[O] with OpTransformer, EstimatorType <: Estimator[ModelType] with OpPipelineStage[O]] extends OpPipelineStageSpec[O, EstimatorType]

Base test class for testing OP estimator instances.

Base test class for testing OP estimator instances. Includes common tests for fitting estimator and verifying the fitted model.

O

output feature type

ModelType

model type produced by this estimator

EstimatorType

type of the estimator being tested
trait OpPipelineStageAsserts extends AppendedClues

Stage assertion for OpPipelineStage
abstract class OpPipelineStageSpec[O <: FeatureType, StageType <: OpPipelineStage[O]] extends FlatSpec with FeatureTypeEquality[O] with TestSparkContext with OpPipelineStageAsserts

Spec for testing OpPipelineStage instances (transformers or estimators).

Spec for testing OpPipelineStage instances (transformers or estimators). Includes common tests for output feature, copy, serialization, json read/write etc.

O

output feature type

StageType

OpPipelineStage type being tested (transformer or estimator)
abstract class OpTransformerSpec[O <: FeatureType, TransformerType <: OpPipelineStage[O] with Transformer with OpTransformer] extends OpPipelineStageSpec[O, TransformerType] with TransformerSpecCommon[O, TransformerType]

Base test class for testing OpPipelineStage instances (transformers or estimators).

Base test class for testing OpPipelineStage instances (transformers or estimators). Includes common tests for schema and data transformations.

O

output feature type

TransformerType

type of the transformer being tested
class Passenger extends SpecificRecordBase with SpecificRecord
class PassengerCSV extends SpecificRecordBase with SpecificRecord
trait PassengerFeaturesTest extends AnyRef
class PassengerProfile extends SpecificRecordBase with SpecificRecord
trait PassengerSparkFixtureTest extends TestSparkContext with PassengerFeaturesTest
case class Prestige(education: Double, income: Double, women: Double, prestige: Double) extends Product with Serializable
trait PrestigeData extends AnyRef
class SparkExample extends SpecificRecordBase with SpecificRecord
class SparkExampleJoin extends SpecificRecordBase with SpecificRecord
abstract class SwTransformerSpec[O <: FeatureType, SparkTransformerType <: Transformer with Params, TransformerType <: OpPipelineStage[O] with Transformer with Params with SparkWrapperParams[SparkTransformerType]] extends OpPipelineStageSpec[O, TransformerType] with TransformerSpecCommon[O, TransformerType]

Base test class for testing Spark transformer wrapper instances, e.g SwUnaryTransformer, SwBinaryTransformer etc.

Base test class for testing Spark transformer wrapper instances, e.g SwUnaryTransformer, SwBinaryTransformer etc. Includes common tests for schema and data transformations.

O

output feature type

SparkTransformerType

type of Spark transformer

TransformerType

type of Spark transformer wrapper being tested, e.g. SwUnaryTransformer, SwBinaryTransformer etc.
trait TempDirectoryTest extends BeforeAndAfterAll

Trait that creates a temporary directory before all tests and deletes it after all.

Trait that creates a temporary directory before all tests and deletes it after all. Modified version of Spark 2.x test suite trait
trait TestCommon extends Matchers with Assertions

Trait with test commons such as Spec and Resource functions
sealed trait TestOpVectorColumnType extends AnyRef

Represents a type of column associated with a feature in a vector metadata.
trait TestSparkContext extends TempDirectoryTest with TestCommon

Trait to enable Spark context for tests Modified version of Spark 2.x test suite trait
trait TestSparkStreamingContext extends TestSparkContext

Trait to enable Spark streaming context for tests

Value Members

object PassengerFeaturesTest

object SparkMatchers

Convenient matchers for spark functionality.

Examples of usage:

 in(myDF) allOf "MyNumericColumn" should beBetween(-1, 1)

 in(myDF) someOf "MyStringColumn" should (
   (x: String) => (x contains "Country") || (x contains "State")
)

in(myDF) noneOf "UserName" shouldContain "Snowden"

object TestFeatureBuilder extends Product with Serializable

Test Feature Builder is a factory for creating datasets and features for tests
object TestOpVectorColumnType
object TestOpVectorMetadataBuilder

Helps construct OpVectorMetadata as expected from a stage
object TestOpWorkflowBuilder

A factory to create OpWorkflow instances for tests

package test

Type Members

trait FeatureAsserts extends Matchers

trait FeatureTestBase extends TestSparkContext with PropertyChecks

trait FeatureTypeEquality[O <: FeatureType] extends AnyRef

final class Gender extends Enum[Gender]

class JoinTestData extends SpecificRecordBase with SpecificRecord

abstract class OpEstimatorSpec[O <: FeatureType, ModelType <: Model[ModelType] with OpPipelineStage[O] with OpTransformer, EstimatorType <: Estimator[ModelType] with OpPipelineStage[O]] extends OpPipelineStageSpec[O, EstimatorType]

trait OpPipelineStageAsserts extends AppendedClues

abstract class OpPipelineStageSpec[O <: FeatureType, StageType <: OpPipelineStage[O]] extends FlatSpec with FeatureTypeEquality[O] with TestSparkContext with OpPipelineStageAsserts

abstract class OpTransformerSpec[O <: FeatureType, TransformerType <: OpPipelineStage[O] with Transformer with OpTransformer] extends OpPipelineStageSpec[O, TransformerType] with TransformerSpecCommon[O, TransformerType]

class Passenger extends SpecificRecordBase with SpecificRecord

class PassengerCSV extends SpecificRecordBase with SpecificRecord

trait PassengerFeaturesTest extends AnyRef

class PassengerProfile extends SpecificRecordBase with SpecificRecord

trait PassengerSparkFixtureTest extends TestSparkContext with PassengerFeaturesTest

case class Prestige(education: Double, income: Double, women: Double, prestige: Double) extends Product with Serializable

trait PrestigeData extends AnyRef

class SparkExample extends SpecificRecordBase with SpecificRecord

class SparkExampleJoin extends SpecificRecordBase with SpecificRecord

trait TempDirectoryTest extends BeforeAndAfterAll

trait TestCommon extends Matchers with Assertions

sealed trait TestOpVectorColumnType extends AnyRef

trait TestSparkContext extends TempDirectoryTest with TestCommon

trait TestSparkStreamingContext extends TestSparkContext

Value Members

object PassengerFeaturesTest

object SparkMatchers

object TestFeatureBuilder extends Product with Serializable

object TestOpVectorColumnType

object TestOpVectorMetadataBuilder

object TestOpWorkflowBuilder

Ungrouped