Class

com.salesforce.op.readers

AggregateAvroReader

Related Doc: package readers

Permalink

class AggregateAvroReader[T <: GenericRecord] extends AvroReader[T] with AggregateDataReader[T]

Data reader for avro events where there might be multiple records for a given key.

Linear Supertypes
AggregateDataReader[T], AggregatedReader[T], AvroReader[T], DataReader[T], ReaderKey[T], Reader[T], ReaderType[T], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AggregateAvroReader
  2. AggregateDataReader
  3. AggregatedReader
  4. AvroReader
  5. DataReader
  6. ReaderKey
  7. Reader
  8. ReaderType
  9. Serializable
  10. Serializable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AggregateAvroReader(readPath: Option[String], key: (T) ⇒ String, aggregateParams: AggregateParams[T])(implicit arg0: ClassTag[T], arg1: scala.reflect.api.JavaUniverse.WeakTypeTag[T])

    Permalink

    readPath

    default path to data

    key

    function for extracting key from avro record

    aggregateParams

    aggregate params

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val aggregateParams: AggregateParams[T]

    Permalink

    aggregate params

    aggregate params

    Definition Classes
    AggregateAvroReaderAggregateDataReader
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def fullTypeName: String

    Permalink

    Full reader input type name

    Full reader input type name

    returns

    full input type name

    Definition Classes
    ReaderType
  11. final def generateDataFrame(rawFeatures: Array[OPFeature], opParams: OpParams = new OpParams())(implicit spark: SparkSession): DataFrame

    Permalink

    Generate the Dataframe that will be used in the OpPipeline calling this method

    Generate the Dataframe that will be used in the OpPipeline calling this method

    rawFeatures

    features to generate from the dataset read in by this reader

    opParams

    op parameters

    spark

    spark instance to do the reading and conversion from RDD to Dataframe

    returns

    A Dataframe containing columns with all of the raw input features expected by the pipeline

    Definition Classes
    AggregatedReaderDataReaderReader
  12. final def generateRow(key: String, records: Seq[T], rawFeatures: Array[OPFeature]): Option[Row]

    Permalink
    Definition Classes
    AggregateDataReaderAggregatedReader
  13. def generateRow(key: String, record: T, rawFeatures: Array[OPFeature]): Option[Row]

    Permalink
    Attributes
    protected
    Definition Classes
    DataReader
  14. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  15. final def getFinalReadPath(params: OpParams): String

    Permalink

    Default method for extracting the path used in read method.

    Default method for extracting the path used in read method. The path is taken in the following order of priority: readerPath, params

    returns

    final path to use

    Attributes
    protected
    Definition Classes
    DataReader
  16. def getGenStage[I](f: OPFeature): FeatureGeneratorStage[I, _ <: FeatureType]

    Permalink
    Attributes
    protected[com.salesforce.op]
    Definition Classes
    Reader
  17. final def getReaderParams(opParams: OpParams): Option[ReaderParams]

    Permalink

    Default method for extracting this reader's parameters from readerParams in OpParams

    Default method for extracting this reader's parameters from readerParams in OpParams

    opParams

    contains map of reader type to ReaderParams instances

    returns

    ReaderParams instance if it exists

    Definition Classes
    ReaderType
  18. final def getSchema(rawFeatures: Array[OPFeature]): StructType

    Permalink

    Derives DataFrame schema for raw features.

    Derives DataFrame schema for raw features.

    rawFeatures

    feature array representing raw feature-data

    returns

    a StructType instance

    Attributes
    protected
    Definition Classes
    DataReader
  19. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  20. final def innerJoin[U](other: DataReader[U], joinKeys: JoinKeys = JoinKeys()): JoinedDataReader[T, U]

    Permalink

    Inner join

    Inner join

    U

    Type of data read by right data reader

    other

    reader from right side of join

    joinKeys

    join keys to use

    returns

    joined reader

    Definition Classes
    Reader
  21. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  22. final def join[U](other: DataReader[U], joinType: JoinType, joinKeys: JoinKeys = JoinKeys()): JoinedDataReader[T, U]

    Permalink

    Join readers

    Join readers

    U

    Type of data read by right data reader

    other

    reader from right side of join

    joinType

    type of join to perform

    joinKeys

    join keys to use

    returns

    joined reader

    Attributes
    protected
    Definition Classes
    Reader
  23. val key: (T) ⇒ String

    Permalink

    function for extracting key from avro record

    function for extracting key from avro record

    Definition Classes
    AvroReader → ReaderKey
  24. final def leftOuterJoin[U](other: DataReader[U], joinKeys: JoinKeys = JoinKeys()): JoinedDataReader[T, U]

    Permalink

    Left Outer join

    Left Outer join

    U

    Type of data read by right data reader

    other

    reader from right side of join

    joinKeys

    join keys to use

    returns

    joined reader

    Definition Classes
    Reader
  25. final def maybeRepartition(data: Dataset[T], params: OpParams): Dataset[T]

    Permalink

    Function to repartition the data based on the op params of this reader

    Function to repartition the data based on the op params of this reader

    data

    dataset

    params

    op params

    returns

    maybe repartitioned dataset

    Attributes
    protected
    Definition Classes
    DataReader
  26. final def maybeRepartition(data: RDD[T], params: OpParams): RDD[T]

    Permalink

    Function to repartition the data based on the op params of this reader

    Function to repartition the data based on the op params of this reader

    data

    rdd

    params

    op params

    returns

    maybe repartitioned rdd

    Attributes
    protected
    Definition Classes
    DataReader
  27. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  28. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  29. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  30. final def outerJoin[U](other: DataReader[U], joinKeys: JoinKeys = JoinKeys()): JoinedDataReader[T, U]

    Permalink

    Outer join

    Outer join

    U

    Type of data read by right data reader

    other

    reader from right side of join

    joinKeys

    join keys to use

    returns

    joined reader

    Definition Classes
    Reader
  31. def read(params: OpParams = new OpParams())(implicit spark: SparkSession): Either[RDD[T], Dataset[T]]

    Permalink

    Function which reads raw data from specified location to use in Dataframe creation, i.e.

    Function which reads raw data from specified location to use in Dataframe creation, i.e. generateDataFrame fun. This function returns either RDD or Dataset of the type specified by this reader. It can be overwritten to carry out any special logic required for the reader (ie filters or joins needed to produce the specified reader type).

    params

    parameters used to carry out specialized logic in reader (passed in from workflow)

    spark

    spark instance to do the reading and conversion from RDD to Dataframe

    returns

    either RDD or Dataset of type T

    Definition Classes
    AvroReaderDataReader
  32. final def readDataset(params: OpParams = new OpParams())(implicit sc: SparkSession, encoder: Encoder[T]): Dataset[T]

    Permalink

    Function which reads raw data from specified location to use in Dataframe creation, i.e.

    Function which reads raw data from specified location to use in Dataframe creation, i.e. generateDataFrame fun. This function returns a Dataset of the type specified by this reader.

    params

    parameters used to carry out specialized logic in reader (passed in from workflow)

    sc

    spark session

    returns

    Dataset of type T

    Definition Classes
    DataReader
  33. val readPath: Option[String]

    Permalink

    default path to data

    default path to data

    Definition Classes
    AvroReaderDataReader
  34. final def readRDD(params: OpParams = new OpParams())(implicit sc: SparkSession): RDD[T]

    Permalink

    Function which reads raw data from specified location to use in Dataframe creation, i.e.

    Function which reads raw data from specified location to use in Dataframe creation, i.e. generateDataFrame fun. This function returns a RDD of the type specified by this reader.

    params

    parameters used to carry out specialized logic in reader (passed in from workflow)

    sc

    spark session

    returns

    RDD of type T

    Definition Classes
    DataReader
  35. implicit val seqEnc: Encoder[Seq[T]]

    Permalink
    Definition Classes
    AggregatedReader
  36. implicit val strEnc: Encoder[String]

    Permalink
    Definition Classes
    AggregatedReader
  37. final def subReaders: Seq[DataReader[_]]

    Permalink

    All the reader's sub readers (used in joins)

    All the reader's sub readers (used in joins)

    returns

    sub readers

    Definition Classes
    DataReaderReader
  38. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  39. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  40. implicit val tupEnc: Encoder[(String, Seq[T])]

    Permalink
    Definition Classes
    AggregatedReader
  41. final def typeName: String

    Permalink

    Short reader input type name

    Short reader input type name

    returns

    short reader input type name

    Definition Classes
    ReaderType
  42. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  43. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  44. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  45. implicit val wtt: scala.reflect.api.JavaUniverse.WeakTypeTag[T]

    Permalink

    Reader type tag

    Reader type tag

    Definition Classes
    AvroReader → ReaderType

Inherited from AggregateDataReader[T]

Inherited from AggregatedReader[T]

Inherited from AvroReader[T]

Inherited from DataReader[T]

Inherited from ReaderKey[T]

Inherited from Reader[T]

Inherited from ReaderType[T]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped