Class

com.salesforce.op.utils.spark

OpSparkListener

Related Doc: package spark

Permalink

class OpSparkListener extends SparkListener

Logs & collects metrics upon completion of Spark application, jobs, stages

Linear Supertypes
SparkListener, SparkListenerInterface, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OpSparkListener
  2. SparkListener
  3. SparkListenerInterface
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OpSparkListener(appName: String, appId: String, runType: String, customTagName: Option[String], customTagValue: Option[String], logStageMetrics: Boolean, collectStageMetrics: Boolean)

    Permalink

    appName

    application name

    appId

    application id

    runType

    OpWorkflowRunType

    customTagName

    tag name printed on log lines

    customTagValue

    the value for the tag printed on log lines

    logStageMetrics

    should log metrics for every stage Note: can increase logging significantly if app has too many stages.

    collectStageMetrics

    should collect metrics for every stage Note: can increase memory usage on the driver if app has too many stages.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val appId: String

    Permalink

    application id

  5. val appName: String

    Permalink

    application name

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. val collectStageMetrics: Boolean

    Permalink

    should collect metrics for every stage Note: can increase memory usage on the driver if app has too many stages.

  9. val customTagName: Option[String]

    Permalink

    tag name printed on log lines

  10. val customTagValue: Option[String]

    Permalink

    the value for the tag printed on log lines

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  15. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  17. val logPrefix: String

    Permalink
  18. val logStageMetrics: Boolean

    Permalink

    should log metrics for every stage Note: can increase logging significantly if app has too many stages.

  19. def metrics: AppMetrics

    Permalink

    All the metrics computed by the spark listener

  20. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  21. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  22. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  23. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit

    Permalink
    Definition Classes
    OpSparkListener → SparkListener → SparkListenerInterface
  24. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit

    Permalink
    Definition Classes
    OpSparkListener → SparkListener → SparkListenerInterface
  25. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  26. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  27. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  28. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  29. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  30. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  31. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  32. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  33. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  34. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit

    Permalink
    Definition Classes
    OpSparkListener → SparkListener → SparkListenerInterface
  35. def onJobStart(jobStart: SparkListenerJobStart): Unit

    Permalink
    Definition Classes
    OpSparkListener → SparkListener → SparkListenerInterface
  36. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  37. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  38. def onOtherEvent(event: SparkListenerEvent): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  39. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  40. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit

    Permalink
    Definition Classes
    OpSparkListener → SparkListener → SparkListenerInterface
  41. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  42. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  43. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  44. def onTaskStart(taskStart: SparkListenerTaskStart): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  45. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit

    Permalink
    Definition Classes
    SparkListener → SparkListenerInterface
  46. val runType: String

    Permalink

    OpWorkflowRunType

  47. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  48. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  49. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  51. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped