distributed_computing:data_processing:spark:log4j [AE Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====== Log4j & Spark ======


Custom properties file for log4j:

<code>
./spark-shell --driver-java-options "-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"
</code>

<code>
--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"
--conf "spark.driver.extraJavaOptions=–Dlog4j.configuration=file:///spark/conf/log4j.properties"
</code>


<code>
spark-submit
  --master spark://127.0.0.1:7077
  --driver-java-options "-Dlog4j.configuration=file:/path/to/log4j-driver.properties -Ddm.logging.level=DEBUG"
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/path/to/log4j-executor.properties -Ddm.logging.name=myapp -Ddm.logging.level=DEBUG"
  ...
</code>
Idea:

Set variables, which are used in the log4j properties.

<code>${sys:PROJECT_HOME}</code>
<code>log4j.appender.rolling.file=/var/log/spark/${dm.logging.name}.log</code>

Still a problem, if a user tries to use a own logger.