Log4j & Spark

Custom properties file for log4j:

./spark-shell --driver-java-options "-Dlog4j.configuration=file:///etc/spark/my-conf/log4j.warnonly.properties"

--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=/tmp/log4j.properties"
--conf "spark.driver.extraJavaOptions=–Dlog4j.configuration=file:///spark/conf/log4j.properties"

spark-submit
  --master spark://127.0.0.1:7077
  --driver-java-options "-Dlog4j.configuration=file:/path/to/log4j-driver.properties -Ddm.logging.level=DEBUG"
  --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/path/to/log4j-executor.properties -Ddm.logging.name=myapp -Ddm.logging.level=DEBUG"
  ...

Idea:

Set variables, which are used in the log4j properties.

${sys:PROJECT_HOME}

log4j.appender.rolling.file=/var/log/spark/${dm.logging.name}.log

Still a problem, if a user tries to use a own logger.