Setup for Raster Processing in macOS or Linux#

The BDT raster processors require additional libararies not included with the BDT jar. This guide describes how to pull these libraries into your environment.

  1. For spark to properly install the required libraries, an ivySettings.xml file must be provided. See the snippet below for what this file should look like. Save this file to an easily accessible place on your machine.

<ivysettings>
    <settings defaultResolver="chain"/>
    <resolvers>
        <chain name="chain">
            <ibiblio name="osgeo" m2compatible="true" root="https://repo.osgeo.org/repository/release" />
            <ibiblio name="central" m2compatible="true" root="https://repo1.maven.org/maven2/" />
        </chain>
    </resolvers>
</ivysettings>

  1. Use this init script for launching a jupyter notebook with BDT:

pyspark \
  --master local[*] \
  --driver-java-options "-XX:+UseCompressedOops -Djava.awt.headless=true" \
  --conf spark.executor.extraJavaOptions="-XX:+UseCompressedOops -Djava.awt.headless=true" \
  --conf spark.sql.execution.arrow.pyspark.enabled=true \
  --conf spark.submit.pyFiles="/<path>/<to>/<bdt_zip>" \
  --jars /<path>/<to>/<bdt_jar> \
  --conf spark.jars.packages="org.geotools:gt-main:24.6,org.geotools:gt-geotiff:24.6,org.geotools:gt-epsg-hsql:24.6" \
  --conf spark.jars.excludes="org.scala-lang:scala-reflect,com.fasterxml.jackson.core:jackson-core" \
  --conf spark.jars.ivySettings="file:///<path-to-ivySettings.xml>"

This init script adds the required libraries for raster processing. Adjust the path to the jar, zip, and ivySettings files to match where they are on your machine.