Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package precisely
    Definition Classes
    com
  • package bigdata
    Definition Classes
    precisely
  • package li
    Definition Classes
    bigdata
  • package spark
    Definition Classes
    li
  • package api

    Provides classes and code snippets to simplify usage of Spatial APIs in Location Intelligence SDK For Big Data.

    Provides classes and code snippets to simplify usage of Spatial APIs in Location Intelligence SDK For Big Data.
    For more information regarding the Usage Guide or API Docs, follow the below links:
    Geo Spatial SDKs
    Location Intelligence SDK For Big Data User Guide

    • Starting a spark session:
    import org.apache.spark.sql.SparkSession
// The config is required to set run the legacy UDFs used in the Spatial APIs.
val session = SparkSession.builder.appName("Example")
    .master("yarn")
    .getOrCreate;
    • Using DownloadManager capable of downloading remote resources (files like TAB, SHAPE, etc.) to a node local path. Supports downloading from HDFS, S3, Google Storage or Local (default).

      NOTE: For Downloading from S3 or Google Storage, you need to either have respective environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or provide configurations in spark session.
    import com.pb.downloadmanager.api.downloaders.LocalFilePassthroughDownloader
import com.pb.downloadmanager.api.downloaders.hadoop.{HDFSDownloader, S3Downloader, GoogleDownloader}
import com.pb.downloadmanager.api.{DownloadManagerBuilder}
val downloadManager = new DownloadManagerBuilder("/home/hadoop/data")
  .addDownloader(new S3Downloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new GoogleDownloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new HDFSDownloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new LocalFilePassthroughDownloader())
  .build()


    • PointInPolygon Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val pointInPolygonDF = SpatialAPI.pointInPolygon(inputDF = inputDF,
            tableFileType = tabFileType,
            tableFilePath = tableFilePath,
            tableFileName = tableFileName,
            libraries = libraries,
            longitude = longitude, latitude = latitude,
            includeEmptySearchResults = true,
            outputFields = outputFields,
            downloadManager = downloadManager
)

// Defaults:
// downloadManager = null, libraries = null, includeEmptySearchResults = true


    • SearchNearest Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val searchNearestDF = SpatialAPI.searchNearest(inputDF = fabricDF,
            tableFileType = tableFileType,
            tableFilePath = tableFilePath,
            tableFileName = tableFileName,
            libraries = libraries,
            maxCandidates = maxCandidates,
            distanceValue = distanceValue,
            distanceUnit = distanceUnit,
            distanceColumnName = distanceColumnName,
            geometryStringType = geometryStringType,
            geometryColumnName = geometryColumnName,
            includeEmptySearchResults = includeEmptySearchResults,
            outputFields = outputFields,
            downloadManager = downloadManager
)

// Defaults:
// distanceColumnName = "distance", downloadManager = null, libraries = null, includeEmptySearchResults = true, maxCandidates = 1000


    • JoinByDistance Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
import com.precisely.bigdata.li.spark.api.util.DistanceJoinOption.DistanceJoinOption
import com.precisely.bigdata.li.spark.api.util.LimitMethods
val joinedDF = SpatialAPI.joinByDistance(df1 = df1,
              df2 = df2,
              df1Longitude = longitude1,
              df1Latitude = latitude1,
              df2Longitude = longitude2,
              df2Latitude = latitude2,
              searchRadius = searchRadius,
              distanceUnit = distanceUnit,
              geoHashPrecision = geoHashPrecision,
              options = Map(
                        DistanceJoinOption.DistanceColumnName -> distanceColumnName,
                        DistanceJoinOption.LimitMatches -> limit,
                        DistanceJoinOption.LimitMethod -> LimitMethods.RowNumber
              )
)

// Defaults:
// geoHashPrecision = 7, options = null


    • HexagonGeneration Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val hexGenDF = SpatialAPI.generateHexagon(
              sparkSession = session,
              minLongitude = minLongitude,
              minLatitude = minLatitude,
              maxLongitude = maxLongitude,
              maxLatitude = maxLatitude,
              hexLevel = hexLevel,
              containerLevel = containerLevel,
              numOfPartitions = numOfPartitions,
              maximumNumOfRowsPerPartition = maxNumberOfRows
)

// Defaults:
// hexLevel = 1, containerLevel = 1, numOfPartitions = 1, maximumNumOfRowsPerPartition = 1


    • Registering the SQL Functions as UDFs:
    import com.precisely.bigdata.li.spark.api.udf.SQLRegistrator
SQLRegistrator.registerAll();
inputDF.createOrReplaceTempView("inputTable")
val pointGeometry = spark.sql("SELECT ST_Point(X, Y, 'epsg:4326') as point_geom, * from inputTable")
val wktGeometry = spark.sql("SELECT ST_GeomFromWKT(WKT) as geom, * from inputTable")

// Currently Available SQL Functions
// ST_Point, ST_GeomFromWKT, ST_GeomFromWKB, ST_GeomFromKML, ST_GeomFromGeoJSON,
// ST_ToGeoJSON, ST_ToKML, ST_ToWKB, ST_ToWKT, ST_Buffer, ST_Union, ST_Transform,
// ST_Intersection, ST_ConvexHull, ST_Within, ST_Disjoint, ST_Intersects,
// ST_IsNullGeom, ST_Overlaps, ST_GeoHash, ST_GeoHashBoundary, ST_HexHash, ST_HexHashBoundary,
// ST_SquareHash, ST_SquareHashBoundary, ST_X, ST_XMax, ST_XMin, ST_Y, ST_YMax, ST_YMin,
// ST_Area, ST_Distance, ST_Length, ST_Perimeter
    Definition Classes
    spark
  • package feature
    Definition Classes
    api
  • package table
    Definition Classes
    api
  • FileTableBuilder
  • GeodatabaseTableBuilder
  • NativeTableBuilder
  • ShapeTableBuilder
  • TableBuilder
  • package udf
    Definition Classes
    api
  • package util
    Definition Classes
    api

package table

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. abstract class FileTableBuilder[A <: FileTableBuilder[A]] extends Serializable
    Attributes
    protected
  2. class GeodatabaseTableBuilder extends FileTableBuilder[GeodatabaseTableBuilder]

    Builder for Geodatabase tables.

    Builder for Geodatabase tables. Use TableBuilder to get this builder.

    returns

    A builder with a reference to the location of the geodatabase and table name.

    Attributes
    protected
  3. class NativeTableBuilder extends FileTableBuilder[NativeTableBuilder]

    Builder for Native TAB tables.

    Builder for Native TAB tables. Use TableBuilder to get this builder.

    returns

    A builder with a reference to the location of the tab files.

    Attributes
    protected
  4. class ShapeTableBuilder extends FileTableBuilder[ShapeTableBuilder]

    Builder for Shapefile tables.

    Builder for Shapefile tables. Use TableBuilder to get this builder.

    returns

    A builder with a reference to the location of the table files.

    Attributes
    protected

Value Members

  1. object TableBuilder

    Contains Builders for all supported table types to be used directly for Spatial operations.

Ungrouped