Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package precisely
    Definition Classes
    com
  • package bigdata
    Definition Classes
    precisely
  • package li
    Definition Classes
    bigdata
  • package spark
    Definition Classes
    li
  • package api

    Provides classes and code snippets to simplify usage of Spatial APIs in Location Intelligence SDK For Big Data.

    Provides classes and code snippets to simplify usage of Spatial APIs in Location Intelligence SDK For Big Data.
    For more information regarding the Usage Guide or API Docs, follow the below links:
    Geo Spatial SDKs
    Location Intelligence SDK For Big Data User Guide

    • Starting a spark session:
    import org.apache.spark.sql.SparkSession
// The config is required to set run the legacy UDFs used in the Spatial APIs.
val session = SparkSession.builder.appName("Example")
    .master("yarn")
    .getOrCreate;
    • Using DownloadManager capable of downloading remote resources (files like TAB, SHAPE, etc.) to a node local path. Supports downloading from HDFS, S3, Google Storage or Local (default).

      NOTE: For Downloading from S3 or Google Storage, you need to either have respective environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or provide configurations in spark session.
    import com.pb.downloadmanager.api.downloaders.LocalFilePassthroughDownloader
import com.pb.downloadmanager.api.downloaders.hadoop.{HDFSDownloader, S3Downloader, GoogleDownloader}
import com.pb.downloadmanager.api.{DownloadManagerBuilder}
val downloadManager = new DownloadManagerBuilder("/home/hadoop/data")
  .addDownloader(new S3Downloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new GoogleDownloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new HDFSDownloader(session.sparkContext.hadoopConfiguration))
  .addDownloader(new LocalFilePassthroughDownloader())
  .build()


    • PointInPolygon Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val pointInPolygonDF = SpatialAPI.pointInPolygon(inputDF = inputDF,
            tableFileType = tabFileType,
            tableFilePath = tableFilePath,
            tableFileName = tableFileName,
            libraries = libraries,
            longitude = longitude, latitude = latitude,
            includeEmptySearchResults = true,
            outputFields = outputFields,
            downloadManager = downloadManager
)

// Defaults:
// downloadManager = null, libraries = null, includeEmptySearchResults = true


    • SearchNearest Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val searchNearestDF = SpatialAPI.searchNearest(inputDF = fabricDF,
            tableFileType = tableFileType,
            tableFilePath = tableFilePath,
            tableFileName = tableFileName,
            libraries = libraries,
            maxCandidates = maxCandidates,
            distanceValue = distanceValue,
            distanceUnit = distanceUnit,
            distanceColumnName = distanceColumnName,
            geometryStringType = geometryStringType,
            geometryColumnName = geometryColumnName,
            includeEmptySearchResults = includeEmptySearchResults,
            outputFields = outputFields,
            downloadManager = downloadManager
)

// Defaults:
// distanceColumnName = "distance", downloadManager = null, libraries = null, includeEmptySearchResults = true, maxCandidates = 1000


    • JoinByDistance Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
import com.precisely.bigdata.li.spark.api.util.DistanceJoinOption.DistanceJoinOption
import com.precisely.bigdata.li.spark.api.util.LimitMethods
val joinedDF = SpatialAPI.joinByDistance(df1 = df1,
              df2 = df2,
              df1Longitude = longitude1,
              df1Latitude = latitude1,
              df2Longitude = longitude2,
              df2Latitude = latitude2,
              searchRadius = searchRadius,
              distanceUnit = distanceUnit,
              geoHashPrecision = geoHashPrecision,
              options = Map(
                        DistanceJoinOption.DistanceColumnName -> distanceColumnName,
                        DistanceJoinOption.LimitMatches -> limit,
                        DistanceJoinOption.LimitMethod -> LimitMethods.RowNumber
              )
)

// Defaults:
// geoHashPrecision = 7, options = null


    • HexagonGeneration Operation:
    import com.precisely.bigdata.li.spark.api.SpatialAPI
val hexGenDF = SpatialAPI.generateHexagon(
              sparkSession = session,
              minLongitude = minLongitude,
              minLatitude = minLatitude,
              maxLongitude = maxLongitude,
              maxLatitude = maxLatitude,
              hexLevel = hexLevel,
              containerLevel = containerLevel,
              numOfPartitions = numOfPartitions,
              maximumNumOfRowsPerPartition = maxNumberOfRows
)

// Defaults:
// hexLevel = 1, containerLevel = 1, numOfPartitions = 1, maximumNumOfRowsPerPartition = 1


    • Registering the SQL Functions as UDFs:
    import com.precisely.bigdata.li.spark.api.udf.SQLRegistrator
SQLRegistrator.registerAll();
inputDF.createOrReplaceTempView("inputTable")
val pointGeometry = spark.sql("SELECT ST_Point(X, Y, 'epsg:4326') as point_geom, * from inputTable")
val wktGeometry = spark.sql("SELECT ST_GeomFromWKT(WKT) as geom, * from inputTable")

// Currently Available SQL Functions
// ST_Point, ST_GeomFromWKT, ST_GeomFromWKB, ST_GeomFromKML, ST_GeomFromGeoJSON,
// ST_ToGeoJSON, ST_ToKML, ST_ToWKB, ST_ToWKT, ST_Buffer, ST_Union, ST_Transform,
// ST_Intersection, ST_ConvexHull, ST_Within, ST_Disjoint, ST_Intersects,
// ST_IsNullGeom, ST_Overlaps, ST_GeoHash, ST_GeoHashBoundary, ST_HexHash, ST_HexHashBoundary,
// ST_SquareHash, ST_SquareHashBoundary, ST_X, ST_XMax, ST_XMin, ST_Y, ST_YMax, ST_YMin,
// ST_Area, ST_Distance, ST_Length, ST_Perimeter
    Definition Classes
    spark
  • package feature
    Definition Classes
    api
  • AttributeConverter
  • AttributeHandler
  • Binary2Binary
  • Boolean2Boolean
  • Byte2Byte
  • Char2String
  • Date2Date
  • DateTime2Date
  • DateTimeZone2Date
  • DbaseDecimal2Decimal
  • Double2Double
  • FeatureGeometry2GeoJson
  • FeatureGeometry2Kml
  • FeatureGeometry2Wkb
  • FeatureGeometry2Wkt
  • Float2Float
  • Int2Integer
  • Long2Long
  • SearchImplicits
  • Short2Short
  • SparkAttribute
  • String2String
  • Style2String
  • Time2Date
  • Timespan2Long
  • package table
    Definition Classes
    api
  • package udf
    Definition Classes
    api
  • package util
    Definition Classes
    api

package feature

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. abstract class AttributeConverter extends Serializable

    This class is used to convert an AttributeType, from an IFeature, to a DataType to use in Spark operations.

  2. class AttributeHandler extends Serializable
  3. trait SearchImplicits extends Serializable
  4. case class SparkAttribute extends Product with Serializable

    A SparkAttribute is meant to hold a single attribute value.

    A SparkAttribute is meant to hold a single attribute value. Null values must use SparkAttribute.setNull

Value Members

  1. object AttributeHandler extends Serializable

    The AttributeHandler contains AttributeConverters to use when converting features of various AttributeTypes to rows of various DataTypes.

    The AttributeHandler contains AttributeConverters to use when converting features of various AttributeTypes to rows of various DataTypes. The default list of converters is appropriate for most cases. Use the constructors to override any converters.

  2. case object Binary2Binary extends AttributeConverter with Product with Serializable
  3. case object Boolean2Boolean extends AttributeConverter with Product with Serializable
  4. case object Byte2Byte extends AttributeConverter with Product with Serializable
  5. case object Char2String extends AttributeConverter with Product with Serializable
  6. case object Date2Date extends AttributeConverter with Product with Serializable
  7. case object DateTime2Date extends AttributeConverter with Product with Serializable
  8. case object DateTimeZone2Date extends AttributeConverter with Product with Serializable
  9. case object DbaseDecimal2Decimal extends AttributeConverter with Product with Serializable
  10. case object Double2Double extends AttributeConverter with Product with Serializable
  11. case object FeatureGeometry2GeoJson extends AttributeConverter with Product with Serializable

    Converts an IFeatureGeometry to a GeoJson string.

  12. case object FeatureGeometry2Kml extends AttributeConverter with Product with Serializable

    Converts an IFeatureGeometry to a KML string.

  13. case object FeatureGeometry2Wkb extends AttributeConverter with Product with Serializable

    Converts an IFeatureGeometry to a Well Known Binary byte array.

  14. case object FeatureGeometry2Wkt extends AttributeConverter with Product with Serializable

    Converts an IFeatureGeometry to a Well Known Text string.

  15. case object Float2Float extends AttributeConverter with Product with Serializable
  16. case object Int2Integer extends AttributeConverter with Product with Serializable
  17. case object Long2Long extends AttributeConverter with Product with Serializable
  18. object SearchImplicits extends SearchImplicits
  19. case object Short2Short extends AttributeConverter with Product with Serializable
  20. object SparkAttribute extends Serializable
  21. case object String2String extends AttributeConverter with Product with Serializable
  22. case object Style2String extends AttributeConverter with Product with Serializable
  23. case object Time2Date extends AttributeConverter with Product with Serializable
  24. case object Timespan2Long extends AttributeConverter with Product with Serializable

Ungrouped