Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package precisely
    Definition Classes
    com
  • package bigdata
    Definition Classes
    precisely
  • package addressing
    Definition Classes
    bigdata
  • package spark
    Definition Classes
    addressing
  • package api

    This section provides code snippets to simplify usage of Geo Addressing SDK for Big Data.

    This section provides code snippets to simplify usage of Geo Addressing SDK for Big Data.


    Geocode Operation Example

    import org.apache.spark.sql.functions._
import com.pb.downloadmanager.api._
import com.pb.downloadmanager.api.downloaders._
import com.pb.downloadmanager.api.downloaders.hadoop._
import com.precisely.addressing.v1.model._
import com.precisely.bigdata.addressing.spark.api._
import com.precisely.addressing.v1._

// DownloadManager copies all the remote resources (HDFS, S3, Google Storage, etc.) to the
// provided local path (e.g. ./downloads) of every worker node.
// This happens in runtime during the first call. Subsequent calls skips downloading process.
val downloadManager = new DownloadManagerBuilder("./downloads")
  .addDownloader(new S3Downloader(spark.sparkContext.hadoopConfiguration))
  .addDownloader(new GoogleDownloader(spark.sparkContext.hadoopConfiguration))
  .addDownloader(new HDFSDownloader(spark.sparkContext.hadoopConfiguration))
  .addDownloader(new LocalFilePassthroughDownloader())
  .build()

// Remove the fields which are not required.
val outputFields = Seq(
    "location.feature.geometry.coordinates.x as LON",
    "location.feature.geometry.coordinates.y as LAT",
    "address.formattedAddress as fullAddress",
    "address.city.longName as locality",
    "address.country.isoAlpha2Code as countryshortname",
    "address.country.name as country",
    "address.placeName as placename",
    "address.admin1.shortName as administrativearealevel1",
    "address.admin2.longName as administrativearealevel2",
    "address.suburb as suburb",
    "address.borough as borough",
    "address.postalCode as postalcode",
    "address.postalCodeExt as postalcodesuffix",
    "customFields['PB_KEY'] as sourceplaceid",
    "address.addressNumber as AddressNumber",
    "address.street as Street",
    "address.unit as Unit",
    "address.unitType as UnitType",
    "address.formattedStreetAddress as StreetAddress",
    "address.formattedLocationAddress as LocationAddress",
    "address.neighborhood as neighborhood",
    "address.floor as floor",
    "score",
    "explanation.source['label'] as label",
    "customFields['PRECISION_CODE'] as PrecisionCode",
    "customFields['MATCH_TYPE'] as MatchType",
    "customFields['LOC_CODE'] as LocationCode",
    "customFields['MATCH_CODE'] as MatchCode",
    "customFields['CBSA_NAME'] as CBSAName",
    "customFields['PREF_CITY'] as PREFCITY",
    "customFields['PB_KEY'] as PBKEY",
    "customFields['DATATYPE_NAME'] as DataTypeName"
)

// You can also use paths from HDFS or Databricks Volumes
// (e.g. hdfs:///addressingDistribution/resources or /Volumes/precisely/default/geo-addressing/addressingDistribution/resources/)
// The reference data is extracted at runtime in every worker node's extractionLocation path.
val geocodeUdf = new AddressingBuilder()
    .withResourcesLocation("s3a:///addressingDistribution/resources")
    .withDownloadManager(downloadManager)
    .withDataLocations("s3a://com.precisely.data/path_to_spd1","s3a://path_to_spd2")
    .withExtractionLocation("./extracted")
    .udfBuilder()
    .withPreferences(new PreferencesBuilder().withReturnAllInfo(true).build())
    .withOutputFields(outputFields: _*)
    .withErrorField("error")
    .withResultAsJSON("jsonOutput")
    .forGeocode()

val geocodeDF = input.withColumn("result", geocodeUdf(map(
  lit("addressLines[0]"), col("address"),
  lit("country"), col("country")
))).select("*", "result.*").drop(colName = "result")


    NOTE: If addressing.yaml is the present at the worker node, dataLocations and extractionLocation is preferred from addressing.yaml over code. In such cases where addressing.yaml is present, you should only provide resourcesLocation.


    Custom Executor Example for Multipass Geocoding:

    import com.precisely.bigdata.addressing.spark.api._
import org.apache.spark.sql.functions._

// Refer to Geocode Operation Example for Full Code.
// Use .forCustomExecutor() API
val customUdf = new AddressingBuilder()
 ...
 .forCustomExecutor(new AddressingExecutor {
      override def execute(input: RequestInput, preferences: Option[Preferences], addressing: Addressing): Response = {

          // First Pass
          val response = addressing.geocode(input.requestAddress(), preferences.orNull)
          // Validate the results.
          if (results.get(0).getCustomFields.get("PRECISION_CODE").toUpperCase().startsWith("S8"))
                return response

          // Second Pass
          // Convert MultiLine Input Address to SingleLine
          val singleLineAddress: RequestAddress = new RequestAddress()
          val filterAddress: String = Seq(inputMultiLineAddress.getStreet,
              inputMultiLineAddress.getCity,
              inputMultiLineAddress.getAdmin1,
              inputMultiLineAddress.getPostalCode
          ).filter(isNotNull).mkString(" ")
          singleLineAddress.setAddressLines(java.util.Collections.singletonList(filterAddress: String))
          singleLineAddress.setCountry(inputMultiLineAddress.getCountry)

          // Geocode with SingleLineAddress and CustomPreferences
          val singleLineResponse: Response = addressing.geocode(singleLineAddress,
                        new PreferencesBuilder().withReturnAllInfo(true).build())

          // Adding a Custom Field
          singleLineResponse.getResults.get(0).getCustomFields.put("TOTAL_MATCHES", singleLineResponse.getResults.size().toString)
          return singleLineResponse
      }
      override def execute(lookupType: LookupType, preferences: Option[Preferences], addressing: Addressing, keyValues: KeyValue*): Response = ???
      override def execute(x: Double, y: Double, country: String, preferences: Option[Preferences], addressing: Addressing): Response = ???
  })

val geocodeDF = input.withColumn("result", customUdf(map(
  lit("addressLines[0]"), col("address"),
  lit("country"), col("country")
))).select("*", "result.*").drop(colName = "result")


    Verify Operation Example:

    import com.precisely.bigdata.addressing.spark.api.AddressingBuilder
import org.apache.spark.sql.functions._

// Refer to Geocode Operation Example for Full Code.
// Use .forVerify() API
val verifyUdf = new AddressingBuilder()
  ...
  .forVerify()

val verifyDf = input.withColumn("result", verifyUdf(map(
  lit("addressLines[0]"), col("address"),
  lit("country"), col("country")
))).select("*", "result.*").drop(colName = "result")


    Reverse Geocode Operation Example:

    import com.precisely.bigdata.addressing.spark.api.AddressingBuilder
import org.apache.spark.sql.functions._

// Refer to Geocode Operation Example for Full Code.
// Use .forReverseGeocode() API
val reverseGeocodeUdf = new AddressingBuilder()
  ...
  .forReverseGeocode()

val reverseGeocodeDf = input.withColumn("result", reverseGeocodeUdf(
  col("x"), col("y"), col("country")
)).select("*", "result.*").drop(colName = "result")


    Lookup Operation Example:

    import com.precisely.bigdata.addressing.spark.api.AddressingBuilder
import org.apache.spark.sql.functions._

// Refer to Geocode Operation Example for Full Code.
// Use .forLookup() API
val lookupUdf = new AddressingBuilder()
  ...
  .forLookup()

val lookUpUdf = input.withColumn("result", lookupUdf(
  col("keyType"), col("key"), col("country")
)).select("*", "result.*").drop(colName = "result")


    Geocode Operation with SQL Queries:

    import com.precisely.bigdata.addressing.spark.api.AddressingBuilder
import org.apache.spark.sql.functions._

// Refer to Geocode Operation Example for Full Code.
// Use .forGeocode(<udf-name>) API
val geocodeUdf = new AddressingBuilder()
  ...
  .forGeocode("PreciselyGeocode")

inputDF.createOrReplaceTempView("inputTable")
val geocodeDF = spark.sql("select *, PreciselyGeocode(map('addressLines[0]', address, 'country', country)) as result from inputTable")
    Definition Classes
    spark
  • AddressingBuilder
  • AddressingExecutor
  • AddressingProvider
  • MapFunctionBuilder
  • RequestInput
  • UDFBuilder

class UDFBuilder extends AnyRef

This class allows you to build a UDF that executes an addressing operation. You can obtain an instance of this builder by starting with an AddressingBuilder.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. UDFBuilder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def forCustomExecutor(addressingExecutor: AddressingExecutor): UserDefinedFunction

    Build a UDF to execute a custom Addressing operation on an address, based on the provided configuration options.

    Build a UDF to execute a custom Addressing operation on an address, based on the provided configuration options. This UDF has a single argument that is a map of String to Column. The String keys are the address fields that are to be populated in the RequestAddress, and the column is the value to populate the fields with.

    This example uses the returned UDF as "customUdf" and adds an "addressing_result" column to the DataFrame

    val resultDataFrame = inputDataFrame.withColumn("addressing_result",
  customUdf(map(
    lit("addressNumber"), col("addressNum"),
    lit("street"), col("street"),
    lit("city"), col("city"),
    lit("admin1"), col("state"),
    lit("postalCode"), col("zip"),
    lit("country"), col("country"))
  ))
    returns

    A UDF that executes a custom Addressing operation.

  9. def forGeocode(udfName: String): UserDefinedFunction

    Register a UDF to geocode an address, based on the provided configuration options.

    Register a UDF to geocode an address, based on the provided configuration options. This UDF has a single argument that is a map of String to Column. The String keys are the address fields that are to be populated in the RequestAddress, and the column is the value to populate the fields with.

    This example uses the registered UDF as "geocode" and adds an "addressing_result" column to the DataFrame:

    val geocodeSqlOutput = spark.sql("select *, geocode(map('addressLines[0]', address, 'country', country)) as addressing_result from inputTable")
    returns

    A UDF that executes the geocode operation of the Addressing API.

  10. def forGeocode(): UserDefinedFunction

    Build a UDF to geocode an address, based on the provided configuration options.

    Build a UDF to geocode an address, based on the provided configuration options. This UDF has a single argument that is a map of String to Column. The String keys are the address fields that are to be populated in the RequestAddress, and the column is the value to populate the fields with.

    This example uses the returned UDF as "addressingUdf" and adds an "addressing_result" column to the DataFrame:

    val resultDataFrame = inputDataFrame.withColumn("addressing_result",
  addressingUdf(map(
    lit("addressNumber"), col("addressNum"),
    lit("street"), col("street"),
    lit("city"), col("city"),
    lit("admin1"), col("state"),
    lit("postalCode"), col("zip"),
    lit("country"), col("country"))
  ))
    returns

    A UDF that executes the geocode operation of the Addressing API.

  11. def forLookup(udfName: String): UserDefinedFunction

    Register a UDF to 'Lookup' a geocoded candidates when given a unique key, based on the provided configuration options.

    Register a UDF to 'Lookup' a geocoded candidates when given a unique key, based on the provided configuration options. This UDF takes two arguments, lookupType Key type- it can be GNAF_PID or PBKEY, Key to search and country.

    This example uses the registered UDF as "lookupUdf" and adds an "addressing_result" column to the DataFrame:

    val lookupSqlOutput = spark.sql("select *, lookupUdf('PB_KEY',key,'USA') as addressing_result from inputTable")
    returns

    A UDF that provides a Key Lookup operation.

  12. def forLookup(): UserDefinedFunction

    Build a UDF to 'lookup' a geocoded candidates when given a unique key, based on the provided configuration options.

    Build a UDF to 'lookup' a geocoded candidates when given a unique key, based on the provided configuration options. This UDF takes two arguments, lookupType Key type- it can be GNAF_PID or PBKEY, Key to search and country.

    This example uses the returned UDF as "addressingUdf" and adds an "addressing_result" column to the DataFrame:

    val resultDataFrame = inputDataFrame.withColumn("addressing_result",
  addressingUdf(lit("PB_KEY"),
  col("key"),
  lit("USA")
  ))
    returns

    A UDF that provides a Key Lookup operation.

  13. def forReverseGeocode(udfName: String): UserDefinedFunction

    Register a UDF to reverse geocode a location, based on the provided configuration options.

    Register a UDF to reverse geocode a location, based on the provided configuration options. This UDF takes X and Y coordinates of location to reverse geocode with optional country parameter.

    This example uses the registered UDF as "reverseGeocode" and adds an "addressing_result" column to the DataFrame:

    val reverseGeocodeSqlOutput = spark.sql("select *, reverseGeocode(x, y, country) as addressing_result from inputTable")
    returns

    A UDF that executes the reverse geocode operation of the Addressing API.

  14. def forReverseGeocode(): UserDefinedFunction

    Build a UDF to reverse geocode a location, based on the provided configuration options.

    Build a UDF to reverse geocode a location, based on the provided configuration options. This UDF takes X and Y coordinates of location to reverse geocode with optional country parameter.

    This example uses the returned UDF as "addressingUdf" and adds an "addressing_result" column to the DataFrame:

    val resultDataFrame = inputDataFrame.withColumn("addressing_result",
  addressingUdf(
    col("x"),
    col("y"),
    col("country"))
  )
    returns

    A UDF that executes the reverse geocode operation of the Addressing API.

  15. def forVerify(udfName: String): UserDefinedFunction

    Register a UDF to verify an address, based on the provided configuration options.

    Register a UDF to verify an address, based on the provided configuration options. This UDF has a single argument that is a map of String to Column. The String keys are the address fields that are to be populated in the RequestAddress, and the column is the value to populate the fields with.

    This example uses the registered UDF as "addressingUdf" and adds an "addressing_result" column to the DataFrame:

    val verifySqlOutput = spark.sql("select *, verify(map('addressLines[0]', address, 'country', country)) as addressing_result from inputTable")
    returns

    A UDF that provides a verify operation.

  16. def forVerify(): UserDefinedFunction

    Build a UDF to verify an address, based on the provided configuration options.

    Build a UDF to verify an address, based on the provided configuration options. This UDF has a single argument that is a map of String to Column. The String keys are the address fields that are to be populated in the RequestAddress, and the column is the value to populate the fields with.

    This example uses the returned UDF as "addressingUdf" and adds an "addressing_result" column to the DataFrame:

    val resultDataFrame = inputDataFrame.withColumn("addressing_result",
  addressingUdf(map(
    lit("addressNumber"), col("addressNum"),
    lit("street"), col("street"),
    lit("city"), col("city"),
    lit("admin1"), col("state"),
    lit("postalCode"), col("zip"),
    lit("country"), col("country"))
  ))
    returns

    A UDF that provides a verify operation.

  17. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  18. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  23. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  28. def withErrorField(errorOutputFieldName: String): UDFBuilder

    Adds an output field for errors.

    Adds an output field for errors. Exceptions during processing of a record are caught and then the message is placed in this output field.

    errorOutputFieldName

    The name for the field in the output that any error message is placed.

  29. def withMultipleResults(): UDFBuilder

    Changes the output of the UDF from a single row of output fields, taken from the top candidate, to an array of rows of output fields, one for each result in the response.

    Changes the output of the UDF from a single row of output fields, taken from the top candidate, to an array of rows of output fields, one for each result in the response. Note: In order to get back multiple results, preferences must be set to allow that.

  30. def withOutputFields(fieldNames: String*): UDFBuilder

    Adds output fields for the UDF.

    Adds output fields for the UDF. Allows you to access any part of the Addressing API Result class. You can use a simple object notation to access sub objects:

    Accessing properties:

    • Use property name, case of first letter is flexible, i.e. "score" would return the value in Result.score
    • Accessing nested properties can be done using a period, i.e. "address.formattedAddress" would return the value in Result.address.formattedAddress

      Accessing Lists and Arrays:
    • Use index values in square brackets, i.e. "addressLines[0]" would return the first element of Result.addressLines

      Accessing Maps:
    • Use square brackets and string literal values, i.e. "parsed['ggs']" would return the value associated with 'ggs' in the Result.parsed map

      While evaluating the object notation any nulls encountered along the path will cause a null value to be returned. A property that isn't found will result in an error.

      The output field notation also allows aliasing:
    • You can use a simple 'as' syntax with an alpha numeric alias, i.e. "address.formattedAddress as address" would have an output field name of 'address'
    • If you need special characters in the output field name you can use a quoted literal for the alias, i.e. "address.formattedAddress as 'formatted_address'" would have an output field name of 'formatted_address'
    fieldNames

    The object notation access field names, which will also be the UDF output field names.

  31. def withPreferences(preferences: Preferences): UDFBuilder

    Sets the preferences that will be used for all geocode calls.

    Sets the preferences that will be used for all geocode calls.

    preferences

    A preferences instance.

  32. def withPreferencesFile(file: String): UDFBuilder

    Sets the location of a yaml configuration file with the desired preferences.

    Sets the location of a yaml configuration file with the desired preferences. Any specific preferences set in this yaml will override the preferences provided by withPreferences

    file

    The location of a yaml configuration file.

  33. def withResultAsJSON(jsonOutputFieldName: String): UDFBuilder

    Adds an output field for Json response.

    Adds an output field for Json response. Exceptions during processing of a record are caught and then the message is placed in the error field.

    jsonOutputFieldName

    The name for the field in the output where that json Response is placed.

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped