System Integrations

Introduction

This page describes how EAS is integrated with other city systems. EAS is built entirely with restful web services. As a result it's fairly easy to integrate with external systems or workflows.

Public External Integration Points

This is a list of the publicized integration services/end points. There are many internal services that we may able to adapt for your specific needs. All of these are now formally under change control. This mean that changes must be coordinated with the stake holders listed on the individual wiki pages.

Contents

Terminology

Extract-Transform-Load (ETL)
Enterprise Addressing System (EAS)
Department of Public Works (DPW)
Department Building Inspection (DBI)
Parcel - includes attribute data as well as geometry
Street segment - includes attribute data as weel as geometry
Address Verification System (AVS): DBIs address system

Parcels

This section describes how we integrate parcel data into EAS.
You can see the details of the ETL in version control here.

Parcel Creation - DPW

Parcel creation occurs at DPW. 
Every week night an automated ETL runs that loads new parcels into EAS.
The DPW process for creating parcels is non-trivial and is documented here (need link).

Parcel Update - DPW

Parcel updates occur at DPW.
Every week night an automated ETL runs that updates EAS parcels from DPW.
We assume that the block and lot fields do not change.
All other fields may be updated, including the geometry.

Parcel Retirement - DPW

Parcel retirement occurs at DPW. 
Parcel retirement is a special kind of update.
Every week night an automated extract-transform-load (ETL) job retires parcels in EAS based on data from DPW.
If a parcel is newly retired, addresses that are linked to the retired parcel are marked as "invalid".
You can read more about invalid addresses here.
The DPW process for retiring parcels is non-trivia and is documented here (need link).

Parcel Provisioning - EAS

In rare cases, an address must be created immediately using a parcel that does not yet officially exist.
In these cases, an EAS Reviewer can "provision" a parcel within EAS.
This creates an unofficial or "provisioned" parcel in EAS.
The EAS Reviewer then requests that the official parcel be created by DPW and must provide documentation from the assessor-recorder to DPW.
This step is currently a manual process and uses email; it should be automated and message driven.
When the official parcel is eventually created by DPW, the nightly ETL uses the block-lot to update the provisioned parcel, and marks the provisioning as complete.

Streets

This section describes how we integrate street data into EAS.
You can see the details of the ETL in version control here.

Street Segment Creation - DPW

Street segment creation occurs at DPW.
Every week night an automated extract-transform-load (ETL) job runs that loads new street segments into EAS.
The DPW process for creating street segment is non-trivial and is documented here (need link).

Street Segment Update - DPW

Street segment updates occur at DPW.
Every week night an automated extract-transform-load (ETL) job updates street segments in EAS from DPW.
A street segment as defined by EAS includes attribute and geometry data.

Street Segment Retirement - DPW

Street segment retirement occurs at DPW. 
Street segment retirement is a special kind of update.
Every week night an automated extract-transform-load (ETL) job runs that retires street segments in EAS based on data from DPW.
If a street segment is newly retired, addresses that are linked to the retired street segment are marked as "invalid".
You can read more about invalid addresses here.
The DPW process for retiring street segments is non-trivial and is documented here (need link).

Street Segment Provisioning - EAS

In very rare cases, an address must be created using a street segment that does not yet officially exist.
In these cases, an SFGIS staff member must "provision" a street segment for EAS.
This process is described here.
This is a manual process; automation may be expensive.
SFGIS staff then requests that the official street segment be created by DPW and must provide documentation to DPW.

Street Name Creation - DPW

Street name creation is automatically handled by the night street ETL.

Street Name Change - DPW

Street name changes are automatically handled by the night street ETL.

Addresses

Invalid Addresses

In EAS, an "invalid" address may be retired or may remain active.
The decision to retire an address is made by an EAS reviewer.
A reviewer may decide to keep an address active, even with a retired parcel, if there is permit activity on that parcel.
However, in most cases the reviewer will retire an "invalid" address.

Address Create/Update/Retire - EAS

Addresses are created, updated, and retired using the EAS web application.
We can automate this with web services as needed.
For an DBI style address to be created, the street segment and parcel must already exist.
This activity diagram shows the simplest example.

Address Change Requests

When an address is created, updated, or retired it must go through a change request process.
A requester submits a change request for review.
A change request can contain an arbitrary collection of creates, updates, and retires.
A reviewer typically approves a change request, but may also reject a change request.
Once the change request is approved, the address becomes visible to all EAS users,
and a change notification is sent to AVS. Read more about the change notification below.
Note that certain parts of the address are not mutable such as the the street and street number.

Address Change Notification - EAS

When a reviewer approves a change request, a change notification message is sent to AVS.
The structure of the message is discussed in detail on this page.
As of Feb 15 2012, EAS has been in production and AVS has become essentially a replication target for EAS.
We are using a point to point web service for this process.
We should consider a message bus architecture so we can more easily support broadcasting.