Page Comparison

Table of Contents

Overview

...

Table of Contents

Overview

• The Bulk Loader is a process that is used to add several new addresses to the EAS at one time.
• There are several stages that make up the Bulk Loader process, outlined below in Summary and Details.

...

...

Anchor
stage1 stage1

Stage 1 - Import and parse reference dataset (Optional)

This optional stage is run once per Bulk Loader process. This stage can be skipped if the reference dataset is already available or if the optional 'filter by reference' step (Step 2.5) is skipped.

...

Anchor
stage3 stage3

Stage 3 - Geocode and filter

•  

  Step 3.1 - Geocode source dataset

...

Anchor
stage4 stage4

Stage 4 - Export shapefile - full set (single batch) or subset (multiple batches)

...

Anchor
stage5 stage5

Stage 5 - Run the Bulk Loader

 For a complete set of steps and background about the Bulk Loader, see also Running the Bulk Loader, a page dedicated to its input, operation and results.

...

•  

  Step 5.2 - Non-production-specific preparation

1. Restore Database

   1. Restore database from latest daily production backup

...

•  

  Step 5.3 - Production-specific preparation

1. Halt Services

   Warning
   title Reason for halting services
   These steps are being performed to facilitate immediate roll-back of the EAS database if the Bulk Load Process ends in failure

   
   

   1. SKIP Turn off the replication server

      1. Disable database replication by shutting down the database service on the replication server (DR PROD DB).

         
         

         Code Block
         language bash title Stop PostgreSQL linenumbers true
         #sudo -u postgres -i #/usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data stop

         
         

   2. Turn off downstream database propagation service(s)

      1. Suspend downstream replication to internal business system database (SF PROD WEB).

         Code Block
         language textbash title stop xmit linenumbers true
         sudo /var/www/html/eas/bin/xmit_change_notifications.bsh stop

         
         

2. Backup Database

   • See also Backup the EAS Databases

...

1. Open a command prompt and change folders:

   Code Block
   language bash linenumbers true
   cd C:\apps\eas_automation\automation\src

   
   

2. Run the step to stage the address records:

   Code Block
   language bash linenumbers true
   python job.py --job stage_bulkload_shapefile --env <environment> --action EXECUTE --v python job.py --job stage_bulkload_shapefile --env SF_DEV --action EXECUTE --v python job.py --job stage_bulkload_shapefile --env SF_QA --action EXECUTE --v python job.py --job stage_bulkload_shapefile --env SF_PROD --action EXECUTE --v

   
   

3. Run the step to bulk load the address records

   
   

   Code Block
   language bash linenumbers true
   python job.py --job bulkload --env <environment> --action EXECUTE --v python job.py --job bulkload --env SF_DEV --action EXECUTE --v python job.py --job bulkload --env SF_QA --action EXECUTE --v python job.py --job bulkload --env SF_PROD --action EXECUTE --v

   
   

4.  To calculate the time it took to run the Bulk Loader look at the timestamps in the output or use a stopwatch or clock to time the operation.

   
   

5. Save Bulk Loader command line output artifact as bulk_loader_CLI_output.txt.

•  

  Step 5.6 - Analysis

  Anchor
  analysis analysis

...

Anchor
stage6 stage6

Stage 6 - Extract results

•  

  Step 6.1 - Archive exceptions

...

1. Archive the entire address_extract table.

   1. Use a query tool such as pgAdmin to query and save the table as a CSV file.

      Code Block
      language sql firstline 1 title address_extract linenumbers true
      SELECT * FROM bulkloader.address_extract;

      
      

   2. Save the file in the network folder dedicated to artifacts for the Bulk Loader iteration.
      1. Save as artifact address_extract.csv
2. Archive the addresses that raised exceptions during the Bulk Loader process

   1. Query subtotals and save as artifact exception_text_counts.csv

      Code Block
      language sql firstline 1 title exception_text_counts linenumbers true
      SELECT exception_text, Count(*) FROM bulkloader.address_extract GROUP BY exception_text ORDER BY exception_text;

      Query all exception text records and save as artifact 

      1. Save artifact as exception_text_counts.csv

   2. Query all exception text records

      Code Block
      language sql firstline 1 title exception_text linenumbers true
      SELECT * FROM bulkloader.address_extract WHERE NOT(exception_text IS NULL) ORDER BY exception_text, id;

      1. Save artifact as exception_text.csv
3. Artifacts
   1. address_extract.csv - Results of every address submitted to the Bulk Loader.
   2. exception_text_counts.csv - Counts of the records that were not loaded due to the error indicated in the 'exception_text' field.
   3. exception_text.csv - Subset of the just the records that were not loaded due to the error indicated in the 'exception_text' field

...

1. Get the unique EAS change_request_id created by the Bulk Load operation. The value of <change_request_id> will be used in the next steps to count addresses added to the EAS.
   

   1. Query the 'public.change_requests' table for the new 'change_request_id' value.

      Code Block
      language sql firstline 1 title change_request_id linenumbers true
      SELECT change_request_id FROM public.change_requests WHERE requestor_comment LIKE 'bulk load change request' ORDER BY change_request_id DESC LIMIT 1;

      Save the file in the network folder dedicated to artifacts for the Bulk Loader iteration.For example, R:\Tec\..\Eas\_Task\path\to\archive\bulkloader_YYYYMMDD\bulkloader\batch_NNN\

      _id DESC LIMIT 1;

      
      

   2. Save artifact as change_request_id.csv
2. Artifacts
   1. change_request_id.csv - The unique EAS change_request_id created by the Bulk Load operation.

...

1. Get all the address records (including units) added to the EAS during the Bulk Loader operation..
   

   1. Query the public.addresses table on the new change_request_id value.

      Code Block
      language sql firstline 1 title addresses linenumbers true
      SELECT * FROM public.addresses WHERE activate_change_request_id = <change_request_id>;

      
      

   2. Save the file in the network folder dedicated to artifacts for the Bulk Loader iteration.For example, R:\Tec\..\Eas\_Task\path\to\archive\bulkloader_YYYYMMDD\bulkloader\batch_NNN\artifact as addresses.csv
2. Extract sample unit address from the output
   1. Pick a random record from the results where unit_num is not NULL. Gather the value in the address_base_id field.
   2. Construct a URL from this value like this: http://eas.sfgov.org/?address=NNNNNN
      • Where NNNNNN is the value from the address_base_id field.
   3. Make note of this URL for use in Step 7 when testing EAS after services are restored.
3.  Artifacts
   1. addresses.csv - All the address records (including units) added to the EAS during the Bulk Loader operation.

...

1. Get all the base records added to the EAS during the Bulk Loader operation.
   

   1. Query the public.address_base table on the new change_request_id value.

      Code Block
      language sql firstline 1 title address_base linenumbers true
      SELECT activate_change_request_id, address_id, public.address_base.* FROM public.address_base, public.addresses WHERE public.address_base.address_base_id = public.addresses.address_base_id AND public.addresses.address_base_flg = TRUE AND public.addresses.activate_change_request_id = <change_request_id>;

      Save the file in the network folder dedicated to artifacts for the Bulk Loader iteration.For example, R:\Tec\..\Eas\_Task\path\to\archive\bulkloader_YYYYMMDD\bulkloader\batch_NNN\

      _request_id>;

      
      

   2. Save artifact as address_base.csv
2. Extract sample base address from the output
   1. Pick a random record from the results. Gather the value in the address_base_id field.
   2. Construct a URL from this value like this: http://eas.sfgov.org/?address=NNNNNN
      • Where NNNNNN is the value from the address_base_id field.
   3. Make note of this URL for use in Step 7 when testing EAS after services are restored.
3. Artifacts
   1. address_base.csv - All the base records added to the EAS during the Bulk Loader operation.

...

1. The number of addresses found in the Step 5.6 (2) should be identical to the number of addresses found in Step 6.3.

2. The number of base addresses found in the Step 5.6 (2) should be identical to the number of base addresses found in Step 6.4.

Anchor
stage7 stage7

Stage 7 - Cleanup and Restoration

•  

  Step 7.

...

•  

  Step 7.1 - Database Cleanup

...

• 1 - Database Cleanup

1. Connect to the database, <environment>_DB, and clear address_extract records from the latest Bulk Loader batch. Make note of final disk usage tally.

TRUNCATE bulkloader.address_extract, bulkloader.blocks_nearest;

VACUUM FULL ANALYZE bulkloader.address_extract;

VACUUM FULL ANALYZE bulkloader.blocks_nearest;

TRUNCATE bulkloader.address_extract, bulkloader.blocks_nearest;

VACUUM FULL ANALYZE bulkloader.address_extract;

date; df /data # 3rd of 3
# Optional step: archive output to 'df.txt' artifact
exit

•  

  Step 7.2 - Clean automation machine

1. Return to automation machine and remove shapefile from 'bulkload_shapefile' folder.
2. Logout of automation machine. 

•  

  Step 7.3 - On Failure Restore Database

1. If the Bulk Loader failed and corrupted any data then restore from the database backup.
   1. Follow these steps to restore from backup.

•  

  Step 7.4 - Restore Services (Production Only)

1. SKIP Turn on production-to-replication service

   • Re-enable database replication by restarting the database service on the replication server (DR PROD DB).

     
     

     Code Block
     language bash title

...

1. • Stop PostgreSQL linenumbers true

...

1. • #sudo

...

1. • -u

...

•  

  Step 7.2 - Clean automation machine

1. Return to automation machine and remove shapefile from 'bulkload_shapefile' folder.
2. Logout of automation machine. 

•  

  Step 7.3 - On Failure Restore Database

1. If the Bulk Loader failed and corrupted any data then restore from the database backup.
   1. Follow these steps to restore from backup.

•  

  Step 7.4 - Restore Services (Production Only)

1. SKIP Turn on production-to-replication service
   • Re-enable database replication by restarting the database service on the replication server (DR PROD DB).

     postgres -i #/usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data start

     
     

2. SKIP Turn on downstream database propagation service(s)

   • Resume downstream replication to internal business system database (SF PROD WEB).

     Code Block
     language bash firstline 1 title start xmit
     #sudo /var/www/html/eas/bin/xmit_change_notifications.bsh start

     
     

•  

  Anchor
  step7.5 step7.5

  Step 7.5 - Enable front-end access to EAS

...

1. Enable web service on <environment>_WEB (SF DEV WEB, SF QA WEB, SF PROD WEB)

   Code Block
   language bash

   titleStop PostgreSQL

   linenumbers true

   #sudo -u postgres -i
   #/usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data start

   SKIP Turn on downstream database propagation service(s)

   Resume downstream replication to internal business system database (SF PROD WEB).

   Code Block
   language text firstline 1 title start xmit
   #sudo /var/www/html/eas/bin/xmit_change_notifications.bsh start

...

Enable web service on <environment>_WEB (SF DEV WEB, SF QA WEB, SF PROD WEB)nable front-end access to EAS

cd /var/www/html
sudo ./set_eas_mode.sh LIVE
exit

...

Browse to website, http://eas.sfgov.org/, and review sample addresses gathered in Step 6.3 and Step 6.4

...

Notify relevant recipients that the Bulk Loader Process is complete

1. cd /var/www/html sudo ./set_eas_mode.sh LIVE exit

   
   

2. Browse to website, http://eas.sfgov.org/, and review sample addresses gathered in Step 6.3 and Step 6.4

3. Notify relevant recipients that the Bulk Loader Process is complete

•  

  Step 7.6 - Archive artifacts

1. List of artifacts
   1. address_base.csv
   2. address_extract.csv
   3. addresses.csv
   4. bulk_loader_CLI_output.txt
   5. change_request_id.csv
   6. df.txt
   7. exception_text.csv
   8. exception_text_counts.csv

2. Contents of progress and summary artifact, BulkLoader_Process_YYYYMMDD.xlsx

   
   

   1. Progress - This sheet contains a table of relevant totals for each batch

      1. Batch number

      2. Batch date

      3. Input record counts

      4. New base record counts

      5. New unit record counts

      6. Sample addresses

         
         

   2. Email Jobs - This sheet contains a table of details related to the weekly 'Address Notification Report' automated email job

      1. Batch range

      2. Record count in batch range

      3. Email Timestamp

      4. Total record counts in email

      5. Subtotal of records generated as a result of the Bulk Loader

      6. Size of email attachment

         
         

   3. Batch N - This sheet tracks the before and after record counts for all tables in the EAS database. There is a sheet for each batch loaded. Within each sheet is a section for the 'before' records, a section for the 'after' record counts, and a 'diff' column showing the change in record counts.

END OF STEPS

Notes

...