Page Comparison

Table of Contents

Overview

...

...

Anchor
stage1 stage1

Stage 1 - Import and parse reference dataset (Optional)

This optional stage is run once per Bulk Loader process. This stage can be skipped if the reference dataset is already available or if the optional 'filter by reference' step (Step 2.5) is skipped.

...

Anchor
stage3 stage3

Stage 3 - Geocode and filter

•  

  Step 3.1 - Geocode source dataset

...

Anchor
stage4 stage4

Stage 4 - Export full set (single batch) or subset (multiple batches)

...

Anchor
stage5 stage5

Stage 5 - Run the Bulk Loader

 For a complete set of steps and background about the Bulk Loader, see also Running the Bulk Loader, a page dedicated to its input, operation and results.

...

1. Halt Services

   Warning
   title Reason for halting services
   These steps are being performed to facilitate immediate roll-back of the EAS database if the Bulk Load Process ends in failure

   
   

   1. Place EAS web application into maintenance mode

      1. Place the Web servers into maintenance mode (SF PROD WEB, DR PROD WEB).

         
         

         Code Block
         language bash title Place EAS web application into maintenance mode linenumbers true
         cd /var/www/html sudo ./set_eas_mode.sh MAINT

         
         

   2. SKIP Turn off the replication server

      1. Disable database replication by shutting down the database service on the replication server (DR PROD DB).

         
         

         Code Block
         language bash title Stop PostgreSQL linenumbers true
         sudo -u postgres -i /usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data stop

         
         

   3. Turn off downstream database propagation service(s)

      1. Suspend downstream replication to internal business system database (SF PROD WEB).

         Code Block
         language text title stop xmit linenumbers true
         sudo /var/www/html/eas/bin/xmit_change_notifications.bsh stop

         
         

2. Backup Database
   1. Make a backup of the EAS database
      • See also Backup the EAS Databases

sudo -u postgres -i
/home/dba/scripts/dbbackup.sh > /var/tmp/dbbackup.log
ls -l /var/tmp # ensure the log file is 0 bytes
ls -la /mnt/backup/pg/daily/easproddb.sfgov.org-* # the timestamp on the last file listed should match timestamp of backup

•  

  Step 5.3 - Database Preparation

   

   

1. Connect to the database, <environment>_DB, and clear any leftover records from previous Bulk Loader batches.

   Code Block
   language sql title TRUNCATE linenumbers true
   TRUNCATE bulkloader.address_extract, bulkloader.blocks_nearest;

   
   

   Code Block
   language sql title VACUUM linenumbers true
   VACUUM FULL ANALYZE bulkloader.address_extract; VACUUM FULL ANALYZE bulkloader.blocks_nearest;

   
   

   
   

2. Make note of EAS record counts before the Bulk Loading operation.

   Code Block
   language sql firstline 1 title Record Counts linenumbers true
   SELECT schemaname,relname,n_live_tup FROM pg_stat_user_tables ORDER BY schemaname,relname,n_live_tup

   1. Save results in artifact as record-counts-before.csv 
   2. Also save results in Excel spreadsheet artifact as bulkloader_results_YYYYMMDD.xlsx (TODO provide path to a template)

3. Make note of the database partition size on the file system at the current point in time.

   Code Block
   language bash firstline 1 title disk usage linenumbers true
   date; df /data

   
   

...

Anchor
stage6 stage6

Stage 6 - Extract results

•  

  Step 6.1 - Archive exceptions

...

1. If the Bulk Loader Process was run on the production server then restore services

   1. SKIP Turn on production-to-replication service

      • Re-enable database replication by restarting the database service on the replication server (DR PROD DB).

        
        

        Code Block
        language bash title Stop PostgreSQL linenumbers true
        sudo -u postgres -i /usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data start

        
        

   2. SKIP Turn on downstream database propagation service(s)

      • Resume downstream replication to internal business system database (SF PROD WEB).

        Code Block
        language text firstline 1 title start xmit
        sudo /var/www/html/eas/bin/xmit_change_notifications.bsh start

        
        

   3. Enable front-end access to EAS

      • Place the Web servers into live mode (SF PROD WEB, DR PROD WEB).

        Code Block
        language bash linenumbers true
        cd /var/www/html sudo ./set_eas_mode.sh LIVE

        
        

...