Table of Contents |
---|
...
Stage Number | Stage | Category | Summary | Environment | Iterations | Estimated Person Time | Estimated Computer Time |
---|---|---|---|---|---|---|---|
1 | Import and parse reference dataset (Optional) | Parsing | This optional step in the Bulk Loader process is to cross check an address for a match in a reference data set. If a source address is found in the reference dataset the address makes it to the next step. If not found the address is put aside in an exclusion set for later review. | Once per Bulk Loader process | 1 hour | 10 minutes | |
2 | Import, parse and filter source dataset | Parsing | Import the dataset destined for the EAS. Parse and filter the set. | Once per Bulk Loader process | 90 minutes | 15 minutes | |
Geocode and filter | Geocoding | Geocode the set and filter further based on the geocoder score and status. | ArcMap | Once per Bulk Loader process | 1 hour | 5 minutes | |
4 | Export full set (single batch) or subset (multiple batches) | Geocoding | For large datasets, create one of many subsets that will be run through the Bulk Loader in multiple batches. | ArcMap | One or more batches for each Bulk Loader process | 30 minutes per batch | 5 minutes per batch |
5 | Bulk Load batch (full set or subset) | Bulk Loading | Run the entire batch or each subset batch through the Bulk Loader. | One or more batches for each Bulk Loader process | 1 hour per batch | 5 minutes per batch | |
6 | Extract results | Bulk Loading | Extract and archive the list of addresses that were added to the EAS . Also archive the unique EAS 'change request id' associated with this batch. Also archive the addresses that were rejected by the Bulk Loader in this batch. | PostgreSQL / pgAdmin | One or more batches for each Bulk Loader process | 1 hour per batch | 5 minutes per batch |
7 | Cleanup and Restoration | Bulk Loading | Clean up database, restore services and in the event of a failure, restore from backup. | PostgreSQL / pgAdmin | One or more batches for each Bulk Loader process | 1 hour per batch | 5 minutes per batch |
...
Anchor | ||||
---|---|---|---|---|
|
This optional stage is run once per Bulk Loader process. This stage can be skipped if the reference dataset is already available or if the optional 'filter by reference' step (Step 2.5) is skipped.
...
Anchorstage3 stage3
Stage 3 - Geocode and filter
stage3 | |
stage3 |
-
Step 3.1 - Geocode source dataset
...
Anchorstage4 stage4
Stage 4 - Export full set (single batch) or subset (multiple batches)
stage4 | |
stage4 |
Note | ||
---|---|---|
| ||
Stages 4, 5 and 6 can be run one time with the results from Stage 3, or they can be run in multiple batches of subsets. A major consideration of when to run the full set at once versus in batches is the number of records being Bulk Loaded. The size of each Bulk Loader operation affects the following aspects of the EAS:
For medium-to-large datasets (input sets with over 1,000 records) it is recommended that the process first be run on a development server to assess the implications of the operation. Where appropriate, perform the Bulk Loading process in batches over several days or weeks. The remaining steps will document a single batch example iteration of a multi-batch process. |
...
Anchorstage5 stage5
Stage 5 - Run the Bulk Loader
stage5 | |
stage5 |
For a complete set of steps and background about the Bulk Loader, see also Running the Bulk Loader, a page dedicated to its input, operation and results.
...
- Restore Database
- Restore database from latest daily production backup
- Halt database and web applicationConnect to the database,
Place EAS web application into maintenance mode
Place the web server into maintenance mode (SF DEV WEB, SF QA WEB).
Code Block language bash title Place EAS web application into maintenance mode linenumbers true cd /var/www/html sudo ./set_eas_mode.sh MAINT
Turn off the database server
Disable database server (SF DEV WEB, SF QA WEB).
Code Block language bash title Stop PostgreSQL linenumbers true sudo -u postgres -i /usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data stop
Step 5.3 - Database Preparation
Step 5.3 - Database Preparation
Connect to the database,
<environment>_DB
, and clear any leftover records from previous Bulk Loader batches.Code Block language sql title truncate linenumbers true TRUNCATE bulkloader.address_extract, bulkloader.blocks_nearest;
Code Block language sql title vacuum linenumbers true VACUUM FULL ANALYZE bulkloader.address_extract;
Code Block language sql title vacuum linenumbers true VACUUM FULL ANALYZE bulkloader.blocks_nearest;
Make note of EAS record counts before the Bulk Loading operation.
Code Block language sql firstline 1 title Record Counts linenumbers true SELECT schemaname,relname,n_live_tup FROM pg_stat_user_tables ORDER BY schemaname,relname,n_live_tup
Make note of the database partition size on the file system.
Code Block language bash firstline 1 title disk usage linenumbers true df /data
...
Anchorstage6 stage6
Stage 6 - Extract results
stage6 | |
stage6 |
-
Step 6.1 - Archive exceptions
...
- If the Bulk Loader Process was run on the production server then restore services
Turn on production-to-replication service
Re-enable database replication by restarting the database service on the replication server (DR PROD DB).
Code Block language bash title Stop PostgreSQL linenumbers true sudo -u postgres -i /usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data start
Turn on downstream database propagation service(s)
Resume downstream replication to internal business system database (SF PROD WEB).
Code Block language text firstline 1 title start xmit sudo /var/www/html/eas/bin/xmit_change_notifications.bsh start
Enable front-end access to EAS
Place the Web servers into live mode (SF PROD WEB, DR PROD WEB).
Code Block language bash linenumbers true cd /var/www/html sudo ./set_eas_mode.sh LIVE
...
- If the Bulk Loader Process was run on a non-production server then restore services
Turn on database service
Code Block sudo -u postgres -i /usr/pgsql-9.0/bin/pg_ctl -D /data/9.0/data startservices (SF DEV WEB, SF QA WEB)language bash title Stop PostgreSQL linenumbers true Enable front-end access to EAS
Code Block language bash linenumbers true cd /var/www/html sudo ./set_eas_mode.sh LIVE
...