3tera Change Control Procedures
- Paul McCullough (Unlicensed)
3tera Change Control Procedures
The following procedures are intended to help facilitate high availability and recoverability from problems that may arise from a scheduled change to an application.
3tera Nomenclature
3tera virtual servers are called "appliances". Appliances can serve as application servers, such as WEB or database servers, or can serve as infrastructure components, such as port forwarding switches or firewalls.
A 3tera catalog contains a list of functional application or infrastructure appliances. When one uses an appliance from a 3tera catalog, it is in an "unbranched" state. It is "linked" and controlled by the catalog. If the appliance in the catalog is updated, that update is propagated to other "unbranched" appliances based on that appliance.
An application is a group of appliances that are functionally linked to provide application services, such as an AJAX stack or standard 3-tier application comprised of WEB and database services.
When an application is copied, migrated or catalog updated, the boot partitions of unbranched appliances are recreated, referencing back to whats most current in the catalog. Configuration changes to unbranched appliances are modified through their respective property pages via the Applogic interface. For instance, basic Apache settings are changed via attributes in the properties sheet of the WEB5 appliance in Applogic. These attributes are applied in unbranched appliances, even when boot partitions are recreated, so this configuration information is not lost when copying or migrating applications or appliances. The relationship between 3tera catalog and unbranched appliance is severed by "branching" an appliance. Any configuration changes made to the boot partition of an unbranched appliance is at risk.
More content on best practices in a 3tera environment.
Staffing
During a scheduled change the following personnel are on call to respond to issues requiring escalation and resolution...
- Operational system administrator
- Application developer/programmer
- Application manager
Contact information is exchanged or made available to all principle parties.
Planning
A written change control will be submitted via STAMP which will include a "back out" plan that is also provided to all principles and formal review between all parties is conducted and validated.
If the application services a client department, a designated representative of the client department is to be given the contact information of the application manager for purposes of scheduled or non-scheduled event notification. Once a non-disruptive time is negotiated, the DT division head is to be notified via e-mail, verbally if possible, and announcement made in the weekly division meeting.
Backup & Restoration
Each application is to have an associated Wiki page detailing the customizations done to specific configuration files and the location of these configuration files. This is a requirement when making changes to configuration file/s on unbranched appliances. (See Best Practices above.) Unbranched Appliances h2. For each application, any configuration changes to unbranched appliances will have a section pertaining to the location, name, and customization of the configuration file.
For example, for Wiki page "3tera - wordpress_mayor (grid sfgov1)" a section appears as...
File: /etc/.../asdf.conf
Add:
Change:
Copy pertinent config files of unbranched appliances to subdirectories on the shared NAS repository in the application named "Repository" under the following directory structure...
/app_name/appliance_name/source_dir_path/config_file Application Backup Copy
For critical application, the application manager may choose to have a functional backup copy of the application. Please note that when copying or migrating an application, boot volumes of unbranched appliances are re-created. Data and configuration changes on these volumes is not migrated or copied. Application staff are to ensure backup copies are operational by applying and testing copied/migrated applications at least 24 hours prior to the change control. If a test of the backup copy is not successfully completed within 24 hours of the change control, the change control is to be rescheduled until testing criteria is met and a successful test of the backup application is completed.
Notification
In the event of an unscheduled outage or delay in timeline:
Contact the Application Manager. The Application Manager may choose to contact the Division Head or customer contact based on the severity on the issue, and has sole discretion to do so.
In the event of a successful scheduled change:
Send txt or e-mail to staff assigned to execute or monitor the change, including the Application Manager.