Initial: WAWF Outage Reportable Notification; INC000005124329 (Critical App)

Application: Wide Area Workflow (WAWF)

Incident Summary: Wide Area Workflow e-Business Suite (WAWF) Invoicing, Receipt, Acceptance and Property Transfer (IRAPT) Application is in a degraded status due to document processing issues.

Operational Impact: Unclassified production environment IRAPT documents are becoming invalid Customers worldwide are experiencing a delay in processing of documents for payment. No known work around exists. Approximately 60 calls received.

Update: The application development team is working with the Program Management Office and the database administrator to resolve the issue.

Event Start Date and Time (Z): 2/27/2017 1222Z
Event End Date and Time (Z):
Related ticket: INC000005123667

*** 2/27/2017 – 10:57am PST Update: The software developers have identified the cause of the issue. A patch is currently being developed and is scheduled to be deployed this tonight 2/27/2017 17:00 MT – 20:00 MT.

*** Update: The software developers have delivered the patch to fix the issue. The software testers need more time to install and test the patch prior to deployment. The patch is expected to be applied tomorrow 3/01/2017 00:00 Zulu – 03:00 Zulu. (2/28 1700-1800MT)

WAWF will be unavailable for the WAWF 5.10 system upgrades on Friday, February 24 from 17:00 MT (19:00 ET). All Portal Systems (iRAPT, EDA, myInvoice, eMIPR, CORT Tool, IUID Registry, NCCS, and MRS tool) will be unavailable during this scheduled Downtime. The WAWF system is due to become available again on Monday, February 27th at 04:00 MT (06:00 ET).
Please plan accordingly.

***Downtime has been completed successfully at 02/26/2017 20:30 MT. All Wide Area Workflow applications are available***

The WAWF PMO has requested that the WAWF application be taken offline to replace the Vote disk on the WAWF cluster. WAWF and all portal systems iRAPT, EDA, CORT, UID, myInvoice, MRS, Contract Closeout, CDR, and EMIPR will be unavailable during this break/fix window.

Break/Fix incident INC000005109002
Related to previous ASI/Change Request CRQ000000627244

Start time 2/19/2017 08:00 MT
End Time – TBD

*** Downtime competed at 21:15 MT. The Vote Disk replacement on the WAWF cluster was completed successfully.

Downtime for: Wide Area Workflow eBusiness Suite .

Reason: Replace the Vote disk on the WAWF cluster. WAWF and all portal systems iRAPT, EDA, CORT, UID, myInvoice, MRS, Contract Closeout, CDR, and EMIPR will be unavailable during this scheduled maintenance window.

Downtime Details:

Occurs Start Date/Time 2/17/2017 18:00 MT
End Date/Time 2/17/2017 23:00 MT

***This downtime has concluded. The Vote disc update was unsuccessful. PMO request downtime be rescheduled for 02/19/2017 08:00 MT ***
***The WAWF application is back online***