Application: WAWF (Wide Area Workflow) Global Exchange Services (GEX)

Incident Summary: WAWF E-Business Suite and GEX applications are unavailable.

Operational Impact: [Outage]An unknown number of users worldwide were unable to process data transactions through the Ogden GEX. No known work around exists for Wide Area Workflow, Defense Travel System and all other customers utilizing the GEX.
An unknown number of WAWF Government and commercial users are unable to access portal applications. Payment transactions are delayed, contract documents are inaccessible and all other portal systems are impacted.

Update: DISA Technicians are investigating the issue. Further updates will be provided as they are available.

Event Start Date and Time (Z): 9/15/2017 1835 Zulu

****** UPDATE 9/15/2017 2142 Zulu:

Incident Summary: During initial investigation of WAWF/GEX connection issue it was discovered that a secondary circuit at Ogden was down, connections between Ogden and several applications became unavailable. Update Ogden site is now in a network HAZCON.

Operational Impact: [HAZCON] An unknown number of users worldwide are unable to access multiple applications going through DECC Ogden. Failover to the primary has somewhat stabilized access.

****** Resolved: (16 SEPT): COMM-IM informed that the DGOC Transport team performed a toggle of the transceiver at Fort Lewis which restored services @ 0145Z 9/16/2017. COMM-IM has verified Ogden Secondary circuit in question is now out of HAZCON and requested to monitor status for 24hrs.

Event End Date and Time (Z): 9/16/2017 0145 Zulu

Interruption of the processing of RFID Combo transactions has been reported. All other EDI transactions are unaffected.

Transactions are being delivered to DLA but not processed. DLA Transaction Services is being consulted.

DLATS Reference No.: INC000000060592
Summary: UDF Combos packet loss

Your reported Incident has been resolved with the following resolution:
channel off on gexb as not processed there before. Started and backlogged sent.

*** RFID Combo processing resumed 4/3/2017 @ 7:14a PT. Unprocessed transactions previously submitted were processed, which may result in spurious notification emails citing duplication of transactions.

The Mil-Pac VAN has been taken offline for a scheduled migration to larger storage volume to allowed increased archival storage.

The IP address of the server will not change. Government NEPs will be unaffected.

Start Time: 3/18/17 – 1400 PST
Estimated End Time: 1800 PST

*** Downtime was completed at 1715 PST
*** Migration to new storage volume was successful.

The Mil-Pac VAN’s front-end server has been taken offline for a scheduled upgrade to its FTP Server.

Start Time: 3/11/17 – 1400 PST
Estimated End Time: 1800 PST

*** Downtime completed at 1720 PST

Initial: WAWF Outage Reportable Notification; INC000005124329 (Critical App)

Application: Wide Area Workflow (WAWF)

Incident Summary: Wide Area Workflow e-Business Suite (WAWF) Invoicing, Receipt, Acceptance and Property Transfer (IRAPT) Application is in a degraded status due to document processing issues.

Operational Impact: Unclassified production environment IRAPT documents are becoming invalid Customers worldwide are experiencing a delay in processing of documents for payment. No known work around exists. Approximately 60 calls received.

Update: The application development team is working with the Program Management Office and the database administrator to resolve the issue.

Event Start Date and Time (Z): 2/27/2017 1222Z
Event End Date and Time (Z):
Related ticket: INC000005123667

*** 2/27/2017 – 10:57am PST Update: The software developers have identified the cause of the issue. A patch is currently being developed and is scheduled to be deployed this tonight 2/27/2017 17:00 MT – 20:00 MT.

*** Update: The software developers have delivered the patch to fix the issue. The software testers need more time to install and test the patch prior to deployment. The patch is expected to be applied tomorrow 3/01/2017 00:00 Zulu – 03:00 Zulu. (2/28 1700-1800MT)

WAWF will be unavailable for the WAWF 5.10 system upgrades on Friday, February 24 from 17:00 MT (19:00 ET). All Portal Systems (iRAPT, EDA, myInvoice, eMIPR, CORT Tool, IUID Registry, NCCS, and MRS tool) will be unavailable during this scheduled Downtime. The WAWF system is due to become available again on Monday, February 27th at 04:00 MT (06:00 ET).
Please plan accordingly.

***Downtime has been completed successfully at 02/26/2017 20:30 MT. All Wide Area Workflow applications are available***

The WAWF PMO has requested that the WAWF application be taken offline to replace the Vote disk on the WAWF cluster. WAWF and all portal systems iRAPT, EDA, CORT, UID, myInvoice, MRS, Contract Closeout, CDR, and EMIPR will be unavailable during this break/fix window.

Break/Fix incident INC000005109002
Related to previous ASI/Change Request CRQ000000627244

Start time 2/19/2017 08:00 MT
End Time – TBD

*** Downtime competed at 21:15 MT. The Vote Disk replacement on the WAWF cluster was completed successfully.

Downtime for: Wide Area Workflow eBusiness Suite .

Reason: Replace the Vote disk on the WAWF cluster. WAWF and all portal systems iRAPT, EDA, CORT, UID, myInvoice, MRS, Contract Closeout, CDR, and EMIPR will be unavailable during this scheduled maintenance window.

Downtime Details:

Occurs Start Date/Time 2/17/2017 18:00 MT
End Date/Time 2/17/2017 23:00 MT

***This downtime has concluded. The Vote disc update was unsuccessful. PMO request downtime be rescheduled for 02/19/2017 08:00 MT ***
***The WAWF application is back online***