Impact: Customer experienced intermittent calls with no audio or dead air on affected calls
Root Cause: An IntelePeer engineer had staged a new audio configuration file for future testing and validation but inadvertently and accidentally deployed the configuration. The file still had to be tested and validated with a destination path still to be defined. The new configuration file was intended to streamline the audio path to continue to improve performance. Because the file was deployed inadvertently, the audio file paths were directed to an errant location which resulted in no audio available creating the dead air effect.
Resolution: Upon immediate alerting of low memory on the storage area networks and customers beginning to report calling issues, Intelepeer engineers immediately worked to identify and re-synch all audio and data files. In certain cases, to facilitate more immediate resolution, audio files were manually synched which restored the service earlier.
Corrective Actions Going Forth:
IntelePeer has implemented a process of checks and balances whereby the individual staging a code change cannot be the same person deploying the code change. The process requires full validation and approval by the QA team before code can be deployed.
IntelePeer is reviewing options to include a systematic addition whereby in the event of errant or no audio file available, the system defaults to a customer approved message/file.