A few weeks ago, I posted about our Shoretel upgrade from version 6.1 to 10.1. Overall, the upgrade was smooth and including an upgrade of the conference bridge hardware and software to version 7. However, there was one little post-upgrade problem. I was unable to view or edit the user configuration for a subset of my users using the Shoretel Director web portal. An “data undefined” error would display in my browser and then once that box was clear, the word undefined appeared in one of the data fields for the user. All other fields were blank and I couldn’t perform any actions like delete, save or reset.
After performing a database repair with our VAR, a ticket was opened with Shoretel directly. A Shoretel engineer looked at the issue, took copies of our database and log history from the upgrade and we were left to wait for a resolution of some sort. The users in question had fully functional phones and voicemail, as well as any other feature they had before the upgrade. Outside of a slowing growing list of tweaks I couldn’t make to those users, the system was perfectly stable.
Because the users had fully functional services, I doubted we were up against any major database corruption. While one could argue that we did an extensive upgrade in one evening (6.1 to 7.5 to 8.5 to 10.1) we didn’t deviate from the standard upgrade process that one could have done over time. While waiting for Shoretel to respond to the escalated ticket, our senior in-house DBA came across some free time and was able to take a look at the MySQL database himself.
The list of affected users spanned departments and had very little in common outright. However, I suspected they had some common component enabled and those settings were causing the new version of the Shoretel Director web portal to choke when loading the information. I’ve noticed that some fields that weren’t required in the past (like Last Name) are now required, so I was hoping it was something along those lines.
I provided my list and my hunch to our DBA who started sorting and running queries on our users table to see what could possibly be mucking up the system. It wasn’t long before he found the culprit – the password hash for the conference bridge for those users in question. For the majority of the users of the conference bridge, I used the same, relatively simple password for every person when setting up their bridge access for the first time. The stored hash for that password, as well as one other password that was used more than once in the system, was causing the problem. Our DBA nulled out the passwords and the user settings were then accessible.
We aren’t sure if it was those two particular passwords or the fact that they were duplicated that was the issue, but we did learn that sometimes knowing your data is more important than anything a vendor could do for you. Because we were familiar with our users, our DBA was able to look for patterns that made sense to us. Our ticket has been with Shoretel for several weeks – it was likely they were looking for a programmatic issue of some kind, because the database was technically sound. Not sure how long it would have taken if our DBA hadn’t had time for a side project.
As a systems administrator, I like to think I can troubleshoot most issues. But database management is an area I don’t spend a lot of time in and I’m thankful for having a great DBA resource sitting nearby. Sometimes being good at your job means recognizing those that do their job well too and making sure they know you wouldn’t be nearly as good without them.
-small.jpg)
No comments:
Post a Comment