Bioware Developer Forum Posts
- Details
- Category: Developer Forum Posts
- Published: Wednesday, October 28th, 2020
- Written by JenHarkness
Originally Posted by JenHarkness ( Original Post ) | 22.10.2020 10:59AM
Hi everyone! I know the guild ships have been disappearing for a lot of guilds lately, and it seems like it's been around forever. The good news is that things should be much better after the release of 6.1.4. Since this has been around a while, I wanted to take a bit of time to give you all a peek behind the curtain to help explain why it was so hard to track down. I also really want you to understand just how much we appreciate your forum posts and bug reports. Even when we don't reply, as a developer my first stop in investigating a backend issue is always looking at logs, and the more detailed the reports are, the easier it is for me to find something.
A bit of background on the Backend
Like many games, we store our persistent character data in a database. To make updates faster, we have a cache layer between the game and the database that batches up updates. Then we write the batch to the database. The benefit is we get better performance, but the drawback is that if there is one bad update in the batch, we abort the entire batch. Most of the time, the effect of aborting is that characters get kicked to character select, which isn't great, but is better than the data being corrupted. The good thing is, these errors are pretty rare, and are always the highest priority for us to fix.
So why might an update fail? Well, one of the nifty things you can do with a database is define a constraint on your data. That's a rule that can never be broken. Now the database is your last line of defense against something going wrong and your data getting corrupted. For this story, the important constraint is on unique character names. If we try to write a batch of updates, and one of the characters in that update has the same name as someone else in the database, the batch gets aborted.
A Victim of our own Success
Now, you all might remember some complaints on the forums lately about character renames not working. Behind the scenes, we have a big list of all the names on each server, so that if someone tries to use a name that's taken, the rename gets rejected before it gets anywhere near the database. Well... how does that list of names get built, you might ask. Basically, before we let you guys into the servers after a restart, the servers ask the database for that list of names so that they can make quick decisions later.
Here's the part where we have become a victim of our own success. It turns out that the programmer who wrote that database query back in 2009 or so added a parameter for how far back in the past to search for taken names. He happened to put 3000 for the default number of days. Yup, that's a bit over 8 years, and we launched a bit less than 9 years ago. It's a bit off because of some database maintenance we did the first year, but you get the point. So anyone who tried to rename a character with one of the names that was last played 8+ years ago would have the rename look like it succeeds, but then the database says no.
Luckily, that was a pretty easy fix, just increase the number. (Although if we start having rename issues in 2093, feel free to yell at me!)
OK.. but why would this have anything to do with guild ships??
Well, it turns out that sometimes the data that stores info on your guild ships was part of the batch that was being kicked out because of a bad rename. Yes, I know the guild ship problems have been happening for years, but that's because anytime we were having database errors, sometimes the guild ships were being caught in the crossfire. However, like I mentioned above, we always fixed the database errors super quick, so people would complain about guild ships, but then we couldn't reproduce it and they died down, because we'd already fixed the database error.
Here's the good news, this time it's really fixed. We've both fixed the rename error, and thanks to the guilds who have given super timely info, we were able to spot the guild ships getting caught in the crossfire. Now that we know what's going on, we were able to make it so that if the guild ship does get batched up with a bad batch, it will be automatically reloaded. So, no more disappearing flagships even if we have database*errors in the future. For realz this time.* (We hope!)
A bit of background on the Backend
Like many games, we store our persistent character data in a database. To make updates faster, we have a cache layer between the game and the database that batches up updates. Then we write the batch to the database. The benefit is we get better performance, but the drawback is that if there is one bad update in the batch, we abort the entire batch. Most of the time, the effect of aborting is that characters get kicked to character select, which isn't great, but is better than the data being corrupted. The good thing is, these errors are pretty rare, and are always the highest priority for us to fix.
So why might an update fail? Well, one of the nifty things you can do with a database is define a constraint on your data. That's a rule that can never be broken. Now the database is your last line of defense against something going wrong and your data getting corrupted. For this story, the important constraint is on unique character names. If we try to write a batch of updates, and one of the characters in that update has the same name as someone else in the database, the batch gets aborted.
A Victim of our own Success
Now, you all might remember some complaints on the forums lately about character renames not working. Behind the scenes, we have a big list of all the names on each server, so that if someone tries to use a name that's taken, the rename gets rejected before it gets anywhere near the database. Well... how does that list of names get built, you might ask. Basically, before we let you guys into the servers after a restart, the servers ask the database for that list of names so that they can make quick decisions later.
Here's the part where we have become a victim of our own success. It turns out that the programmer who wrote that database query back in 2009 or so added a parameter for how far back in the past to search for taken names. He happened to put 3000 for the default number of days. Yup, that's a bit over 8 years, and we launched a bit less than 9 years ago. It's a bit off because of some database maintenance we did the first year, but you get the point. So anyone who tried to rename a character with one of the names that was last played 8+ years ago would have the rename look like it succeeds, but then the database says no.
Luckily, that was a pretty easy fix, just increase the number. (Although if we start having rename issues in 2093, feel free to yell at me!)
OK.. but why would this have anything to do with guild ships??
Well, it turns out that sometimes the data that stores info on your guild ships was part of the batch that was being kicked out because of a bad rename. Yes, I know the guild ship problems have been happening for years, but that's because anytime we were having database errors, sometimes the guild ships were being caught in the crossfire. However, like I mentioned above, we always fixed the database errors super quick, so people would complain about guild ships, but then we couldn't reproduce it and they died down, because we'd already fixed the database error.
Here's the good news, this time it's really fixed. We've both fixed the rename error, and thanks to the guilds who have given super timely info, we were able to spot the guild ships getting caught in the crossfire. Now that we know what's going on, we were able to make it so that if the guild ship does get batched up with a bad batch, it will be automatically reloaded. So, no more disappearing flagships even if we have database*errors in the future. For realz this time.* (We hope!)
