Hello, this weekend I was doing something that should be a simple task of increasing the huge pages configuration in order to increase SGA in some of my client’s Production databases.
The process involves just a few steps and is very well described here by Tim Hall, so I won’t re-explain them. Basically, you should (considering you are using Oracle Linux, for other distributions the steps may be slightly different):
- Check the current configuration. Backup the files you are going to change (including DB spfile).
- Calculate and set the new number of huge pages you need at the OS level, and make it effective.
- Calculate and set the memlock user limit (ulimit) at the OS level.
- Increase the SGA and (optional) set USE_LARGE_PARGES to ONLY and restart the DB to make these changes effective.
As I said, this should be pretty straightforward, but in our case it wasn’t.
For some reason we didn’t know at that time, even doing all the configurations correctly and using the very same values and steps we’ve used successfully in other servers, it did not work at first.
When we tried to restart the DB with the increased SGA, we were receiving the following error message:
The message shows that Oracle was unable to allocate all the shared memory for the SGA.
Well, to make a long story short, after a lot of research, “ipcs” and restarts of servers to make sure there was no “trash” memory chunks lost, Oracle SR and tries, we determined that the problem was that the ulimit for memlock was not being respected. The value shown when we issued “ulimit -l” had nothing to do with the value we’ve put for memlock in the /etc/security/limits.conf file.
Then we realized that in Doc ID 361468.1 there is the following statement:
“These settings can also be done in a *.conf file under /etc/security/limits.d/ directory. Also if there is a *.conf file in /etc/security/limits.d/ that has settings for memlock, setting it up in /etc/security/limits.conf may not work. Make sure you check the files /etc/security/limits.d/*.conf files.”
So, our problem was simply there were different *conf files in that folder, with different ulimit values set, and this was messing up our configuration.
Interestingly, the same files with the same configurations were in several servers, but it the issues occurred in some servers but not in others. And, if you see the names of the files (related to oracle preinstall configuration), I would guess those files were put into the server during the process of installation of new versions of the DB software.
So, at this point I don’t know for sure what/who created the files and put them there, and what is the order they are validated that made it a problem in some servers but not in others. Date order maybe? Let me know in the comments if you know it.
Anyway, this post is an advisory: make sure you don’t have undesired *conf files in /etc/security/limits.d/ that could mess up your configuration. Just rename them (making sure it won’t break anything) for something not ending with “.conf” and they won’t be used.
That’s it, see you next time.
EDUARDO CLARO 