Ulimit conflict with PAM and Systemd

Mydbops

Dec 10, 2017

Mins to Read

All

As a part of the MySQL Support, we had a support request from a client.The issue is DB server runs out of open files limit, though it is configured. It causes the DB hang and crash at times. Sometimes they can’t able to fix. So we plan to write our experience with configuring. We believe this article can help in configuring appropriate Ulimit value without any obstacles.

Let us jump to the subject.

What are the errors we might face while ulimit is not properly configured?

Too many open files
System unable to allocate necessary resources for the monitor thread
can’t create new thread, closing connection

The above-shared list is just sample’s, maybe people who currently reading this blog may also face issue related to ulimit, for that they may have different debug message too.

What is ulimit in a simple term?

User limits command, limit the use of system-wide resources.

Where are the Ulimit Configuration file located and how it is loaded?

The ulimit resource configuration is located from the below file

/etc/security/limits.conf

/etc/security/limits.d/*.conf

By default /etc/security/limits.conf is loaded but it can be override by the configuration reside on /etc/security/limit.d/*conf.

ulimit resource limitation based on the Linux user. Every individual /etc/security/limit.d/*.conf are read and the file is parsed one after another in the order of “C” locale and concatenate together in the order of parsing.

As I have specified above, the file loaded from /etc/security/limit.d/*.conf in the order of “C” locale and apply, but there will be one exception. For example, we have two configurations in /etc/security/limit.d/ folder.

alpha.conf
mysql.conf

both files configured for resource limitation for mysql user(domain). Only mysql.conf applied because the file is in case the domain is the same or more specific.

How the default soft or hard limit for the number of user’s processes is applied?

The system wide configuration file /etc/security/limits.d/90-nproc.conf (RHEL5, RHEL6), /etc/security/limits.d/20-nproc.conf (RHEL7) specifies the default nproc limits as:

 
*      soft    nproc     4096 root        soft    nproc     unlimited

**How PAM Modules related to /etc/security/limit.d/*.conf and /etc/security/limits.conf ?**

The pam_limits PAM module sets limits on the system resources that can be obtained in a user session.

The /etc/pam.d/system-auth file is used by Red-Hat and like systems to group together common security policies. It is often included in other /etc/pam.d policy files where those common policies are required.

When accessing a system via ssh through sshd the /etc.pam.d/sshd policy file is consulted. This file includes /etc/pam.d/system-auth so your changes to /etc/pam.d/system-auth are valid.

The file /etc/pam.d/login is consulted when you log in via the /bin/login program, therefore, any changes to it only affect /bin/login.

login – rules for local (console login)

system-auth – common rules many services

password-auth – common rules for many remote services

sshd – rules for SSHD daemon only

How PAM create Obstacle while applying resource limit and How we mitigate?

After we allocate resource limit to the user, we try to login into the user account and check for the update resource information. But unfortunately, the new updated information not updated on the user login.

 
session requires pam_limits.so

This happen because of pam module not properly configured to load the /etc/security/limit.d/.conf* and /etc/security/limits.conf . We need to apply the above line into the /etc/pam.d/login or system-auth or password-auth or sshd file.

In most cases pam_limits.so will be enabled by default. In some cases, it is not so we need to append session required pam_limits line on appropriate pam config file related to the login program. This modification required logout.

Below I have shown, how to configure pam_limits.so module in system-auth configuration and applying ulimit value to the user:

Environment:

Os: Centos 7.4

Kernel: 3.10.0-693.11.1.el7.x86_64

Current Status:

 
[root@centos7 ~]# id uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Logging in as mysql (nologin)

 
root@centos7 ~]# su -s /bin/bash mysql
[mysql@centos7 root]$ id uid=1001(mysql) gid=1001(mysql) groups=1001(mysql) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Validating the ulimit for mysql user after logging in

 
[mysql@centos7 root]$ ulimit -a 
core file size          (blocks, -c) 0 
data seg size           (kbytes, -d) 
unlimited scheduling priority             (-e) 0 
file size               (blocks, -f) 
unlimited pending signals                 (-i) 1870 
max locked memory       (kbytes, -l) 64 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 1024 
pipe size            (512 bytes, -p) 8 
POSIX message queues     (bytes, -q) 819200 
real-time priority              (-r) 0 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 1870 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited

Trying to increase ulimit value for nofile and nproc by creating mysql.conf under /etc/security/limit.d/ dir:

 
[root@centos7 ~]# vim /etc/security/limits.d/mysql.conf 
mysql   soft    nofile  55000 
mysql   hard    nofile  655000 
mysql   soft    nproc   55000 
mysql   hard    nproc   655000

Now login and check the current ulimit value for mysql user:

 
[root@centos7 ~]# su -s /bin/bash mysql
[mysql@centos7 root]$ ulimit -a 
core file size          (blocks, -c) 0 
data seg size           (kbytes, -d) unlimited 
scheduling priority             (-e) 0 
file size               (blocks, -f) unlimited 
pending signals                 (-i) 1870 
max locked memory       (kbytes, -l) 64 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 1024 
pipe size            (512 bytes, -p) 8 
POSIX message queues     (bytes, -q) 819200 
real-time priority              (-r) 0 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 1870 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited

Ulimit value was not updated still, Now let us check the pam module:

 
[root@centos7 ~]# vim /etc/pam.d/system-auth 
#%PAM-1.0 # This file is auto-generated. 
# User changes will be destroyed the next time authconfig is run. 
auth        required      pam_env.so 
auth        required      pam_faildelay.so delay=2000000 
auth        sufficient    pam_unix.so nullok try_first_pass 
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_successauth        required      pam_deny.so 

account     required      pam_unix.so 
account     sufficient    pam_localuser.so 
account     sufficient    pam_succeed_if.so uid < 1000 quiet 
account     required      pam_permit.so 

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= 
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok 
password    required      pam_deny.so 

session     optional      pam_keyinit.so revoke 
#session     required      pam_limits.so 
session     optional      pam_systemd.so 
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid 
session     required      pam_unix.so

We see observe that pam_limits.so is not enabled (commented).Now let us enable pam_limits.so and validate it.

 
[root@centos7 ~]# cat /etc/pam.d/system-auth | grep "pam_limits.so"
session     required      pam_limits.so

 
[root@centos7 ~]# su -s /bin/bash mysql
[mysql@centos7 root]$ ulimit -n 55000
[mysql@centos7 root]$ ulimit -a core 
file size          (blocks, -c) 0 
data seg size           (kbytes, -d) unlimited 
scheduling priority             (-e) 0 
file size               (blocks, -f) unlimited 
pending signals                 (-i) 1870 
max locked memory       (kbytes, -l) 64 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 55000 
pipe size            (512 bytes, -p) 8 POSIX 
message queues     (bytes, -q) 819200 
real-time priority              (-r) 0 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 55000 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited

Successfully we have updated the resource limit for the user.

After changing the ulimit settings, you must restart the process (mysqld) to take advantage of the modified settings. You can use the /proc file system to see the current limitations on a running process.

Conflict between ulimit and Systemd

Now mostly every Linux distribution moving from init or upstart to Systemd. Systemd is very mature layer between kernel and application. Unfortunately not every User and OS is fully integrated with it.

When service started using systemd, it won’t consider the ulimit value defined for the process owner. Systemd is providing an option to set a limit over process using systemd variable likes LimitNOFILE and LimitNPROC.

How to configure systemd to limit open files and process?

Login into MySQL user and checking the openfile limit and process limit for the MySQL user.

 
[root@centos7 ~]# su -s /bin/bash mysql
[mysql@centos7 root]$ ulimit -a 
core file size          (blocks, -c) 0 
data seg size           (kbytes, -d) unlimited 
scheduling priority             (-e) 0 
file size               (blocks, -f) unlimited 
pending signals                 (-i) 1870 
max locked memory       (kbytes, -l) 64 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 55000 
pipe size            (512 bytes, -p) 8 
POSIX message queues     (bytes, -q) 819200 
real-time priority              (-r) 0 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 55000 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited

Validating current status of Mysql process using systemctl.

 
[root@centos7 ~]#systemctl status mysqld mysqld.service 
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2017-09-13 09:32:39 IST; 2 months 27 days ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
 Main PID: 9131 (mysqld)
   CGroup: /system.slice/mysqld.service
           └─9131 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Check owner of the mysql process.

 
[root@centos7 ~]#ps -aux | grep -i mysqld | grep -iv "grep" 
mysql     9131 54.0 82.3 8209676 6730120 ?     Sl   Sep13 68933:28 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

Finding openfile limit and process limit for MySQL process.

 
[root@centos7 ~]# cat /proc/`pidofmysqld`/limits 
Limit                     Soft Limit           Hard Limit           Units 
Max cpu time              unlimited            unlimited            seconds 
Max file size             unlimited            unlimited            bytes 
Max data size             unlimited            unlimited            bytes 
Max stack size            8388608              unlimited            bytes 
Max core file size        0                    unlimited            bytes 
Max resident set          unlimited            unlimited            bytes 
Max processes             1870                 1870                 processes 
Max open files           5000                5000                 files 
Max locked memory         65536                65536                bytes 
Max address space         unlimited            unlimited            bytes 
Max file locks            unlimited            unlimited            locks 
Max pending signals       1870                 1870                 signals 
Max msgqueue size         819200               819200               bytes 
Max nice priority         0                    0 
Max realtime priority     0                    0 
Max realtime timeout      unlimited            unlimited            us

Even Though MySQL User(Domain) process and openfile soft limit were set to 55000, it is not affecting MySQL process it is still the older value of 1870 and 5000. This because of MySQL service is started using systemctl.

Fixing systemd MySQL service file

Open the systemd config for MySQL service and append the limits LimitNOFILE=55000 and LimitPROC=55000 under service category.

 
[root@centos7 ~]# vim /usr/lib/systemd/system/mysqld.service

[Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=mysql.service

[Service]
User=mysql
Group=mysql
LimitNOFILE=55000
LimitNPROC=55000

Reload the daemon and Restart the MySQL service.

 
[root@centos7 ~]# systemctl daemon-reload 
[root@centos7 ~]# systemctl restart mysqld
[root@centos7 system]# cat /proc/`pidof mysqld`/limits 
Limit                     Soft Limit           Hard Limit           Units 
Max cpu time              unlimited            unlimited            seconds 
Max file size             unlimited            unlimited            bytes 
Max data size             unlimited            unlimited            bytes 
Max stack size            8388608              unlimited            bytes 
Max core file size        0                    unlimited            bytes 
Max resident set          unlimited            unlimited            bytes 
Max processes             55000                55000                processes 
Max open files            55000                55000                files 
Max locked memory         65536                65536                bytes 
Max address space         unlimited            unlimited            bytes 
Max file locks            unlimited            unlimited            locks 
Max pending signals       1870                 1870                 signals 
Max msgqueue size         819200               819200               bytes 
Max nice priority         0                    0 
Max realtime priority     0                    0 
Max realtime timeout      unlimited            unlimited            us

Sometimes we get a conflict between unit file located in /etc/systemd/system/multi-user.target.wants/mysqld.service if you still face issues with resource limits have a look at this file too.

Hope this helps in modifying the ulimit values for a user and process.

Linux

No items found.

About the Author

Mydbops

Ulimit conflict with PAM and Systemd​​​

Let us jump to the subject.

What are the errors we might face while ulimit is not properly configured?

The above-shared list is just sample’s, maybe people who currently reading this blog may also face issue related to ulimit, for that they may have different debug message too.

What is ulimit in a simple term?

User limits command, limit the use of system-wide resources.

Where are the Ulimit Configuration file located and how it is loaded?

The ulimit resource configuration is located from the below file

/etc/security/limits.conf

/etc/security/limits.d/*.conf

By default /etc/security/limits.conf is loaded but it can be override by the configuration reside on /etc/security/limit.d/*conf.

ulimit resource limitation based on the Linux user. Every individual /etc/security/limit.d/*.conf are read and the file is parsed one after another in the order of “C” locale and concatenate together in the order of parsing.

As I have specified above, the file loaded from /etc/security/limit.d/*.conf in the order of “C” locale and apply, but there will be one exception. For example, we have two configurations in /etc/security/limit.d/ folder.

both files configured for resource limitation for mysql user(domain). Only mysql.conf applied because the file is in case the domain is the same or more specific.

How the default soft or hard limit for the number of user’s processes is applied?

The system wide configuration file /etc/security/limits.d/90-nproc.conf (RHEL5, RHEL6), /etc/security/limits.d/20-nproc.conf (RHEL7) specifies the default nproc limits as:

How PAM Modules related to /etc/security/limit.d/*.conf and /etc/security/limits.conf ?

The pam_limits PAM module sets limits on the system resources that can be obtained in a user session.

The /etc/pam.d/system-auth file is used by Red-Hat and like systems to group together common security policies. It is often included in other /etc/pam.d policy files where those common policies are required.

When accessing a system via ssh through sshd the /etc.pam.d/sshd policy file is consulted. This file includes /etc/pam.d/system-auth so your changes to /etc/pam.d/system-auth are valid.

The file /etc/pam.d/login is consulted when you log in via the /bin/login program, therefore, any changes to it only affect /bin/login.

login – rules for local (console login)

system-auth – common rules many services

password-auth – common rules for many remote services

sshd – rules for SSHD daemon only

How PAM create Obstacle while applying resource limit and How we mitigate?

After we allocate resource limit to the user, we try to login into the user account and check for the update resource information. But unfortunately, the new updated information not updated on the user login.

This happen because of pam module not properly configured to load the /etc/security/limit.d/*.conf and /etc/security/limits.conf . We need to apply the above line into the /etc/pam.d/login or system-auth or password-auth or sshd file.

In most cases pam_limits.so will be enabled by default. In some cases, it is not so we need to append session required pam_limits line on appropriate pam config file related to the login program. This modification required logout.

Below I have shown, how to configure pam_limits.so module in system-auth configuration and applying ulimit value to the user:

Environment:

Os: Centos 7.4

Kernel: 3.10.0-693.11.1.el7.x86_64

Current Status:

Logging in as mysql (nologin)

Validating the ulimit for mysql user after logging in

Trying to increase ulimit value for nofile and nproc by creating mysql.conf under /etc/security/limit.d/ dir:

Now login and check the current ulimit value for mysql user:

Ulimit value was not updated still, Now let us check the pam module:

We see observe that pam_limits.so is not enabled (commented).Now let us enable pam_limits.so and validate it.

Successfully we have updated the resource limit for the user.

After changing the ulimit settings, you must restart the process (mysqld) to take advantage of the modified settings. You can use the /proc file system to see the current limitations on a running process.

Conflict between ulimit and Systemd

Now mostly every Linux distribution moving from init or upstart to Systemd. Systemd is very mature layer between kernel and application. Unfortunately not every User and OS is fully integrated with it.

When service started using systemd, it won’t consider the ulimit value defined for the process owner. Systemd is providing an option to set a limit over process using systemd variable likes LimitNOFILE and LimitNPROC.

How to configure systemd to limit open files and process?

Login into MySQL user and checking the openfile limit and process limit for the MySQL user.

Validating current status of Mysql process using systemctl.

Check owner of the mysql process.

Finding openfile limit and process limit for MySQL process.

Even Though MySQL User(Domain) process and openfile soft limit were set to 55000, it is not affecting MySQL process it is still the older value of 1870 and 5000. This because of MySQL service is started using systemctl.

Fixing systemd MySQL service file

Open the systemd config for MySQL service and append the limits LimitNOFILE=55000 and LimitPROC=55000 under service category.

Reload the daemon and Restart the MySQL service.

Sometimes we get a conflict between unit file located in /etc/systemd/system/multi-user.target.wants/mysqld.service if you still face issues with resource limits have a look at this file too.

Hope this helps in modifying the ulimit values for a user and process.

About the Author

Topic covered

Continue Reading

MySQL Clone Plugin Speed Test

Scale Your Database Traffic with Read/Write Splitting using MySQL Router: A Mydbops Webinar Recap

MySQL Release Cycle Revolution: Unleashing a New Era of Innovation and Stability

Mysqldump​ vs Mysqlpump vs Mydumper

Subscribe Now!

Ulimit conflict with PAM and Systemd

**How PAM Modules related to /etc/security/limit.d/*.conf and /etc/security/limits.conf ?**

This happen because of pam module not properly configured to load the /etc/security/limit.d/.conf* and /etc/security/limits.conf . We need to apply the above line into the /etc/pam.d/login or system-auth or password-auth or sshd file.

Mysqldump vs Mysqlpump vs Mydumper