# Bug OP-14398 - Storages automatic project folders is exhausting file descriptors on workers

Project: OpenProject
Status: closed
URL: https://community.openproject.org/work_packages/OP-14398

## Open points

_No work packages_

## Details

| Attribute | Value |
| --- | --- |
| Date |  |
| Priority | Medium |
| Module |  |
| Target versions | 13.3.1 |
| Observed in versions | 13.3.0 |
| Sprint | 13.3.1 |
| Backlog bucket |  |
| Bug found in version (deprecated) | 13.3.x |
| Found in mobile app version |  |
| Follow-up feedback e-mail |  |
| Requires doc change | No |
| Accessibility | No |
| Pull preview |  |
| List |  |
| Labels |  |

## People

| Attribute | Value |
| --- | --- |
| Assignee |  |
| Accountable |  |
| Product manager |  |
| Developers | OP Dev Titan Team |
| Designer |  |
| Ordered by |  |
| Signed off by |  |
| Interest by |  |

## Planning

| Attribute | Value |
| --- | --- |
| Relevance |  |

## Costs

| Attribute | Value |
| --- | --- |
| Votes |  |
| Work |  |
| Labor costs |  |

## Description

### Error report

Happens on community.openproject.org.

We experienced the following error on our worker process: `Errno::EMFILE (Too many open files - getaddrinfo)`

As there are too many open files, some things do not work properly like mails not being sent, and we need to restart the worker process to start afresh.

### Analysis

There is one Nextcloud storage configured with automatic management. The host is [https://nextcloud.openproject.org](https://nextcloud.openproject.org/) which resolves to 193.142.157.16.

Every 5 minutes the `Storages::ManageNextcloudIntegrationCronJob` is triggered and do DAV requests on [nextcloud.openproject.org](https://nextcloud.openproject.org/).

Every time this job runs, a new socket is opened and never closed. It stays in the **CLOSE\_WAIT** state, meaning the server end has closed the connection, and the socket is waiting for the client end to close the connection.

In our code, we are using HTTP with persistent plugin. The [persistent plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Persistent.html) states this:

> ## When you should use this
> 
> So why isn’t this turned on by default? Keeping connections alive for the duration of a process alocates resources indefinitely, and in some edge cases, you might hit limitations of your OS (such as [EMFILE](https://www.man7.org/linux/man-pages/man3/errno.3.html)).
> 
> You should therefore use it:
> 
>   - If you’re connecting always to a known and bound set of origins;
>   - If your process is long lived;
> 
> You’re not recommended to use it:
> 
>   - If the number of origins you’re connecting to is unbounded (example: web scraping/web crawler);
>   - If your process is short lived;

And that's exactly what we are experiencing: the connections are kept alive and we have EMFILE errors.

### Current workaround

Cyril raised the limit of maximum open files on the system.

This can only be a temporary workaround.

### Detailed diagnosis

There are many open files. Finding where they come from:

``` text
root@worker-54968d8954-r7rxv:/app# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
app            1       0  0 Feb19 ?        00:00:00 /bin/bash -e /app/docker/prod/worker
app           28       1  4 Feb19 ?        00:48:33 /app/vendor/bundle/ruby/3.2.0/bin/rake jobs:work

root@worker-54968d8954-r7rxv:/app# su - app -c "ls /proc/1/fd -l"
total 0
lrwx------ 1 app app 64 Feb 19 16:25 0 -> /dev/null
l-wx------ 1 app app 64 Feb 19 16:25 1 -> 'pipe:[36045]'
l-wx------ 1 app app 64 Feb 19 16:25 2 -> 'pipe:[36046]'
lr-x------ 1 app app 64 Feb 20 08:18 255 -> /app/docker/prod/worker
# => this one is fine

root@worker-54968d8954-r7rxv:/app# su - app -c "ls /proc/28/fd -l"
total 0
lrwx------ 1 app app 64 Feb 20 08:19 0 -> /dev/null
l-wx------ 1 app app 64 Feb 20 08:19 1 -> 'pipe:[36045]'
lrwx------ 1 app app 64 Feb 20 08:19 10 -> 'socket:[36941]'
lrwx------ 1 app app 64 Feb 20 08:19 100 -> 'socket:[546712]'
lrwx------ 1 app app 64 Feb 20 08:19 101 -> 'socket:[554716]'
lrwx------ 1 app app 64 Feb 20 08:19 102 -> 'socket:[559388]'
[...]
lrwx------ 1 app app 64 Feb 20 09:51 194 -> 'socket:[1302881]'
lrwx------ 1 app app 64 Feb 20 09:51 195 -> 'socket:[1311134]'
lrwx------ 1 app app 64 Feb 20 09:51 196 -> 'socket:[1317039]'
lrwx------ 1 app app 64 Feb 20 09:51 197 -> 'socket:[1321923]'
# => almost 200 sockets open. that's too many.

root@worker-54968d8954-r7rxv:/app# su - app -c "netstat -tonp"
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name     Timer
tcp        1      0 100.64.9.137:39786      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:56382      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:48878      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:42264      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:49158      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:41598      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:40900      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:60416      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:50014      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:58772      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        0      0 100.64.9.137:36266      172.16.0.2:5432         ESTABLISHED -                    keepalive (4971.68/0/0)
tcp        1      0 100.64.9.137:52684      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:37610      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
[...]
[...]
tcp        1      0 100.64.9.137:48422      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
tcp        1      0 100.64.9.137:43982      193.142.157.16:443      CLOSE_WAIT  28/rake jobs:work    off (0.00/0/0)
```

Almost all open files are socket connections, and these socket connections are established with 193.142.157.16 and are in CLOSE\_WAIT state.

To find exactly when and how that happens, I ran a small script to have every second: the time, the number of CLOSED\_WAIT connections, the non-CLOSED\_WAIT connection to 193.142.157.16:

``` text
root@worker-54968d8954-r7rxv:/app# while true ; do date ; netstat -ton | grep CLOSE_WAIT |wc -l ; netstat -ton | grep -v CLOSE_WAIT | grep 193.142.157.16 ; echo ; sleep 1; done

[...]

Tue Feb 20 09:15:01 UTC 2024
183

Tue Feb 20 09:15:02 UTC 2024
183

Tue Feb 20 09:15:03 UTC 2024
183

Tue Feb 20 09:15:04 UTC 2024
183

Tue Feb 20 09:15:05 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:15:06 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:15:07 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:15:08 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

[...]

Tue Feb 20 09:16:19 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:16:20 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:16:21 UTC 2024
183
tcp        0      0 100.64.9.137:34124      193.142.157.16:443      ESTABLISHED off (0.00/0/0)

Tue Feb 20 09:16:22 UTC 2024
184

Tue Feb 20 09:16:23 UTC 2024
184

Tue Feb 20 09:16:24 UTC 2024
184
```

The connection is established at 09:15:05, and closed by server at 09:16:21

looking at worker logs:

``` text
I, [2024-02-20T09:15:05.354539 #28]  INFO -- : [ActiveJob] [Storages::ManageNextcloudIntegrationCronJob] [1bbeb26f-57d9-4007-88fb-70135b80a220] Performing Storages::ManageNextcloudIntegrationCronJob (Job ID: 1bbeb26f-57d9-4007-88fb-70135b80a220) from DelayedJob(default) enqueued at 2023-11-07T13:40:38.000000000Z
I, [2024-02-20T09:15:31.455423 #28]  INFO -- : [ActiveJob] [Storages::ManageNextcloudIntegrationCronJob] [1bbeb26f-57d9-4007-88fb-70135b80a220] Performed Storages::ManageNextcloudIntegrationCronJob (Job ID: 1bbeb26f-57d9-4007-88fb-70135b80a220) from DelayedJob(default) in 26100.92ms
```

Job started at 09:15:05, and finishes 26 seconds later at 09:15:31.

As the server closed the connection at 09:16:21, it means it waits for 50 seconds before closing the connection on its end.

[HTTPX Persistent plugin doc](https://honeyryderchuck.gitlab.io/httpx/wiki/Persistent.html) indicates that "If the connection goes down (because the peer closed it after a period of inactivity), the connection will be reopened and requests will be retried in the new connection".

I think HTTPX should take care of closing the old connection, so that may be a bug from their end.

Nevertheless, we should revisit our usage of persistent connections to make sure open files leaking does not happen.
