Ok, this is frustrating... the link here:

https://docs.openproject.org/installation-and-operations/configuration/ssl/

is apparently outdated...

I had no problem with the commands, everything seemed to work, but there is no /etc/letsencrypt/live directory, there appears to be only one .pem cert, that was installed to /etc/letsencrypt/keys

So... when reconfiguring openproject, do I skip the two entries asking for the individual cert/key, and only put in this path/to/file for the bnundled .pem?

And, I was still getting the errors, until I remembered...

We use Cloudflare for DNS, and I had set up the sub-domain A record in proxied mode...

Disabling that got rid of the error... and now I have the /live directory under /etc/letsencrypt

I just tested, and we now have access to it from outside t he office on SSL.

yay!

Thanks so much for your help! I probably wouldn't have gotten it running otherwise.

Ok, so, now the problem is, this system is extremely, maddeningly slow to load every single page. It literally takes 20 seconds or so when moving between pages. This is not usable, so I need to figure out what is wrong. There is nothing in the apache2 logs, syslog, or any other logs that I can see...

What are your system specs in terms of memory?

Can you ps aux | grep open to see how many processes are running? OpenProject logs are accessible via openproject logs.

VMWare ESXi VM, one CPU/core, 8GB Ram

root@deb-projects:~# ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root         1  0.0  0.1 103776 10004 ?        Ss   Aug06   0:01 /sbin/init

root         2  0.0  0.0      0     0 ?        S    Aug06   0:00 [kthreadd]

root         3  0.0  0.0      0     0 ?        I<   Aug06   0:00 [rcu_gp]

root         4  0.0  0.0      0     0 ?        I<   Aug06   0:00 [rcu_par_gp]

root         6  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/0:0H-kblockd]

root         7  0.0  0.0      0     0 ?        I    Aug06   0:00 [kworker/u2:0-events_unbound]

root         8  0.0  0.0      0     0 ?        I<   Aug06   0:00 [mm_percpu_wq]

root         9  0.0  0.0      0     0 ?        S    Aug06   0:00 [ksoftirqd/0]

root        10  0.0  0.0      0     0 ?        I    Aug06   0:03 [rcu_sched]

root        11  0.0  0.0      0     0 ?        I    Aug06   0:00 [rcu_bh]

root        12  0.0  0.0      0     0 ?        S    Aug06   0:00 [migration/0]

root        13  0.0  0.0      0     0 ?        I    Aug06   0:30 [kworker/0:1-events]

root        14  0.0  0.0      0     0 ?        S    Aug06   0:00 [cpuhp/0]

root        15  0.0  0.0      0     0 ?        S    Aug06   0:00 [kdevtmpfs]

root        16  0.0  0.0      0     0 ?        I<   Aug06   0:00 [netns]

root        17  0.0  0.0      0     0 ?        S    Aug06   0:00 [kauditd]

root        18  0.0  0.0      0     0 ?        S    Aug06   0:00 [khungtaskd]

root        19  0.0  0.0      0     0 ?        S    Aug06   0:00 [oom_reaper]

root        20  0.0  0.0      0     0 ?        I<   Aug06   0:00 [writeback]

root        21  0.0  0.0      0     0 ?        S    Aug06   0:00 [kcompactd0]

root        22  0.0  0.0      0     0 ?        SN   Aug06   0:00 [ksmd]

root        23  0.0  0.0      0     0 ?        SN   Aug06   0:00 [khugepaged]

root        24  0.0  0.0      0     0 ?        I<   Aug06   0:00 [crypto]

root        25  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kintegrityd]

root        26  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kblockd]

root        27  0.0  0.0      0     0 ?        I<   Aug06   0:00 [edac-poller]

root        28  0.0  0.0      0     0 ?        I<   Aug06   0:00 [devfreq_wq]

root        29  0.0  0.0      0     0 ?        S    Aug06   0:00 [watchdogd]

root        30  0.0  0.0      0     0 ?        S    Aug06   0:00 [kswapd0]

root        48  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kthrotld]

root        49  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/24-pciehp]

root        50  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/25-pciehp]

root        51  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/26-pciehp]

root        52  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/27-pciehp]

root        53  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/28-pciehp]

root        54  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/29-pciehp]

root        55  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/30-pciehp]

root        56  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/31-pciehp]

root        57  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/32-pciehp]

root        58  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/33-pciehp]

root        59  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/34-pciehp]

root        60  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/35-pciehp]

root        61  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/36-pciehp]

root        62  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/37-pciehp]

root        63  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/38-pciehp]

root        64  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/39-pciehp]

root        65  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/40-pciehp]

root        66  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/41-pciehp]

root        67  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/42-pciehp]

root        68  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/43-pciehp]

root        69  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/44-pciehp]

root        70  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/45-pciehp]

root        71  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/46-pciehp]

root        72  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/47-pciehp]

root        73  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/48-pciehp]

root        74  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/49-pciehp]

root        75  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/50-pciehp]

root        76  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/51-pciehp]

root        77  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/52-pciehp]

root        78  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/53-pciehp]

root        79  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/54-pciehp]

root        80  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/55-pciehp]

root        81  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ipv6_addrconf]

root        83  0.0  0.0      0     0 ?        I    Aug06   0:00 [kworker/u2:1-events_unbound]

root        92  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kstrp]

root       135  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_0]

root       136  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_0]

root       137  0.0  0.0      0     0 ?        I<   Aug06   0:00 [vmw_pvscsi_wq_0]

root       140  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ata_sff]

root       142  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_1]

root       144  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_1]

root       146  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_2]

root       148  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_2]

root       191  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/0:1H-kblockd]

root       221  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/u3:0]

root       223  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/sda1-8]

root       224  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ext4-rsv-conver]

root       260  0.0  0.1  27612  9408 ?        Ss   Aug06   0:00 /lib/systemd/systemd-journald

root       277  0.0  0.0  22064  4860 ?        Ss   Aug06   0:00 /lib/systemd/systemd-udevd

root       370  0.0  0.0      0     0 ?        I<   Aug06   0:00 [nfit]

root       371  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ttm_swap]

root       373  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/16-vmwgfx]

root       430  0.0  0.1  48220 10872 ?        Ss   Aug06   0:00 /usr/bin/VGAuthService

systemd+   431  0.0  0.0  93084  6556 ?        Ssl  Aug06   0:00 /lib/systemd/systemd-timesyncd

root       432  0.0  0.1  48496 11460 ?        Ss   Aug06   0:26 /usr/bin/vmtoolsd

root       435  0.0  0.0   8504  2820 ?        Ss   Aug06   0:00 /usr/sbin/cron -f

root       437  0.0  0.0   5260   684 ?        Ss   Aug06   0:00 /bin/sleep infinity

root       439  0.0  0.0   5260   684 ?        Ss   Aug06   0:00 /bin/sleep infinity

openpro+   440  0.0  3.8 472796 311672 ?       Ssl  Aug06   0:41 /opt/openproject/vendor/bundle/ruby/2.6.0/bin/rake jobs:work

root       443  0.0  0.0   5260   744 ?        Ss   Aug06   0:00 /bin/sleep infinity

openpro+   444  0.0  0.0   6728  3172 ?        Ss   Aug06   0:00 /bin/bash -e ./packaging/scripts/web

root       445  0.0  0.0 225824  4320 ?        Ssl  Aug06   0:00 /usr/sbin/rsyslogd -n -iNONE

root       450  0.0  0.0  19392  7296 ?        Ss   Aug06   0:00 /lib/systemd/systemd-logind

memcache   454  0.0  0.2 422560 16992 ?        Ssl  Aug06   0:13 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid

message+   461  0.0  0.0   8972  3688 ?        Ss   Aug06   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

root       475  0.0  0.0  15852  6600 ?        Ss   Aug06   0:00 /usr/sbin/sshd -D

root       479  0.0  0.0   5612  1588 tty1     Ss+  Aug06   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux

openpro+   537  0.0  4.0 508032 333404 ?       Sl   Aug06   0:23 unicorn master --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

postgres   542  0.0  0.3 213156 26912 ?        S    Aug06   0:01 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/post

postgres   544  0.0  0.1 213300  8404 ?        Ss   Aug06   0:00 postgres: 10/main: checkpointer process

postgres   545  0.0  0.0 213156  7444 ?        Ss   Aug06   0:00 postgres: 10/main: writer process

postgres   546  0.0  0.1 213156  9832 ?        Ss   Aug06   0:00 postgres: 10/main: wal writer process

postgres   547  0.0  0.0 213592  6788 ?        Ss   Aug06   0:00 postgres: 10/main: autovacuum launcher process

postgres   548  0.0  0.0  68348  5048 ?        Ss   Aug06   0:01 postgres: 10/main: stats collector process

postgres   549  0.0  0.0 213448  5444 ?        Ss   Aug06   0:00 postgres: 10/main: bgworker: logical replication launcher

root       550  0.0  0.4  49544 37900 ?        Ss   Aug06   0:01 /usr/sbin/apache2 -k start

www-data   552  0.0  0.3  48968 30760 ?        S    Aug06   0:00 /usr/sbin/apache2 -k start

www-data   553  0.0  0.5 796448 44600 ?        Sl   Aug06   0:01 /usr/sbin/apache2 -k start

www-data   554  0.0  0.5 796520 43632 ?        Sl   Aug06   0:01 /usr/sbin/apache2 -k start

root       622  0.0  0.0  16632  7852 ?        Ss   Aug06   0:00 sshd: mf-admin [priv]

mf-admin   625  0.0  0.1  21024  8344 ?        Ss   Aug06   0:00 /lib/systemd/systemd --user

mf-admin   626  0.0  0.0  23096  2244 ?        S    Aug06   0:00 (sd-pam)

mf-admin   639  0.0  0.0  16916  5844 ?        S    Aug06   0:00 sshd: mf-admin@pts/0

mf-admin   640  0.0  0.0   7912  4660 pts/0    Ss   Aug06   0:00 -bash

root       643  0.0  0.0   9768  3476 pts/0    S    Aug06   0:00 su -

root       644  0.0  0.0   8372  5216 pts/0    S    Aug06   0:00 -bash

postgres   660  0.0  0.2 215196 21708 ?        Ss   Aug06   0:08 postgres: 10/main: openproject openproject 127.0.0.1(54126) idle

openpro+   662  0.0  4.1 508032 335436 ?       Sl   Aug06   0:02 unicorn worker[0] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   663  0.0  4.1 508152 337996 ?       Sl   Aug06   0:02 unicorn worker[1] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   666  0.0  4.1 508032 336084 ?       Sl   Aug06   0:03 unicorn worker[2] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   667  0.0  4.1 508032 336908 ?       Sl   Aug06   0:02 unicorn worker[3] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

root      4442  0.0  0.0      0     0 ?        I    09:27   0:00 [kworker/0:0-ata_sff]

postgres  4472  0.0  0.2 215808 22428 ?        Ss   09:32   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54450) idle

root      4475  0.0  0.0      0     0 ?        I    09:33   0:00 [kworker/0:2-ata_sff]

postgres  4476  0.0  0.2 215164 20344 ?        Ss   09:33   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54456) idle

postgres  4485  0.0  0.2 214844 16772 ?        Ss   09:34   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54470) idle

postgres  4486  0.0  0.2 214896 18604 ?        Ss   09:34   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54474) idle

root      4495  0.0  0.0  10632  3160 pts/0    R+   09:35   0:00 ps aux

root@deb-projects:~#

Aaargh, sorry, you said to run it through grep...

root@deb-projects:# ps aux | grep open
openpro+   440  0.0  3.8 472796 311672 ?       Ssl  Aug06   0:41 /opt/openproject/vendor/bundle/ruby/2.6.0/bin/rake jobs:work
openpro+   444  0.0  0.0   6728  3172 ?        Ss   Aug06   0:00 /bin/bash -e ./packaging/scripts/web
openpro+   537  0.0  4.0 508032 333404 ?       Sl   Aug06   0:23 unicorn master --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
postgres   660  0.0  0.2 215196 21708 ?        Ss   Aug06   0:08 postgres: 10/main: openproject openproject 127.0.0.1(54126) idle
openpro+   662  0.0  4.1 508032 335436 ?       Sl   Aug06   0:02 unicorn worker[0] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   663  0.0  4.1 508152 337996 ?       Sl   Aug06   0:02 unicorn worker[1] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   666  0.0  4.1 508032 336084 ?       Sl   Aug06   0:03 unicorn worker[2] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   667  0.0  4.1 508032 336908 ?       Sl   Aug06   0:02 unicorn worker[3] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
root      4523  0.0  0.0   6076   880 pts/0    S+   09:43   0:00 grep open
root@deb-projects:#

Ok, so, what would I be looking for in the OpenProject logs?

Your output looks fine. You should check what's happening in the Network tab of the Firefox or Chrome developer console, to see if the slowness is due to a client-side or server-side issue.

Ok, just figured out, the slowness is only when accessing from the internal LAN. Accessing from my phone over the internet it is fine...

Hmmmmmmmmm.... I wonder if the way I ended up doing this is the issue...

IWe use Windows Server (2016) for DNS. I created a sub-domain - atl.mediafiche.com, and then an A record pointing to the local server IP for OpenProject.

Then, I did the same thing in Cloudflare, but pointed it to a secondary public IP, and routed that traffic through my firewall to the OP server.

I'll start poking around, but I'm not sure why that would be a problem...

Ok, any ideas why it would be dog slow when accessing internally, but fine when accessing over the internet? I've checked everything I can think of... all internal DNS pointers resolve properly, no conflicts

I see messages like these - and wondering about the 'duration' times being seen... the messages only present in the logs when the page finally loads in the browser, nothing gets added when I click ro while I'm waiting...

Aug  7 16:15:26 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:26.117046 #665]  INFO -- : method=GET path=/admin/design/upsale format=html controller=CustomStylesController action=upsale status=200 duration=36.65 view=28.09 db=2.88 user=4

Aug  7 16:15:28 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:28.827210 #664]  INFO -- : method=GET path=/admin/enumerations format=html controller=EnumerationsController action=index status=200 duration=117.03 view=97.00 db=14.03 user=4

Aug  7 16:15:30 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:30.854229 #665]  INFO -- : method=GET path=/settings format=html controller=SettingsController action=show status=200 duration=58.62 view=38.28 db=8.17 user=4

Aug  7 16:15:34 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:34.058227 #664]  INFO -- : method=GET path=/users format=html controller=UsersController action=index status=200 duration=67.52 view=54.14 db=6.57 user=4

Aug  7 16:16:39 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:16:39.166950 #668]  INFO -- : method=GET path=/users/4/edit format=html controller=UsersController action=edit status=200 duration=366.30 view=328.46 db=19.14 user=4

Aug  7 16:17:01 deb-projects CRON[1527]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Aug  7 16:19:05 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:19:05.837829 #662]  INFO -- : method=GET path=/admin/groups format=html controller=GroupsController action=index status=200 duration=60.72 view=36.14 db=9.96 user=4

Aug  7 16:19:54 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:19:54.536292 #664]  INFO -- : method=GET path=/admin format=html controller=AdminController action=index status=200 duration=37.91 view=28.74 db=3.06 user=4

Are those duration, view and db numbers normal?

The durations are in milliseconds, so they definitely look fine. You should try to see if there are any warnings in the Apache log (/var/log/apache2/error*.log or through journalctl).

If I understand correctly, you have both a public IP and a private IP linked to that VM, and the traffic is slow via the private IP. It might be that Apache performs some reverse lookup on the source private IP, and that lookup times out. You should check whether host YOUR_PRIVATE_IP returns something when run from your VM (YOURPRIVATEIP being the IP of your internal computer from which you are trying to connect). Just a wild guess but worth a check.

The public IP is routed through an SNAT on my Watchguard that routes it to the internal private IP of 192.168.1.165

root@deb-projects:# host 192.168.1.165
165.1.168.192.in-ad
r.arpa domain name pointer projects.atl.mediafiche.com.
root@deb-projects:#

Which is the correct FQDN hostname

Nothing in apache2 error logs since noon today, and those were expected, from when I was setting it up...

And nothing of note in journalctl

Also...

root@deb-projects:~# nslookup projects.atl.mediafiche.com
Server:         192.168.1.170
Address:        192.168.1.170#53

Name:   projects.atl.mediafiche.com
Address: 192.168.1.165

root@deb-projects:~# nslookup 192.168.1.165
165.1.168.192.in-addr.arpa      name = projects.atl.mediafiche.com.

And yes, 192.168.1.170 is my internal DNS server

I don't know if it is significant, but sometimes - not all the time - when the delay is occurring, I see this ain the status area of the web browser (lower left corner) for many many seconds:

Transferring from secure.gravatar.com...

I'm seeing this more and more... maybe significant...

I've looked everywhere and can't see where or even if it is possible to totally disable gravatars to see if this is  the issue... so, is it possible, and if so, how/where?

Thanks

Ok, this really peeves me off...

With the issues I had initially getting LetsEncrypt SSL Cert installed, I THOUGHT I had tested using a different browser after I got SSL working but had this slow access. Apparently I didn't.

The problem was I was using Firefox - and it recently switched on SecureDNS by default, which bypasses the local DNS Server for all by localhost by default.

Simply adding my local domain to about:config > > network.trr.excluded-domains fixed it right up.

Now to go see if Firefox's current GPO support includes changes to this setting...