Oh, the file that was created is:

/etc/letsencryot/keys/0000_key-certbot.pem

And I just tried and am unable to skip the prompt for the SSL certificate

Hello,

Could you let me know which distribution you are using? Or paste the output of cat /etc/os-release?

As far as I know Let's Encrypt files are supposed to be under /etc/letsencrypt/live. Can you also paste the output of certbot --version ?

Thank you

Regarding the prompt for the SSL certificate, please make sure you run openproject reconfigure so that you are asked again whether you want to setup SSL support.

root@projects:/home/mf-admin# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@projects:/home/mf-admin#

certbot --version gives command not found - maybe it isn't in the path?

I poked around, and found /usr/local/bin/certbot-auto, and tried that command with the --version, but it went through an upgrade process instead of just showing me the version:

root@projects:/home/mf-admin# /usr/local/bin/certbot-auto --version
Upgrading certbot-auto 1.6.0 to 1.7.0...
Replacing certbot-auto...
Creating virtual environment...
Installing Python packages...
Installation succeeded.
certbot 1.7.0
root@projects:/home/mf-admin#

Like I said, I installed it following the instructions at the link I provided.

Thanks for y our help!

I did run openproject reconfigure, but wasn't sure if I was supposed to skip everything until I got to the part where I enable SSL - or if I was supposed to re-enter the exact same setup as I did before...

It would be much better if there was a flag like "openporoject reconfigure -SSL-ONLY" or something, so it wouldn't touch the existing config and would only do what is necessary to enable SSL.

I think I'm going to have to nuke and reinstall from scrathch. Not a huge deal, but I'd prefer not to, since I did spend a couple of hours setting things up...

I'm kicking myself now for not doing a DB dump/backup before trying this... my bad... and I know better.

Any ideas? I'm getting ready to nuke the Debian install and reinstall from scratch, but would like to enable SSL from the start this time - but still don't have proper instructions for defining the Certs...

Hi Charles, I tested on a Debian 10 yesterday and don't get the same output as you. It does create 2 files in /etc/letsencrypt/live/my.domain.com/ (fullchain.pem and privkey.pem). The I just need to openproject reconfigure, hit ENTER (previously configured values are kept by default) until I get to the SSL wizard, and fill in the requested info with those 2 paths.

Could you try requesting certificates from certbot-auto again and pasting the full output? At the end of a successful run it should display the path towards the 2 files I mentioned above, which are all you need to fill in when configuring OpenProject.

Well, its too late, I think I'd already clobbered the database anyway - so, I just nuked and reinstalled debian from scratch, getting ready to install openproject again...

Do I just enter the path, or do I also specify the filename too... so, is it:

/etc/letsencrypt/live/mydomain.com/

or

/etc/letsencrypt/live/my.domain.com/privkey.pem

?

Also, if I'm not mistaken, it asks for THREE paths, the first two are for CERT, then KEY, and the last is something about a bundled .pem,

Thanks for your help!

Oh, and please confirm - you were using the exact same commands from the link I posted?

You need to enter the full path. I just opened a PR to improve the doc: https://github.com/opf/openproject/pull/8550/files. Let me know if it helps. Direct link to updated file is at https://github.com/opf/openproject/blob/491161f9aca9f41b93be571531b23ca5e1efaf87/docs/installation-and-operations/configuration/ssl/README.md#create-a-free-ssl-certificate-using-lets-encrypt.

And yes I did enter the exact same commands as indicated in the doc.

Ok, thanks so much for the fast responses! Excellent support, I must say.

I'm installing openproject now, will let you know how it goes.

Ok, same problem... there is no /etc/letsencrypt/live directory created...

Question... I'm doing the letsencrypt step BEFORE I run openproject configure.

At the end of the letsencrypt step I get this (obviously, since the web server i sn't set up, this is going to fail):

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for projects.atl.mediafiche.com
Using the webroot path /opt/openproject/public for all unmatched domains.
Waiting for verification...
Challenge failed for domain projects.atl.mediafiche.com <-- in red
http-01 challenge for projects.atl.mediafiche.com
Cleaning up challenges
Some challenges have failed. <-- in red

IMPORTANT NOTES:
- The following errors were reported by the server:

  Domain: projects.atl.mediafiche.com
  Type:   tls
  Detail: Fetching
  https://projects.atl.mediafiche.com/.well-known/acme-challenge/LnCbUsmDRNQl_wVYvoLV1CARCHB3_iYVV1h5TQHVSX4:
  remote error: tls: handshake failure

  To fix these errors, please make sure that your domain name was
  entered correctly and the DNS A/AAAA record(s) for that domain
  contain(s) the right IP address. Additionally, please check that
  you have an up-to-date TLS configuration that allows the server to
  communicate with the Certbot client.
- Your account credentials have been saved in your Certbot
  configuration directory at /etc/letsencrypt. You should make a
  secure backup of this folder now. This configuration directory will
  also contain certificates and private keys obtained by Certbot so
  making regular backups of this folder is ideal.
root@deb-projects:/home/mf-admin# 

So... what am I missing??

For Let's Encrypt to work, you need to have a web server running at projects.atl.mediafiche.com. The easiest way to do this is to configure OpenProject without SSL so that you have something running at http://projects.atl.mediafiche.com. Then Let's Encrypt will be able to perform the challenge, and then you reconfigure

Here is the contents of /etc/letsencrypt:

root@deb-projects:/home/mf-admin# ls -al /etc/letsencrypt/
total 28
drwxr-xr-x  7 root root 4096 Aug  6 08:51 .
drwxr-xr-x 74 root root 4096 Aug  6 08:50 ..
drwx------  3 root root 4096 Aug  6 08:50 accounts
drwxr-xr-x  2 root root 4096 Aug  6 08:51 csr
drwx------  2 root root 4096 Aug  6 08:51 keys
drwxr-xr-x  2 root root 4096 Aug  6 08:51 renewal
drwxr-xr-x  5 root root 4096 Aug  6 08:50 renewal-hooks
root@deb-projects:/home/mf-admin#

Ah, ok - maybe add one more change to the doc, and clarify  that...

Thanks very much... so, after I configure Openproject, should I just re-run the letsencrypt commands?

Yes I'll add this to the doc. And yes you just have to re-run the certbot-auto command after the first configuration. Make sure you no longer have any failures in the log output.

Okay, thanks again!

Ok, so I configured openproject the same way I did before, but now I'm getting a 404 error... so I initiated openproject reconfigure, and... if I follow your instructions to just hit ENTER on everything - the first choice is 'Instal;la new postgresql server..." - won't this clobber the existing DB?

Confused...

reconfigure will display everything from the wizard, even things you already set up (that's why it's called reconfigure). So in your case, just hit ENTER on all choices that you've already selected before and that you don't want to change (previously entered passwords will not be displayed, but the underlying value will still be the same if you hit ENTER). It will not clobber the existing DB. Your idea of being able to selectively ask for a specific wizard screen is interesting, I'll see if it can be easily implemented.

Ok, something isn't right... I must have done something differently the first tim I installed Debian...

When I do a fresh install of Debian, should I install apache as one of  the base packages? I can't remember if I did or not the first time, all I know is, I had no problem getting the non-ssl version up and running, and now, it keeps redirecting me to https:, and giving me a 404 access denied error:

Secure Connection Failed

An error occurred during a connection to projects.atl.mediafiche.com. Cannot communicate securely with peer: no common encryption algorithm(s).

Error code: SSL_ERROR_NO_CYPHER_OVERLAP

Ideas? I'm about ready to nukle again and choose to include Apache when I install Debian...

Your process should be as follow, once you have a fresh debian install running:

1. apt-get install openproject
2. openproject configure, no SSL support. OpenProject should then be accessible on http://you-domain.com. If not, then something isn't right with your configuration.
3. install certbot-auto, run it and get your certificate issued.
4. openproject reconfigure, hit ENTER until you get to the SSL wizard, select Yes, then enter the certificate, key, and CA bundle details as per the doc.

That's all you should have to do. The SSL_ERROR_NO_CYPHER_OVERLAP error you're getting looks like a client error, i.e. your browser may not recognise the SSL ciphers used by the server? Which browser are you using, and which version?

Firefox 64bit on Win10 64bit, latest version

Chrome gives a different error:

Forbidden

You don't have permission to access this resource.

Apache/2.4.38 (Debian) Server at projects.atl.mediafiche.com Port 80

So, for some reason, Firefox is forcing me to SSL - and I did find references to two different setting to change in about:config to disable this, but neither worked... maybe there is one I'm missing

Ok, got it working... at least partially.

I'm not sure if I misunderstood the instructions (probably), or if this is a bug, but can you check me please...

What I had done is defined a Server Path Prefix:

atl.mediafiche.com/openproject

So, I ran openproject reconfigure, and simply deleted this and left it blank, and it is now working (in Chrome on Windows Desktop) on http://

I still can't get there from my phone (which is accessing from outside our LAN), but hopefully after I get SSL enabled, it will work...

Thanks again for your help!

And, I was still getting the errors, until I remembered...

We use Cloudflare for DNS, and I had set up the sub-domain A record in proxied mode...

Disabling that got rid of the error... and now I have the /live directory under /etc/letsencrypt

I just tested, and we now have access to it from outside t he office on SSL.

yay!

Thanks so much for your help! I probably wouldn't have gotten it running otherwise.

Ok, so, now the problem is, this system is extremely, maddeningly slow to load every single page. It literally takes 20 seconds or so when moving between pages. This is not usable, so I need to figure out what is wrong. There is nothing in the apache2 logs, syslog, or any other logs that I can see...

What are your system specs in terms of memory?

Can you ps aux | grep open to see how many processes are running? OpenProject logs are accessible via openproject logs.

VMWare ESXi VM, one CPU/core, 8GB Ram

root@deb-projects:~# ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root         1  0.0  0.1 103776 10004 ?        Ss   Aug06   0:01 /sbin/init

root         2  0.0  0.0      0     0 ?        S    Aug06   0:00 [kthreadd]

root         3  0.0  0.0      0     0 ?        I<   Aug06   0:00 [rcu_gp]

root         4  0.0  0.0      0     0 ?        I<   Aug06   0:00 [rcu_par_gp]

root         6  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/0:0H-kblockd]

root         7  0.0  0.0      0     0 ?        I    Aug06   0:00 [kworker/u2:0-events_unbound]

root         8  0.0  0.0      0     0 ?        I<   Aug06   0:00 [mm_percpu_wq]

root         9  0.0  0.0      0     0 ?        S    Aug06   0:00 [ksoftirqd/0]

root        10  0.0  0.0      0     0 ?        I    Aug06   0:03 [rcu_sched]

root        11  0.0  0.0      0     0 ?        I    Aug06   0:00 [rcu_bh]

root        12  0.0  0.0      0     0 ?        S    Aug06   0:00 [migration/0]

root        13  0.0  0.0      0     0 ?        I    Aug06   0:30 [kworker/0:1-events]

root        14  0.0  0.0      0     0 ?        S    Aug06   0:00 [cpuhp/0]

root        15  0.0  0.0      0     0 ?        S    Aug06   0:00 [kdevtmpfs]

root        16  0.0  0.0      0     0 ?        I<   Aug06   0:00 [netns]

root        17  0.0  0.0      0     0 ?        S    Aug06   0:00 [kauditd]

root        18  0.0  0.0      0     0 ?        S    Aug06   0:00 [khungtaskd]

root        19  0.0  0.0      0     0 ?        S    Aug06   0:00 [oom_reaper]

root        20  0.0  0.0      0     0 ?        I<   Aug06   0:00 [writeback]

root        21  0.0  0.0      0     0 ?        S    Aug06   0:00 [kcompactd0]

root        22  0.0  0.0      0     0 ?        SN   Aug06   0:00 [ksmd]

root        23  0.0  0.0      0     0 ?        SN   Aug06   0:00 [khugepaged]

root        24  0.0  0.0      0     0 ?        I<   Aug06   0:00 [crypto]

root        25  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kintegrityd]

root        26  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kblockd]

root        27  0.0  0.0      0     0 ?        I<   Aug06   0:00 [edac-poller]

root        28  0.0  0.0      0     0 ?        I<   Aug06   0:00 [devfreq_wq]

root        29  0.0  0.0      0     0 ?        S    Aug06   0:00 [watchdogd]

root        30  0.0  0.0      0     0 ?        S    Aug06   0:00 [kswapd0]

root        48  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kthrotld]

root        49  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/24-pciehp]

root        50  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/25-pciehp]

root        51  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/26-pciehp]

root        52  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/27-pciehp]

root        53  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/28-pciehp]

root        54  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/29-pciehp]

root        55  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/30-pciehp]

root        56  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/31-pciehp]

root        57  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/32-pciehp]

root        58  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/33-pciehp]

root        59  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/34-pciehp]

root        60  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/35-pciehp]

root        61  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/36-pciehp]

root        62  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/37-pciehp]

root        63  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/38-pciehp]

root        64  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/39-pciehp]

root        65  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/40-pciehp]

root        66  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/41-pciehp]

root        67  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/42-pciehp]

root        68  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/43-pciehp]

root        69  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/44-pciehp]

root        70  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/45-pciehp]

root        71  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/46-pciehp]

root        72  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/47-pciehp]

root        73  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/48-pciehp]

root        74  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/49-pciehp]

root        75  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/50-pciehp]

root        76  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/51-pciehp]

root        77  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/52-pciehp]

root        78  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/53-pciehp]

root        79  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/54-pciehp]

root        80  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/55-pciehp]

root        81  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ipv6_addrconf]

root        83  0.0  0.0      0     0 ?        I    Aug06   0:00 [kworker/u2:1-events_unbound]

root        92  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kstrp]

root       135  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_0]

root       136  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_0]

root       137  0.0  0.0      0     0 ?        I<   Aug06   0:00 [vmw_pvscsi_wq_0]

root       140  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ata_sff]

root       142  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_1]

root       144  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_1]

root       146  0.0  0.0      0     0 ?        S    Aug06   0:00 [scsi_eh_2]

root       148  0.0  0.0      0     0 ?        I<   Aug06   0:00 [scsi_tmf_2]

root       191  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/0:1H-kblockd]

root       221  0.0  0.0      0     0 ?        I<   Aug06   0:00 [kworker/u3:0]

root       223  0.0  0.0      0     0 ?        S    Aug06   0:00 [jbd2/sda1-8]

root       224  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ext4-rsv-conver]

root       260  0.0  0.1  27612  9408 ?        Ss   Aug06   0:00 /lib/systemd/systemd-journald

root       277  0.0  0.0  22064  4860 ?        Ss   Aug06   0:00 /lib/systemd/systemd-udevd

root       370  0.0  0.0      0     0 ?        I<   Aug06   0:00 [nfit]

root       371  0.0  0.0      0     0 ?        I<   Aug06   0:00 [ttm_swap]

root       373  0.0  0.0      0     0 ?        S    Aug06   0:00 [irq/16-vmwgfx]

root       430  0.0  0.1  48220 10872 ?        Ss   Aug06   0:00 /usr/bin/VGAuthService

systemd+   431  0.0  0.0  93084  6556 ?        Ssl  Aug06   0:00 /lib/systemd/systemd-timesyncd

root       432  0.0  0.1  48496 11460 ?        Ss   Aug06   0:26 /usr/bin/vmtoolsd

root       435  0.0  0.0   8504  2820 ?        Ss   Aug06   0:00 /usr/sbin/cron -f

root       437  0.0  0.0   5260   684 ?        Ss   Aug06   0:00 /bin/sleep infinity

root       439  0.0  0.0   5260   684 ?        Ss   Aug06   0:00 /bin/sleep infinity

openpro+   440  0.0  3.8 472796 311672 ?       Ssl  Aug06   0:41 /opt/openproject/vendor/bundle/ruby/2.6.0/bin/rake jobs:work

root       443  0.0  0.0   5260   744 ?        Ss   Aug06   0:00 /bin/sleep infinity

openpro+   444  0.0  0.0   6728  3172 ?        Ss   Aug06   0:00 /bin/bash -e ./packaging/scripts/web

root       445  0.0  0.0 225824  4320 ?        Ssl  Aug06   0:00 /usr/sbin/rsyslogd -n -iNONE

root       450  0.0  0.0  19392  7296 ?        Ss   Aug06   0:00 /lib/systemd/systemd-logind

memcache   454  0.0  0.2 422560 16992 ?        Ssl  Aug06   0:13 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 -P /var/run/memcached/memcached.pid

message+   461  0.0  0.0   8972  3688 ?        Ss   Aug06   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

root       475  0.0  0.0  15852  6600 ?        Ss   Aug06   0:00 /usr/sbin/sshd -D

root       479  0.0  0.0   5612  1588 tty1     Ss+  Aug06   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux

openpro+   537  0.0  4.0 508032 333404 ?       Sl   Aug06   0:23 unicorn master --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

postgres   542  0.0  0.3 213156 26912 ?        S    Aug06   0:01 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/post

postgres   544  0.0  0.1 213300  8404 ?        Ss   Aug06   0:00 postgres: 10/main: checkpointer process

postgres   545  0.0  0.0 213156  7444 ?        Ss   Aug06   0:00 postgres: 10/main: writer process

postgres   546  0.0  0.1 213156  9832 ?        Ss   Aug06   0:00 postgres: 10/main: wal writer process

postgres   547  0.0  0.0 213592  6788 ?        Ss   Aug06   0:00 postgres: 10/main: autovacuum launcher process

postgres   548  0.0  0.0  68348  5048 ?        Ss   Aug06   0:01 postgres: 10/main: stats collector process

postgres   549  0.0  0.0 213448  5444 ?        Ss   Aug06   0:00 postgres: 10/main: bgworker: logical replication launcher

root       550  0.0  0.4  49544 37900 ?        Ss   Aug06   0:01 /usr/sbin/apache2 -k start

www-data   552  0.0  0.3  48968 30760 ?        S    Aug06   0:00 /usr/sbin/apache2 -k start

www-data   553  0.0  0.5 796448 44600 ?        Sl   Aug06   0:01 /usr/sbin/apache2 -k start

www-data   554  0.0  0.5 796520 43632 ?        Sl   Aug06   0:01 /usr/sbin/apache2 -k start

root       622  0.0  0.0  16632  7852 ?        Ss   Aug06   0:00 sshd: mf-admin [priv]

mf-admin   625  0.0  0.1  21024  8344 ?        Ss   Aug06   0:00 /lib/systemd/systemd --user

mf-admin   626  0.0  0.0  23096  2244 ?        S    Aug06   0:00 (sd-pam)

mf-admin   639  0.0  0.0  16916  5844 ?        S    Aug06   0:00 sshd: mf-admin@pts/0

mf-admin   640  0.0  0.0   7912  4660 pts/0    Ss   Aug06   0:00 -bash

root       643  0.0  0.0   9768  3476 pts/0    S    Aug06   0:00 su -

root       644  0.0  0.0   8372  5216 pts/0    S    Aug06   0:00 -bash

postgres   660  0.0  0.2 215196 21708 ?        Ss   Aug06   0:08 postgres: 10/main: openproject openproject 127.0.0.1(54126) idle

openpro+   662  0.0  4.1 508032 335436 ?       Sl   Aug06   0:02 unicorn worker[0] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   663  0.0  4.1 508152 337996 ?       Sl   Aug06   0:02 unicorn worker[1] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   666  0.0  4.1 508032 336084 ?       Sl   Aug06   0:03 unicorn worker[2] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

openpro+   667  0.0  4.1 508032 336908 ?       Sl   Aug06   0:02 unicorn worker[3] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production

root      4442  0.0  0.0      0     0 ?        I    09:27   0:00 [kworker/0:0-ata_sff]

postgres  4472  0.0  0.2 215808 22428 ?        Ss   09:32   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54450) idle

root      4475  0.0  0.0      0     0 ?        I    09:33   0:00 [kworker/0:2-ata_sff]

postgres  4476  0.0  0.2 215164 20344 ?        Ss   09:33   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54456) idle

postgres  4485  0.0  0.2 214844 16772 ?        Ss   09:34   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54470) idle

postgres  4486  0.0  0.2 214896 18604 ?        Ss   09:34   0:00 postgres: 10/main: openproject openproject 127.0.0.1(54474) idle

root      4495  0.0  0.0  10632  3160 pts/0    R+   09:35   0:00 ps aux

root@deb-projects:~#

Aaargh, sorry, you said to run it through grep...

root@deb-projects:# ps aux | grep open
openpro+   440  0.0  3.8 472796 311672 ?       Ssl  Aug06   0:41 /opt/openproject/vendor/bundle/ruby/2.6.0/bin/rake jobs:work
openpro+   444  0.0  0.0   6728  3172 ?        Ss   Aug06   0:00 /bin/bash -e ./packaging/scripts/web
openpro+   537  0.0  4.0 508032 333404 ?       Sl   Aug06   0:23 unicorn master --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
postgres   660  0.0  0.2 215196 21708 ?        Ss   Aug06   0:08 postgres: 10/main: openproject openproject 127.0.0.1(54126) idle
openpro+   662  0.0  4.1 508032 335436 ?       Sl   Aug06   0:02 unicorn worker[0] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   663  0.0  4.1 508152 337996 ?       Sl   Aug06   0:02 unicorn worker[1] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   666  0.0  4.1 508032 336084 ?       Sl   Aug06   0:03 unicorn worker[2] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
openpro+   667  0.0  4.1 508032 336908 ?       Sl   Aug06   0:02 unicorn worker[3] --config-file config/unicorn.rb --host 127.0.0.1 --port 6000 --env production
root      4523  0.0  0.0   6076   880 pts/0    S+   09:43   0:00 grep open
root@deb-projects:#

Ok, so, what would I be looking for in the OpenProject logs?

Your output looks fine. You should check what's happening in the Network tab of the Firefox or Chrome developer console, to see if the slowness is due to a client-side or server-side issue.

Ok, just figured out, the slowness is only when accessing from the internal LAN. Accessing from my phone over the internet it is fine...

Hmmmmmmmmm.... I wonder if the way I ended up doing this is the issue...

IWe use Windows Server (2016) for DNS. I created a sub-domain - atl.mediafiche.com, and then an A record pointing to the local server IP for OpenProject.

Then, I did the same thing in Cloudflare, but pointed it to a secondary public IP, and routed that traffic through my firewall to the OP server.

I'll start poking around, but I'm not sure why that would be a problem...

Ok, any ideas why it would be dog slow when accessing internally, but fine when accessing over the internet? I've checked everything I can think of... all internal DNS pointers resolve properly, no conflicts

I see messages like these - and wondering about the 'duration' times being seen... the messages only present in the logs when the page finally loads in the browser, nothing gets added when I click ro while I'm waiting...

Aug  7 16:15:26 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:26.117046 #665]  INFO -- : method=GET path=/admin/design/upsale format=html controller=CustomStylesController action=upsale status=200 duration=36.65 view=28.09 db=2.88 user=4

Aug  7 16:15:28 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:28.827210 #664]  INFO -- : method=GET path=/admin/enumerations format=html controller=EnumerationsController action=index status=200 duration=117.03 view=97.00 db=14.03 user=4

Aug  7 16:15:30 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:30.854229 #665]  INFO -- : method=GET path=/settings format=html controller=SettingsController action=show status=200 duration=58.62 view=38.28 db=8.17 user=4

Aug  7 16:15:34 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:15:34.058227 #664]  INFO -- : method=GET path=/users format=html controller=UsersController action=index status=200 duration=67.52 view=54.14 db=6.57 user=4

Aug  7 16:16:39 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:16:39.166950 #668]  INFO -- : method=GET path=/users/4/edit format=html controller=UsersController action=edit status=200 duration=366.30 view=328.46 db=19.14 user=4

Aug  7 16:17:01 deb-projects CRON[1527]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Aug  7 16:19:05 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:19:05.837829 #662]  INFO -- : method=GET path=/admin/groups format=html controller=GroupsController action=index status=200 duration=60.72 view=36.14 db=9.96 user=4

Aug  7 16:19:54 deb-projects openproject-web-1.service[446]: I, [2020-08-07T16:19:54.536292 #664]  INFO -- : method=GET path=/admin format=html controller=AdminController action=index status=200 duration=37.91 view=28.74 db=3.06 user=4

Are those duration, view and db numbers normal?

The durations are in milliseconds, so they definitely look fine. You should try to see if there are any warnings in the Apache log (/var/log/apache2/error*.log or through journalctl).

If I understand correctly, you have both a public IP and a private IP linked to that VM, and the traffic is slow via the private IP. It might be that Apache performs some reverse lookup on the source private IP, and that lookup times out. You should check whether host YOUR_PRIVATE_IP returns something when run from your VM (YOURPRIVATEIP being the IP of your internal computer from which you are trying to connect). Just a wild guess but worth a check.

The public IP is routed through an SNAT on my Watchguard that routes it to the internal private IP of 192.168.1.165

root@deb-projects:# host 192.168.1.165
165.1.168.192.in-ad
r.arpa domain name pointer projects.atl.mediafiche.com.
root@deb-projects:#

Which is the correct FQDN hostname

Nothing in apache2 error logs since noon today, and those were expected, from when I was setting it up...

And nothing of note in journalctl

Also...

root@deb-projects:~# nslookup projects.atl.mediafiche.com
Server:         192.168.1.170
Address:        192.168.1.170#53

Name:   projects.atl.mediafiche.com
Address: 192.168.1.165

root@deb-projects:~# nslookup 192.168.1.165
165.1.168.192.in-addr.arpa      name = projects.atl.mediafiche.com.

And yes, 192.168.1.170 is my internal DNS server

I don't know if it is significant, but sometimes - not all the time - when the delay is occurring, I see this ain the status area of the web browser (lower left corner) for many many seconds:

Transferring from secure.gravatar.com...

I'm seeing this more and more... maybe significant...

I've looked everywhere and can't see where or even if it is possible to totally disable gravatars to see if this is  the issue... so, is it possible, and if so, how/where?

Thanks

Ok, this really peeves me off...

With the issues I had initially getting LetsEncrypt SSL Cert installed, I THOUGHT I had tested using a different browser after I got SSL working but had this slow access. Apparently I didn't.

The problem was I was using Firefox - and it recently switched on SecureDNS by default, which bypasses the local DNS Server for all by localhost by default.

Simply adding my local domain to about:config > > network.trr.excluded-domains fixed it right up.

Now to go see if Firefox's current GPO support includes changes to this setting...