I have just migrated my installations to a new server, from Ubuntu 14 to Ubuntu 16. I run an NGINX proxy, which receives incoming connections and forwards them to an Apache backend. Everything worked well on the Ubuntu 14 install. However, on the new server (exactly the same setup, except running Ubuntu 16) I get a 500 error when generating PDF’s. This occurs whether I am generating on e-mail, guest URL or direct from the dashboard. Each time, there is a long delay and then a 500 error.
I have enabled debug, but this doesn’t appear to be producing anything helpful in investigating the issue. In the Apache log (which also logs NGINX errors), I see a timeout between NGINX and Apache. I have put a copy of all three logs (Debug, Apache Access and Apache Error) on IP paste.
Server is running:
OS: Ubuntu 16.04
NGINX: 1.13.8
Apache: 2.4.18
PHP: 7.0.22-0ubuntu0.16.04.1
PHP has all required modules enabled (gd, json, mbstring, mcrypt, mysqli, recode, xmlrpc). The remaining requirements (hash, openssl and zlib) are built in to php7.0 as standard.
I have checked file permissions, and the web server uesr has the relevant rights and ownership of all files.
I’m not using fcgi by default, but I have tried its sister setting, proxy_read_timeout. When enabled, it is impossible to access the site - it will try to load then throw a 500. NGINX is running as a fast front-end cache. I did try FCGI briefly to rule out Apache and still received the error. I will try fcgi again with the fastcgi_read_timeout et, but I’m not hopeful.
All of the site config settings are the same between the two servers. I’ve created a test server running 14.04, without the NGINX element (and used the production server’s NGINX to proxy the connection). Running direct to the test server (no NGINX), it works. Connecting via the NGINX proxy (running on 16.04), the 500 error re-occurs.
I shall continue my investigations. Next step is to try a direct LAMP on 16.04. That will rule out an OS level issue, but I am now pretty convinced it is NGINX at fault.
Further investigation is leading me to believe the overall problem is not OS or NGINX based. On my test server, I am encountering extremely slow PDF generation time. When I use a custom PDF css including a background, I eventually hit another error:
I’m now pointing the finger at mPDF as being the root cause. Paste of Debug Output from test server.
Long PDF generation times may be possible in dev environments due to the extensive logging of mPDF. However, your production server / environment should run pretty fast.
Whether Debug is enabled or not, PDF generation seems to take exactly 120s. I’ve just tested this with a fresh install and no modifications anywhere and using the stock templates - PDF generation still takes 120s. Server wise, both my sandbox server and production servers have many times more power and resource compared the server I have migrated from. Something isn’t right, it’s just working out what that is - and more importantly how to fix it!
edited to add: I also tried using the latest (dev) version of mPDF to no avail - still 120 seconds!
the strangest thing is occurring, though. I fired a up a fresh server (Ubuntu 16) running Apache only. I ran a a completely fresh install of IP, and PDF generated in under 5 seconds as expected. I added a fresh second site, with my existing IP database and ran the tables update. I set the Incoive templates to the default IP ones. PDF generation takes exactly 2 minutes, with or without debug mode enabled. The first IP install also now takes 2 minutes to generate invoices. I really can’t wrap my head around it!
Execution time across the board is excellent - it’s only PDF generation that is causing an issue. Every other action is <1s in terms of execution time.
I have just found the culprit, but have no idea how to resolve the issue!
The issue occurs when HTTPS is enabled on IP; when HTTPS is disabled in ipconfig.php, PDFs render in the expected timeframe (<1s).
The current version on Ubuntu systems is 1.0.2g which is end of life. I will try updating openssl on my dev server as running IP without SSL wrappers is bad news. I will post an update when I’ve finished.
OK, So I’ve tried upgrading to the most recent LTS version of openssl; the problem persists. HTTPS enabled - 2 mins to generate PDF. HTTP only: <1s to generate PDF.
This is using php7.1 - basically the same result as php7.0. I’ve upgraded the system version of openssl also. php7.2 can’t be used, as the mcrypt extension has been removed.
I have always used https for IP. I’m running a number of other SaaS solutions also, none of which are having issues running under https. The problem only surfaced when I had to retire my original IP platform. With the new server, I have moved to the most up to date releases of all components based on the Ubuntu main repo.
Much in the same way it is helpful to know what the problematic configuration is, It might be helpful to know what the working configurations are. Component release levels vary so widely across different platforms and even between different patch levels on the same OS. What’s working by default one day might, very well, be completely broken the next!
Can you make a little test for me with a PHP5.6 (last is 5.6.33) - with and without https of course -
I just want to eliminate something new in PHP7 which can cause the bug
TL/DR: I've fixed the issue. It was embarrassingly simple:
Added a hosts file entry for the FDQN pointing at the LAN address.
There are no firewalls that would affect the traffic; if it were firewall it would affect HTTP and HTTPS equally. I’ve just tried PHP5.6 for fun (expecting other PHP errors as 1.5x is supposed to be 7+ only), but…
DEBUG - 2018-01-25 21:58:50 --> Total execution time: 121.0206
As I expected, no joy. You got me thinking, though. In relation to the previous link you sent (mPDF render failure re. firewall), there is something funky going on with mPDF. I’ve just solved the problem across the board (production, dev - both Ubuntu14x, Ubuntu16x, BSD, Ubuntu with NGINX and Apache).
It was simple… Hosts file entry with the LAN address on the server for the FQDN of the IP instance.
What is happening? When mPDF is loading image assets (Invoice Logo), instead of pulling the asset from a local path, it is using the FQDN. The really odd behaviour is the difference between HTTP and HTTPS. Why it takes almost exactly 120 seconds to generate on HTTPS and <1s on HTTP is open for debate. The best solution would be for mPDF to retrieve assets from the filepath, and not over HTTPx, but that’s a debate for another forum!
Thank you everyone for your suggestions, pointers - and more importantly the time you have taken to assist.
d