Troubleshooting Unresponsive WordPress Sites On Lightsail
It has been a while since I’ve written a technical post here. A new startup venture, Rally Creek USA – an off grid and tiny home endeavor, has been taking a lot of time time over the past 6 months. While that project has a lot of technical aspects, especially in the solar power generation side of things, it is less time on the keyboard and more time outside building things and trying to not get electrocuted. However, as a tech guy you know I still have a lot of web and code stuff attached to that project. That is where this post comes from.
At Rally Creek USA, we are running a stack of web presence servers on AWS Lightsail. With the latest Lightsail updates, starting up a WordPress server is easier-than-ever. While not as sophisticated as WordPress presence dedicated services like Kinsta, it does provide a “one stop shop” that blends nicely into my other tech ventures like Store Locator Plus® that are still running full SaaS services on the AWS stack.
There is one issue lately that is driving us crazy — unresponsive WordPress sites on Lightsail. In particular the Rally Creek USA site is especially vulnerable thanks to some poor software design in the WordPress and JetPack mobile apps that Automattic and WordPress dev teams are putting out. Turns out the in-app media file upload service hammers the crap out of server resources even when uploading a single image. The process starts multiple threads to push an image, often tapping out the PHP child processes; Likely an over-aggressive timeout issue in their ack/nack loop on the curl calls … purely conjecture but plausible.
Troubleshooting Unresponsive WordPress Sites On Lightsail
After logging into AWS Lightsail and rebooting the server every time we try to upload more than one image per five minute window, it was time to investigate. WordPress on Lightsail is run via server images created by Bitnami. Bitnami hacks the shit out of a standard LAMP stack in an effort to be flexible and all-things-to-all-people. They do a decent job, but that means debugging problems means finding Bitnami’s “special sauce” and deconstructing it to find out what is really going on. Here are the errors we started with as they were barfed out into the Apache error logs:
Apache Error Log
---
No such file or directory:
AH02454: FCGI: attempt to connect to Unix domain socket /opt/bitnami/php/var/run/www.sock (www-fpm:8000) failed
AH01079: failed to make connection to backend: httpd-UDS
[Thu May 16 09:55:33.090401 2024] [proxy:error] [pid 71976:tid 140392793589504] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /opt/bitnami/php/var/run/www.sock (www-fpm:8000) failed
[Thu May 16 09:55:33.090458 2024] [proxy_fcgi:error] [pid 71976:tid 140392793589504] [client 54.242.141.242:55418] AH01079: failed to make connection to backend: httpd-UDS
...45x
[Thu May 16 10:40:23.967757 2024] [proxy_fcgi:error] [pid 72376:tid 140391458199296] [client 185.180.143.141:37998] AH01079: failed to make connection to backend: httpd-UDS
[Thu May 16 10:40:37.518564 2024] [authz_core:error] [pid 71971:tid 140392548726528] [client 185.180.143.141:48924] AH01630: client denied by server configuration: /opt/bitnami/apache/cgi-bin
---
No such file or directory:
AH02454: FCGI: attempt to connect to Unix domain socket /opt/bitnami/php/var/run/www.sock (www-fpm:8000) failed
AH01079: failed to make connection to backend: httpd-UDS
36x...
[Thu May 16 11:07:53.608112 2024] [proxy:error] [pid 72376:tid 140392793589504] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /opt/bitnami/php/var/run/www.sock (www-fpm:8000) failed
[Thu May 16 11:07:53.608146 2024] [proxy_fcgi:error] [pid 72376:tid 140392793589504] [client 52.167.144.20:6554] AH01079: failed to make connection to backend: httpd-UDS
---
Shutdown
[Thu May 16 11:09:42.611991 2024] [mpm_event:notice] [pid 71969:tid 140393354558720] AH00491: caught SIGTERM, shutting down
---
Startup
AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
AH01071: Got error 'PHP message: PHP Notice: PHP Request Startup: file created in the system's temporary directory in Unknown on line 0
[Thu May 16 11:10:14.774838 2024] [ssl:warn] [pid 746:tid 140494208154880] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Thu May 16 11:10:14.785722 2024] [ssl:warn] [pid 747:tid 140494208154880] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Thu May 16 11:10:14.788056 2024] [mpm_event:notice] [pid 747:tid 140494208154880] AH00489: Apache/2.4.57 (Unix) OpenSSL/1.1.1w configured -- resuming normal operations
[Thu May 16 11:10:14.788104 2024] [core:notice] [pid 747:tid 140494208154880] AH00094: Command line: '/opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf'
[Thu May 16 11:16:01.392591 2024] [proxy_fcgi:error] [pid 751:tid 140493354600192] [client 69.1.8.178:50747] AH01071: Got error 'PHP message: PHP Notice: PHP Request Startup: file created in the system's temporary directory in Unknown on line 0'
PHP FPM LOG
[15-May-2024 15:47:37] WARNING: [pool www] server reached pm.max_children setting (30), consider raising it
[15-May-2024 17:00:57] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawni
ng 8 children, there are 19 idle, and 25 total children
[15-May-2024 17:00:58] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawni
ng 16 children, there are 19 idle, and 26 total children
[15-May-2024 17:01:03] WARNING: [pool www] child 62369 exited on signal 9 (SIGKILL) after 22237.765674 seconds from start
[15-May-2024 17:01:05] WARNING: [pool www] child 62623 exited on signal 9 (SIGKILL) after 20398.649721 seconds from start
[15-May-2024 17:03:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawni
ng 8 children, there are 19 idle, and 25 total childrenTools For Debugging Unresponsive WordPress Sites On Lightsail
There are some tools and files that can help get to the source of the issue. In our case it appears to be a server that is too small (memory-wise in particular) and possibly with PHP configured in a non-optimal way. To find that issue you need to know about Bitnami Diagnostics, Apache Log Files, PHP Log Files, and PHP configuration files.
All of these tools require you login to your Lightsail console and assume you know basic Linux commands. If not, hire a technical professional to help as you can easily FUBAR your entire configuration and need to start over. You should also take a snapshot of your server using the AWS Lightsail console tools before messing things up — even if you do know what you are doing. It is easier to restore a snapshot than rebuild an entire website from scratch.
Bitnami Diagnostics
The first tool at your disposal is the Bitnami diagnostic tool. This will check your configuration and highlight any errors. It will also note any errors in your Apache log files. This is the same tool you need to use if submitting a ticket to Bitnami.
sudo /opt/bitnami/bndiagnostic-toolApache Error Log
To delve deeper into errors thrown by web coding or Apache configuration, look into the Apache error log. It lives in the custom Bitnami stack directory. This is where most of the errors in our log post above come from.
sudo cat ~/stack/apache/logs/error_logPHP Log
While there were some interesting things in the Apache log, most of the items are warnings related to Bitnami’s creative solution to dynamic web hosting configuration and SSL certificates. The errors are related to a failed PHP socket which is the communication channel between Apache and PHP. From past experience, this typically indicates an issue in the PHP engine not Apache or web coding. To look into the PHP logs you need to go here:
sudo cat /opt/bitnami/php/logs/php-fpm.logOur errors are noted in the log about at the top of this article, near the bottom. It is indicating not enough spare servers running. Generally that means heavy traffic for the size of the server. In our case our fledgling Rally Creek USA blog only has a few hundred visitors at MOST each day — no nearly enough to tax the configuration we are using. Turns out the culprit is that damn Jetpack or WordPress mobile app image uploader. Any time we post more than one image per minute using either app the website gets hammered with I/O requests and eats up all the PHP server sockets. While we cannot fix that we can try to boost our PHP server instances in PHP-FPM before buying a larger server to let Jetpack be a pig for 15 minutes every few days.
For that we need to delve into the PHP configuration file.
PHP Configuration File
If you do need (want) to update the default Bitnami PHP settings, you’ll need to edit the php.ini file and restart services afterwards. As noted above, take a snapshot first in case things go off-the-rails.
You can find and edit the php.ini file here:
/opt/bitnami/php/etc/php/ini/php.iniI am not going to get into how or what to edit — that requires expertise that is covered in other articles. If you do not know what you are doing, hire a professional system admin.
Restarting The LAMP Stack On AWS Lightsail
After updating the configuration, restart the LAMP stack. This can also help reset an unresponsive WordPress site on LIghtsail befre doing a full system reboot:
sudo /opt/bitnami/ctlscript.sh restartHope this article helps you debug your AWS Lightsail WordPress issues. Good luck!