问题标签 [laravel-horizon]

我们的 Laravel 应用程序在 Kubernetes 中有两个不同的 pod，

• 一个在端口 80 上运行的 apache，（CMD /usr/sbin/apache2ctl -D FOREGROUND）
• 和另一个正在运行的工作人员（Laravel Horizo​​n）（CMD php /var/www/artisan horizo​​n）

问题是当我检查地平线仪表板时，它显示“活动”，我可以在“待定作业”部分看到作业，但它们从未真正执行。他们只是无所事事地坐在那里。

现在，当我在运行 apache 的 pod 中进行 SSH 并手动运行命令“php artisan horizo​​n”时，它实际上会执行所有待处理的作业。

我已经确保了以下几点：

1. 两个 pod 都连接到同一个 Redis 数据库服务
2. 两个 pod 的 Horizo​​n 前缀相同

我在 Laravel Horizo​​n 的排队时间有问题..

案例：我有一个考试监考软件，每 30 秒从约 1500 名学生的网络摄像头拍摄照片。然后我把它放在队列中，通过 websocket 向老师广播最新的图像。

我的服务器有 32 个 CPU 和 192GB RAM。

这是我的地平线配置：

我从系统中收到了一些类似的电子邮件。像这样从 1000 秒到 >30k 秒不等。

即使我放了 100k maxProcesses，队列处理也感觉很慢。

有人可以帮我弄这个吗？提前致谢..

编辑：为了使问题更清楚，这里是摘要：

• 我的服务器有 32 个 CPU 和 192GB RAM (Ubuntu 18.04)。

• 我在 Horizo​​n 配置中分配了 10 万个最大进程（我昨天什至将其更改为 1.000.000 个最大进程）

• 但是队列处理没有我想象的那么快。其中一个队列甚至有 30k+ 秒的等待时间。

• 如何让队列处理更快？我错过了任何地平线/服务器配置吗？一些限制配置可能在 PHP 限制/Redis 限制/任何限制中？

Sorry for the title which might sound like an "allready answered" topic but I believe my case is unique.

Also, this is my first post so I apologize if I am not on the proper channel as I am not sure wether my problem is on the server administration side or the Laravel's configuration one.

I am trying to get some fresh ideas on how to resolve an issue with Horizon / Predis / HAProxy which I thought was fixed but is showing up again.

Some details on environment

• 2x Apache servers : PHP Version 7.2.29-1+ubuntu18.04.1+deb.sury.org+1
  • thread safe is disabled and we use FPM
• 2x Redis servers using a simple master-slave setup (no high availability, no sentinel) : redis version 4.0.9
• load balancing with HAProxy version 1.9

Libraries

• laravel/framework: 6.14.0
• laravel/horizon": 3.7.2
• redis/predis: 1.1.1

Horizon configuration

The Horizon daemon is managed through Supervisor.

This is the Redis client configuration in config/database.php:

This the Redis connection configuration in config/queue.php:

As you can see there are two connections defined for the same physical Redis server. The application uses queues for 2 different types of jobs:

• Fast / short processes like broadcasting, notifications or some Artisan commands calls. These use the first connection configuration with low timeout setting.
• Long running processes which essentially query large amounts of data on a Snowflake DB (cloud based SQL like DB) and / or update / inserts documents on a Solr server. These processes use the 2nd connection configuration as they can take quite some time to complete (usually around 20 minutes for the one combining read from Snowflake and write to Solr)

The application is a business webapp meant for private use by my company and the load is rather small (around 200 jobs queued / day) but the long running processes are critical to the business : job failure or double run is not acceptable.

This is the config/horizon.php file:

Initial problem <solved>

When we went live at the beginning of the year we immediately hit a problem with the jobs running on the long-running-queue connection:

Error while reading line from the server. [tcp://redis_host:6379] errors started popping left and right.

These translated into jobs being stuck in pending state, until they finally ended up being marked as failed although the tasks had in reality succeeded.

At the time the application's long running processes were limited to the Snowflake SELECT queries.

After going through the numerous posts about it on Laravel Horizon's github issues as well as SO's topics and testing the suggestions without luck we finally figured out that the culprit was our load balancer closing the connection after 90 seconds.

Redis has a tcp-keepalive default config parameter of 300 secs so we tweaked the HAProxy's configuration to close at 310 secs and - poof! -, everything worked fine for a while.

This is HAProxy's configuration for the application nowadays:

New problem (initial reborn ?)

Coming back a few months later the application has evolved and we now have a job which reads and yields from Snowflake in batch to build a Solr update query. The Solr client is solarium/solarium and we use the addBuffered plugin.

This worked flawlessly on our pre-production environment which doesn't have load balancing.

So next we moved to the production environment and the Redis connection issues rose again unexpectedly, except this time we've got the HAProxy setup properly.

Monitoring the keys in Redis we can see that these jobs get indeed reserved but end up in the delayed state after some time, waiting to be tried again once the job's timeout is reached.

This is a real problem as we end up going through the job's max tries count until it eventually gets marked as failed, running it x times because it never gets the complete flag, putting unecessary stress on the environment and consuming resources when in fact the job DID succeed at first try.

This is what we get from HAProxy's logs:

Jun 26 11:35:43 apache_host haproxy[215280]: 127.0.0.1:42660 [26/Jun/2020:11:29:02.454] PROD-redis PROD-redis/redis_host 1/0/401323 61 cD 27/16/15/15/0 0/0

Jun 26 11:37:18 apache_host haproxy[215280]: 127.0.0.1:54352 [26/Jun/2020:11:28:23.409] PROD-redis PROD-redis/redis_host 1/0/535191 3875 cD 24/15/14/14/0 0/0

The cD part is the interesting information, as per haProxy's documentation:

c : the client-side timeout expired while waiting for the client to send or receive data.

D : the session was in the DATA phase.

There are more logs like this and there is no obvious pattern in the delay between the connection established and the moment it closes as you can see from dates.

Before getting there we have :

• switched to a Redis version 5.0.3 server: same issue.
• removed HAProxy from the equation and established direct connection between the client and Redis : works flawlessly.

I'm a bit at a loss as to how to figure out and fix the issue for good. Going back to the HAProxy log concerning client-side timeout, I wonder what could possibly be wrong about the client configuration and what I should try next.

Maybe someone here will come with a suggestion ? Thank you for reading.

大家好，我知道这是一个非常基本和通用的问题，但别无选择，只能问，突出显示的选项，这些是什么以及它们如何出现以及它们代表什么。这是从网络文章中获取的一般图像。

我正在尝试在本地使用 Horizo​​n，在我的 Ubuntu 20.04 / Laravel 7 / PHP 7.4

安装成功，但是当我尝试通过 http://localhost:8000/horizo​​n 访问地平线时，我得到了 404。

和这个问题很相似

我试图清除我的路线，检查我的路线是否存在

给我：

我不知道该怎么办。任何想法 ？

我已经在子文件夹中安装了 Laravel，并正在尝试安装 Horizo​​n。路由到“test.com/sub-folder/horizo​​n”后，所有损坏的设计以及内部链接都指向主域而不是主域没有子文件夹。

经过搜索，似乎是github issue中已经报告的已知问题

当 Laravel 安装在子文件夹中时，是否有任何解决方法可以使 Horizo​​n 工作？

我编写了新的时间表并将其拉到测试服务器上，它运行良好，但是当我将它拉到生产环境时它不起作用。我在 laravel-horizo​​n 中查看了指标，并没有新的 Job。所以我决定我有一些进程为此我将我的 Horizo​​n.php 进程从 3 更改为 5，但它更改为 10。此外，我清除了我的配置和缓存，指标变为空，而且我的一些旧工作现在也不起作用.

我收到以下错误：

Argument 2 passed to Laravel\Horizon\ProvisioningPlan::__construct() must be of the type array, null given, called in /Users/camilo/Documents/development/laravel/via-express/vendor/laravel/horizon/src/ProvisioningPlan.php on line 58

我不知道为什么，地平线安装是新鲜的。我运行了以下命令：

此外，在互联网上找不到与此错误相关的任何内容。有任何想法吗？

我派遣工作在我的地平线队列上做一些工作。似乎相同的作业总是在 60 秒的运行时失败，对我来说这看起来像是一个超时问题：

我通过 cron 计划调度了大约 10 个工作，而这 4 个总是失败。

这是我的config/horizon.php（只是有趣的部分）：

如您所见，我设置了 300 秒的超时，但作业总是在 60 秒标记处失败。

我使用这个入口点命令在我的 docker 容器中启动 Horizo​​n 队列处理器：php /path/to/artisan horizon

为了调试问题，我手动实例化了作业类，以查看它在哪里失败，如下所示：

这似乎花费了超过 60 秒的时间，但它确实完成了。

所以我的问题是 - 你如何正确设置 laravel Horizo​​n 的超时时间？

我正在使用 laravel 8.x 和最新版本的 Horizo​​n 包。

我在 laravel.log 中有很多错误，每 4-5 秒出现 2 个错误。

这是错误：

这是我的配置：

在我的 .env 中：

这是堆栈：

• 地平线版本：5.6.0
• Laravel 版本：8.17.0
• PHP版本：7.4.3
• Redis 驱动和版本：predis/phpredis 5.0.7 / 1.1.6
• 数据库驱动程序和版本：Postgres 12

我应该怎么做才能摆脱它们？