Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(core): Introduce concurrency control for main mode #9453

Merged
merged 71 commits into from
Jun 12, 2024

Conversation

ivov
Copy link
Contributor

@ivov ivov commented May 17, 2024

This PR introduces concurrency control for main mode. This limits how many production executions are allowed to run at the same time in main mode, preventing event loop overload and so improving reliability.

Concurrency control applies only in main mode and only to production executions (i.e. in webhook and trigger modes), not to executions in manual, retry, error, integrated, cli or internal modes.

Summary of changes

Queuing

  • Implement ConcurrencyControlService.
  • Reuse Adi's work in ConcurrencyQueue.
  • Block or let through executions at ActiveExecutions.
  • On releasing an enqueued execution, reset startedAt timestamp.
  • On cancelling an enqueued execution, remove from queue.
  • On retrying an enqueued execution, block retry attempt.
  • On deleting an enqueued execution via Public API, remove from queue.
  • On deleting an enqueued execution via Internal API, remove from queue.
  • On instance startup post crash, exclude enqueued executions from crash recovery.
  • On instance startup post crash, run all enqueued executions.
  • Serve executions prioritizing running over new
  • Ensure queue mode is unaffected by concurrency control.

Envs, logs, tests

  • Add new env var N8N_CONCURRENCY_PRODUCTION_LIMIT.
  • Add logs on throttling and releasing executions.
  • Add tests for ConcurrencyControlService and ConcurrencyQueue.
  • Add telemetry to monitor throttling frequency.

Discussion: https://www.notion.so/n8n/Concurrency-limits-for-main-mode-8b642df5e98a479cadd1cfc4950d27a1

Follow-up to: #8458

@n8n-assistant n8n-assistant bot added core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team labels May 17, 2024
krynble
krynble previously approved these changes Jun 10, 2024
Copy link
Contributor

@krynble krynble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

⚠️ Some Cypress E2E specs are failing, please fix them before merging

krynble
krynble previously approved these changes Jun 12, 2024
Copy link
Contributor

@krynble krynble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

⚠️ Some Cypress E2E specs are failing, please fix them before merging

krynble
krynble previously approved these changes Jun 12, 2024
Copy link

cypress bot commented Jun 12, 2024

4 flaky tests on run #5452 ↗︎

0 388 0 0 Flakiness 4

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ivov 🗃️ e2e/*
Project: n8n Commit: fcda46eafc
Status: Passed Duration: 04:56 💡
Started: Jun 12, 2024 1:00 PM Ended: Jun 12, 2024 1:05 PM
Flakiness  5-ndv.cy.ts • 2 flaky tests

View Output Video

Test Artifacts
NDV > should not retrieve remote options when required params throw errors Screenshots Video
NDV > Stop listening for trigger event from NDV Screenshots Video
Flakiness  10-undo-redo.cy.ts • 1 flaky test

View Output Video

Test Artifacts
Undo/Redo > should undo/redo adding nodes Test Replay Screenshots Video
Flakiness  20-workflow-executions.cy.ts • 1 flaky test

View Output Video

Test Artifacts
Current Workflow Executions > should auto load more items if there is space and auto scroll Test Replay Screenshots Video

Review all test suite changes for PR #9453 ↗︎

Copy link
Contributor

✅ All Cypress E2E specs passed

Copy link
Contributor

⚠️ Some Cypress E2E specs are failing, please fix them before merging

Copy link
Contributor

✅ All Cypress E2E specs passed

@ivov ivov merged commit 7973423 into master Jun 12, 2024
31 checks passed
@ivov ivov deleted the introduce-concurrency-control-for-main-mode branch June 12, 2024 13:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants