Skip to content

e2e: qa alldevices test: minimize latency between qa clients and devices #2537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
nikw9944 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

e2e: qa alldevices test: minimize latency between qa clients and devices #2537

nikw9944 wants to merge 4 commits into from

Conversation

@nikw9944
Copy link
Contributor

@nikw9944 nikw9944 commented Dec 30, 2025

Summary of Changes

  • Update the QA alldevices test to minimize latency between qa clients and devices

Process:

  1. When the alldevices test starts, call GetLatency for each client
  2. Create a list of devices for each host.
    1. If there are multiple hosts with <50ms latency for that device, assign the device to the host with the fewest devices
    2. Otherwise, associate each device with the host with the best latency
  3. Loop through n batches, where n is the largest number of devices associated with a host
    1. Look at each host
      1. If there are >= 2 hosts, and the current host has 0 devices left to test, remove the current host
      2. If there are 2 hosts, and the current host has 0 devices left to test, test the previously tested device again
    2. Test all hosts in current batch

Next steps:

  1. Once we add more QA clients, we can further decrease the latency threshold.

@nikw9944 nikw9944 linked an issue Dec 30, 2025 that may be closed by this pull request
qa: update alldevices test to only connect qa hosts to devices in the same continent #2518
Open
@nikw9944 nikw9944 marked this pull request as ready for review December 30, 2025 21:03
@nikw9944
Copy link
Contributor Author

nikw9944 commented Dec 30, 2025

Output:

...
    qa_test.go:67: 2025-12-30T19:31:37.664Z INF Step 1. Collect `doublezero latency` for each client                                                                                                                                                
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF Step 2. Create a list of devices for each client.                                                                                                                                                   
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF     1. If there are multiple clients with <50ms latency for that device, assign the device to the client with the fewest devices                                                                    
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF     2. Otherwise, associate each device with the client with the lowest latency                                                                                                                     
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF Assigning devices to clients based on latency                                                                                                                                                       
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF Client device assignments:                                                                                                                                                                          
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF                                                                                                                                                                                                     
        |          | ams-mn-qa01              | fra-mn-qa01              | lon-mn-qa01              | nyc-mn-qa01              | sfo-mn-qa01             | sgp-mn-qa01             |                                                                
        |----------|--------------------------|--------------------------|--------------------------|--------------------------|-------------------------|-------------------------|                                                                
        | Batch 1  | ce2-dzd-001 (16.0ms)     | frankry (2.1ms)          | lon001-dz002 (1.5ms)     | nyc001-dz002 (2.4ms)     | dz-slc-sw01 (19.9ms)    | dgt-dzd-sin-sg3 (4.2ms) |                                                                
        | Batch 2  | dz-ld4-sw01 (14.6ms)     | dub001-dz001 (23.2ms)    | mrs001-dz002 (17.1ms)    | dgt-dzd-nyc-ny2 (3.6ms)  | dz-chi-sw01 (47.1ms)    | bom001-dz001 (77.6ms)   |                                                                
        | Batch 3  | dgt-dzd-ams-ams1 (2.8ms) | dz-sxb-01 (5.1ms)        | cherydam (8.4ms)         | dgt-dzd-ash-dc3 (6.6ms)  | dgt-dzd-lax-la2 (8.5ms) | bom001-dz002 (68.2ms)   |                                                                
        | Batch 4  | lts-dzd-001 (7.9ms)      | dz-mad-01 (29.7ms)       | bdc-dzd-001 (19.2ms)     | dz-tor1-sw01 (13.4ms)    | sea001-dz001 (19.6ms)   | dz-ty9-sw01 (79.2ms)    |                                                                
        | Batch 5  | dgt-dzd-lon-ths (6.1ms)  | swerry (29.6ms)          | dgt-dzd-fra-fr5 (14.6ms) | was001-dz001 (6.2ms)     | tyo001-dz002 (105.7ms)  | hkg001-dz002 (31.4ms)   |                                                                
        | Batch 6  | dz-muc-01 (17.7ms)       | dgt-dzd-dub-db2 (19.2ms) | dz-mrs-01 (24.4ms)       | dfw001-dz002 (34.6ms)    | dz-sea10-sw01 (17.9ms)  | tyo002-dz002 (73.1ms)   |                                                                
        | Batch 7  | dz-ams-01 (2.1ms)        | cherlita (24.7ms)        | dz-fra-01 (14.0ms)       | dz-ny7-sw01 (1.8ms)      | dz-lax-sw01 (12.6ms)    | tyo001-dz001 (74.2ms)   |                                                                
        | Batch 8  | swdzd01-lon2 (8.1ms)     | ams001-dz002 (13.3ms)    | dub001-dz002 (11.8ms)    | dz-dc10-sw01 (7.2ms)     | sjc001-dz002 (1.8ms)    | dz-sg1-sw01 (3.8ms)     |                                                                
        | Batch 9  | fra001-dz002 (7.0ms)     | mrs001-dz001 (18.4ms)    | dz-waw-01 (28.6ms)       | sao001-dz001 (111.9ms)   | chi001-dz001 (46.6ms)   | sin001-dz002 (2.9ms)    |                                                                
        | Batch 10 |                          | dz-fr5-sw01 (13.4ms)     |                          | dz-ch2-sw01 (19.0ms)     | lax001-dz002 (12.4ms)   | hkg001-dz001 (37.4ms)   |                                                                
        | Batch 11 |                          |                          |                          | sao001-dz002 (111.0ms)   | dfw001-dz001 (44.2ms)   |                         |                                                                
        | Batch 12 |                          |                          |                          | nyc002-dz002 (2.3ms)     | sea001-dz002 (19.2ms)   |                         |                                                                
        | Batch 13 |                          |                          |                          | dgt-dzd-dal-da3 (34.7ms) | sjc001-dz001 (4.1ms)    |                         |                                                                
        | Batch 14 |                          |                          |                          | nyc001-dz001 (3.0ms)     |                         |                         |                                                                
        | Batch 15 |                          |                          |                          | was001-dz002 (7.1ms)     |                         |                         |                                                                
        | Batch 16 |                          |                          |                          | chi001-dz002 (22.0ms)    |                         |                         |                                                                
                                                                                                                                                                                                                                                    
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF Planning to test deviceCount=67 clientCount=6 totalBatches=16                                                                                                                                       
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF Step 3. Loop through n batches, where n is the largest number of devices associated with a client                                                                                                   
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF     1. For each client                                                                                                                                                                              
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF         1. If there are >= 2 clients, and the current client has 0 devices left to test, remove the current client from testing                                                                     
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF         2. If there are 2 clients, and the current client has 0 devices left to test, test the previously tested device again                                                                       
    qa_test.go:67: 2025-12-30T19:31:39.951Z INF     2. Test all clients in current batch                                                                                                                                                            
=== RUN   TestQA_AllDevices_UnicastConnectivity/batch_1                                                                                                                                                                                             
...
    qa_test.go:67: 2025-12-30T19:58:42.849Z INF Test results:                                                                                                                                                                                       
    qa_test.go:67: 2025-12-30T19:58:42.849Z INF                                                                                                                                                                                                     
        |          | ams-mn-qa01                    | fra-mn-qa01                    | lon-mn-qa01                    | nyc-mn-qa01                    | sfo-mn-qa01                   | sgp-mn-qa01                   |                            
        |----------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|-------------------------------|-------------------------------|                            
        | Batch 1  | ce2-dzd-001 (16.0ms) 25/25     | frankry (2.1ms) 25/25          | lon001-dz002 (1.5ms) 25/25     | nyc001-dz002 (2.4ms) 25/25     | dz-slc-sw01 (19.9ms) 25/25    | dgt-dzd-sin-sg3 (4.2ms) 25/25 |                            
        | Batch 2  | dz-ld4-sw01 (14.6ms) 25/25     | dub001-dz001 (23.2ms) 25/25    | mrs001-dz002 (17.1ms) 25/25    | dgt-dzd-nyc-ny2 (3.6ms) 25/25  | dz-chi-sw01 (47.1ms) 25/25    | bom001-dz001 (77.6ms) 25/25   |                            
        | Batch 3  | dgt-dzd-ams-ams1 (2.8ms) 25/25 | dz-sxb-01 (5.1ms) 25/25        | cherydam (8.4ms) 25/25         | dgt-dzd-ash-dc3 (6.6ms) 25/25  | dgt-dzd-lax-la2 (8.5ms) 25/25 | bom001-dz002 (68.2ms) 25/25   |                            
        | Batch 4  | lts-dzd-001 (7.9ms) 0/0        | dz-mad-01 (29.7ms) 0/0         | bdc-dzd-001 (19.2ms) 0/0       | dz-tor1-sw01 (13.4ms) 0/0      | sea001-dz001 (19.6ms) 0/0     | dz-ty9-sw01 (79.2ms) 0/0      |                            
        | Batch 5  | dgt-dzd-lon-ths (6.1ms) 25/25  | swerry (29.6ms) 25/25          | dgt-dzd-fra-fr5 (14.6ms) 25/25 | was001-dz001 (6.2ms) 25/25     | tyo001-dz002 (105.7ms) 25/25  | hkg001-dz002 (31.4ms) 25/25   |                            
        | Batch 6  | dz-muc-01 (17.7ms) 25/25       | dgt-dzd-dub-db2 (19.2ms) 25/25 | dz-mrs-01 (24.4ms) 25/25       | dfw001-dz002 (34.6ms) 25/25    | dz-sea10-sw01 (17.9ms) 25/25  | tyo002-dz002 (73.1ms) 25/25   |                            
        | Batch 7  | dz-ams-01 (2.1ms) 25/25        | cherlita (24.7ms) 25/25        | dz-fra-01 (14.0ms) 25/25       | dz-ny7-sw01 (1.8ms) 25/25      | dz-lax-sw01 (12.6ms) 25/25    | tyo001-dz001 (74.2ms) 25/25   |                            
        | Batch 8  | swdzd01-lon2 (8.1ms) 25/25     | ams001-dz002 (13.3ms) 25/25    | dub001-dz002 (11.8ms) 25/25    | dz-dc10-sw01 (7.2ms) 25/25     | sjc001-dz002 (1.8ms) 25/25    | dz-sg1-sw01 (3.8ms) 25/25     |                            
        | Batch 9  | fra001-dz002 (7.0ms) 25/25     | mrs001-dz001 (18.4ms) 25/25    | dz-waw-01 (28.6ms) 25/25       | sao001-dz001 (111.9ms) 25/25   | chi001-dz001 (46.6ms) 25/25   | sin001-dz002 (2.9ms) 25/25    |                            
        | Batch 10 |                                | dz-fr5-sw01 (13.4ms) 15/15     |                                | dz-ch2-sw01 (19.0ms) 15/15     | lax001-dz002 (12.4ms) 15/15   | hkg001-dz001 (37.4ms) 15/15   |                            
        | Batch 11 |                                |                                |                                | sao001-dz002 (111.0ms) 5/5     | dfw001-dz001 (44.2ms) 5/5     |                               |                            
        | Batch 12 |                                |                                |                                | nyc002-dz002 (2.3ms) 5/5       | sea001-dz002 (19.2ms) 5/5     |                               |                            
        | Batch 13 |                                |                                |                                | dgt-dzd-dal-da3 (34.7ms) 5/5   | sjc001-dz001 (4.1ms) 20/20    |                               |                            
        | Batch 14 |                                |                                |                                | nyc001-dz001 (3.0ms) 5/5       |                               |                               |                            
        | Batch 15 |                                |                                |                                | was001-dz002 (7.1ms) 5/5       |                               |                               |                            
        | Batch 16 |                                |                                |                                | chi001-dz002 (22.0ms) 5/5      |                               |                               |

@nikw9944
Copy link
Contributor Author

nikw9944 commented Dec 30, 2025

infra PR: https://github.com/malbeclabs/infra/pull/451

snormore
snormore approved these changes Dec 30, 2025
View reviewed changes
Copy link
Contributor

@snormore snormore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@snormore snormore snormore approved these changes

@packethog packethog Awaiting requested review from packethog

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

qa: update alldevices test to only connect qa hosts to devices in the same continent

3 participants

@nikw9944 @snormore