Conversation
rawahars
force-pushed
the
main
branch
3 times, most recently
from
a7f450e to
5760cdd
Compare
* HvSocket support for containers Applications connecting from the host into the container should use container-specific VMID. This ID will need to be the same as the container's VMID inside the guest, which is calculated by HCS/GCS like it's done in this PR by `HCSIDToGUID`. To allow the container ID to work with HvSocket on the host, we need to set up an AddressInfo mapping to tell HvSocket to redirect the call into the UVM, which is done in this PR by default for all WCOW containers. Add internal `hvsocketaddr.exe` tool that clients can use to generate VM ID for container. Add a generic function for creating HvSocket address info mapping. export a function that creates a mapping for containers only. --------- Signed-off-by: Maksim An <maksiman@microsoft.com> Co-authored-by: Kevin Parsons <kevpar@microsoft.com>
* feature: cross-container named pipes
Add new "uvm://" mount prefix to support cross-container
pipes for Xenon WCOW containers. For now, it's a WCOW-only
feature, while the Linux work is being prototyped.
Additionally, extend the logic of `GetContainerPipeMapping` to
also handle cross-container pipes within the UVM. The syntax
similar to sandbox mounts:
```
{
"host_path": "uvm://\\\\.\\pipe\\uvmPipe",
"container_path": "\\\\.\\pipe\\containerPipe"
}
```
Containers sharing the pipe need to have the same "host_path".
refactor how named pipes are parsed and added for WCOW.
`setupMounts` will now try to parse mount source as a named pipe
for both process isolated and hyper-v isolated containers.
The mapped pipes will be tracked under `namedPipeMounts` and
later added to HCS container doc.
go mod tidy in test directory
---------
Signed-off-by: Maksim An <maksiman@microsoft.com>
Signed-off-by: Maksim An <maksiman@microsoft.com>
tooling: allow pause container to be run in privileged mode
Currently WCOW UVM only support booting with VmbFS and legacy layers. However, we are adding support for booting the UVM with BlockCIM layers. This commit updates the WCOWBootFiles struct to support different boot configurations. Signed-off-by: Amit Barve <ambarve@microsoft.com>
Initial changes to allow creating confidential WCOW UVMs. uvmboot tool is also updated for easier command line testing of confidential UVMs. Signed-off-by: Amit Barve <ambarve@microsoft.com>
* Fix `golang.org/x/crypto` & `/net` vulnerabilities Update `golang.org/x/crypto` and`golang.org/x/net` to fix reported vulnerabilies. (This update requires `go1.23`, so updated that in `go.mod`). Also update other `golang.org/x/` modules. PRs: - 2418 - 2417 - 2415 - 2414 - 2411 - 2409 - 2408 - 2396 - 2395 NOTE: **This commit only has updates to `go.mod`.** Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * `go.sum` and vendor updates Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
…icrosoft#2405) This PR fixes a discrepancy between user info handling between GCS and rego policy enforcer. For example, GCS doesn't require the user/group to exist in container's /etc/passwd and /etc/group and has a fallback to UID and GID 0, when the user is absent. Rego enforcer's `GetUserInfo`, however, always tries to lookup user/group in /etc/passwd and /etc/group and returns an error when the UID doesn't exist. This behavior is inconsistent with non confidential LCOW workloads and fixed in this PR. To avoid circular imports, the spec.go and spec_devices.go under `internal/guest/runtime/hcsv2` have been moved under `internal/guest/spec` and the dependent code updated accordingly. As a result a bunch of methods are now exported, but still under `internal`, so this shouldn't cause problems. User parsing has been updated and split into `ParseUserStr`, which returns UID and GID for a given `username` string and `SetUserStr`, which just sets the UID and GID for the OCI process. Rego enforcer's `GetUserInfo` now prioritizes the result of `ParseUserStr` and fallbacks to the previous behavior of UID/GID lookup in container's filesystem. Signed-off-by: Maksim An <maksiman@microsoft.com>
* Use multi-error for annotation processing Return all errors from `"internal/oci".ProcessAnnotations`. Update associated tests to check for expected errors. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: always check `errors.Is`; fix `subtest` bug Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
* Add `rootfs.exe` tool to merge tar image layers Add tool to merge layer tarball files, which is used for creating WCOW uVM images with additional features added, without first needing to create a dedicated container image. This basically amounts to the append and flatten container image operations from the [crane](github.com/google/go-containerregistry/cmd/crane) tool, with some specific changes: - use `"path"` and not `"path/filepath"` for path manipulation, which avoids paths with `\` separators; - append a trailing `/` to directory paths (since different `tar` implementations may append them); and - overwrite the owner UID and GID to zero, to avoid any user issues. The tool avoids needing to unpack tar files within the same directory (as is done for LCOW) to combine the deltas with tar images. Based on 2024 Hackathon to replace current LCOW uVM rootfs creation, before the switch to Azure Linux. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: remove leftover code, fix linter, extra comment Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Adds the Go wrappers on windows device query APIs to allow finding a disk device by the provided LUN. This is mostly required inside the GCS where shim includes the LUN at which a disk is attaches in the request and then the GCS needs to be able to find that disk to use it further. Signed-off-by: Amit Barve <ambarve@microsoft.com>
This commit adds a new tool that can take a valid Windows container image tarball and extracts all the UtilityVM files from that tarball into a block CIM. The end result should be a block CIM that has all the files necessary to successfully boot & run a UtilityVM. Signed-off-by: Amit Barve <ambarve@microsoft.com>
* Add `uvm://` mount support for LCOW Allow privileged LCOW containers to mount the uVM's filesystem. This supports cadvisor and related tools to monitor the pod and associated containers without needing to run directly within the uVM. See: https://github.com/google/cadvisor Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: path name Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
* Store extraction error for `LazyImageLayers` Have `test\internal\layers.LazyImageLayers` always fail if image extraction fails initially. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * Build required binaries Add build stages for various binaries to ensure they compile without issue. Alphabetize build steps to make lookup easier. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
* Switch to `golangci-lint` v2 v1 is deprecated, and `golangci/golangci-lint-action@v8` requires v2.1+. Use `golangci-lint migrate` to switch to v2 settings and remove outdated/deprecated settings. Ignore `nil` pointer lints for tests, since the linter doesn't handle `t.Fatal` appropriately. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * Fix linter errors Various new lint issues, mostly staticcheck quick fixes (via `golangci-ling --fix`): - embedded field selector - simplify conditionals via De Morgan - replace `strings.Replace(_, _, _, -1)` with `strings.ReplaceAll()` - switch `if/else-if` to `switch` - move conditional into infinite `for` loops - omit unnecessary type declaration Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
For environment variables in the rego framework. Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com>
* Trim LCOW `GetProperties` response Zero out the Linux `GetProperties` `Blkio` field, since it scales with the number of container layers attacked to the uVM. Additionally empty the `Rdma` and `Network` fields, in case they can also grow without bound. None of the fields are used in any code paths in the AzCRI, or exposed elsewhere. Clarify comment about the maximum message size, to reflect that it mirrors and HCS value and is not arbitrary. Additionally, don't quit the receive loop if the message size is too large, since that brings the bridge down with it. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: undo receive loop changes Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
* Swap `EvalSymlinks` with `ResolvePath` Redo PR 1644, which swapped builtin `"path/filepath".EvalSymlinks` with `"github.com/Microsoft/go-winio/pkg/fs".ResolvePath`, since the later is able to handle deeply nested symlinks and (as of [go1.23](golang.org/doc/go1.23#pathfilepathpkgpathfilepath)), mountpoints. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: update CIM test code Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Empty path was being added to lookup paths for `LinuxBootFiles`. Worked since the path was often just `C:\ContainerPlat`, but fix it regardless. Also add check to make sure the `LinuxBootFiles` path found is also a directory. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Break out `pkg\annotations\annotations.go` constants into sections for easier searching and readability. Deprecate `AnnotationExpansions` and instead provide `AnnotationExpansionMap()`, which returns the same value, but provides a new copy every call, so the `map` cannot be modified. Cannot delete it or change its type, since it is technically public. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
* Omnibus dependabot update Combined dependency upgrades that update both `go.mod` and `test/go.mod` so that CI passes. dependabot PRs: - 2348 - 2385 - 2397 - 2401 - 2402 - 2435 - 2443 - 2445 - 2446 - 2447 - 2452 - 2453 - 2463 Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * go mod tidy and vendor Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * Regenerate protbuf files Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
When a kernel is built with virtio-vsock we encounter a kernel panic in our init script, while trying to read entropy from the host. Parsing through the boot logs, it seems like the hv_sock transport is not being initialized: ``` [ 0.712310] NET: Registered PF_VSOCK protocol family [ 0.716225] hv_vmbus: registering driver hv_sock [ 0.719551] hv_vmbus: unregistering driver hv_sock [ 0.723281] IPI shorthand broadcast: enabled ``` Disabling `virtio_vsock_init` seems to fix the problem: ``` [ 0.829397] NET: Registered PF_VSOCK protocol family [ 0.831728] initcall virtio_vsock_init blacklisted [ 0.835999] hv_vmbus: registering driver hv_sock [ 0.837906] IPI shorthand broadcast: enabled ``` For kernels built without virtio-vsock, skipping `virtio_vsock_init` is a no-op: ``` [ 0.539320] NET: Registered PF_VSOCK protocol family [ 0.541417] hv_vmbus: registering driver hv_sock [ 0.549999] IPI shorthand broadcast: enabled ``` Signed-off-by: Maksim An <maksiman@microsoft.com>
Block CIMs can now provide integrity checking (via a hash/Merkel tree, similar to dm-verity on Linux). A block CIM written with integrity checking enabled is called a verified CIM. A verified CIM is written once and then sealed to prevent any further modifications. When such a CIM is sealed it returns a digest of its contents. Such a CIM can then be mounted by passing in this digest. Every read on that mounted volume will then be verified against this digest to ensure the integrity of the contents of that CIM. Signed-off-by: Amit Barve <ambarve@microsoft.com>
Signed-off-by: Pooja Mahadev Soundalgekar <pomahade@microsoft.com>
Move inject and load fragment into the securitypolicy pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com>
C-WCOW: SecurityPolicy and sidecar fixes
add gcs-sidecar build step in CI. Signed-off-by: Maksim An <maksiman@microsoft.com>
Adding support for WCOW UVM log forward service
vsmb share redirector start and bind to vmbus
VM version 12 is the minimum version that supports the various SNP features. Signed-off-by: Sunil Muthuswamy <sunilmut@microsoft.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.41.0 to 0.45.0. - [Commits](golang/crypto@v0.41.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Log final LCOW OCI spec; always call go in Makefile Use an `io.MultiWriter` to capture the OCI spec as written to the `config.json` file that is provided to runc so it can be logged, and avoid needing to marshal the spec twice. Also, add a `FORCE` target to the Makefile to always run the `go build` recipe for Go binaries. This fixes a bug where make will not rebuild the binaries and instead `make clean` must be called to remove them before building. See: www.gnu.org/software/make/manual/html_node/Force-Targets.html Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: scrub logged OCI spec Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Add handling of Extensible Virtual Disks mounts for LCOW. Prior to this change, EVDs were supported only for WCOW. The expectation is that the EVD will be presented to container as a block device. Signed-off-by: Maksim An <maksiman@microsoft.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3 to 4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v3...v4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: '4' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…rosoft#2550) Bumps [github.com/opencontainers/runc](https://github.com/opencontainers/runc) from 1.3.0 to 1.3.3. - [Release notes](https://github.com/opencontainers/runc/releases) - [Changelog](https://github.com/opencontainers/runc/blob/v1.3.3/CHANGELOG.md) - [Commits](opencontainers/runc@v1.3.0...v1.3.3) --- updated-dependencies: - dependency-name: github.com/opencontainers/runc dependency-version: 1.3.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Restore runc kill all behavior for init processes PR microsoft#2538 removed the `runc kill --all` flag, when signaling containers. However, when cleaning up after the init process exists, the `--all` flag is still needed to remove any potentially orphaned processes when using runc before v1.2. See: opencontainers/runc@f8ad20f#diff-ade6035c3e554d7627cdc368b27f475fc0dad83e02382a1dea9cae9b75871087 Additionally, switch to using error strings directly from runc code in `internal\guest\runtime\runc\utils.go`: they have been available since runc v1.1.0-rc.1. See: opencontainers/runc#3033 Also, add logic to match on container not/still running error strings and return them for `Kill`, since returning `ERROR_VMCOMPUTE_SYSTEM_ALREADY_STOPPED` (`0xc0370110`) when killing a stopped container is expected behavior and handled appropriately in `"cmd/containerd-shim-runhcs-v1".(*hcsExec).Kill()`. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * go mod tidy and vendor; fix lint issues Fix lint errors (introduced by go1.24): ``` printf: non-constant format string in call to github.com/pkg/errors.Wrapf (govet) ``` Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Fix bug where `internal/guest/runtime/runc/container.go` code assumed that the logfile passed to runc would contain an error without checking. This can result in scenarios where `cmd.Run` (or `cmd.CombinedOutput`) returns a non-nil `err` but (due to a runc's failure to start or write to the log file, or the JSON is invalid) `runcErr` is nil and therefore the error returned by `errors.Wrapf` is also nil. Those scenarios can ultimately panic since it violates invariants where a nil error is assumed to mean a successful operation or a usable return value. Fix this by guarding on `runcErr == nil` and warn in those situations. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
…oft#2544) * C-WCOW: Unify data structures and reuse for C-LCOW and C-WCOW Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Move gcs-sidecar confidential options to securitypolicy pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Move security policy config from LCOW gcs Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Move InjectFragment to securitypolicy pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Move securitycontext dir to securitypolicy pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: PspDriver and hostdata changes in SecurityPolicy pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Cleanup confidential options Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * C-WCOW: Hardware checks and reports in separate pkg Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> * Prevent panic from interface case --------- Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com> Co-authored-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
switch to CimWriter.dll and regenerate syscalls Update CIMFS image creation syscalls to use CimWriter.dll. Add `CimFsSupported` utility to check for CIM FS dlls. Update `pkg/cimfs/cimfs.go` to use `CimFsSupported` together with build number check. Signed-off-by: Maksim An <maksiman@microsoft.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v5...v6) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
… in old policies (microsoft#2540) * rego: Allow sending SIGTERM and SIGKILL to the container init process in old policies We used to allow SIGTERM/SIGKILL the container init process even if the container's signals list is empty due to a bug fixed in microsoft#2538. However, because our tooling has been generating policies with an empty signals list, we need to special case this for old policies to maintain backwards compatibility. Update framework.rego to have SIGTERM and SIGKILL as default kill signals for init process for framework API versions "0.4.1" and below. Newer policies must explicitly have these signals present, otherwise sending signal will be denied. Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com> * Fix missing denial reason when a signal request to a non-init process is denied This happens if the container.signals list contains relevant signals, but the process's signals list does not allow the signal. Old: {"decision":"deny","input":{"argList":["/bin/sleep","infinity"],"containerID":"0971693a04cdd4f2eeefc569754b5cd8046ec0b7c7ed6899bb3dec0dd45ba735","isInitProcess":false,"rule":"signal_container_process","signal":9},"reason":{"errors":[]}} Now: {"decision":"deny","input":{"argList":["/bin/sleep","infinity"],"containerID":"3873bfc939e2415892b5b74a7b1dbade0f7222e266df43df85968ddda59be56e","isInitProcess":false,"rule":"signal_container_process","signal":9},"reason":{"errors":["target isn't allowed to receive the signal"]}} Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> --------- Signed-off-by: Tingmao Wang <tingmaowang@microsoft.com> Co-authored-by: Maksim An <maksiman@microsoft.com>
* Use `tool` directives in `go.mod` Remove `tools/` package and instead mark tools directly in `go.mod` with `tool` directive (support added in go1.24). This has the added benefit of making tools indirect dependencies. See: https://go.dev/ref/mod#go-mod-file-tool Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> * PR: forgot test/go.mod Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com> --------- Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
Signed-off-by: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Drop logs when iterating over host devices from info to trace. Add more logs (and span) to `AddAssignedDevice` function for tracing where time is spent when assigning devices. Increase `AddAssignedDevice` timeout from 10 seconds to 60 to allow more time when adding a large number of devices or waiting on host to make them available. Add log when adding OCI runtime hook for nvidia tool. Signed-off-by: Hamza El-Saawy <hamzaelsaawy@microsoft.com>
This implementation was added for hcsshim + HvLite (OpenHCL). The idea was that we could use different virt stacks, as long as they implement the remotevm APIs. That initiative is dormant as of present and this codepath is no-op/dead code. We can likely re-use and re-purpose the interfaces created in the package. But keeping the obsolete code demand maintenance cost too when we make changes to the interface. If we need to support this functionality in the future, this commit can be reverted. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
The Kubernetes project has migrated all container images to a new community-owned registry: registry.k8s.io instead of k8s.gcr.io. This commit changes the same. Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.