feat: support returnDistinctValues

[jmackenzieGA]

I get an error trying to convert a featureLayer to an sf object? Any suggestions?

library(arcgislayers)
url <- "https://mapprod3.environment.nsw.gov.au/arcgis/rest/services/Planning/EPI_Primary_Planning_Layers/MapServer/2"
fs <- arc_open(url)
ls <- list_fields(fs)
sf <- arc_select(fs, fields = c("LAY_NAME","LAY_CLASS"))

Error in rlang::set_names():
! x must be a vector
Run rlang::last_trace() to see where the error occurred.

I am able to create a featureLayer object from the featureServer URL, and I can list the field names. I tried using the 'fields' flag to limit what is read in, but there seems to be an issue with my featureLayer object when i run arc_select( )?

For this exercise, my goal is just to wrangle unique values from one field in the attribute table, but in general, I am hoping for a generic solution to wrangle all available features & attributes from a large collection of publicly available featureServer URLs.

[JosiahParry]

Okay, so you didn't copy the whole error 😉

There is an issue in the requests somewhere along the way. This took a minute or so on my side which suggests this is a very big feature service and you want to bring all of it into memory.

I'd recommend you don't do that and only bring down the bits and pieces you need. As you can see there are nearly 71k features that you are reading into memory with this. My guess is that the featureservice sees this as a potential attack and is rate limiting you.

httr2::request(url) |> 
  httr2::req_url_path_append("query") |> 
  httr2::req_url_query(where = "1=1", outFields = "*", returnCountOnly = "true", f = "json") |> 
  httr2::req_perform() |> 
  httr2::resp_body_json()

#> $count
#> [1] 70797

[jmackenzieGA]

thanks for the feedback, josiah. not sure how i managed to crop of the error message, so apologies for any misdirection.

often i can spatially restrict queries to an AOI, but for this task, i need to extract a unique list of available state-wide planning zones (from the field 'LAY_NAME'). i couldn't figure out how to drop the geometries and just work with the table, so i tried pulling everything local.

following your httr example, i'll see if i can pipe the wrangling server-side, and avoid both rate limiting and unnecessary downloads )

[JosiahParry]

I’d strongly suggest you don’t do that! You’ll be inadvertently reinventing this package. Do you need the geometries immediately?

Do you have more detail to the problem you can provide? I’d be happy to help you through this :)

[jmackenzieGA]

the arcgislayers package is great, so definitely not looking to re-invent anything!

for this task, i just need a list of unique values from a single field in the attribute table, not any geometries.

in the big picture, our organisation is hoping to rely more on feature services for access to authoritative datasets. i'm in the process of assessing pros and cons. feature services are great for visualisation, and some types queries are well supported. however, for large-scale spatial analysis, if rate limiting is common (and highly variable) amongst host settings, downloading static datasets may still be required? i'm optimistic this will change. for example, with imagery, using STACS, it's easy to select large AOIs, bands, time periods and project CRS all server side, then just pull down only what's required.

[JosiahParry]

Why don't you try this:

library(arcgislayers)

furl <- "https://mapprod3.environment.nsw.gov.au/arcgis/rest/services/Planning/EPI_Primary_Planning_Layers/MapServer/2"

flayer <- arc_open(furl)

res <- arc_select(
  flayer,
  geometry = FALSE,
  fields = c("LAY_NAME", "LAY_CLASS")
)

dplyr::count(res, LAY_NAME, LAY_CLASS, sort = TRUE)
#> LAY_NAME                                     LAY_CLASS     n
#> 1      Zone                             Public Recreation 17047
#> 2      Zone                    Environmental Conservation  8172
#> 3      Zone                                Infrastructure  6600
#> 4      Zone                      Environmental Management  4576
#> 5      Zone                            Primary Production  3192
#> 6      Zone                               Rural Landscape  3093
#> 7      Zone                       Low Density Residential  3037
#> 8      Zone                    Medium Density Residential  2605
#> 9      Zone                                  Local Centre  2504
#> 10     Zone            National Parks and Nature Reserves  2233
#> 11     Zone                           General Residential  1560
#> 12     Zone                         Large Lot Residential  1556
#> 13     Zone                          Environmental Living  1453

[jmackenzieGA]

thanks, josiah!

the 'geometry=FALSE' flag for the arc_select( ) function is just what i needed for restricting the query to table attributes )

[JosiahParry]

I'm glad that helped! There is also the possibility of using returnDistinctValues argument in the rest API but that doesn't work.

So I'm going to reopen this issue and rename it.

library(arcgislayers)

furl <- "https://mapprod3.environment.nsw.gov.au/arcgis/rest/services/Planning/EPI_Primary_Planning_Layers/MapServer/2"

flayer <- arc_open(furl)

res <- arc_select(
  flayer,
  geometry = FALSE,
  fields = "LAY_CLASS",
  returnDistinctValues = "true"
)

this is something that should work. we could support it directly by adding a new argument to the function

[jmackenzieGA]

even better, josiah! thanks again )