What does understanding code structure and ranking results by relevance actually mean?

[m-kru]

The cs is described as follows:

CLI code search tool that understands code structure and ranks results by relevance. No indexing required

I did a little test today. I cloned the Linux repository and started searching some text patterns. However, not some random patterns. I was searching texts which represent types, mostly structures, for example, eventfd_ctx.

To be honest, I am a little bit disappointed.

1. I thought that the tool is "smart" enough to automatically discover type definitions and place them the highest in the results. However, type definitions are buried down in the results, and I still have to manually skim through the results. The tool ranks higher local variables definitions than global type definition.
2. cs is not aware that the found snippet is a type definition, and it would be nice to extend the printed snippet so that the full definition is displayed. I still have to manually open the file.

These 2 things started me wondering. Maybe I don't understand the idea of cs, and the requirements the tool tries to meet.

[boyter]

1. Not types. That would require a full AST parse of the code. What cs is aware of is, the following,

• If the code is code, a comment or a string
• The relative complexity of the file
• Where complexity hotspots within the file are (although its not using them yet)

It has no concept of types. To do so would require tree-sitter or some such, and probably an index. Note I am looking into this and there may be a way to integrate this information, but for right now its just too much to do. If you need this sort of thing you need a full index system such as hound, sourcegraph etc...

2. cs has no awareness of types.

Now, I suspect may be running on the release version, not the code sitting in master. I should note that the master code is where the extra logic is living, so its possible you dont have access to it at all. Version 2.0.0 was the foundation which has allowed all of the above.

I will be writing a blog post up on all of this at some point in the next week or so. For now however, consider it a ranked grep, but with the ability to do boolean queries, and some knowledge of the structure of code allowing for searching of just strings, comments or code as needed.

[boyter]

I have updated cs so it is now aware of types to an extent... https://github.com/boyter/cs/blob/master/pkg/ranker/declarations.go where it knows about some languages and can use that to dial in things if you select that mode

cs "handleRequest" --only-declarations 

So smarter than before, but certainly not perfect. If you have a language not in the list please submit a PR.

[m-kru]

Well, of course I understand cs does not build AST, this is not the point.
Moreover, I think it should not build AST, as what makes cs unique is its alternative approach, with much lower complexity, yet with potentially extremely useful output.

However, in each language there are some specific characters/tokens that can be used to rank a search as more relevant. For example, in C, whenever there is { after the search text, or } before the search text, this means this is probably a definition, which is more relevant than just a regular use.

Compare the following search results:

linux/fs/eventfd.c
30:struct eventfd_ctx {

linux/drivers/vhost/vhost.h
89:	struct eventfd_ctx *ctx;

You don't need AST to rank the first one higher.

Being fully aware of type definitions is a different story. Much harder, and I can live with an alternative solution where I can open my editor of choice based on the search result by just pressing 'enter' on the selected result. Unfortunately, currently also no possible.

[boyter]

Are you using the last build from master?

As for opening via enter, that should still work,

cat `cs`

Works for me on every version.

[m-kru]

I tried master today. It behaves the same. Still type definition is buried down in the result. Enter does not open the file, it only prints the file path to the stdout. The only thing that differs is highlighting of language keywords.

[boyter]

Use F5 to open the file. Enter will return the path to stdout as you say for backtick

vi `cs`