Feature/filesystem vcs ignoring

[Deepam02]

closes #439

The filesystem tools now respect .gitignore rules.
.git directories and ignored files are automatically excluded from the following operations by default:

• search_files
• search_files_content
• list_directory
• list_directory_with_sizes
• directory_tree

[Deepam02]

I have fixed the lint issues.

[rumpl]

The library you are using for gitignore is very old and not maintained any more, could we use go-git? Here's a piece of code that an agent gave me, it seems to work really well, might need some cleanup but it shows the idea of how to check if something is ignored or not by git:

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/go-git/go-billy/v5/util"
	"github.com/go-git/go-git/v5"
	"github.com/go-git/go-git/v5/plumbing/format/gitignore"
)

type IgnoreChecker struct {
	repo     *git.Repository
	patterns []gitignore.Pattern
	matcher  gitignore.Matcher
}

func NewIgnoreChecker(repo *git.Repository) (*IgnoreChecker, error) {
	w, err := repo.Worktree()
	if err != nil {
		return nil, err
	}

	var patterns []gitignore.Pattern

	patterns, err = gitignore.ReadPatterns(w.Filesystem, nil)
	if err != nil {
		return nil, err
	}

	matcher := gitignore.NewMatcher(patterns)

	return &IgnoreChecker{
		repo:     repo,
		patterns: patterns,
		matcher:  matcher,
	}, nil
}

func (ic *IgnoreChecker) IsIgnored(path string, isDir bool) bool {
	return ic.matcher.Match([]string{path}, isDir)
}

func (ic *IgnoreChecker) WalkRepository() error {
	w, err := ic.repo.Worktree()
	if err != nil {
		return err
	}

	fs := w.Filesystem

	return util.Walk(fs, "", func(path string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		}

		if path == ".git" {
			return filepath.SkipDir
		}

		isDir := info.IsDir()
		ignored := ic.IsIgnored(path, isDir)

		status := "Tracked"
		if ignored {
			status = "Ignored"
		}

		fmt.Printf("[%s] %s\n", status, path)

		// Skip ignored directories
		if ignored && isDir {
			return filepath.SkipDir
		}

		return nil
	})
}

func main() {
	repo, err := git.PlainOpen("<PATH_TO_YOUR_REPOSITORY>")
	if err != nil {
		log.Fatal(err)
	}

	checker, err := NewIgnoreChecker(repo)
	if err != nil {
		log.Fatal(err)
	}

	if err := checker.WalkRepository(); err != nil {
		log.Fatal(err)
	}
}

[rumpl]

The filesystem toolset has the add_allowed_directory, when a dir is added we should add a matcher for it I think

[Deepam02]

Good catch! Currently matchers are only loaded at initialization. I can easily add matcher loading in addAllowedDirectory() by extracting the loading logic into a helper method

[Deepam02]

@rumpl Sorry about the test failures, I have fixed the issues now and everything should be working fine.

Is the rest of the PR good?

[rumpl]

@Deepam02 i’ll try to find the time tomorrow to take a close look (I’ll be at a conference all day). Looks good from afar though. Thanks for all the contributikns