Skip to content

Commit e4dfcf1

Browse files
juanjemdIosjuanjemdIos
committed
bug extracting readthedocs badges with regexp. Documentation with support languages. Fixes #860, Fixes #862, Fixes #857 
1 parent acde842 commit e4dfcf1

File tree

15 files changed

+1531
-108
lines changed

15 files changed

+1531
-108
lines changed

‎docs/readmefile.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
The following metadata fields can be extracted from a readme.md file.
2+
Unlike others files formats (pom, cargo, cabal...), README documents do not follow a formal specification. They are free‑form text files, usually written in markdown or restructuredtext, and their structure varies widely across projects. SOMEF applies heuristics to identify common sections (e.g., Title, Description, Installation, Usage, License...) and extracts metadata accordingly.
3+
4+
| Software metadata category | SOMEF metadata JSON path | README.MD metadata file field |
5+
|--------------------------------|----------------------------------------|----------------------------------------|
6+
| acknowledgement | acknowledgement[i].result.value | hearders with acknowledgement |
7+
| citation | citation[i].result.value | headers with citation, reference, cite. Extract bibtext **(1)** |
8+
| contact | contact[i].result.value | headers with contact |
9+
| contributing_guidelines | contributing_guidelines[i].result.value | headers with contributing |
10+
| contributors | contributors[i].result.value | headers with contributor |
11+
| description | description[i].result.value | headers with description, introduction, basics, initiation, overview |
12+
| documentation | documentation[i].result.value | github or gitlab url documentation **(2)**, headers with documentation, readthedocs same name project, readthedocs in badges, wiki links in badges and text |
13+
| download | download[i].result.value | headers with download |
14+
| executable_example | executable_example[i].result.value | extracts Binder from badgets **(3)** |
15+
| faq | faq[i].result.value | headers with faq, errors, problems |
16+
| full_title | full_title[i].result.value | extract full title **(4)** |
17+
| homepage | homepage[i].result.value | homepage from badgets **(5)** |
18+
| identifier | idenfier[i].result.value | extract from badgets directly or get from zenodo with latest doi **(6)**, swh identifiers **(7)** |
19+
| images | images[i].result.value | other images in the README apart from the logo |
20+
| installation | installation[i].result.value | headers with installation, install, setup, prepare, preparation, manual, guide |
21+
| license | license[i].result.value | headers with license |
22+
| logo | logo[i].result.value | look images in badges and text **(8)** |
23+
| package_distribution | package_distribution[i].result.value | Pypi or latest Pypi version in badges **(9)** |
24+
| related_documentation | dorelated_documentationumentation[i].result.value | readthedocs diferent name project |
25+
| run | run[i].result.value | headers with run, execute |
26+
| readme_url | readme_url[i].result.value | url in raw githubuser content **(10)** |
27+
| related_papers | related_papers[i].result.value | look for arXiv reference in all the text **(11)** |
28+
| repository_status | repository_status[i].result.value | badges with Project status **(12)** |
29+
| requirements | requirements[i].result.value | headers with requirement, prerequisite, dependency, dependent |
30+
| support | support[i].result.value | headers with support, help, report |
31+
| support_channels | support_channels[i].result.value | extract information of gitter, reddit and discord in badges and text **(13)** |
32+
| usage | usage[i].result.value | headers with usage, example, implement, implementation, demo, tutorial, start, started |
33+
34+
35+
------
36+
37+
**(1)**
38+
- Example:
39+
```bib
40+
@inproceedings{garijo2017widoco,
41+
title={WIDOCO: a wizard for documenting ontologies},
42+
author={Garijo, Daniel},
43+
booktitle={International Semantic Web Conference},
44+
pages={94--102},
45+
year={2017},
46+
organization={Springer, Cham},
47+
doi = {10.1007/978-3-319-68204-4_9},
48+
funding = {USNSF ICER-1541029, NIH 1R01GM117097-01},
49+
url={http://dgarijo.com/papers/widoco-iswc2017.pdf}
50+
}
51+
```
52+
- Result:
53+
```
54+
{
55+
"result": {
56+
"value": "@inproceedings{garijo2017widoco,\n url = {http://dgarijo.com/papers/widoco-iswc2017.pdf},\n funding = {USNSF ICER-1541029, NIH 1R01GM117097-01},\n doi = {10.1007/978-3-319-68204-4_9},\n organization = {Springer, Cham},\n year = {2017},\n pages = {94--102},\n booktitle = {International Semantic Web Conference},\n author = {Garijo, Daniel},\n title = {WIDOCO: a wizard for documenting ontologies},\n}",
57+
"type": "Text_excerpt",
58+
"format": "bibtex",
59+
"doi": "10.1007/978-3-319-68204-4_9",
60+
"title": "WIDOCO: a wizard for documenting ontologies",
61+
"author": "Garijo, Daniel",
62+
"url": "http://dgarijo.com/papers/widoco-iswc2017.pdf"
63+
},
64+
}
65+
```
66+
67+
68+
**(2)**
69+
- Example if github:
70+
```
71+
f"https://github.com/{owner}/{repo_name}/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}"
72+
```
73+
- Example if gitlab:
74+
```
75+
f"https://{domain_gitlab}/{owner}/{repo_name}/-/tree/{urllib.parse.quote(repo_default_branch)}/{docs_path}"
76+
```
77+
78+
**(3)**
79+
- Example: `[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/user/repo/HEAD)`
80+
- Result: `"value": "https://mybinder.org/v2/gh/user/repo/HEAD"`
81+
82+
**(4)**
83+
- Example: `# WIzard for DOCumenting Ontologies (WIDOCO)`
84+
- Result:
85+
```
86+
"full_title": [
87+
{
88+
"result": {
89+
"type": "String",
90+
"value": "WIzard for DOCumenting Ontologies (WIDOCO)"
91+
},
92+
"confidence": 1,
93+
"technique": "regular_expression",
94+
"source": "https://raw.githubusercontent.com/dgarijo/Widoco/master/README.md"
95+
}
96+
]
97+
```
98+
99+
**(5)**
100+
- Example: `[![Project homepage](https://img.shields.io/badge/homepage-project-blue)](https://myproject.org)`
101+
- Result: `"value": "https://myproject.org"`
102+
103+
104+
**(6)**
105+
- Example: `[![DOI](https://zenodo.org/badge/11427075.svg)](https://doi.org/10.5281/zenodo.11093793)`
106+
- Result: `"value": "https://doi.org/10.5281/zenodo.11093793"`
107+
108+
**(7)**
109+
- Example: `[![SWH](https://archive.softwareheritage.org/badge/swh:1:dir:40d462bbecefc3a9c3e810567d1f0d7606e0fae7/)](https://archive.softwareheritage.org/swh:1:dir:40d462bbecefc3a9c3e810567d1f0d7606e0fae7;origin=...)`
110+
- Result: ` "value": "https://archive.softwareheritage.org/swh:1:dir:40d462bbecefc3a9c3e810567d1f0d7606e0fae7",`
111+
112+
113+
**(8)**
114+
- Example: `![Logo](src/main/resources/logo/logo2.png)`
115+
- Result: `"value": "https://raw.githubusercontent.com/dgarijo/Widoco/master/src/main/resources/logo/logo2.png"``
116+
117+
**(9)**
118+
- Example: `[![PyPI](https://badge.fury.io/py/somef.svg)](https://badge.fury.io/py/somef) `
119+
- Result: `"value": "https://pypi.org/project/somef"`
120+
121+
122+
**(10)**
123+
- Example:
124+
```
125+
[Yulun Zhang](http://yulunzhang.com/), [Yapeng Tian](http://yapengtian.org/), [Yu Kong](http://www1.ece.neu.edu/~yukong/), [Bineng Zhong](https://scholar.google.de/citations?user=hvRBydsAAAAJ&hl=en), and [Yun Fu](http://www1.ece.neu.edu/~yunfu/), "Residual Dense Network for Image Super-Resolution", CVPR 2018 (spotlight), [[arXiv]](https://arxiv.org/abs/1802.08797)
126+
```
127+
- Result: `"value": "https://arxiv.org/abs/1802.08797"`
128+
129+
130+
**(11)**
131+
- Example:
132+
```
133+
f"https://raw.githubusercontent.com/{owner}/{repo_name}/{repo_ref}/{urllib.parse.quote(partial)}"
134+
```
135+
136+
**(12)**
137+
- Example:
138+
```
139+
[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
140+
```
141+
- Result:
142+
```
143+
"value": "https://www.repostatus.org/#active",
144+
"description": "Active \u2013 The project has reached a stable, usable state and is being actively developed."
145+
```
146+
147+
**(13)**
148+
- Example:
149+
```
150+
[![Gitter chat](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/myproject/community)
151+
[Reddit](https://www.reddit.com/r/myproject)
152+
[Discord](https://discord.com/invite/xyz789)
153+
```
154+
- Result:
155+
```
156+
"value": "https://gitter.im/myproject/community"
157+
....
158+
"value": "https://www.reddit.com/r/myproject"
159+
.....
160+
"value": "https://discord.com/invite/xyz789"
161+
```
162+
163+

docs/requerimentstxt.md renamed to docs/requirementstxt.md

File renamed without changes.

‎docs/supported_languages.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
2+
SOMEF recognizes the programming languages used in a software repository by inspecting
3+
well-known configuration files, dependency descriptors and executable artifacts.
4+
To know more about the extraction details for each type of file, click on it.
5+
6+
7+
8+
| Language | Supported Files |
9+
|-----------|----------------------------|
10+
| Haskell | [`*.cabal`](./cabal.md) |
11+
| Java | [`pom.xml`](./pom.md) |
12+
| JavaScript | [`package.json`](./packagejson.md), [`bower.json`](./bower.md) |
13+
| Julia | [`Project.toml`](./julia.md) |
14+
| PHP | [`composer.json`](./composer.md) |
15+
| Python | [`setup.py`](./setuppy.md), [`pyproject.toml`](./pyprojecttoml.md), [`requirements.txt`](./requirementstxt.md) |
16+
| R | [`DESCRIPTION`](./description.md) |
17+
| Ruby | [`*.gemspec`](./gemspec.md) |
18+
| Rust | [`Cargo.toml`](./cargo.md) |
19+
20+
---
21+
22+
SoMEF also detects the following files to recognize build instructions, workflows or executable examples:
23+
24+
25+
| Language | Supported Files | Software metadata category |
26+
|-----------|------------------------------------|-----------------------------|
27+
| Docker | `Dockerfile`, `docker-compose.yml` | has_built_file
28+
| Jupyter Notebook | `*.ipynb` | executable_example |
29+
| Ontologies | `*.ttl`, `*.owl`, `*.nt`, `*.xml`, `*.jsonld` | ontologies |
30+
| Shell | `*.sh` | has_script_file |
31+
| YAML | `*.yml`, `*.yaml` | continuous_integration, workflows
32+
33+

‎docs/supported_metadata_files.md‎

Lines changed: 1 addition & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -2,82 +2,6 @@
22

33
This project supports extracting metadata from specific types of files commonly used to declare authorship and contribution in open source repositories.
44

5-
## Supported files of authors.
6-
7-
The following filenames are recognized and processed automatically:
8-
9-
* `AUTHORS`
10-
* `AUTHORS.md`
11-
* `AUTHORS.txt`
12-
13-
These files are expected to be located at the root of the repository. Filenames are matched case-insensitively.
14-
15-
## Purpose and Format
16-
17-
These files typically contain a list of individuals and/or organizations that have contributed to the project. While there is no universal standard for formatting, a widely referenced convention is Google's guidance:
18-
19-
🔗 [Google Open Source: Authors Files Protocol](https://opensource.google/documentation/reference/releasing/authors/)
20-
21-
The content may be structured as:
22-
23-
* Simple plain text, with one contributor per line.
24-
* Markdown-formatted text (`.md` files).
25-
* Lines including contributor names, emails (e.g., `Name <email>`), and sometimes affiliations.
26-
27-
### Examples of Valid Entries
28-
29-
```text
30-
Jane Doe <[email protected]>
31-
John Smith
32-
Acme Corporation <[email protected]>
33-
Google Inc.
34-
```
35-
36-
### Examples of NON Valid Entries
37-
38-
```text
39-
JetBrains <>
40-
Microsoft
41-
Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung
42-
scrawl - Top contributor
43-
Tom
44-
```
45-
## What Is Read vs. Discarded
46-
47-
When processing these files, the parser will:
48-
49-
**Include** lines that:
50-
51-
* Contain person names, optionally with emails (`Name <email>`).
52-
* Clearly refer to organizations (e.g., "Google LLC", "OpenAI Inc.").
53-
54-
**Discard** lines that:
55-
56-
* Are headers, decorative separators, or markdown formatting (`#`, `*`, `=`, etc.).
57-
* Contain only URLs or links.
58-
* Are single words with no email and no organizational keyword (e.g., `JetBrains <>`).
59-
* Are markdown or structured noise (`---`, `{}`, etc.).
60-
* Contain more than four words and are not recognized as organizations — to avoid capturing generic or descriptive sentences (e.g., This line not is an author).
61-
62-
### Special Cases
63-
64-
* Entries with only a first name and an email are accepted but must not assign an empty `last_name`.
65-
* Lines starting with `-` or `*` are considered lists, but only parsed if the content matches expected author patterns.
66-
* Blocks enclosed in `{}` are stripped before parsing.
67-
* Any line matching known organization suffixes (`Inc.`, `LLC`, `Ltd.`, `Corporation`) is treated as an organization, even if no email is present.
68-
* Some organization names (e.g., Open Source Initiative) may be mistakenly treated as person names if they do not contain a company designator or email. To improve detection, it is recommended to use names like Open Source Initiative Inc.
69-
* In such cases, only the meaningful part (typically the name) is extracted before any descriptive annotations.
70-
For example, the line:
71-
Tom Smith (Tom) - Project leader 2010-2018
72-
Will be interpreted as:
73-
{
74-
"type": "Person",
75-
"name": "Tom Smith",
76-
"value": "Tom Smith",
77-
"given_name": "Tom",
78-
"last_name": "Smith"
79-
}
80-
815

826
## Supported Metadata Files in SOMEF
837

@@ -90,6 +14,7 @@ SOMEF can extract metadata from a wide range of files commonly found in software
9014
| `bower.json` | JavaScript (Bower) | Package descriptor used for configuring packages that can be used as a dependency for Bower-managed front-end projects. | <div align="center">[🔍](./bower.md)</div>| [📄](https://github.com/bower/spec/blob/master/json.md)| |[Example](https://github.com/juanjemdIos/somef/blob/master/src/somef/test/test_data/repositories/js-template/bower.json) |
9115
| `package.json` | JavaScript / Node.js | Defines metadata, scripts, and dependencies for Node.js projects | <div align="center">[🔍](./packagejson.md)| [📄](https://docs.npmjs.com/cli/v10/configuring-npm/package-json)| 10.9.4|[Example](https://github.com/npm/cli/blob/latest/package.json) |
9216
| `codemeta.json` | JSON-LD | Metadata file for research software using JSON-LD vocabulary | <div align="center">[🔍](./codemetajson.md)</div> | [📄](https://github.com/codemeta/codemeta/blob/master/crosswalk.csv)| [v3.0](https://w3id.org/codemeta/3.0)|[Example](https://github.com/codemeta/codemeta/blob/master/codemeta.json) |
17+
| `readme.me` | Markdown | Main documentation file of repository | <div align="center">[🔍](./readmefile.md)</div>| | |[Example](https://github.com/KnowledgeCaptureAndDiscovery/somef/blob/master/README.md) |
9318
| `composer.json` | PHP | Manifest file serves as the package descriptor used in PHP projects. | <div align="center">[🔍](./composer.md)</div>| [📄](https://getcomposer.org/doc/04-schema.md)| [2.8.12](https://getcomposer.org/changelog/2.8.12)|[Example](https://github.com/composer/composer/blob/main/composer.json) |
9419
| `juliaProject.toml` | Python | Defines the package metadata and dependencies for Julia projects, used by the Pkg package manager.| <div align="center">[🔍](./julia.md)</div>| [📄](https://docs.julialang.org/en/v1/)| |[Example](https://github.com/JuliaLang/TOML.jl/blob/master/Project.toml) |
9520
| `pyproject.toml` | Python | Modern Python project configuration file used by tools like Poetry and Flit | <div align="center">[🔍](./pyprojecttoml.md)</div>| [📄](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)| |[Example](https://github.com/KnowledgeCaptureAndDiscovery/somef/blob/master/pyproject.toml) |

‎mkdocs.yml‎

Lines changed: 16 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -4,25 +4,23 @@ nav:
44
- Install: install.md
55
- Usage: usage.md
66
- Output: output.md
7-
- Supported formats:
8-
- Authors: author.md
9-
- Bower: bower.md
10-
- Cabal: cabal.md
11-
- Cargo: cargo.md
12-
- Codemeta: codemetajson.md
13-
- Composer: composer.md
14-
- Description: description.md
15-
- Gemespec: gemspec.md
16-
- Julia Projects: julia.md
17-
- Package JSON: packagejson.md
18-
- Pom: pom.md
19-
- Pyproject: pyprojecttoml.md
20-
- Requirements: requirementstxt.md
21-
- Setup: setuppy.md
22-
7+
- Supported metadata files: supported_metadata_files.md
8+
- Supported languages: supported_languages.md
9+
# - Authors: author.md
10+
# - Bower: bower.md
11+
# - Cabal: cabal.md
12+
# - Cargo: cargo.md
13+
# - Codemeta: codemetajson.md
14+
# - Composer: composer.md
15+
# - Description: description.md
16+
# - Gemespec: gemspec.md
17+
# - Julia Projects: julia.md
18+
# - Package JSON: packagejson.md
19+
# - Pom: pom.md
20+
# - Pyproject: pyprojecttoml.md
21+
# - Requirements: requirementstxt.md
22+
# - Setup: setuppy.md
2323

24-
- Contributing: contributing.md
25-
- Changelog: changelog.md
2624
theme:
2725
name: material
2826

0 commit comments

Comments
 (0)