Skip to content

Move testdata module into testFixtures in the core. #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JohnLCaron merged 1 commit into from
May 24, 2025
Merged

Move testdata module into testFixtures in the core. #136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .idea/gradle.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 26 additions & 13 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,26 @@
# netchdf
_last updated: 5/22/2025_
_last updated: 5/24/2025_

This is a rewrite in Kotlin of parts of the devcdm and netcdf-java libraries.

The intention is to create a maintainable, read-only, pure JVM library allowing full access to
netcdf3, netcdf4, hdf4, hdf5, hdf-eos2 and hdf-eos5 data files.
netcdf3, netcdf4, hdf4, hdf5, hdf-eos2, and hdf-eos5 data files.

Please contact me if you'd like to help out. Especially needed are test datasets from all the important data archives!!

## Building

* Download Java 21 JDK and set JAVA_HOME.
* Download git and add to PATH.

cd <your_build_dir>
git clone https://github.com/JohnLCaron/netchdf.git
cd netchdf
./gradlew clean assemble

Also see:
* [Building and Running ncdump](clibs/Readme.md)

### Why this library?

There is so much important scientific data stored in the NetCDF and HDF file formats, that those formats will
Expand Down Expand Up @@ -87,7 +100,6 @@ Currently that is Gradle 8.14.
For now, you must download and build the library yourself. Eventually we will publish it to Maven Central.
The IntelliJ IDE is highly recommended for all JVM development.


### Scope

Our goal is to give read access to all the content in NetCDF, HDF5, HDF4, and HDF-EOS files.
Expand Down Expand Up @@ -128,16 +140,17 @@ only supported for test purposes.
Currently we have ~1500 test files:

````
hdf4 = 205 files
hdf-eos2 = 267 files
hdf5 = 113 files
hdf-eos5 = 18 files
netcdf3 = 664 files
netcdf3.2 = 81 files
netcdf3.5 = 1 files
netcdf4 = 121 files
````

hdf-eos2 = 267 files
hdf-eos5 = 18 files
hdf4 = 205 files
hdf5 = 113 files
netcdf3 = 664 files
netcdf3.2 = 81 files
netcdf3.5 = 1 files
netcdf4 = 121 files

total # files = 1470
````
We need to get representative samples of recent files for improved testing and code coverage.

### Data Model notes
Expand Down
15 changes: 15 additions & 0 deletions cli/Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Netchdf Command Line Interface
5/24/2025

## Building

cd <your_build_dir>
clone https://github.com/JohnLCaron/netchdf.git
cd netchdf
./gradlew clean assemble
./gradlew :cli:uberJar

## Running ncdump

cd cli/build/libs
java -jar netchdf-uber.jar --filename your_file
31 changes: 31 additions & 0 deletions cli/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
plugins {
alias(libs.plugins.kotlin.jvm)
}

dependencies {
api(project(":core"))

implementation(libs.kotlinx.cli)
implementation(libs.oshai.logging)
implementation(libs.logback.classic)
}

kotlin {
jvmToolchain(21)
}

tasks.register<Jar>("uberJar") {
archiveClassifier = "uber"
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
archiveBaseName.set("netchdf")

manifest {
attributes("Main-Class" to "com.sunya.netchdf.cli.ncdump")
}

from(sourceSets.main.get().output)
dependsOn(configurations.runtimeClasspath)
from({
configurations.runtimeClasspath.get().filter { it.name.endsWith("jar") }.map { zipTree(it) }
})
}
0 ...in/kotlin/com/sunya/netchdf/cli/ncdump.kt → ...in/kotlin/com/sunya/netchdf/cli/ncdump.kt
View file
Open in desktop
File renamed without changes.
5 changes: 3 additions & 2 deletions clibs/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
plugins {
alias(libs.plugins.kotlin.jvm)
id ("java-test-fixtures")
}

dependencies {
api(project(":core"))
implementation(project(":testdata"))
// implementation(project(":testdata"))
implementation(libs.oshai.logging)
implementation(libs.kotlinx.coroutines.core)

// testImplementation(libs.guava)
testImplementation(kotlin("test"))
testImplementation(libs.bundles.jvmtest)
testImplementation(libs.kotest.property)
testImplementation(testFixtures(project(":core")))
}

kotlin {
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/java/com/sunya/netchdf/netcdf4/TestNetcdfClib.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import com.sunya.netchdf.netcdfClib.ffm.netcdf_h;
import org.junit.jupiter.api.Test;
import com.sunya.testdata.TestFilesKt;
import com.sunya.netchdf.testdata.TestFilesKt;

public class TestNetcdfClib {

Expand Down
4 changes: 2 additions & 2 deletions clibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibExtra.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
package com.sunya.netchdf

import com.sunya.cdm.util.Stats
import com.sunya.testdata.NetchdfExtraFiles
import com.sunya.netchdf.testdata.NetchdfExtraFiles
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.testdata.testData
import com.sunya.netchdf.testdata.testData
import org.junit.jupiter.api.AfterAll
import java.util.stream.Stream

Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/NetchdfClibTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import com.sunya.cdm.util.nearlyEquals
import com.sunya.netchdf.hdf4Clib.Hdf4ClibFile
import com.sunya.netchdf.hdf5Clib.Hdf5ClibFile
import com.sunya.netchdf.netcdfClib.NClibFile
import com.sunya.testdata.*
import com.sunya.netchdf.testdata.*
import org.junit.jupiter.api.AfterAll
import org.junit.jupiter.api.BeforeAll
import org.junit.jupiter.api.Disabled
Expand Down
4 changes: 2 additions & 2 deletions clibs/src/test/kotlin/com/sunya/netchdf/hdf4/H4Ccompare.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ import com.sunya.cdm.array.ArrayUByte
import com.sunya.cdm.util.Stats
import com.sunya.netchdf.*
import com.sunya.netchdf.hdf4Clib.Hdf4ClibFile
import com.sunya.testdata.H4Files
import com.sunya.netchdf.testdata.H4Files
import org.junit.jupiter.api.AfterAll
import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.testdata.testData
import com.sunya.netchdf.testdata.testData
import org.junit.jupiter.api.Assertions
import java.util.*
import java.util.stream.Stream
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/hdf5/H5dataTiming.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import com.sunya.netchdf.netcdf3.Netcdf3File
import com.sunya.netchdf.openNetchdfFile
import com.sunya.netchdf.netcdfClib.NClibFile
import org.junit.jupiter.api.Test
import com.sunya.testdata.testData
import com.sunya.netchdf.testdata.testData
import kotlin.system.measureNanoTime
import kotlin.test.assertTrue

Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/hdf5/Hdf5Compare.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import com.sunya.netchdf.compareCdlWithClib
import com.sunya.netchdf.compareDataWithClib
import com.sunya.netchdf.compareSelectedDataWithClib
import com.sunya.netchdf.netcdfClib.NClibFile
import com.sunya.testdata.*
import com.sunya.netchdf.testdata.*
import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/hdf5/Hdf5openTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import com.sunya.netchdf.compareCdlWithClib
import com.sunya.netchdf.compareDataWithClib
import com.sunya.netchdf.readNetchdfData
import com.sunya.netchdf.showNcHeader
import com.sunya.testdata.*
import com.sunya.netchdf.testdata.*

import org.junit.jupiter.api.AfterAll
import org.junit.jupiter.api.BeforeAll
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/hdf5Clib/H5CopenTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import com.sunya.netchdf.compareCdlWithClib
import com.sunya.netchdf.compareDataWithClib
import com.sunya.netchdf.openNetchdfFile
import com.sunya.netchdf.readNetchdfData
import com.sunya.testdata.*
import com.sunya.netchdf.testdata.*

import org.junit.jupiter.api.AfterAll
import org.junit.jupiter.api.BeforeAll
Expand Down
4 changes: 2 additions & 2 deletions clibs/src/test/kotlin/com/sunya/netchdf/netcdf3/N3dataCompare.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ import org.junit.jupiter.api.Test
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.testdata.N3Files
import com.sunya.testdata.testData
import com.sunya.netchdf.testdata.N3Files
import com.sunya.netchdf.testdata.testData
import java.util.*
import java.util.stream.Stream

Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/netcdf3/N3headerCompare.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.netchdf.netcdfClib.NClibFile
import com.sunya.testdata.N3Files
import com.sunya.netchdf.testdata.N3Files
import java.util.*
import java.util.stream.Stream
import kotlin.test.assertEquals
Expand Down
4 changes: 2 additions & 2 deletions clibs/src/test/kotlin/com/sunya/netchdf/netcdf3/N3ncdumpTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import kotlin.test.assertTrue
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.testdata.testData
import com.sunya.testdata.testFilesIn
import com.sunya.netchdf.testdata.testData
import com.sunya.netchdf.testdata.testFilesIn
import java.io.File

// doesnt work because of differences in the value printout.
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/netcdf4/N4strictTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package com.sunya.netchdf.netcdf4
import com.sunya.netchdf.netcdfClib.NClibFile
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test
import com.sunya.testdata.testData
import com.sunya.netchdf.testdata.testData
import kotlin.test.assertEquals

// trying to compare output with ncdump - h; probably get rid of
Expand Down
2 changes: 1 addition & 1 deletion clibs/src/test/kotlin/com/sunya/netchdf/netcdf4/N4versionTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.Arguments
import org.junit.jupiter.params.provider.MethodSource
import com.sunya.netchdf.netcdfClib.NClibFile
import com.sunya.testdata.N4Files
import com.sunya.netchdf.testdata.N4Files
import java.util.stream.Stream
import kotlin.test.Test
import kotlin.test.assertTrue
Expand Down
2 changes: 1 addition & 1 deletion clibs/testStatus.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
5/17/2025
5/23/2025

dir failed passed time
hdf4 6 4256 3 hr 13min
Expand Down
23 changes: 7 additions & 16 deletions core/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
plugins {
alias(libs.plugins.kotlin.jvm)
id ("java-test-fixtures")
}

dependencies {
implementation(libs.oshai.logging)
implementation(libs.kotlinx.coroutines.core)

testImplementation(project(":testdata"))
testFixturesImplementation(libs.bundles.jvmtest)
testFixturesImplementation(libs.kotest.property)
testFixturesImplementation(libs.kotlinx.coroutines.core)
// runTest() for running suspend functions in tests
testFixturesImplementation(libs.kotlinx.coroutines.test)

testImplementation(kotlin("test"))
testImplementation(libs.bundles.jvmtest)
testImplementation(libs.kotest.property)
Expand All @@ -29,18 +35,3 @@ kotlin {
jvmToolchain(21)
}

tasks.register<Jar>("uberJar") {
archiveClassifier = "uber"
duplicatesStrategy = DuplicatesStrategy.EXCLUDE

//manifest {
// attributes("Main-Class" to "com.sunya.netchdf.ncdump")
//}

from(sourceSets.main.get().output)
dependsOn(configurations.runtimeClasspath)
from({
configurations.runtimeClasspath.get().filter { it.name.endsWith("jar") }.map { zipTree(it) }
})
}

4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayDouble.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayFloat.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayInt.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayLong.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayShort.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
4 changes: 2 additions & 2 deletions core/src/test/kotlin/com/sunya/cdm/array/TestArrayString.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ package com.sunya.cdm.array
import com.sunya.cdm.api.*
import com.sunya.cdm.layout.IndexND
import com.sunya.cdm.layout.IndexSpace
import com.sunya.testdata.propTestSlowConfig
import com.sunya.testdata.runTest
import com.sunya.netchdf.testdata.propTestSlowConfig
import com.sunya.netchdf.testdata.runTest
import io.kotest.property.Arb
import io.kotest.property.arbitrary.int
import io.kotest.property.checkAll
Expand Down
Loading