Fix build caching: remove generateSecretKeys dependency from test#1028
Fix build caching: remove generateSecretKeys dependency from test#1028jacopoc merged 4 commits intoapache:trunkfrom
Conversation
The generateSecretKeys task writes new random keys to security.properties on every build, causing cascading cache misses for compileTestGroovy, checkstyleMain, checkstyleTest, and test. Use a git hook to remove the test dependency and provide fixed keys instead. Upstream fix: apache/ofbiz-framework#1028
The test task previously depended on generateSecretKeys, which writes new random values to security.properties on every build. Since this file is part of the main resource source set, it causes cascading cache misses for compileTestGroovy, checkstyleMain, checkstyleTest, and test. Instead, provide a test-specific security.properties with fixed keys in framework/security/src/test/resources/. This shadows the main config on the test classpath, so unit tests that need JWT keys (e.g. ModelFormTest) still work. The generateSecretKeys task remains available for manual use and for the loadAll task.
d92eb60 to
4054eb6
Compare
|
Thanks for the quick fix in #1029! It does resolve the issue for local build cache reuse (two consecutive builds on the same machine) — the first build generates keys, the second sees them and skips. However, it doesn't fully address cross-machine build cache reuse. On a fresh checkout the keys are empty, so This PR complements #1029 by removing |
There was a problem hiding this comment.
I like the idea of creating a copy of security.properties that is only used by tests, however it would be nice to keep it minimal (i.e., do not include any property that is not required for testing).
There was a problem hiding this comment.
A minimal security.properties file like the following should be enough for the test task to succeed:
##############################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
###############################################################################
####
# OFBiz Security Settings
####
# -- Security key used to encrypt and decrypt the autogenerated password in forgot password functionality.
# Read Passwords and JWT (JSON Web Tokens) usage documentation to choose the way you want to store this key
# The key must be 512 bits (ie 64 chars) as we use HMAC512 to create the token, cf. OFBIZ-12724
# Run './gradlew generateSecretKeys' to generate a cryptographically secure random key.
login.secret_key_string=ZdxxqjX87e6CR9Rf5pAzA+GTX4DP2T0H3bimGaDGA59OytU/tVQ7AC7N9PUA9e17
# -- The secret key for the JWT token signature.
# Read Passwords and JWT (JSON Web Tokens) usage documentation to choose the way you want to store this key
# The key must be 512 bits (ie 64 chars) as we use HMAC512 to create the token, cf. OFBIZ-12724
# Run './gradlew generateSecretKeys' to generate a cryptographically secure random key.
security.token.key=TzWl/rNavz7VYMc6sxDpf8Yon6NnkbUfrL4JrscJBdTEx3vtwQQ7Mt0wk/TtZXUn
#-- To accept the execution on some groovy script who match the deniedScriptletsTokens regExp, put their hash here.
#-- like allowedScriptletHashes={SHA}59f8ab616b3878ddf825ea50c13ce603a3a6c5a9,{SHA}59f5ab516b3878ddf825ea50c13ce603a3a6c5a9
allowedScriptletHashes= {SHA}4e025676cfa6df142e3457099271ecdcd1c1f5f9,{SHA}d8451d7509ae73421974f47752b6e9eef7503041,{SHA}edf12cf95597d52eacc14020a85a8df2abb34ab7
#-- RegExp to secure groovy script execution. If the regExp match a script, it would be disabled and OFBiz run nothing.
#-- In this case, you will have on log the original script with it hash. The hash can be added on allowedScriptletHashes
#-- properties to accept it on the next execution.
deniedScriptletsTokens=java\\s*\.|import\\s|embed[^\\w]|process[^\\w]|class[^\\w]|require[^\\w]\
|\.\\s*.exec.*[\(|\\s]|\.\\s*calc.*[\(|\\s]|\.\\s*.eval.*[\(|\\s]|Eval\\s*\.|\\s+File\
|System\\s*\.|\.\\s*codehaus|\.\\s*groovy[^:]|\.\\s*runtime\|groovyx\\s*\.
#-- If you want to deactivate the security control on each groovy script set to false.
# Warn ensure to be sure on what you do because this can open the door for code injection
useDeniedScriptletsTokens=true
|
@ribafish Thank you again for your work. I really like the goal of properly supporting cross-machine build cache (and making better use of the Develocity cache, see #1033).
we are not able to run instead
which would be our preferred setup. Frankly speaking, I’m not sure what issues are preventing us from running these tasks... it still needs to be investigated. Hence, ideally, we should make the Any advice would be more than welcome! |
Use exact property set suggested by jacopoc: JWT keys, groovy scriptlet security tokens, and allowed scriptlet hashes.
Summary
The
testtask depends ongenerateSecretKeys, which writes new random values toframework/security/config/security.propertieson every build. Since this file is part of the main resource source set (config/dirs are included viagetDirectoryInActiveComponentsIfExists('config')), changing it causes cascading Gradle build cache misses for::compileTestGroovy— classpath includes main resources:checkstyleMain— classpath includessourceSets.main.output:checkstyleTest— same reason:test— classpath + test classes differThis PR:
dependsOn 'generateSecretKeys'from thetesttask — the mainsecurity.propertiesis no longer mutated duringbuildsecurity.propertiesinframework/security/src/test/resources/with fixed keys — this shadows the main config on the test classpath so unit tests that require JWT keys (e.g.ModelFormTest) continue to workThe
generateSecretKeystask is unchanged and remains available for manual use (./gradlew generateSecretKeys) and as a dependency ofloadAll.