Module Engine Secret
Overview
The engine_secret
module is responsible for orchestrating secret scanning within the DevSecOps Engine Tools platform. It automates the execution of secret scanning tools, processes scan configurations, manages exclusions, and integrates results for further risk analysis and reporting.
Configuration Structure
The module is configured through two main JSON files located in example_remote_config_local/engine_sast/engine_secret/
:
ConfigTool.json
Main configuration file that defines scanning behavior, tool versions, and security thresholds.
{
"IGNORE_SEARCH_PATTERN": "(.*test.*)",
"MESSAGE_INFO_ENGINE_SECRET": "engine_secret run successfully",
"THRESHOLD": {
"VULNERABILITY": {
"Critical": 99,
"High": 99,
"Medium": 99,
"Low": 99
},
"COMPLIANCE": {
"Critical": 0
}
},
"TARGET_BRANCHES": ["trunk", "develop"],
"trufflehog": {
"VERSION": "3.88.31",
"EXCLUDE_PATH": [".git", "_venv"],
"EXCLUDE_DETECTORS": ["aws", "userflow"],
"NUMBER_THREADS": 4,
"FILTER_ENTROPY": 3.0,
"ENABLE_CUSTOM_RULES": false,
"EXTERNAL_DIR_OWNER": "ExternalOrg",
"EXTERNAL_DIR_REPOSITORY": "DevSecOps_Checks",
"APP_ID_GITHUB": "",
"INSTALLATION_ID_GITHUB": "",
"USE_EXTERNAL_CHECKS_GIT": false,
"USE_EXTERNAL_CHECKS_DIR": false,
"RULES": {
"MISCONFIGURATION_SCANNING": {
"References": "https://reference.url/",
"Mitigation": "Make sure you only enable the Spring Boot Actuator endpoints that you really need and restrict access to these endpoints."
}
}
},
"gitleaks": {
"VERSION": "8.21.1",
"EXCLUDE_PATH": [".git"],
"NUMBER_THREADS": 4,
"ALLOW_IGNORE_LEAKS": false,
"ENABLE_CUSTOM_RULES": false,
"EXTERNAL_DIR_OWNER": "ExternalOrg",
"EXTERNAL_DIR_REPOSITORY": "DevSecOps_Checks",
"APP_ID_GITHUB": "",
"INSTALLATION_ID_GITHUB": "",
"USE_EXTERNAL_CHECKS_GIT": false,
"USE_EXTERNAL_CHECKS_DIR": false
}
}
Configuration Parameters
General Configuration
- IGNORE_SEARCH_PATTERN: Regex pattern to exclude files/folders from scanning (e.g.,
"(.*test.*)"
ignores test files) - MESSAGE_INFO_ENGINE_SECRET: Success message displayed when engine completes successfully
- TARGET_BRANCHES: Array of branch names that should be scanned for secrets (e.g.,
["trunk", "develop"]
)
Threshold Configuration
- THRESHOLD.VULNERABILITY: Maximum allowed vulnerabilities by severity level:
Critical
: Maximum 99 critical vulnerabilities allowed (high tolerance for secrets)High
: Maximum 99 high severity vulnerabilitiesMedium
: Maximum 99 medium severity vulnerabilitiesLow
: Maximum 99 low severity vulnerabilities
- THRESHOLD.COMPLIANCE: Maximum allowed compliance issues:
Critical
: Maximum 0 critical compliance issues (zero tolerance)
Trufflehog Tool Configuration
- VERSION: Trufflehog version to use (e.g.,
"3.88.31"
) - EXCLUDE_PATH: Array of paths to exclude from scanning (e.g.,
[".git", "_venv"]
) - EXCLUDE_DETECTORS: Array of detector names to disable (e.g.,
["aws", "userflow"]
) - NUMBER_THREADS: Number of threads for parallel processing (e.g.,
4
) - FILTER_ENTROPY: Minimum entropy threshold for secret detection (e.g.,
3.0
) - ENABLE_CUSTOM_RULES: Boolean flag to enable custom detection rules
- External Rules Configuration:
EXTERNAL_DIR_OWNER
: External organization for custom rulesEXTERNAL_DIR_REPOSITORY
: Repository containing custom rulesAPP_ID_GITHUB
: GitHub App ID for authenticationINSTALLATION_ID_GITHUB
: GitHub App installation IDUSE_EXTERNAL_CHECKS_GIT
: Enable external checks from Git repositoryUSE_EXTERNAL_CHECKS_DIR
: Enable external checks from directory
- RULES: Custom rule definitions with references and mitigation guidance
Gitleaks Tool Configuration
- VERSION: Gitleaks version to use (e.g.,
"8.21.1"
) - EXCLUDE_PATH: Array of paths to exclude from scanning (e.g.,
[".git"]
) - NUMBER_THREADS: Number of threads for parallel processing (e.g.,
4
) - ALLOW_IGNORE_LEAKS: Boolean flag to allow ignoring specific leaks
- ENABLE_CUSTOM_RULES: Boolean flag to enable custom detection rules
- External Rules Configuration: Same structure as Trufflehog for consistency
Exclusions.json
Defines exclusion rules for repositories and specific secret scanning findings.
Structure
{
"All": {
"TRUFFLEHOG": []
},
"Repository_test": {
"TRUFFLEHOG": [
{
"id": "SECRET_SCANNING",
"where": "azure_api/secretos_azure_api.txt",
"create_date": "30042024",
"expired_date": "undefined",
"hu": "12345",
"reason": "false_positive"
},
{
"id": "SECRET_SCANNING",
"where": "keys_test.txt",
"create_date": "30042024",
"expired_date": "undefined",
"hu": "12345",
"reason": "false_positive"
}
]
}
}
Exclusion Types
- All: Global exclusions applied to all repositories
- Repository-specific: Exclusions for specific repositories (e.g.,
"Repository_test"
) - Tool-specific exclusions: Organized by scanning tool:
TRUFFLEHOG
: Exclusions for Trufflehog findingsGITLEAKS
: Exclusions for Gitleaks findings (if needed)
Exclusion Fields
Each exclusion entry contains:
id
: Type of secret scanning finding (e.g.,"SECRET_SCANNING"
)where
: Specific file path where the exclusion appliescreate_date
: Date when exclusion was created (format: DDMMYYYY)expired_date
: Expiration date for the exclusion ("undefined"
for permanent)hu
: Human user identifier for audit trailreason
: Justification for exclusion (e.g.,"false_positive"
,"test_data"
)
Main Responsibilities
- Secret Scanning Orchestration: Executes secret scanning tools (Trufflehog, Gitleaks) on source code and pull requests
- Configuration Management: Loads and processes scan configurations and exclusions from remote repositories
- Pull Request Analysis: Identifies and filters files changed in pull requests for targeted secret scanning
- Exclusions Management: Applies exclusion rules based on configuration and DevSecOps policy with audit trail
- Result Processing: Aggregates and normalizes findings for risk evaluation and reporting
- Threshold Enforcement: Validates findings against configured vulnerability and compliance thresholds
- Branch-specific Scanning: Focuses scanning on specified target branches for efficiency
Key Components
runner_secret_scan.py
: Main entry point for secret scan orchestrationentry_point_tool.py
: Initializes the secret scanning engine and triggers the scan processsecret_scan.py
: Core use case for executing the scan, handling configuration, exclusions, and result aggregation- Adapters: Integrations for secret scanning tools (Trufflehog, Gitleaks) and Git operations
Supported Tools and Features
- Trufflehog: Advanced secret detection with entropy filtering and custom detector exclusions
- Gitleaks: Fast and accurate secret detection with comprehensive rule sets
- Pull Request Scanning: Supports scanning only files changed in pull requests for efficiency
- Configurable Exclusions: Supports exclusion of files/folders and custom ignore patterns with expiration and audit trail
- Thresholds and Policies: Handles custom thresholds and build-breaking policies for different severity levels
- Multi-threading: Parallel processing support for improved scanning performance
- External Rules Integration: Support for custom rules from external repositories via GitHub Apps
Example Usage
The secret scanning engine is typically invoked as part of the overall DevSecOps pipeline, after code changes are detected:
devsecops-engine-tools \
--platform_devops github \
--remote_config_source github \
--remote_config_repo devsecops-config \
--module engine_secret \
--tool trufflehog \
--folder_path path/to/source
Tool-specific Usage
Trufflehog Scanning
devsecops-engine-tools \
--platform_devops azure \
--remote_config_source azure \
--remote_config_repo devsecops-config \
--module engine_secret \
--tool trufflehog \
Gitleaks Scanning
devsecops-engine-tools \
--platform_devops github \
--remote_config_source loca \
--remote_config_repo devsecops-config \
--module engine_secret \
--tool gitleaks \
--folder_path src/
Configuration Guidelines
Adding Exclusions
- Add exclusions to
Exclusions.json
with specific file paths when possible - Include creation date, expiration date, and human user identifier for audit compliance
- Provide clear reason for exclusion (false_positive, test_data, etc.)
- Review and clean expired exclusions regularly
- Use repository-specific exclusions instead of global ones when possible
Tuning Detection Parameters
- Entropy Filtering: Adjust
FILTER_ENTROPY
in Trufflehog configuration to reduce false positives - Detector Exclusions: Add specific detectors to
EXCLUDE_DETECTORS
if they generate excessive false positives - Path Exclusions: Configure
EXCLUDE_PATH
to skip non-relevant directories - Threading: Adjust
NUMBER_THREADS
based on available system resources
Managing Custom Rules
- Enable
ENABLE_CUSTOM_RULES
for advanced detection scenarios - Configure external repository access via GitHub Apps authentication
- Maintain custom rules in dedicated repository for version control
- Document custom rules with references and mitigation guidance
Extensibility
- New secret scanning tools can be added by extending the adapters and use cases
- Custom detection rules can be defined and loaded from external repositories
- Supports integration with various version control and CI/CD platforms
- Exclusion logic can be extended for additional use cases and audit requirements
- Tool-specific configurations can be expanded without code changes
Testing
- Unit tests are provided in the
test/
directory, covering orchestration logic, configuration parsing, and exclusion handling - Integration tests validate tool execution and result processing workflows
- Test data exclusions should be properly documented and maintained