v0.4.0 - Azure Monitor Pipeline & Code Refactoring

🚀 Release Notes

🌟 Major Changes

🛡️ Microsoft XDR Pipeline (formerly Microsoft 365 Defender)

• 🔄 Microsoft 365 Defender pipeline renamed to Microsoft XDR
• ⚠️ Users should migrate to the new Microsoft XDR pipeline

🆕 Azure Monitor Pipeline (NEW!)

• 🧪 New Azure Monitor pipeline introduced (alpha status)
• 🗃️ Supports field mappings for SecurityEvents and SigninLogs tables
• 📊 All 698 Azure Monitor tables supported in final queries

🔍 Enhanced Sentinel ASIM Pipeline (Beta)

• 🔑 Additional field mappings added (beta status)

📈 Expanded Table Support

• Microsoft XDR: 38 tables
• Sentinel ASIM: 8 tables
• Azure Monitor: 698 tables

🏗️ Codebase Refactoring

• 🧱 Improved organization and structure
• 🔄 Better sharing of components across pipelines

✨ New Features

🎛️ Custom Table Name Support

• 🆕 Set custom table names with query_table parameter
• 🐍 Configurable via YAML or Python

🔀 Flexible Rule Category Handling

• 🚫 "Unsupported rule category" error suppressed when the following conditions are met:
  • Rule category is absent or category not in mappings.py for each pippeline
• A valid table is supplied via query_table param

🛠️ Technical Improvements

📜 Table Generation Scripts

• 🤖 New scripts in utils folder
• 🔄 Auto-populate valid tables and field schema in tables.py for each pipeline

🗺️ Field Mappings

• 🔨 Ongoing improvements for all pipelines

📊 Rule-to-Table Mapping

• 🚧 Work in progress on advanced mapping methods

📚 Documentation

• 📝 Updated README with query_table usage
• 💡 New examples for YAML and Python implementations
• FAQ/Troubleshooting section

⚠️ Deprecation Notices

• 🚫 Microsoft 365 Defender pipeline is deprecated
• 🔜 Users should migrate to Microsoft XDR pipeline

🔮 Future Work

• 🔍 Expanding field mappings across pipelines
• 🧠 Developing sophisticated rule-to-table mapping
• 🔧 Refining Azure Monitor and Sentinel ASIM pipelines

📘 Please refer to the updated README for detailed usage instructions and examples of the new features.

v0.3.3 - SentinelASIM Pipeline Bug Fixes

🚀 Release Notes: pySigma Kusto Backend v0.3.2

🐛 Bug Fixes

🛡️ Sentinel ASIM Pipeline

• Resolved issues with the sentinelasim pipeline, improving its stability and reliability.
  • Fixes #25

🧪 Testing Improvements

📊 Sentinel ASIM Pipeline

• Added basic test coverage for the sentinelasim pipeline, enhancing our ability to catch potential issues early.

🔮 Ongoing Development

We're actively working on expanding and refining our pipeline support:

• The sentinelasim pipeline and other pipelines are under active development.
• We're continuously adding more tables and features to these pipelines.
• We're also working on refactoring the codebase to make it more organized and easier to contribute to
• Your patience and feedback during this process are greatly appreciated!

v0.3.1 - Now with more Kusto!

🚀 Release Notes: pySigma Kusto Backend v0.3.1

🏷️ Repository Rename

We've renamed the repository from pySigma-backend-microsoft365defender to pySigma-backend-kusto. This change reflects our expanded focus on supporting the Kusto Query Language (KQL) as a backend for various platforms.

🆕 New Features

📊 Sentinel ASIM Pipeline

• Introduced the sentinel_asim pipeline, providing initial beta support for the Sentinel Advanced Security Information Model (ASIM) tables. (Thanks @adonm!)

🛠️ Backend Enhancements

• Renamed Microsoft365DefenderBackend to KustoBackend for clarity and broader applicability.
• The microsoft_365_defender_pipeline is no longer automatically applied by the backend, allowing for more flexible configurations.

🔧 Microsoft 365 Defender Pipeline Improvements

• Table renaming is now performed within the pipeline rather than the backend.
• Added the ability to override Sigma Rule category -> table name mappings with a custom query_table parameter.

📘 Usage Example

For details on using custom table names, see the README.

🔮 Future Plans

We're actively working on expanding support for additional KQL-based platforms. Stay tuned for updates on Microsoft XDR and other integrations!

🙏 Acknowledgements

Special thanks to the SigmaHQ Discord community for their valuable input on the repository direction.

v0.2.6

What's Changed

• Update microsoft365defender.py by @adonm in #19
  • Ignores the "Initiated" field in network connections as for endpoints almost all events are outbound
• Increased pinned certifi dependency version

New Contributors

• @adonm made their first contribution in #19

Full Changelog: v0.2.4...v0.2.5

v0.2.4: Wildcard escaping fixes

• Fixed issue #13 where '*' character was being escaped incorrectly CommandLine strings
• Fixed issue #14 where Sigma schema wildcards ('*', '?') in the middle of a string would create nonsense queries
  • Since KQL does not use wildcards, anytime a wildcard value is seen inside a string (not at the beginning or end) from the Sigma Rule, we now split it by the wildcard and use a contains for each substring.
  • Example: a CommandLine field with a value of advfirewall firewall add rule name=Dropbox dir=in action=allow "program=?:\Program Files (x86)\Dropbox\Client\Dropbox.exe" enable=yes profile=Any will be converted to (ProcessCommandLine contains "advfirewall firewall add rule name=Dropbox dir=in action=allow \"program=" and ProcessCommandLine contains ":\\Program Files (x86)\\Dropbox\\Client\\Dropbox.exe\" enable=yes profile=Any")

v0.2.3

• Fixed issue #11 by adding more verbose error handling and hash algorithm parsing in 'Hashes' field
• Loosened pySigma pinned version to allow compatibility with pySigma >= 0.11.0

v0.2.2

• Increased supported pySigma version to >= 0.9.0, <= 0.10.6

v0.2.1

• Pinned certifi version to 2023.07.22 to fix CVE-2023-37920

v0.2.0

Support for the ParentImage Field!

• The pipeline and backend now support the argument transform_parent_image (bool, defaults to True)
  • If set, this will map the ParentImage Sysmon field to InitiatingProcessParentFileName M365 field, and extract the parent process name from the full path as the value for all supported rule categories except process_creation.
  • This is because the Microsoft 365 Defender table schema does not have a InitiatingProcessParentFolderPath field where the entire path can be used. Previously, having the ParentImage field in a rule would cause an invalid field error for non-process_creation rules.
  • For usage and more information, see the README

v0.1.3

• Changed backend negation logic to fix #7 issue
• Changed backend parenthesize attribute to True to fix #8 issue
• README and formatting edits