{ "@context": [ "https://www.w3.org/ns/credentials/v2" ], "type": [ "VerifiableCredential", "BlogPostCredential" ], "id": "urn:uuid:d91179d2-a5e2-47a8-9f09-9fce5d72852b", "issuer": "did:webvh:QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr:mjendza.net", "validFrom": "2026-09-09T15:18:31Z", "credentialSubject": { "title": "You ran Maester 200 times. Now what?", "author": "Mateusz Jendza", "body": "![Observe, detect and prevent — operating Maester across tenants](/images/maester-vnext/big-picture.svg)\n\n[Maester](https://maester.dev) is an amazing tool. Point it at an Entra ID tenant. A few minutes later you have one self-contained HTML report. It tells you which of about 280 security controls hold and which do not: CIS, CISA/SCuBA, EIDSCA, ORCA, plus any custom Pester test you write yourself.\n\nRun it once and it is excellent.\n\nRun it every night, across several tenants, and you learn something else. **A report is not the same thing as a practice.** After a few months you have a blob container with hundreds of files, three per run:\n\n- `.html` — easy for a human to read,\n- `.json` — easy to process programmatically,\n- `.md` — easy to attach to a triage record.\n\nThree questions come up in every governance review, and a folder of reports answers none of them:\n\n- `Is this tenant getting better or worse?` Each report is one point in time. To compare two you open two 4 MB HTML files side by side and squint.\n- `Did anyone act on last week's findings?` Maester marks some tests `Investigate`: it could not decide, so a human has to look. That is a to-do list with nowhere to record that the to-do got done.\n- `Which runs are even comparable?` A drift check is eight tests. A baseline sweep is 280. Both land in the same folder, sorted by timestamp, and a \"pass rate over time\" chart that mixes them is noise.\n\nThese are not tooling questions. They are the ordinary questions the owner of a tenant has to answer. So we built an operating layer on top, in three stages: \n- observe the posture over time,\n- detect what changed,\n- prevent it from coming back. \n\nIt does not replace the Maester report — that stays exactly as it is, one click away.\n\n>💡Note: First, how to extend Maester with your own tests. Second, what to build on top of the results.\n\n## Extending Maester: write your own test\n\nMaester runs on [Pester](https://pester.dev), so a custom test is a normal Pester file named `*.Tests.ps1`, placed in a `Custom` folder next to the built-in tests. Here is a real one, checking that no application secret is expired or about to expire:\n\n```powershell\nDescribe \"MJ.CTS\" -Tag \"CTS.1003\", \"MJ\", \"Assessment\" {\n BeforeAll {\n . $PSScriptRoot/Modules/Test-MtApplicationSecretExpiry.ps1\n }\n\n It \"CTS.1003: Application secrets should not be expired or expiring soon\" {\n $result = Test-MtApplicationSecretExpiry\n\n if ($null -ne $result) {\n $result | Should -Be $true -Because \"expired secrets cause application outages.\"\n }\n }\n}\n```\n\nThe `if ($null -ne $result)` guard is not noise. `$null` means *\"this test could not run at all\"*, and asserting on it would turn a missing connection into a failed control.\n\nThe logic lives in a separate function so it can be reused and unit tested. Inside it, three Maester helpers do the work a plain Pester test cannot:\n\n```powershell\n# 1. Do not fail when we are simply not connected.\nif (-not (Test-MtConnection Graph)) {\n Add-MtTestResultDetail -SkippedBecause NotConnectedGraph\n return $null\n}\n\n# 2. Put a readable explanation in the report, as Markdown.\nAdd-MtTestResultDetail -Result $markdownTable `\n -Description \"Application secrets should not expire within 30 days.\"\n\n# 3. Hand the decision to a human when the tool cannot make it.\nAdd-MtTestResultDetail -TestInvestigate `\n -Result \"Two report-only policies overlap; confirm which one applies.\"\n```\n\nThat third one matters later. `-TestInvestigate` is how a test says *\"I found something, but a person has to judge it.\"*\n\nRun your own tests, all of them or one area by tag:\n\n```powershell\nInvoke-Maester -Path ./Custom -DisableTelemetry -OutputFolder ./test-results\nInvoke-Maester -Path ./Custom -Tag \"Applications\" -OutputFolder ./test-results\n```\n\n**Tag on purpose.** Tags are how you select tests later and, as you will see below, how a whole run gets grouped. Give a `Describe` block a tag that names the family of checks, and put a stable id in the `It` name.\n\n**Severity is configuration, not code.** A test can carry a `Severity:High` tag, but `Custom/maester-config.json` can set severity per test id, and the config file wins. So you can lower the severity of a control you have consciously accepted without editing anybody's test.\n\nThat is extension inside Maester. Now the part that happens after the run.\n\n## The loop, not the pipeline\n\nRun on a schedule, upload to Azure Blob Storage (one container per tenant), then review. The third step is the one most setups skip, and it is the one that feeds back into the first: reviewing produces a decision, and a decision that is not written down did not happen.\n\nHere is a single run in the viewer:\n\n![A single Maester run — series badge, pass rate over decided tests, per-framework blocks, and the original report one click away](/images/maester-vnext/single-view.png)\n\nEverything on that screen answers a review question instead of describing a file: which series the run belongs to (*Full baseline*), who ran it, and the result per framework block. \"86%\" is a number nobody can act on. \"The Maester/Entra block is at 0%\" is.\n\nThe pass rate is calculated over decided tests only. Skipped and not-run tests are excluded from the denominator instead of quietly counting as passes. That is the difference between a posture number and a flattering one. The undecided tests do not disappear either — they become the work queue below.\n\n## Observe — is this tenant getting better or worse?\n\n![Observing tenant posture over time](/images/maester-vnext/detect.png)\n\nA single report tells you what is true tonight. Posture is a trend, and a trend needs measurements of the same thing.\n\n### Runs are comparable only if they ran the same tests\n\nA drift check and a baseline sweep are different kinds of run, and the storage container does not know that. Sorted by timestamp they interleave, and a pass-rate chart drawn over the mix is worse than no chart at all.\n\nSo runs are grouped into series: `baseline`, `drift`, `custom`, and whatever else you invent. Every trend is drawn per series.\n\nThe nice part is where the series comes from. Nobody has to tag anything by hand, and no naming convention has to be enforced on whoever schedules the runs. Maester 2.x already writes the resolved Pester configuration into the run JSON:\n\n```json\n\"PesterConfig\": {\n \"Run\": { \"Path\": [\"./Custom\"] },\n \"Filter\": { \"Tag\": [\"DIFF\"] }\n}\n```\n\nThat is a record of 'what was run', which is exactly what we want to group by. Better still, `PesterConfig` sits before `Tests` in the JSON, so the small ranged read the run list already does contains it — no extra download.\n\nSo grouping works on every run already in storage. No change to the runner, no re-upload, no migration, nothing new for an operator to remember. Instead of one line jumping between an 8-test drift run and a 280-test sweep, you get one sparkline per series, and \"we improved this quarter\" becomes a claim you can show instead of assert.\n\nIf you want to override that, or you still have Maester 1.x runs that record no `PesterConfig`, stamp `suite = ` as blob metadata at upload time. Runs already in storage can be placed the same way, without re-running anything.\n\n### One trap worth naming\n\nThe drift tests live inside the `./Custom` folder. So a drift run and a custom run have the **same `-Path`**, and only `-Tag DIFF` separates them.\n\nGet the match order wrong and every drift run quietly merges into \"custom checks\". There is no error anywhere, and the chart still looks fine. A wrong grouping is worse than no grouping.\n\n### One container per tenant\n\nMulti-tenant is the normal case for anyone running this as a service, and the boundary is the storage container: one per tenant, named with the tenant GUID. Access is granted per container. \"Who can see this tenant's posture\" becomes an RBAC question with an auditable answer, not a filter inside an application.\n\n## Detect — what changed, and what is new versus recurring\n\nTrends tell you the direction. Operations needs the delta.\n\n### Drift, as a scheduled question\n\nNext to the nightly baseline there is a **drift check**. Export the directory twice with [EntraExporter](https://github.com/microsoft/EntraExporter), compare the two JSON trees, and fail on any difference.\n\n![Drift detection — did anything move, and did anyone tell us](/images/maester-vnext/drift-detection.png)\n\nThis is also a custom Maester test, and a good example of how far you can push one. It uses Pester's `BeforeDiscovery` to build the test list from the folder structure, so **you define new drift checks by creating folders**, not by writing code:\n\n```\ndrift/Applications//\n baseline.json the older export\n current.json the newer export\n settings.json optional, narrows what counts as a difference\n```\n\nSimplified, to show the shape:\n\n```powershell\nBeforeDiscovery {\n $driftFolders = Get-ChildItem -Path $driftRoot -Directory\n}\n\nDescribe \"MJ.CTS.Governance\" -ForEach $driftFolders {\n It \"MT1060.<_.Name>.4: Drift all values in '<_.Name>' match\" `\n -Tag \"DIFF\", \"MT1060\", \"MT1060.$($_.Name)\" {\n $issues = Compare-MtJsonObject -Baseline $baselineData `\n -Current $currentData `\n -Settings $settingsObject\n $issues.Count | Should -Be 0\n }\n}\n```\n\nThat is assertion `.4` of four. In the real test, `BeforeAll` loads both files and runs the comparison once per folder, and the result is split across `.1` baseline is valid JSON, `.2` current is valid JSON, `.3` no property went missing, `.4` no value changed. `Compare-MtJsonObject` ships with Maester 2.x.\n\n```powershell\nInvoke-Maester -Path ./Custom -Tag DIFF `\n -SkipGraphConnect -DisableTelemetry -OutputFolder ./test-results\n```\n\nIt answers a different question from the baseline sweep. Not \"are we compliant\" but \"did anything move, and did anyone tell us\". As its own series it gets its own trend line: a flat line is a quiet tenant, a spike is a change window you either knew about or did not.\n\nTwo flags in that command are really governance decisions.\n\n> **The comparison needs no permissions.** `-SkipGraphConnect` works because it reads two exported files from disk and never calls Graph. The check that tells you the tenant moved does not itself need standing access to the tenant.\n\n> **A known deviation is documented, not silenced.** Each compared object can carry a `settings.json` that narrows what counts as a difference — the fields you have consciously accepted as varying. That is an exception register next to the evidence, which is a very different thing from commenting out a failing test.\n\n### Investigate is a work queue the tool cannot close\n\nMaester marks a test `Investigate` when it cannot decide: the policy exists but two report-only rules overlap, or an exclusion looks deliberate and somebody should confirm it. It is a real work queue, and it is stateless. Every run regenerates it from scratch, so last week's judgement calls come back looking identical to tonight's new ones.\n\nSo each `Investigate` test becomes a tickable work item:\n\n![The triage board — one item ticked and dated, the rest still pending](/images/maester-vnext/investigated.png)\n\nThat turns \"some tests need a human\" into something a review meeting can run on. One of three looked at, dated and struck through. Two still open, with severity attached so you know which to take first. Next month, \"we reviewed that\" has a record behind it instead of a memory.\n\nThe state lives in a small file next to the run it belongs to. The ticked `MT.1017` above is this entry:\n\n```json\n{\n \"version\": 1,\n \"run\": \"TestResults-2026-05-21-170322\",\n \"tasks\": { \"MT.1017\": { \"status\": \"completed\", \"updatedAt\": \"…\" } }\n}\n```\n\nOnly completed items are stored. No entry means pending, so a run nobody touched and a run with no file at all behave identically. There is no initialisation step, no \"create the task list\" action for someone to forget, and no way for the list to drift out of sync with the tests it describes. The run JSON itself is never modified — the original artifacts stay untouched, which is the whole reason they are worth keeping as evidence.\n\nBe honest about the cost. Recording decisions **makes the tool read-write**, which the original design ruled out. It needs *Storage Blob Data Contributor* instead of *Reader*, and anyone who can reach the app can tick items. If your evidence chain requires an untouchable audit trail, turn triage off and keep the decisions somewhere that has approvals.\n\n## Prevent — closing the loop\n\nMost assessment tooling stops at observe and detect. That is why the same finding comes back quarter after quarter. Prevention is not a feature of a viewer — it is what you do with the output.\n\n**A finding fixed by hand comes back.** The durable fix is to move the control into code and let the audit prove it stayed there: identities and policy provisioned through Terraform with a PR-based approval trail, then continuously audited by the same Maester tests that found the gap. That is the loop described in [Governance Entra ID with Backstage and Maester](https://mjendza.net/post/governance-entra-id-backstage-maester/), and the pipeline shape is in [Entra ID Four Musketeers](https://mjendza.net/post/entra-musketeers/).\n\n**The drift series is the enforcement signal.** Once a control is in code, any change to it outside the pipeline is drift by definition. The interesting metric stops being the pass rate and becomes the number of changes that reached the tenant without a pull request.\n\n**The tooling is in scope too.** An assessment platform holds a complete description of every tenant's weaknesses, so it is a target in its own right. RBAC data-plane roles, no SAS tokens. The Maester HTML is third-party content, so it is always rendered in an iframe with `sandbox=\"allow-scripts\"` and never with `allow-same-origin` next to it — together the two defeat the sandbox. And the run JSON is sensitive: it carries the runner's UPN, the tenant name and the tenant's full security posture, so keep the app behind Entra ID authentication or on `localhost`.\n\n## Where this is today\n\nRun list, run detail, series grouping, comparison, test history and investigation tracking are implemented and in use. The drift check works, but its CI workflow still runs without `-Tag DIFF` and uploads to GitHub artifacts instead of the storage account, so those runs never reach the viewer. Both are one-line pipeline fixes, and a good illustration of the point above: the grouping is only as good as the invocation.\n\n## Summary\n\n- **Maester is extensible by design.** A custom test is a normal Pester file plus a few helpers: `Add-MtTestResultDetail` for readable output, `-SkippedBecause` for an honest skip, `-TestInvestigate` to hand a decision to a human.\n- **Tags and paths are metadata, not decoration.** They select which tests run, and afterwards they are what lets you group the runs.\n- **Use the metadata that is already there.** Maester has been recording its own invocation in every output file since version 2.0. The process change you do not have to roll out is the one that actually happens.\n- **A report is not a workflow.** It tells you what is true, not what changed or what anyone decided about it. That needs somewhere to live, and it does not have to be big. Store it to analyze later.\n- **Prevent, or you will read the same finding next quarter.** Move the control into code with a PR trail, let the same Maester tests prove it stayed there, and count the changes that reached the tenant without a pull request.\n\nLinks and connected resources:\n- Maester: https://maester.dev\n- Pester: https://pester.dev\n- EntraExporter: https://github.com/microsoft/EntraExporter\n- My post about governing Entra ID with Backstage and Maester: https://mjendza.net/post/governance-entra-id-backstage-maester/\n- My post about running Maester, EntraExporter and ZeroTrustAssessment from GitHub Actions: https://mjendza.net/post/entra-musketeers/", "datePublished": "2026-09-09", "url": "/post/maester-200x", "description": "How to extend Maester with your own tests, and what to build on top of the results — observe posture over time, detect what changed, and prevent it from coming back.", "tags": [ "Maester", "Entra-Id", "Security", "Azure", "IAM", "Assessment", "Tools" ] }, "didLog": [ { "versionId": "1-QmSxeBTasSjnf9tNPskJNcg3hxC7oiap1xzcDFzTcspmzQ", "versionTime": "2026-03-12T22:30:56Z", "parameters": { "method": "did:webvh:1.0", "scid": "QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr", "updateKeys": [ "did:key:z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2#z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2" ], "portable": false, "nextKeyHashes": [], "watchers": [], "witness": {}, "deactivated": false }, "state": { "@context": [ "https://www.w3.org/ns/did/v1", "https://w3id.org/security/multikey/v1" ], "id": "did:webvh:QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr:mjendza.net", "controller": "did:webvh:QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr:mjendza.net", "verificationMethod": [ { "type": "Multikey", "publicKeyMultibase": "z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2", "purpose": "assertionMethod", "id": "did:webvh:QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr:mjendza.net#arRc5Er2" } ], "authentication": [], "assertionMethod": [ "did:webvh:QmTVQnV3qGxWzWmnmWJAy1zkYswgbUmE95K5qodmAizVfr:mjendza.net#arRc5Er2" ], "keyAgreement": [], "capabilityDelegation": [], "capabilityInvocation": [] }, "proof": [ { "type": "DataIntegrityProof", "cryptosuite": "eddsa-jcs-2022", "verificationMethod": "did:key:z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2#z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2", "created": "2026-03-12T22:30:56Z", "proofPurpose": "assertionMethod", "proofValue": "z3KyoDrjFzZbrH8Y36NHT5k9X5hBiQAD8nScjiZdZXRfauQQei4E8KKSwmxaWb8ZhEwrYgXVgs6jf5cm4T4XgjuKV" } ] } ], "proof": { "type": "DataIntegrityProof", "cryptosuite": "eddsa-jcs-2022", "verificationMethod": "did:key:z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2#z6MksoqpqENZmzzA4nhCPkfcbWtRHVegGV38Yqu2arRc5Er2", "created": "2026-09-09T15:18:31Z", "proofPurpose": "assertionMethod", "proofValue": "z2Dy76czJMwjByTfGfB6h2wUqFGtXDJ3SGt7Egs45nzPmAaipto9xmNdefUPSHxd7ECB5kjZY4U5DAqo2uFToBi1H" } }