DLL Behavior Trace: Investigating Supported Native x86 and x64 DLLs

What DLL Behavior Trace is

DLL Behavior Trace is an Enterprise analysis workflow for a supported standalone native Windows DLL. It begins with read-only PE inspection and, only after explicit analyst approval, can load the DLL inside a matching-bitness controlled host while Malware INFO records supported runtime evidence.

The workflow is intended to answer a focused question: what did this DLL do during this specific, bounded invocation under the measured collection coverage?

It is not a malware verdict. It is not automatic containment authority. It is not a complete record of everything the DLL could do in another process, configuration, or environment.

Critical safety warning: Execute an unknown DLL only in an authorized disposable isolated analysis VM with a recoverable clean snapshot. The VM must contain no trusted network access, credentials, business shares, connected devices, customer evidence, or sensitive data. DLL Behavior Trace does not make unknown code safe.

Supported and unsupported DLL types

The current controlled workflow accepts a DLL only when preflight identifies all of the following:

• An existing native Windows PE DLL.
• x86 or x64 machine architecture.
• A file between 1 byte and 100 MiB.
• A location on a local fixed disk.
• A selected DLL that is not a reparse point or symbolic-link target.
• A stable SHA-256 identity that can be revalidated before execution.

The workflow rejects:

• Managed .NET DLLs with CLR metadata.
• ARM64 and other unsupported machine architectures.
• Kernel drivers such as .sys files.
• Invalid, truncated, empty, oversized, or non-DLL PE files.
• DLLs selected through a rejected reparse-point boundary.
• Files whose identity changes between inspection and controlled execution.

A .dll extension alone does not identify an execution model. Resource-only DLLs may contain no useful executable behavior. Plugins, codecs, shell extensions, browser modules, COM servers, service DLLs, and side-loaded DLLs may require a particular host or callback contract that a standalone load cannot reproduce.

Start with static inspection

Use Inspect DLL before selecting any execution mode. Static inspection does not load the DLL. It records bounded evidence such as:

• SHA-256, file size, and x86/x64 architecture.
• Standard and delay-load imported modules and APIs.
• Named and ordinal exports.
• Forwarded-export metadata.
• Read-only dependency candidates in checked locations.
• TLS callback addresses, mapped sections, and validation state.
• Partial or truncated inventory diagnostics.

These results are capability and readiness evidence.

An imported API shows that the PE import structures reference that API. It does not prove the DLL called it. An export name is an interface clue, not proof of its function signature, purpose, or intent. A validated TLS callback entry shows that the static callback address maps to the reported PE location; it does not prove maliciousness or complete runtime capture.

Dependency resolution has a similarly narrow meaning. A candidate found in the DLL directory or matching Windows system directory can help the analyst prepare the lab, but it does not guarantee that Windows side-by-side policy, loader behavior, or DLL initialization will succeed.

Understanding the four invocation modes

DLL Behavior Trace supports exactly four invocation modes.

LoadOnly

LoadOnly is the narrowest dynamic option, but it still executes code. The controlled host loads the DLL, which can trigger loader-managed TLS callbacks and DllMain process-attach logic.

This mode can reveal initialization activity such as dependency loading, environment checks, memory operations, thread creation, file or Registry access, and network setup when those actions cross supported observed boundaries. It may miss the main capability if the DLL waits for an export call, configuration file, named event, timer, service message, plugin callback, or user interaction.

Never interpret a short LoadOnly timeline as proof that the DLL is inactive.

Rundll32Export

Rundll32Export calls one exact, case-sensitive named export using the supported Rundll32-compatible callback contract.

Static PE metadata cannot prove that an arbitrary export has that signature. Use this mode only after independent evidence confirms compatibility. Suitable evidence may include a captured command, trusted vendor documentation, an installer or persistence record, or reverse-engineering confirmation of the callback contract.

Do not guess from names such as Start, Run, Init, or Main. Ordinal-only exports are not invoked, forwarded exports are not invoked, and a case-mismatched name is rejected. Calling an export with the wrong convention or parameter contract can crash the host, corrupt state, or trigger an unintended operation.

RegisterServer

RegisterServer requires the exact DllRegisterServer export and invokes the conventional COM registration entry point. Registration code can create or modify CLSID, InprocServer32, TypeLib, shell-extension, browser-helper, persistence, or vendor-specific Registry state. It can also perform other actions chosen by the DLL.

Use this mode only when observing self-registration is necessary to the investigation. Compare Registry and other evidence before and after the operation in the disposable VM.

UnregisterServer

UnregisterServer requires the exact DllUnregisterServer export. It is not a safe undo button. A DLL can remove unrelated values, leave partial state, or execute arbitrary logic. Successful unregistration also does not prove that every registration change was reversed.

Registration and COM activation are separate questions. Neither COM mode proves how a class behaves when a legitimate client later activates it.

Controlled benign fixture walkthrough

The repository includes one purpose-built benign C fixture, BenignDllTraceFixture.c, for controlled validation. The source defines:

• A TLS process-attach callback that writes a TLS_PROCESS_ATTACH marker.
• DllMain process-attach logic that writes a DLL_PROCESS_ATTACH marker.
• A Rundll32-compatible RunFixture export that records its bounded argument.
• DllRegisterServer and DllUnregisterServer exports that record benign markers.
• A forwarded export and an ordinal-only export used to verify fail-closed selection.

The focused integration harness builds the same source as x86 and x64, then uses the corresponding host for each architecture. It verifies that:

1. LoadOnly reaches both the benign TLS callback and DllMain.
2. Rundll32Export passes the expected argument to RunFixture.
3. The COM registration and unregistration entry points are reached.
4. A changed hash, wrong bitness, wrong export-name case, forwarded export, and failing COM result are rejected or reported as failures.

This fixture proves the bounded host invocation contract. It is not malware, does not represent every DLL type, and does not prove installed-service, full API coverage, memory-capture, export-package, or production behavior. The compiled fixture DLL is not included in this article.

What evidence the analyst should watch

Begin with collection health before interpreting behavior.

Session and coverage

Confirm that the effective mode is Deep Trace and that the matching collector became active before the controlled host resumed. Coverage should distinguish:

• Configured: a supported boundary the collector is designed to watch.
• Installed: observation was available for that boundary in this session.
• Observed: the target reached the boundary during the recorded window.
• Unavailable: a known collection gap prevented observation.

Installed without Observed means the monitored boundary was not seen during the measured window. Unavailable means the system cannot make that observation. Neither state proves that the underlying behavior did not occur through another API, a direct system call, custom code, an unsupported boundary, or activity outside the recorded window.

Static and runtime evidence

Keep the evidence classes separate:

• Static evidence: imports, exports, dependencies, TLS metadata, sections, and file identity.
• Observed runtime evidence: supported API boundaries and typed results captured within the verified session identity and coverage window.
• Correlated evidence: Guard, Event Log, or other supporting observations that are explicitly marked as not being exact API-call proof.
• Derived or inferred evidence: behavior sequences, resource relationships, or hypotheses calculated from cited observations.
• Gap or unavailable coverage: evidence the collector could not obtain or validate.

High confidence in collection means the product is confident about the stated observation. It does not convert a capability, inference, or single event into proof of malware or attacker intent.

Timeline, resources, memory, and diagnostics

Review the API timeline and Resource Lineage for supported loader, file, Registry, process, thread, memory, network, and cleanup activity. Check whether results say attempted, failed, successful, or unknown rather than assuming that an API name alone proves an effect.

When an eligible policy-bound memory snapshot exists, treat it as a separate evidence path. Captured bytes prove that those bytes were present when read within the recorded scope. They do not prove which API created, decrypted, protected, or executed them.

Use collector and host diagnostics when rows are missing. A loader failure, export failure, unavailable boundary, dropped row, partial inventory, or early host exit changes what the trace can support.

How to interpret sparse or incomplete results

A quiet result has several possible explanations:

• The DLL performed only minimal initialization.
• Required configuration or companion files were absent.
• The selected export or mode did not match the real contract.
• The important behavior occurred before collector readiness or after the bounded window.
• Execution moved to another process.
• The DLL used unsupported APIs, direct system calls, inlined code, or custom cryptography.
• A dependency or loader condition prevented the expected path.
• Coverage was partial, degraded, or unavailable.
• The DLL depends on an authentic host environment.

Therefore, "few events" means only that few supported observations were retained in that session. It does not mean inactive, clean, benign, or safe.

Host-dependent and side-loaded DLL behavior

Standalone loading is often the wrong model for a plugin, service DLL, or side-loaded component.

A plugin may depend on application-specific interfaces, a working directory, configuration, or a callback supplied by the legitimate process. A service DLL may expect svchost.exe, Service Control Manager configuration, and a ServiceMain contract. A side-loaded DLL may be selected only because a particular executable searches a particular directory while companion files are present.

When host context is part of the evidence, preserve the original isolated case layout and trace the authentic host EXE when authorized and safe. The host process, command line, current directory, companion files, service configuration, and process identity may all affect the behavior.

An isolated standalone trace remains useful for a first load or a confirmed export contract, but it cannot override stronger evidence from the authentic host context.

Evidence export and Final Report to CISO

After a trace reaches a terminal state and final checksum refresh completes, Export Verified ZIP creates a deliberate analyst copy of the evidence. The export process verifies the session identity and checksum inventory, rejects unsafe paths, re-hashes inventory-listed files, applies file and total-size bounds, and commits the destination only after successful verification.

The extracted package can include:

• The original checksum-covered JSON and JSONL evidence.
• sha256.txt and export-report.json.
• A readable offline index.html and per-source report pages.
• Coverage, timeline, behavior, diagnostics, and available memory-artifact references.
• A normalized api-trace-handoff.json for further correlation.

The original structured evidence remains authoritative. HTML is a readable view, not a replacement evidence source.

Final Report to CISO can consume the supported normalized handoff as part of a complete verified evidence package. It preserves source identity, observed versus supporting evidence, coverage gaps, contradictions, and limitations for executive review. Report generation does not confirm malware, infer attacker intent automatically, or authorize quarantine, termination, deletion, blocking, or another response action.

Safety and operational limitations

The matching-bitness host, suspended start, collector-readiness gate, bounded execution window, and kill-on-close Job Object reduce observation and cleanup risk. They are not sandbox containment and do not provide rollback.

A DLL can still:

• Create, modify, delete, or encrypt files.
• Change the Registry, services, persistence, or security settings.
• Create or inject processes and threads.
• Access credentials, case data, connected devices, shares, or networks.
• Crash or hang applications or Windows.
• Detect analysis, exploit a vulnerability, or affect reachable systems.

The Job Object helps terminate the controlled host at the cleanup boundary. It does not reverse file, Registry, service, process, device, credential, or network effects that occurred while the DLL was running.

Do not execute an unknown DLL on an employee workstation, administrator workstation, customer endpoint, server, domain controller, trusted air-gapped environment, or VM connected to business resources. Begin with static inspection, define the unresolved question, and execute only when the expected evidence justifies the risk.

Practical checklist

Before execution:

1. Preserve the original DLL and companion files with path, SHA-256, source, timestamps, signer, and case context.
2. Run read-only static inspection and confirm native x86 or x64 architecture.
3. Review imports, exports, forwarders, dependencies, TLS callbacks, and partial-inventory diagnostics.
4. Decide whether standalone loading matches the real execution model.
5. Select LoadOnly unless independent evidence supports a more specific contract.
6. Confirm the exact case-sensitive export and callback signature before Rundll32 mode.
7. Treat COM registration and unregistration as system-changing operations.
8. Prepare an authorized disposable isolated VM with a clean snapshot, no credentials, no shares, no sensitive data, and no trusted network.
9. Read the complete warning and approve only the single intended invocation.

During and after execution:

1. Confirm collector readiness and the effective coverage window.
2. Separate static, observed, correlated, inferred, and unavailable evidence.
3. Review outcomes, Resource Lineage, memory artifacts, dropped events, and diagnostics.
4. Do not interpret sparse evidence as inactivity or missing evidence as benign behavior.
5. Use the authentic host EXE when plugin, service, side-loading, or companion-file context is essential.
6. Export the verified terminal package and preserve its limitations for analyst and CISO review.
7. Make verdict and response decisions through the authorized investigation workflow, never from a DLL name, export name, API row, or score alone.