test: pagfault.c File Reference

Main Page | Class Hierarchy | Class List | File List | Class Members | File Members

# pagfault.c File Reference

`#include "mi.h"`

Go to the source code of this file.

## Defines

#define | STATUS_ISSUE_PAGING_IO (0xC0033333)
#define | STATUS_PTE_CHANGED 0x87303000
#define | STATUS_REFAULT 0xC7303001
#define | MI_PROTOTYPE_WSINDEX ((ULONG)-1)
#define | MMMAX_INPAGE_SUPPORT 4

## Functions

VOID | MiHandleBankedSection (IN PVOID VirtualAddress, IN PMMVAD Vad)
NTSTATUS | MiCompleteProtoPteFault (IN BOOLEAN StoreInstruction, IN PVOID FaultingAddress, IN PMMPTE PointerPte, IN PMMPTE PointerProtoPte)
NTSTATUS | MiDispatchFault (IN BOOLEAN StoreInstruction, IN PVOID VirtualAddress, IN PMMPTE PointerPte, IN PMMPTE PointerProtoPte, IN PEPROCESS Process, OUT PLOGICAL ApcNeeded)
NTSTATUS | MiResolveDemandZeroFault (IN PVOID VirtualAddress, IN PMMPTE PointerPte, IN PEPROCESS Process, IN ULONG PrototypePte)
NTSTATUS | MiResolveTransitionFault (IN PVOID FaultingAddress, IN PMMPTE PointerPte, IN PEPROCESS CurrentProcess, IN ULONG PfnLockHeld, OUT PLOGICAL ApcNeeded)
NTSTATUS | MiResolvePageFileFault (IN PVOID FaultingAddress, IN PMMPTE PointerPte, OUT PMMINPAGE_SUPPORT *ReadBlock, IN PEPROCESS Process)
NTSTATUS | MiResolveProtoPteFault (IN BOOLEAN StoreInstruction, IN PVOID FaultingAddress, IN PMMPTE PointerPte, IN PMMPTE PointerProtoPte, OUT PMMINPAGE_SUPPORT *ReadBlock, IN PEPROCESS Process, OUT PLOGICAL ApcNeeded)
NTSTATUS | MiResolveMappedFileFault (IN PVOID FaultingAddress, IN PMMPTE PointerPte, OUT PMMINPAGE_SUPPORT *ReadBlock, IN PEPROCESS Process)
NTSTATUS | MiWaitForInPageComplete (IN PMMPFN Pfn2, IN PMMPTE PointerPte, IN PVOID FaultingAddress, IN PMMPTE PointerPteContents, IN PMMINPAGE_SUPPORT InPageSupport, IN PEPROCESS CurrentProcess)
PMMPTE | MiFindActualFaultingPte (IN PVOID FaultingAddress)
PMMPTE | MiCheckVirtualAddress (IN PVOID VirtualAddress, OUT PULONG ProtectCode)
NTSTATUS FASTCALL | MiCheckPdeForPagedPool (IN PVOID VirtualAddress)
NTSTATUS FASTCALL | MiCheckPdeForSessionSpace (IN PVOID VirtualAddress)
VOID | MiInitializePfn (IN PFN_NUMBER PageFrameIndex, IN PMMPTE PointerPte, IN ULONG ModifiedState)
VOID | MiInitializeReadInProgressPfn (IN PMDL Mdl, IN PMMPTE BasePte, IN PKEVENT Event, IN WSLE_NUMBER WorkingSetIndex)
VOID | MiInitializeTransitionPfn (IN PFN_NUMBER PageFrameIndex, IN PMMPTE PointerPte, IN WSLE_NUMBER WorkingSetIndex)
VOID | MiInitializeCopyOnWritePfn (IN PFN_NUMBER PageFrameIndex, IN PMMPTE PointerPte, IN WSLE_NUMBER WorkingSetIndex, IN PVOID SessionPointer)
BOOLEAN | MmIsAddressValid (IN PVOID VirtualAddress)
VOID | MiInitializePfnForOtherProcess (IN PFN_NUMBER PageFrameIndex, IN PMMPTE PointerPte, IN PFN_NUMBER ContainingPageFrame)
VOID | MiAddValidPageToWorkingSet (IN PVOID VirtualAddress, IN PMMPTE PointerPte, IN PMMPFN Pfn1, IN ULONG WsleMask)
PMMINPAGE_SUPPORT | MiGetInPageSupportBlock (VOID)
VOID | MiFreeInPageSupportBlock (IN PMMINPAGE_SUPPORT Support)
VOID | MiFlushInPageSupportBlock ()
NTSTATUS | MiSessionCopyOnWrite (IN PMM_SESSION_SPACE SessionSpace, IN PVOID FaultingAddress, IN PMMPTE PointerPte)

## Variables

MMPTE | MmSharedUserDataPte
PVOID | MmSpecialPoolStart
PVOID | MmSpecialPoolEnd
MMINPAGE_SUPPORT_LIST | MmInPageSupportList

## Define Documentation

#define MI_PROTOTYPE_WSINDEX ((ULONG)-1)

| Definition at line 43 of file pagfault.c. Referenced by MiInitializeReadInProgressPfn(), MiInitializeTransitionPfn(), MiResolveMappedFileFault(), and MiResolvePageFileFault().

#define MMMAX_INPAGE_SUPPORT 4

| Referenced by MiFlushInPageSupportBlock().

#define STATUS_ISSUE_PAGING_IO (0xC0033333)

| Definition at line 34 of file pagfault.c. Referenced by MiDispatchFault(), MiResolveMappedFileFault(), and MiResolvePageFileFault().

#define STATUS_PTE_CHANGED 0x87303000

| Definition at line 35 of file pagfault.c. Referenced by MiDispatchFault(), and MiWaitForInPageComplete().

#define STATUS_REFAULT 0xC7303001

| Definition at line 36 of file pagfault.c. Referenced by MiDispatchFault(), MiResolveDemandZeroFault(), MiResolveMappedFileFault(), MiResolvePageFileFault(), MiResolveProtoPteFault(), MiResolveTransitionFault(), and MiWaitForInPageComplete().

## Function Documentation

VOID MiAddValidPageToWorkingSet | ( | IN PVOID | VirtualAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PMMPFN | Pfn1,
|  | IN ULONG | WsleMask
| ) |

Definition at line 4056 of file pagfault.c.

References ASSERT, _MMWSLE::e1, FALSE, MI_IS_PAGE_TABLE_ADDRESS, MI_IS_PROCESS_SPACE_ADDRESS, MI_IS_SESSION_ADDRESS, MI_IS_SESSION_PTE, MI_IS_SYSTEM_CACHE_ADDRESS, MI_SET_PTE_IN_WORKING_SET, MiLocateAndReserveWsle(), MiUpdateWsle(), MmSessionSpace, MmSystemCacheWs, MmSystemCacheWsle, MmWsle, PERFINFO_ADDTOWS, PsGetCurrentProcess, _MMWSLENTRY::SameProtectAsProto, _MMWSLE::u1, _EPROCESS::UniqueProcessId, _EPROCESS::Vm, _MM_SESSION_SPACE::Vm, _MMSUPPORT::VmWorkingSetList, _MM_SESSION_SPACE::Wsle, and WSLE_NUMBER.

Referenced by MiAllocatePoolPages(), MiCompleteProtoPteFault(), MiDispatchFault(), MiMakeSpecialPoolPagable(), MiResolveDemandZeroFault(), MiResolveTransitionFault(), and MiSessionCommitPageTables().

    04065                    :
    04066
    04067     This routine adds the specified virtual address into the
    04068     appropriate working set list.
    04069
    04070 Arguments:
    04071
    04072     VirtualAddress - Supplies the address to add to the working set list.
    04073
    04074     PointerPte - Supplies a pointer to the pte that is now valid.
    04075
    04076     Pfn1 - Supplies the PFN database element for the physical page
    04077            mapped by the virtual address.
    04078
    04079     WsleMask - Supplies a mask (protection and flags) to OR into the
    04080                working set list entry.
    04081
    04082 Return Value:
    04083
    04084     None.
    04085
    04086 Environment:
    04087
    04088     Kernel mode, APCs disabled, working set lock.  PFN lock NOT held.
    04089
    04090 --*/
    04091
    04092 {
    04093     WSLE_NUMBER WorkingSetIndex;
    04094     PEPROCESS Process;
    04095     PMMSUPPORT WsInfo;
    04096     PMMWSLE Wsle;
    04097
    04098     ASSERT (MI_IS_PAGE_TABLE_ADDRESS(PointerPte));
    04099     ASSERT (PointerPte->u.Hard.Valid == 1);
    04100
    04101     if (MI_IS_SESSION_ADDRESS (VirtualAddress) || MI_IS_SESSION_PTE (VirtualAddress)) {
    04102         //
    04103         // Current process's session space working set.
    04104         //
    04105
    04106         WsInfo = &MmSessionSpace->Vm;
    04107         Wsle = MmSessionSpace->Wsle;
    04108     }
    04109     else if (MI_IS_PROCESS_SPACE_ADDRESS(VirtualAddress)) {
    04110
    04111         //
    04112         // Per process working set.
    04113         //
    04114
    04115         Process = PsGetCurrentProcess();
    04116         WsInfo = &Process->Vm;
    04117         Wsle = MmWsle;
    04118
    04119         PERFINFO_ADDTOWS(Pfn1, VirtualAddress, Process->UniqueProcessId)
    04120     } else {
    04121
    04122         //
    04123         // System cache working set.
    04124         //
    04125
    04126         WsInfo = &MmSystemCacheWs;
    04127         Wsle = MmSystemCacheWsle;
    04128
    04129         PERFINFO_ADDTOWS(Pfn1, VirtualAddress, (HANDLE) -1);
    04130     }
    04131
    04132     WorkingSetIndex = MiLocateAndReserveWsle (WsInfo);
    04133     MiUpdateWsle (&WorkingSetIndex,
    04134                   VirtualAddress,
    04135                   WsInfo->VmWorkingSetList,
    04136                   Pfn1);
    04137     Wsle[WorkingSetIndex].u1.Long |= WsleMask;
    04138
    04139 #if DBG
    04140     if (MI_IS_SYSTEM_CACHE_ADDRESS(VirtualAddress)) {
    04141         ASSERT (MmSystemCacheWsle[WorkingSetIndex].u1.e1.SameProtectAsProto);
    04142     }
    04143 #endif //DBG
    04144
    04145     MI_SET_PTE_IN_WORKING_SET (PointerPte, WorkingSetIndex);
    04146
    04147     KeFillEntryTb ((PHARDWARE_PTE)PointerPte, VirtualAddress, FALSE);
    04148     return;
    04149 }

NTSTATUS FASTCALL MiCheckPdeForPagedPool | ( | IN PVOID | VirtualAddress | ) |

Definition at line 3125 of file pagfault.c.

References FALSE, MI_IS_KERNEL_PAGE_TABLE_ADDRESS, MI_IS_SESSION_ADDRESS, MI_IS_SESSION_PTE, MI_WRITE_VALID_PTE, MiCheckPdeForSessionSpace(), MiGetPdeAddress, MiGetPteAddress, MiGetVirtualAddressMappedByPte, MiHydra, MMPTE, MmSystemPagePtes, MmSystemRangeStart, NTSTATUS(), PDE_PER_PAGE, TRUE, and _MMPTE::u.

Referenced by MiCheckProtoPtePageState(), MiCloneProcessAddressSpace(), MiDeletePte(), MiDeleteSystemPagableVm(), MiEliminateWorkingSetEntry(), MiHandleForkTransitionPte(), MiInitializeCopyOnWritePfn(), MiInitializePfn(), MiInitializeReadInProgressPfn(), MiInitializeTransitionPfn(), and MmAccessFault().

    03131                    :
    03132
    03133     This function copies the Page Table Entry for the corresponding
    03134     virtual address from the system process's page directory.
    03135
    03136     This allows page table pages to be lazily evaluated for things
    03137     like paged pool and per-session mappings.
    03138
    03139 Arguments:
    03140
    03141     VirtualAddress - Supplies the virtual address in question.
    03142
    03143 Return Value:
    03144
    03145     Either success or access violation.
    03146
    03147 Environment:
    03148
    03149     Kernel mode, DISPATCH level or below.
    03150
    03151 --*/
    03152 {
    03153     PMMPTE PointerPde;
    03154     PMMPTE PointerPte;
    03155     NTSTATUS status;
    03156
    03157     if (MiHydra == TRUE) {
    03158
    03159         if (MI_IS_SESSION_ADDRESS (VirtualAddress) == TRUE) {
    03160
    03161             //
    03162             // Virtual address in the session space range.
    03163             //
    03164
    03165             return MiCheckPdeForSessionSpace (VirtualAddress);
    03166         }
    03167
    03168         if (MI_IS_SESSION_PTE (VirtualAddress) == TRUE) {
    03169
    03170             //
    03171             // PTE for the session space range.
    03172             //
    03173
    03174             return MiCheckPdeForSessionSpace (VirtualAddress);
    03175         }
    03176     }
    03177
    03178     status = STATUS_SUCCESS;
    03179
    03180     if (MI_IS_KERNEL_PAGE_TABLE_ADDRESS(VirtualAddress)) {
    03181
    03182         //
    03183         // PTE for paged pool.
    03184         //
    03185
    03186         PointerPde = MiGetPteAddress (VirtualAddress);
    03187         status = STATUS_WAIT_1;
    03188     } else if (VirtualAddress < MmSystemRangeStart) {
    03189
    03190         return STATUS_ACCESS_VIOLATION;
    03191
    03192     } else {
    03193
    03194         //
    03195         // Virtual address in paged pool range.
    03196         //
    03197
    03198         PointerPde = MiGetPdeAddress (VirtualAddress);
    03199     }
    03200
    03201     //
    03202     // Locate the PDE for this page and make it valid.
    03203     //
    03204
    03205     if (PointerPde->u.Hard.Valid == 0) {
    03206         PointerPte = MiGetVirtualAddressMappedByPte (PointerPde);
    03207 #if !defined (_X86PAE_)
    03208         MI_WRITE_VALID_PTE (PointerPde,
    03209                             MmSystemPagePtes [((ULONG_PTR)PointerPde &
    03210                                ((sizeof(MMPTE) * PDE_PER_PAGE) - 1)) / sizeof(MMPTE)]);
    03211 #else
    03212         MI_WRITE_VALID_PTE (PointerPde,
    03213                             MmSystemPagePtes [((ULONG_PTR)PointerPde &
    03214                                (PD_PER_SYSTEM * (sizeof(MMPTE) * PDE_PER_PAGE) - 1)) / sizeof(MMPTE)]);
    03215 #endif
    03216         KeFillEntryTb ((PHARDWARE_PTE)PointerPde, PointerPte, FALSE);
    03217     }
    03218     return status;
    03219 }

NTSTATUS FASTCALL MiCheckPdeForSessionSpace | ( | IN PVOID | VirtualAddress | ) |

Definition at line 3224 of file pagfault.c.

References ASSERT, DbgPrint, FALSE, Index, MI_IS_SESSION_ADDRESS, MI_IS_SESSION_PTE, MiGetPdeAddress, MiGetPdeSessionIndex, MiGetPteAddress, MiGetVirtualAddressMappedByPte, MiHydra, MmSessionSpace, _MM_SESSION_SPACE::PageTables, TRUE, and _MMPTE::u.

Referenced by MiCheckPdeForPagedPool(), and MmAccessFault().

    03230                    :
    03231
    03232     This function copies the Page Table Entry for the corresponding
    03233     session virtual address from the current session's data structures.
    03234
    03235     This allows page table pages to be lazily evaluated for session mappings.
    03236     The caller must check for the current process having a session space.
    03237
    03238 Arguments:
    03239
    03240     VirtualAddress - Supplies the virtual address in question.
    03241
    03242 Return Value:
    03243
    03244     STATUS_WAIT_1 - The mapping has been made valid, retry the fault.
    03245
    03246     STATUS_SUCCESS - Did not handle the fault, continue further processing.
    03247
    03248     !STATUS_SUCCESS - An access violation has occurred - raise an exception.
    03249
    03250 Environment:
    03251
    03252     Kernel mode, DISPATCH level or below.
    03253
    03254 --*/
    03255
    03256 {
    03257     PMMPTE PointerPde;
    03258     PVOID  SessionVirtualAddress;
    03259     ULONG  Index;
    03260
    03261     //
    03262     // Caller should have checked for this.
    03263     //
    03264
    03265     ASSERT (MiHydra == TRUE);
    03266
    03267     //
    03268     // First check whether the reference was to a page table page which maps
    03269     // session space.  If so, the PDE is retrieved from the session space
    03270     // data structure and made valid.
    03271     //
    03272
    03273     if (MI_IS_SESSION_PTE (VirtualAddress) == TRUE) {
    03274
    03275         //
    03276         // Verify that the current process has a session space.
    03277         //
    03278
    03279         PointerPde = MiGetPdeAddress (MmSessionSpace);
    03280
    03281         if (PointerPde->u.Hard.Valid == 0) {
    03282
    03283 #if DBG
    03284             DbgPrint("MiCheckPdeForSessionSpace: No current session for PTE %p\n",
    03285                 VirtualAddress);
    03286
    03287             DbgBreakPoint();
    03288 #endif
    03289             return STATUS_ACCESS_VIOLATION;
    03290         }
    03291
    03292         SessionVirtualAddress = MiGetVirtualAddressMappedByPte ((PMMPTE) VirtualAddress);
    03293
    03294         PointerPde = MiGetPteAddress (VirtualAddress);
    03295
    03296         if (PointerPde->u.Hard.Valid == 1) {
    03297
    03298             //
    03299             // The PDE is already valid - another thread must have
    03300             // won the race.  Just return.
    03301             //
    03302
    03303             return STATUS_WAIT_1;
    03304         }
    03305
    03306         //
    03307         // Calculate the session space PDE index and load the
    03308         // PDE from the session space table for this session.
    03309         //
    03310
    03311         Index = MiGetPdeSessionIndex (SessionVirtualAddress);
    03312
    03313         PointerPde->u.Long = MmSessionSpace->PageTables[Index].u.Long;
    03314
    03315         if (PointerPde->u.Hard.Valid == 1) {
    03316             KeFillEntryTb ((PHARDWARE_PTE)PointerPde, VirtualAddress, FALSE);
    03317             return STATUS_WAIT_1;
    03318         }
    03319
    03320 #if DBG
    03321         DbgPrint("MiCheckPdeForSessionSpace: No Session PDE for PTE %p, %p\n",
    03322             PointerPde->u.Long, SessionVirtualAddress);
    03323
    03324         DbgBreakPoint();
    03325 #endif
    03326         return STATUS_ACCESS_VIOLATION;
    03327     }
    03328
    03329     if (MI_IS_SESSION_ADDRESS (VirtualAddress) == FALSE) {
    03330
    03331         //
    03332         // Not a session space fault - tell the caller to try other handlers.
    03333         //
    03334
    03335         return STATUS_SUCCESS;
    03336     }
    03337
    03338     //
    03339     // Handle PDE faults for references in the session space.
    03340     // Verify that the current process has a session space.
    03341     //
    03342
    03343     PointerPde = MiGetPdeAddress (MmSessionSpace);
    03344
    03345     if (PointerPde->u.Hard.Valid == 0) {
    03346
    03347 #if DBG
    03348         DbgPrint("MiCheckPdeForSessionSpace: No current session for VA %p\n",
    03349             VirtualAddress);
    03350
    03351         DbgBreakPoint();
    03352 #endif
    03353         return STATUS_ACCESS_VIOLATION;
    03354     }
    03355
    03356     PointerPde = MiGetPdeAddress (VirtualAddress);
    03357
    03358     if (PointerPde->u.Hard.Valid == 0) {
    03359
    03360         //
    03361         // Calculate the session space PDE index and load the
    03362         // PDE from the session space table for this session.
    03363         //
    03364
    03365         Index = MiGetPdeSessionIndex (VirtualAddress);
    03366
    03367         PointerPde->u.Long = MmSessionSpace->PageTables[Index].u.Long;
    03368
    03369         if (PointerPde->u.Hard.Valid == 1) {
    03370
    03371             KeFillEntryTb ((PHARDWARE_PTE)PointerPde,
    03372                            MiGetPteAddress(VirtualAddress),
    03373                            FALSE);
    03374
    03375             return STATUS_WAIT_1;
    03376         }
    03377
    03378 #if DBG
    03379         DbgPrint("MiCheckPdeForSessionSpace: No Session PDE for VA %p, %p\n",
    03380             PointerPde->u.Long, VirtualAddress);
    03381
    03382         DbgBreakPoint();
    03383 #endif
    03384
    03385         return STATUS_ACCESS_VIOLATION;
    03386     }
    03387
    03388     //
    03389     // Tell the caller to continue with other fault handlers.
    03390     //
    03391
    03392     return STATUS_SUCCESS;
    03393 }
    #endif

PMMPTE MiCheckVirtualAddress | ( | IN PVOID | VirtualAddress,
|  | OUT PULONG | ProtectCode
| ) |

Definition at line 2880 of file pagfault.c.

References _IMAGE_ENTRY_IN_SESSION::Address, ASSERT, DISPATCH_LEVEL, _MMVAD::FirstPrototypePte, _MM_SESSION_SPACE::ImageList, KeFlushEntireTb(), KeLowerIrql(), KeRaiseIrql(), _IMAGE_ENTRY_IN_SESSION::LastAddress, MI_GET_PROTECTION_FROM_VAD, MI_IS_PAGE_TABLE_ADDRESS, MI_IS_SESSION_ADDRESS, MI_VA_TO_VPN, MiGetProtoPteAddress, MiGetPteAddress, MiHandleBankedSection(), MiLocateAddress(), MM_EXECUTE_WRITECOPY, MM_LARGE_PAGES, MM_NOACCESS, MM_PAGED_POOL_START, MM_READONLY, MM_READWRITE, MM_SESSION_SPACE_WS_LOCK_ASSERT, MM_UNKNOWN_PROTECTION, MmPagedPoolInfo, MmSessionSpace, MmSharedUserDataPte, NULL, PAGE_ALIGN, PAGE_SHIFT, _IMAGE_ENTRY_IN_SESSION::PrototypePtes, _MMVAD::StartingVpn, TRUE, _MMPTE::u, _SUBSECTION::u, _MMVAD::u, _MMVAD::u2, _MMVAD::u4, and ZeroPte.

Referenced by MiFindActualFaultingPte(), MiWaitForInPageComplete(), and MmAccessFault().

    02887                    :
    02888
    02889     This function examines the virtual address descriptors to see
    02890     if the specified virtual address is contained within any of
    02891     the descriptors.  If a virtual address descriptor is found
    02892     which contains the specified virtual address, a PTE is built
    02893     from information within the virtual address descriptor and
    02894     returned to the caller.
    02895
    02896 Arguments:
    02897
    02898     VirtualAddress - Supplies the virtual address to locate within
    02899                      a virtual address descriptor.
    02900
    02901 Return Value:
    02902
    02903     Returns the PTE which corresponds to the supplied virtual address.
    02904     If no virtual address descriptor is found, a zero pte is returned.
    02905
    02906 Environment:
    02907
    02908     Kernel mode, APCs disabled, working set mutex held.
    02909
    02910 --*/
    02911
    02912 {
    02913     PMMVAD Vad;
    02914     PMMPTE PointerPte;
    02915     PLIST_ENTRY NextEntry;
    02916     PIMAGE_ENTRY_IN_SESSION Image;
    02917
    02918     if (VirtualAddress <= MM_HIGHEST_USER_ADDRESS) {
    02919
    02920 #if defined(MM_SHARED_USER_DATA_VA)
    02921
    02922         if (PAGE_ALIGN(VirtualAddress) == (PVOID) MM_SHARED_USER_DATA_VA) {
    02923
    02924             //
    02925             // This is the page that is double mapped between
    02926             // user mode and kernel mode.  Map in as read only.
    02927             // On MIPS this is hardwired in the TB.
    02928             //
    02929
    02930             *ProtectCode = MM_READONLY;
    02931             return &MmSharedUserDataPte;
    02932         }
    02933
    02934 #endif
    02935
    02936         Vad = MiLocateAddress (VirtualAddress);
    02937         if (Vad == (PMMVAD)NULL) {
    02938
    02939             *ProtectCode = MM_NOACCESS;
    02940             return NULL;
    02941         }
    02942
    02943         //
    02944         // A virtual address descriptor which contains the virtual address
    02945         // has been located.  Build the PTE from the information within
    02946         // the virtual address descriptor.
    02947         //
    02948
    02949 #ifdef LARGE_PAGES
    02950
    02951         if (Vad->u.VadFlags.LargePages == 1) {
    02952
    02953             KIRQL OldIrql;
    02954             PSUBSECTION Subsection;
    02955
    02956             //
    02957             // The first prototype PTE points to the subsection for the
    02958             // large page mapping.
    02959
    02960             Subsection = (PSUBSECTION)Vad->FirstPrototypePte;
    02961
    02962             ASSERT (Subsection->u.SubsectionFlags.LargePages == 1);
    02963
    02964             KeRaiseIrql (DISPATCH_LEVEL, &OldIrql);
    02965             KeFlushEntireTb (TRUE, TRUE);
    02966             KeFillLargeEntryTb ((PHARDWARE_PTE)(Subsection + 1),
    02967                                  VirtualAddress,
    02968                                  Subsection->StartingSector);
    02969
    02970             KeLowerIrql (OldIrql);
    02971             *ProtectCode = MM_LARGE_PAGES;
    02972             return NULL;
    02973         }
    02974 #endif //LARGE_PAGES
    02975
    02976         if (Vad->u.VadFlags.PhysicalMapping == 1) {
    02977
    02978             //
    02979             // This is a banked section.
    02980             //
    02981
    02982             MiHandleBankedSection (VirtualAddress, Vad);
    02983             *ProtectCode = MM_NOACCESS;
    02984             return NULL;
    02985         }
    02986
    02987         if (Vad->u.VadFlags.PrivateMemory == 1) {
    02988
    02989             //
    02990             // This is a private region of memory.  Check to make
    02991             // sure the virtual address has been committed.  Note that
    02992             // addresses are dense from the bottom up.
    02993             //
    02994
    02995             if (Vad->u.VadFlags.UserPhysicalPages == 1) {
    02996
    02997                 //
    02998                 // These mappings only fault if the access is bad.
    02999                 //
    03000
    03001                 ASSERT (MiGetPteAddress(VirtualAddress)->u.Long == ZeroPte.u.Long);
    03002                 *ProtectCode = MM_NOACCESS;
    03003                 return NULL;
    03004             }
    03005
    03006             if (Vad->u.VadFlags.MemCommit == 1) {
    03007                 *ProtectCode = MI_GET_PROTECTION_FROM_VAD(Vad);
    03008                 return NULL;
    03009             }
    03010
    03011             //
    03012             // The address is reserved but not committed.
    03013             //
    03014
    03015             *ProtectCode = MM_NOACCESS;
    03016             return NULL;
    03017
    03018         } else {
    03019
    03020             //
    03021             // This virtual address descriptor refers to a
    03022             // section, calculate the address of the prototype PTE
    03023             // and construct a pointer to the PTE.
    03024             //
    03025             //*******************************************************
    03026             //*******************************************************
    03027             // well here's an interesting problem, how do we know
    03028             // how to set the attributes on the PTE we are creating
    03029             // when we can't look at the prototype PTE without
    03030             // potentially incurring a page fault.  In this case
    03031             // PteTemplate would be zero.
    03032             //*******************************************************
    03033             //*******************************************************
    03034             //
    03035
    03036             if (Vad->u.VadFlags.ImageMap == 1) {
    03037
    03038                 //
    03039                 // PTE and proto PTEs have the same protection for images.
    03040                 //
    03041
    03042                 *ProtectCode = MM_UNKNOWN_PROTECTION;
    03043             } else {
    03044                 *ProtectCode = MI_GET_PROTECTION_FROM_VAD(Vad);
    03045             }
    03046             PointerPte = (PMMPTE)MiGetProtoPteAddress(Vad,
    03047                                                 MI_VA_TO_VPN (VirtualAddress));
    03048             if (PointerPte == NULL) {
    03049                 *ProtectCode = MM_NOACCESS;
    03050             }
    03051             if (Vad->u2.VadFlags2.ExtendableFile) {
    03052
    03053                 //
    03054                 // Make sure the data has been committed.
    03055                 //
    03056
    03057                 if ((MI_VA_TO_VPN (VirtualAddress) - Vad->StartingVpn) >
    03058                     (ULONG_PTR)((Vad->u4.ExtendedInfo->CommittedSize - 1)
    03059                                                  >> PAGE_SHIFT)) {
    03060                     *ProtectCode = MM_NOACCESS;
    03061                 }
    03062             }
    03063             return PointerPte;
    03064         }
    03065
    03066     } else if (MI_IS_PAGE_TABLE_ADDRESS(VirtualAddress)) {
    03067
    03068         //
    03069         // The virtual address is within the space occupied by PDEs,
    03070         // make the PDE valid.
    03071         //
    03072
    03073         if (((PMMPTE)VirtualAddress >= MiGetPteAddress (MM_PAGED_POOL_START)) &&
    03074             ((PMMPTE)VirtualAddress <= MmPagedPoolInfo.LastPteForPagedPool)) {
    03075
    03076             *ProtectCode = MM_NOACCESS;
    03077             return NULL;
    03078         }
    03079
    03080         *ProtectCode = MM_READWRITE;
    03081         return NULL;
    03082     }
    03083     else if (MI_IS_SESSION_ADDRESS (VirtualAddress) == TRUE) {
    03084
    03085         //
    03086         // See if the session space address is copy on write.
    03087         //
    03088
    03089         MM_SESSION_SPACE_WS_LOCK_ASSERT ();
    03090
    03091         PointerPte = NULL;
    03092         *ProtectCode = MM_NOACCESS;
    03093
    03094         NextEntry = MmSessionSpace->ImageList.Flink;
    03095
    03096         while (NextEntry != &MmSessionSpace->ImageList) {
    03097
    03098             Image = CONTAINING_RECORD(NextEntry, IMAGE_ENTRY_IN_SESSION, Link);
    03099
    03100             if ((VirtualAddress >= Image->Address) && (VirtualAddress <= Image->LastAddress)) {
    03101                 PointerPte = Image->PrototypePtes +
    03102                     (((PCHAR)VirtualAddress - (PCHAR)Image->Address) >> PAGE_SHIFT);
    03103                 *ProtectCode = MM_EXECUTE_WRITECOPY;
    03104                 break;
    03105             }
    03106
    03107             NextEntry = NextEntry->Flink;
    03108         }
    03109
    03110         return PointerPte;
    03111     }
    03112
    03113     //
    03114     // Address is in system space.
    03115     //
    03116
    03117     *ProtectCode = MM_NOACCESS;
    03118     return NULL;
    03119 }

NTSTATUS MiCompleteProtoPteFault | ( | IN BOOLEAN | StoreInstruction,
|  | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PMMPTE | PointerProtoPte
| ) |

Definition at line 1762 of file pagfault.c.

References APC_LEVEL, ASSERT, _SUBSECTION::ControlArea, DbgPrint, FALSE, _FILE_OBJECT::FileName, _CONTROL_AREA::FilePointer, MI_GET_PAGE_FRAME_FROM_PTE, MI_GET_PROTECTION_FROM_SOFT_PTE, MI_IS_SESSION_IMAGE_ADDRESS, MI_MAKE_VALID_PTE, MI_PFN_ELEMENT, MI_PTE_LOOKUP_NEEDED, MI_SET_PTE_DIRTY, MI_STARTING_OFFSET, MI_WRITE_VALID_PTE, MiAddValidPageToWorkingSet(), MiGetPteAddress, MiGetSubsectionAddress, MiGetVirtualAddressMappedByPte, MiHydra, MiReleasePageFileSpace(), MM_PFN_LOCK_ASSERT, MM_PROTECTION_WRITE_MASK, NULL, _MMPFN::OriginalPte, PERFINFO_SOFTFAULT, PsGetCurrentProcess, PsGetCurrentThread, TRUE, _CONTROL_AREA::u, _MMPTE::u, _MMPFN::u1, _MMWSLE::u1, _MMPFN::u2, _MMPFN::u3, and UNLOCK_PFN.

Referenced by MiDispatchFault(), and MiResolveProtoPteFault().

    01771                    :
    01772
    01773     This routine completes a prototype PTE fault.  It is invoked
    01774     after a read operation has completed bringing the data into
    01775     memory.
    01776
    01777 Arguments:
    01778
    01779     StoreInstruction - Supplies TRUE if the instruction is trying
    01780                        to modify the faulting address (i.e. write
    01781                        access required).
    01782
    01783     FaultingAddress - Supplies the faulting address.
    01784
    01785     PointerPte - Supplies the PTE for the faulting address.
    01786
    01787     PointerProtoPte - Supplies a pointer to the prototype PTE to fault in,
    01788                       NULL if no prototype PTE exists.
    01789
    01790 Return Value:
    01791
    01792     status.
    01793
    01794 Environment:
    01795
    01796     Kernel mode, PFN lock held.
    01797
    01798 --*/
    01799 {
    01800     MMPTE TempPte;
    01801     MMWSLE ProtoProtect;
    01802     PFN_NUMBER PageFrameIndex;
    01803     PMMPFN Pfn1;
    01804     PMMPFN Pfn2;
    01805     PMMPTE ContainingPageTablePointer;
    01806 #if defined(_PREFETCH_)
    01807     PFILE_OBJECT FileObject;
    01808     LONGLONG FileOffset;
    01809     PSUBSECTION Subsection;
    01810 #endif
    01811
    01812     MM_PFN_LOCK_ASSERT();
    01813
    01814     PageFrameIndex = MI_GET_PAGE_FRAME_FROM_PTE (PointerProtoPte);
    01815     Pfn1 = MI_PFN_ELEMENT(PageFrameIndex);
    01816     Pfn1->u3.e1.PrototypePte = 1;
    01817
    01818 #if defined(_PREFETCH_)
    01819
    01820     //
    01821     // Capture prefetch fault information.
    01822     //
    01823
    01824     FileObject = NULL;
    01825
    01826     if (CCPF_IS_PREFETCHER_ACTIVE()) {
    01827
    01828         if (FaultingAddress < MM_HIGHEST_USER_ADDRESS) {
    01829
    01830             TempPte = Pfn1->OriginalPte;
    01831
    01832             if (TempPte.u.Soft.Prototype == 1) {
    01833
    01834                 Subsection = MiGetSubsectionAddress (&TempPte);
    01835
    01836                 if (Subsection->ControlArea->u.Flags.Image) {
    01837
    01838                     FileObject = Subsection->ControlArea->FilePointer;
    01839                     ASSERT (FileObject->FileName.Length > 0);
    01840
    01841                     FileOffset = MI_STARTING_OFFSET (Subsection,
    01842                                                      PointerProtoPte);
    01843                 }
    01844             }
    01845         }
    01846     }
    01847
    01848 #endif
    01849
    01850     //
    01851     // Prototype PTE is now valid, make the PTE valid.
    01852     //
    01853
    01854     ASSERT (PointerProtoPte->u.Hard.Valid == 1);
    01855
    01856     //
    01857     // A PTE just went from not present, not transition to
    01858     // present.  The share count and valid count must be
    01859     // updated in the page table page which contains this
    01860     // PTE.
    01861     //
    01862
    01863     ContainingPageTablePointer = MiGetPteAddress(PointerPte);
    01864     Pfn2 = MI_PFN_ELEMENT(ContainingPageTablePointer->u.Hard.PageFrameNumber);
    01865     Pfn2->u2.ShareCount += 1;
    01866
    01867     ProtoProtect.u1.Long = 0;
    01868     if (PointerPte->u.Soft.PageFileHigh == MI_PTE_LOOKUP_NEEDED) {
    01869
    01870         //
    01871         // The protection code for the prototype PTE comes from this
    01872         // PTE.
    01873         //
    01874
    01875         ProtoProtect.u1.e1.Protection = MI_GET_PROTECTION_FROM_SOFT_PTE(PointerPte);
    01876
    01877     } else {
    01878
    01879         //
    01880         // Take the protection from the prototype PTE.
    01881         //
    01882
    01883         ProtoProtect.u1.e1.Protection = MI_GET_PROTECTION_FROM_SOFT_PTE(&Pfn1->OriginalPte);
    01884         ProtoProtect.u1.e1.SameProtectAsProto = 1;
    01885
    01886         if (StoreInstruction == TRUE && (ProtoProtect.u1.e1.Protection & MM_PROTECTION_WRITE_MASK) == 0) {
    01887
    01888             //
    01889             // This is the errant case where the user is trying to write
    01890             // to a readonly subsection in the image.  Since we're more than
    01891             // halfway through the fault, take the easy way to clean this up -
    01892             // treat the access as a read for the rest of this trip through
    01893             // the fault.  We'll then immediately refault when the instruction
    01894             // is rerun (because it's really a write), and then we'll notice
    01895             // that the user's PTE is not copy-on-write (or even writable!)
    01896             // and return a clean access violation.
    01897             //
    01898
    01899 #if DBG
    01900             DbgPrint("MM: user tried to write to a readonly subsection in the image! %p %p %p\n",
    01901                 FaultingAddress,
    01902                 PointerPte,
    01903                 PointerProtoPte);
    01904 #endif
    01905             StoreInstruction = FALSE;
    01906         }
    01907     }
    01908
    01909     MI_MAKE_VALID_PTE (TempPte,
    01910                        PageFrameIndex,
    01911                        ProtoProtect.u1.e1.Protection,
    01912                        PointerPte);
    01913
    01914     //
    01915     // If this is a store instruction and the page is not copy on
    01916     // write, then set the modified bit in the PFN database and
    01917     // the dirty bit in the PTE.  The PTE is not set dirty even
    01918     // if the modified bit is set so writes to the page can be
    01919     // tracked for FlushVirtualMemory.
    01920     //
    01921
    01922     if ((StoreInstruction) && (TempPte.u.Hard.CopyOnWrite == 0)) {
    01923
    01924 #if DBG
    01925         if (MiHydra == TRUE) {
    01926
    01927             PVOID Va;
    01928
    01929             Va = MiGetVirtualAddressMappedByPte (PointerPte);
    01930
    01931             //
    01932             // Session space backed by the filesystem should not be writable.
    01933             //
    01934
    01935             ASSERT (!MI_IS_SESSION_IMAGE_ADDRESS (Va));
    01936         }
    01937 #endif
    01938
    01939         Pfn1->u3.e1.Modified = 1;
    01940         MI_SET_PTE_DIRTY (TempPte);
    01941         if ((Pfn1->OriginalPte.u.Soft.Prototype == 0) &&
    01942                       (Pfn1->u3.e1.WriteInProgress == 0)) {
    01943              MiReleasePageFileSpace (Pfn1->OriginalPte);
    01944              Pfn1->OriginalPte.u.Soft.PageFileHigh = 0;
    01945         }
    01946     }
    01947
    01948     MI_WRITE_VALID_PTE (PointerPte, TempPte);
    01949
    01950     if (Pfn1->u1.Event == NULL) {
    01951         Pfn1->u1.Event = (PVOID)PsGetCurrentThread();
    01952     }
    01953
    01954     UNLOCK_PFN (APC_LEVEL);
    01955
    01956     PERFINFO_SOFTFAULT(Pfn1, FaultingAddress, PERFINFO_LOG_TYPE_PROTOPTEFAULT);
    01957
    01958     MiAddValidPageToWorkingSet (FaultingAddress,
    01959                                 PointerPte,
    01960                                 Pfn1,
    01961                                 (ULONG) ProtoProtect.u1.Long);
    01962
    01963 #if defined(_PREFETCH_)
    01964
    01965     //
    01966     // Log prefetch fault information now that thw PFN lock has been released
    01967     // and the PTE has been made valid.  This minimizes PFN lock contention,
    01968     // allows CcPfLogPageFault to allocate (and fault on) pool, and allows other
    01969     // threads in this process to execute without faulting on this address.
    01970     //
    01971     // Note that the process' working set mutex is still held so any other
    01972     // faults or operations on user addresses by other threads in this process
    01973     // will block for the duration of this call.
    01974     //
    01975
    01976     if (FileObject != NULL) {
    01977         CcPfLogPageFault (FileObject, FileOffset, PsGetCurrentProcess());
    01978     }
    01979
    01980 #endif
    01981
    01982     ASSERT (PointerPte == MiGetPteAddress(FaultingAddress));
    01983
    01984     return STATUS_SUCCESS;
    01985 }

NTSTATUS MiDispatchFault | ( | IN BOOLEAN | StoreInstruction,
|  | IN PVOID | VirtualAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PMMPTE | PointerProtoPte,
|  | IN PEPROCESS | Process,
|  | OUT PLOGICAL | ApcNeeded
| ) |

Definition at line 68 of file pagfault.c.

References ActiveAndValid, APC_LEVEL, _ETHREAD::ApcNeeded, ASSERT, _MMINPAGE_SUPPORT::BasePte, _MDL::ByteCount, DbgPrint, _MMINPAGE_SUPPORT::Event, FALSE, _FILE_OBJECT::FileName, _MMINPAGE_SUPPORT::FilePointer, FreePageList, HYDRA_PROCESS, IoPageRead(), _MMINPAGE_SUPPORT::IoStatus, KeDelayExecutionThread(), KeEnterCriticalRegion, KeLeaveCriticalRegion, KernelMode, KeSetEvent(), LOCK_PFN, LOCK_SESSION_SPACE_WS, LOCK_SYSTEM_WS, _MMINPAGE_SUPPORT::Mdl, MI_IS_PHYSICAL_ADDRESS, MI_IS_SESSION_ADDRESS, MI_MAGIC_AWE_PTEFRAME, MI_MAKE_TRANSITION_PTE_VALID, MI_PFN_ELEMENT, MI_REMOVE_LOCKED_PAGE_CHARGE, MI_SET_PTE_DIRTY, MI_WRITE_VALID_PTE, MiAddValidPageToWorkingSet(), MiCompleteProtoPteFault(), MiDecrementReferenceCount(), MiFlushInPageSupportBlock(), MiGetPdeAddress, MiGetPteAddress, MiHydra, MiInsertPageInList(), MiResolveDemandZeroFault(), MiResolvePageFileFault(), MiResolveProtoPteFault(), MiResolveTransitionFault(), MiRestoreTransitionPte(), MiUnlinkPageFromList(), MiWaitForInPageComplete(), MM_DBG_PAGEFAULT, MmHardFaultNotifyRoutine, MmIsRetryIoStatus, MmPageLocationList, MmShortTime, _ETHREAD::NestedFaultCount, NT_SUCCESS, NTSTATUS(), NULL, _MMINPAGE_SUPPORT::Page, PAGE_SIZE, PERFINFO_DISPATCHFAULT_DECL, PERFINFO_HARDFAULT, PERFINFO_HARDFAULT_INFO, PERFINFO_HARDFAULT_IOTIME, _MMINPAGE_SUPPORT::Pfn, PHARD_FAULT_NOTIFY_ROUTINE, PsGetCurrentThread, _MMPFN::PteFrame, _MMINPAGE_SUPPORT::ReadOffset, StandbyPageList, STATUS_ISSUE_PAGING_IO, STATUS_PTE_CHANGED, STATUS_REFAULT, TRUE, _MMPTE::u, _MMPFN::u1, _MMPFN::u2, _MMPFN::u3, UNLOCK_PFN, UNLOCK_SESSION_SPACE_WS, UNLOCK_SYSTEM_WS, and UNLOCK_WS.

Referenced by MmAccessFault().

    00079                    :
    00080
    00081     This routine dispatches a page fault to the appropriate
    00082     routine to complete the fault.
    00083
    00084 Arguments:
    00085
    00086     StoreInstruction - Supplies TRUE if the instruction is trying
    00087                        to modify the faulting address (i.e. write
    00088                        access required).
    00089
    00090     VirtualAddress - Supplies the faulting address.
    00091
    00092     PointerPte - Supplies the PTE for the faulting address.
    00093
    00094     PointerProtoPte - Supplies a pointer to the prototype PTE to fault in,
    00095                       NULL if no prototype PTE exists.
    00096
    00097     Process - Supplies a pointer to the process object.  If this
    00098               parameter is NULL, then the fault is for system
    00099               space and the process's working set lock is not held.
    00100               If this parameter is HYDRA_PROCESS, then the fault is for session
    00101               space and the process's working set lock is not held - rather
    00102               the session space's working set lock is held.
    00103
    00104     ApcNeeded - Supplies a pointer to a location set to TRUE if an I/O
    00105                 completion APC is needed to complete partial IRPs that
    00106                 collided.
    00107
    00108                 It is the caller's responsibility to initialize this (usually
    00109                 to FALSE) on entry.  However, since this routine may be called
    00110                 multiple times for a single fault (for the page directory,
    00111                 page table and the page itself), it is possible for it to
    00112                 occasionally be TRUE on entry.
    00113
    00114                 If it is FALSE on exit, no completion APC is needed.
    00115
    00116 Return Value:
    00117
    00118     status.
    00119
    00120 Environment:
    00121
    00122     Kernel mode, working set lock held.
    00123
    00124 --*/
    00125
    00126 {
    00127     MMPTE TempPte;
    00128     NTSTATUS status;
    00129     PMMINPAGE_SUPPORT ReadBlock;
    00130     MMPTE SavedPte;
    00131     PMMINPAGE_SUPPORT CapturedEvent;
    00132     KIRQL OldIrql;
    00133     PPFN_NUMBER Page;
    00134     PFN_NUMBER PageFrameIndex;
    00135     LONG NumberOfBytes;
    00136     PMMPTE CheckPte;
    00137     PMMPTE ReadPte;
    00138     PMMPFN PfnClusterPage;
    00139     PMMPFN Pfn1;
    00140     PHARD_FAULT_NOTIFY_ROUTINE NotifyRoutine;
    00141     KIRQL PreviousIrql;
    00142     LOGICAL WsLockChanged;
    00143     PETHREAD CurrentThread;
    00144
    00145     PERFINFO_DISPATCHFAULT_DECL();
    00146
    00147     WsLockChanged = FALSE;
    00148
    00149 ProtoPteNotResident:
    00150
    00151     if (PointerProtoPte != NULL) {
    00152
    00153         //
    00154         // Acquire the PFN lock to synchronize access to prototype PTEs.
    00155         // This is required as the working set lock will not prevent
    00156         // multiple processes from operating on the same prototype PTE.
    00157         //
    00158
    00159         LOCK_PFN (OldIrql);
    00160
    00161         //
    00162         // Make sure the protoptes are in memory.  For
    00163         // user mode faults, this should already be the case.
    00164         //
    00165
    00166         if (!MI_IS_PHYSICAL_ADDRESS(PointerProtoPte)) {
    00167             CheckPte = MiGetPteAddress (PointerProtoPte);
    00168
    00169             if (CheckPte->u.Hard.Valid == 0) {
    00170
    00171                 ASSERT (Process == NULL || (MiHydra == TRUE && Process == HYDRA_PROCESS));
    00172
    00173                 //
    00174                 // The page that contains the prototype PTE is not in memory.
    00175                 //
    00176
    00177                 VirtualAddress = PointerProtoPte;
    00178                 PointerPte = CheckPte;
    00179                 PointerProtoPte = NULL;
    00180                 UNLOCK_PFN (OldIrql);
    00181
    00182                 if (Process == HYDRA_PROCESS) {
    00183                     //
    00184                     // We were called while holding this session space's
    00185                     // working set lock.  But we need to fault in a
    00186                     // prototype PTE which is in system paged pool. This
    00187                     // must be done under the system working set lock.
    00188                     //
    00189                     // So we release the session space WSL lock and get
    00190                     // the system working set lock.  When done
    00191                     // we return STATUS_MORE_PROCESSING_REQUIRED
    00192                     // so our caller will call us again to handle the
    00193                     // actual prototype PTE fault.
    00194                     //
    00195
    00196                     ASSERT (MiHydra == TRUE);
    00197                     UNLOCK_SESSION_SPACE_WS (APC_LEVEL);
    00198
    00199                     //
    00200                     // Lock the system working set for paged pool
    00201                     //
    00202
    00203                     LOCK_SYSTEM_WS (PreviousIrql);
    00204
    00205                     //
    00206                     // System working set is locked so set Process to show it.
    00207                     //
    00208
    00209                     Process = NULL;
    00210
    00211                     WsLockChanged = TRUE;
    00212
    00213                     ASSERT (MI_IS_SESSION_ADDRESS (VirtualAddress) == FALSE);
    00214                 }
    00215                 else {
    00216                     ASSERT (Process == NULL);
    00217                 }
    00218
    00219                 goto ProtoPteNotResident;
    00220             }
    00221         }
    00222
    00223         if (PointerPte->u.Hard.Valid == 1) {
    00224
    00225             //
    00226             // PTE was already made valid by the cache manager support
    00227             // routines.
    00228             //
    00229
    00230             UNLOCK_PFN (OldIrql);
    00231
    00232             if (WsLockChanged == TRUE) {
    00233                 UNLOCK_SYSTEM_WS (APC_LEVEL);
    00234                 LOCK_SESSION_SPACE_WS (PreviousIrql);
    00235             }
    00236
    00237             return STATUS_SUCCESS;
    00238         }
    00239
    00240         ReadPte = PointerProtoPte;
    00241
    00242         PERFINFO_HARDFAULT_INFO(PointerProtoPte);
    00243
    00244         status = MiResolveProtoPteFault (StoreInstruction,
    00245                                          VirtualAddress,
    00246                                          PointerPte,
    00247                                          PointerProtoPte,
    00248                                          &ReadBlock,
    00249                                          Process,
    00250                                          ApcNeeded);
    00251         //
    00252         // Returns with PFN lock released.
    00253         //
    00254
    00255         ASSERT (KeGetCurrentIrql() == APC_LEVEL);
    00256
    00257     } else {
    00258
    00259         TempPte = *PointerPte;
    00260         ASSERT (TempPte.u.Long != 0);
    00261
    00262         if (TempPte.u.Soft.Transition != 0) {
    00263
    00264             //
    00265             // This is a transition page.
    00266             //
    00267
    00268             status = MiResolveTransitionFault (VirtualAddress,
    00269                                                PointerPte,
    00270                                                Process,
    00271                                                FALSE,
    00272                                                ApcNeeded);
    00273
    00274         } else if (TempPte.u.Soft.PageFileHigh == 0) {
    00275
    00276             //
    00277             // Demand zero fault.
    00278             //
    00279
    00280             status = MiResolveDemandZeroFault (VirtualAddress,
    00281                                                PointerPte,
    00282                                                Process,
    00283                                                FALSE);
    00284         } else {
    00285
    00286             //
    00287             // Page resides in paging file.
    00288             //
    00289
    00290             ReadPte = PointerPte;
    00291             LOCK_PFN (OldIrql);
    00292             status = MiResolvePageFileFault (VirtualAddress,
    00293                                              PointerPte,
    00294                                              &ReadBlock,
    00295                                              Process);
    00296         }
    00297     }
    00298
    00299     ASSERT (KeGetCurrentIrql() == APC_LEVEL);
    00300
    00301     if (NT_SUCCESS(status)) {
    00302
    00303         if (WsLockChanged == TRUE) {
    00304             UNLOCK_SYSTEM_WS (APC_LEVEL);
    00305             LOCK_SESSION_SPACE_WS (OldIrql);
    00306         }
    00307
    00308         return status;
    00309     }
    00310
    00311     if (status == STATUS_ISSUE_PAGING_IO) {
    00312
    00313         SavedPte = *ReadPte;
    00314
    00315         CapturedEvent = (PMMINPAGE_SUPPORT)ReadBlock->Pfn->u1.Event;
    00316
    00317         CurrentThread = NULL;
    00318
    00319         if (Process == HYDRA_PROCESS) {
    00320             UNLOCK_SESSION_SPACE_WS(APC_LEVEL);
    00321         }
    00322         else if (Process != NULL) {
    00323
    00324             //
    00325             // APCs must be explicitly disabled to prevent suspend APCs from
    00326             // interrupting this thread before the I/O has been issued.
    00327             // Otherwise a shared page I/O can stop any other thread that
    00328             // references it indefinitely until the suspend is released.
    00329             //
    00330
    00331             CurrentThread = PsGetCurrentThread();
    00332
    00333             ASSERT (CurrentThread->NestedFaultCount <= 2);
    00334             CurrentThread->NestedFaultCount += 1;
    00335
    00336             KeEnterCriticalRegion();
    00337             UNLOCK_WS (Process);
    00338         }
    00339         else {
    00340             UNLOCK_SYSTEM_WS(APC_LEVEL);
    00341         }
    00342
    00343 #if DBG
    00344         if (MmDebug & MM_DBG_PAGEFAULT) {
    00345             DbgPrint ("MMFAULT: va: %p size: %lx process: %s file: %Z\n",
    00346                 VirtualAddress,
    00347                 ReadBlock->Mdl.ByteCount,
    00348                 Process == HYDRA_PROCESS ? (PUCHAR)"Session Space" : (Process ? Process->ImageFileName : (PUCHAR)"SystemVa"),
    00349                 &ReadBlock->FilePointer->FileName
    00350             );
    00351         }
    00352 #endif //DBG
    00353
    00354         PERFINFO_HARDFAULT(VirtualAddress, ReadBlock);
    00355
    00356 #if defined(_PREFETCH_)
    00357
    00358         //
    00359         // Assert no reads issued here are marked as prefetched.
    00360         //
    00361
    00362         ASSERT (ReadBlock->PrefetchMdl == NULL);
    00363
    00364 #endif
    00365
    00366         //
    00367         // Issue the read request.
    00368         //
    00369
    00370         status = IoPageRead ( ReadBlock->FilePointer,
    00371                              &ReadBlock->Mdl,
    00372                              &ReadBlock->ReadOffset,
    00373                              &ReadBlock->Event,
    00374                              &ReadBlock->IoStatus);
    00375         if (!NT_SUCCESS(status)) {
    00376
    00377             //
    00378             // Set the event as the I/O system doesn't set it on errors.
    00379             //
    00380
    00381
    00382             ReadBlock->IoStatus.Status = status;
    00383             ReadBlock->IoStatus.Information = 0;
    00384             KeSetEvent (&ReadBlock->Event,
    00385                         0,
    00386                         FALSE);
    00387         }
    00388
    00389         //
    00390         // Wait for the I/O operation.
    00391         //
    00392
    00393         status = MiWaitForInPageComplete (ReadBlock->Pfn,
    00394                                           ReadPte,
    00395                                           VirtualAddress,
    00396                                           &SavedPte,
    00397                                           CapturedEvent,
    00398                                           Process);
    00399
    00400         if (CurrentThread != NULL) {
    00401             KeLeaveCriticalRegion();
    00402
    00403             ASSERT (CurrentThread->NestedFaultCount <= 3);
    00404             ASSERT (CurrentThread->NestedFaultCount != 0);
    00405
    00406             CurrentThread->NestedFaultCount -= 1;
    00407
    00408             if ((CurrentThread->ApcNeeded == 1) &&
    00409                 (CurrentThread->NestedFaultCount == 0)) {
    00410                 *ApcNeeded = TRUE;
    00411                 CurrentThread->ApcNeeded = 0;
    00412             }
    00413         }
    00414
    00415         PERFINFO_HARDFAULT_IOTIME();
    00416
    00417         //
    00418         // MiWaitForInPageComplete RETURNS WITH THE WORKING SET LOCK
    00419         // AND PFN LOCK HELD!!!
    00420         //
    00421
    00422         //
    00423         // This is the thread which owns the event, clear the event field
    00424         // in the PFN database.
    00425         //
    00426
    00427         Pfn1 = ReadBlock->Pfn;
    00428         Page = &ReadBlock->Page[0];
    00429         NumberOfBytes = (LONG)ReadBlock->Mdl.ByteCount;
    00430         CheckPte = ReadBlock->BasePte;
    00431
    00432         while (NumberOfBytes > 0) {
    00433
    00434             //
    00435             // Don't remove the page we just brought in to
    00436             // satisfy this page fault.
    00437             //
    00438
    00439             if (CheckPte != ReadPte) {
    00440                 PfnClusterPage = MI_PFN_ELEMENT (*Page);
    00441                 ASSERT (PfnClusterPage->PteFrame == Pfn1->PteFrame);
    00442 #if DBG
    00443                 if (PfnClusterPage->u3.e1.InPageError) {
    00444                     ASSERT (status != STATUS_SUCCESS);
    00445                 }
    00446 #endif //DBG
    00447                 if (PfnClusterPage->u3.e1.ReadInProgress != 0) {
    00448
    00449                     ASSERT (PfnClusterPage->PteFrame != MI_MAGIC_AWE_PTEFRAME);
    00450                     PfnClusterPage->u3.e1.ReadInProgress = 0;
    00451
    00452                     if (PfnClusterPage->u3.e1.InPageError == 0) {
    00453                         PfnClusterPage->u1.Event = (PKEVENT)NULL;
    00454                     }
    00455                 }
    00456                 MI_REMOVE_LOCKED_PAGE_CHARGE(PfnClusterPage, 9);
    00457                 MiDecrementReferenceCount (*Page);
    00458             } else {
    00459                 PageFrameIndex = *Page;
    00460             }
    00461
    00462             CheckPte += 1;
    00463             Page += 1;
    00464             NumberOfBytes -= PAGE_SIZE;
    00465         }
    00466
    00467         if (status != STATUS_SUCCESS) {
    00468
    00469             MI_REMOVE_LOCKED_PAGE_CHARGE(MI_PFN_ELEMENT(PageFrameIndex), 9);
    00470             MiDecrementReferenceCount (PageFrameIndex);
    00471
    00472             if (status == STATUS_PTE_CHANGED) {
    00473
    00474                 //
    00475                 // State of PTE changed during I/O operation, just
    00476                 // return success and refault.
    00477                 //
    00478
    00479                 UNLOCK_PFN (APC_LEVEL);
    00480
    00481                 if (WsLockChanged == TRUE) {
    00482                     UNLOCK_SYSTEM_WS (APC_LEVEL);
    00483                     LOCK_SESSION_SPACE_WS (OldIrql);
    00484                 }
    00485
    00486                 return STATUS_SUCCESS;
    00487
    00488             }
    00489
    00490             //
    00491             // An I/O error occurred during the page read
    00492             // operation.  All the pages which were just
    00493             // put into transition should be put onto the
    00494             // free list if InPageError is set, and their
    00495             // PTEs restored to the proper contents.
    00496             //
    00497
    00498             Page = &ReadBlock->Page[0];
    00499
    00500             NumberOfBytes = ReadBlock->Mdl.ByteCount;
    00501
    00502             while (NumberOfBytes > 0) {
    00503
    00504                 PfnClusterPage = MI_PFN_ELEMENT (*Page);
    00505
    00506                 if (PfnClusterPage->u3.e1.InPageError == 1) {
    00507
    00508                     if (PfnClusterPage->u3.e2.ReferenceCount == 0) {
    00509
    00510                         PfnClusterPage->u3.e1.InPageError = 0;
    00511
    00512                         //
    00513                         // Only restore the transition PTE if the address
    00514                         // space still exists.  Another thread may have
    00515                         // deleted the VAD while this thread waited for the
    00516                         // fault to complete - in this case, the frame
    00517                         // will be marked as free already.
    00518                         //
    00519
    00520                         if (PfnClusterPage->u3.e1.PageLocation != FreePageList) {
    00521                             ASSERT (PfnClusterPage->u3.e1.PageLocation ==
    00522                                                             StandbyPageList);
    00523                             MiUnlinkPageFromList (PfnClusterPage);
    00524                             MiRestoreTransitionPte (*Page);
    00525                             MiInsertPageInList (MmPageLocationList[FreePageList],
    00526                                                 *Page);
    00527                         }
    00528                     }
    00529                 }
    00530                 Page += 1;
    00531                 NumberOfBytes -= PAGE_SIZE;
    00532             }
    00533             UNLOCK_PFN (APC_LEVEL);
    00534
    00535             if (WsLockChanged == TRUE) {
    00536                 UNLOCK_SYSTEM_WS (APC_LEVEL);
    00537                 LOCK_SESSION_SPACE_WS (OldIrql);
    00538             }
    00539
    00540             if (status == STATUS_REFAULT) {
    00541
    00542                 //
    00543                 // The I/O operation to bring in a system page failed
    00544                 // due to insufficent resources.  Delay a bit, then
    00545                 // return success and refault.
    00546                 //
    00547
    00548                 KeDelayExecutionThread (KernelMode, FALSE, (PLARGE_INTEGER)&MmShortTime);
    00549                 return STATUS_SUCCESS;
    00550             }
    00551
    00552             return status;
    00553         }
    00554
    00555         //
    00556         // PTE is still in transition state, same protection, etc.
    00557         //
    00558
    00559         ASSERT (Pfn1->u3.e1.InPageError == 0);
    00560
    00561         if (Pfn1->u2.ShareCount == 0) {
    00562             MI_REMOVE_LOCKED_PAGE_CHARGE (Pfn1, 9);
    00563         }
    00564
    00565         Pfn1->u2.ShareCount += 1;
    00566         Pfn1->u3.e1.PageLocation = ActiveAndValid;
    00567
    00568         MI_MAKE_TRANSITION_PTE_VALID (TempPte, ReadPte);
    00569         if (StoreInstruction && TempPte.u.Hard.Write) {
    00570             MI_SET_PTE_DIRTY (TempPte);
    00571         }
    00572         MI_WRITE_VALID_PTE (ReadPte, TempPte);
    00573
    00574         if (PointerProtoPte != NULL) {
    00575
    00576             //
    00577             // The prototype PTE has been made valid, now make the
    00578             // original PTE valid.
    00579             //
    00580
    00581             if (PointerPte->u.Hard.Valid == 0) {
    00582 #if DBG
    00583                 NTSTATUS oldstatus = status;
    00584 #endif //DBG
    00585
    00586                 //
    00587                 // PTE is not valid, continue with operation.
    00588                 //
    00589
    00590                 status = MiCompleteProtoPteFault (StoreInstruction,
    00591                                                   VirtualAddress,
    00592                                                   PointerPte,
    00593                                                   PointerProtoPte);
    00594
    00595                 //
    00596                 // Returns with PFN lock released!
    00597                 //
    00598
    00599 #if DBG
    00600                 if (PointerPte->u.Hard.Valid == 0) {
    00601                     DbgPrint ("MM:PAGFAULT - va %p  %p  %p  status:%lx\n",
    00602                         VirtualAddress, PointerPte, PointerProtoPte, oldstatus);
    00603                 }
    00604 #endif //DBG
    00605             }
    00606         } else {
    00607
    00608 #if PFN_CONSISTENCY
    00609             if (MiGetPteAddress(ReadPte) == MiGetPdeAddress(PTE_BASE)) {
    00610                 Pfn1->u3.e1.PageTablePage = 1;
    00611             }
    00612 #endif
    00613
    00614             if (Pfn1->u1.Event == 0) {
    00615                 Pfn1->u1.Event = (PVOID)PsGetCurrentThread();
    00616             }
    00617
    00618             UNLOCK_PFN (APC_LEVEL);
    00619             MiAddValidPageToWorkingSet (VirtualAddress,
    00620                                         ReadPte,
    00621                                         Pfn1,
    00622                                         0);
    00623         }
    00624
    00625         //
    00626         // Note this routine could release and reacquire the PFN lock!
    00627         //
    00628
    00629         LOCK_PFN (OldIrql);
    00630         MiFlushInPageSupportBlock();
    00631         UNLOCK_PFN (APC_LEVEL);
    00632
    00633         if (status == STATUS_SUCCESS) {
    00634             status = STATUS_PAGE_FAULT_PAGING_FILE;
    00635         }
    00636         NotifyRoutine = MmHardFaultNotifyRoutine;
    00637         if (NotifyRoutine) {
    00638             (*NotifyRoutine) (
    00639                 ReadBlock->FilePointer,
    00640                 VirtualAddress
    00641                 );
    00642         }
    00643     }
    00644
    00645     //
    00646     // Stop high priority threads from consuming the CPU on collided
    00647     // faults for pages that are still marked with inpage errors.  All
    00648     // the threads must let go of the page so it can be freed and the
    00649     // inpage I/O reissued to the filesystem.
    00650     //
    00651
    00652     if (MmIsRetryIoStatus(status)) {
    00653         KeDelayExecutionThread (KernelMode, FALSE, (PLARGE_INTEGER)&MmShortTime);
    00654         status = STATUS_SUCCESS;
    00655     }
    00656
    00657     if ((status == STATUS_REFAULT) ||
    00658         (status == STATUS_PTE_CHANGED)) {
    00659         status = STATUS_SUCCESS;
    00660     }
    00661
    00662     ASSERT (KeGetCurrentIrql() == APC_LEVEL);
    00663
    00664     if (WsLockChanged == TRUE) {
    00665         UNLOCK_SYSTEM_WS (APC_LEVEL);
    00666         LOCK_SESSION_SPACE_WS (OldIrql);
    00667     }
    00668
    00669     return status;
    00670 }

PMMPTE MiFindActualFaultingPte | ( | IN PVOID | FaultingAddress | ) |

Definition at line 2749 of file pagfault.c.

References MI_IS_PHYSICAL_ADDRESS, MI_PTE_LOOKUP_NEEDED, MiCheckVirtualAddress(), MiGetPdeAddress, MiGetPpeAddress, MiGetPteAddress, MiPteToProto, NULL, and _MMPTE::u.

Referenced by MiWaitForInPageComplete().

    02755                    :
    02756
    02757     This routine locates the actual PTE which must be made resident in order
    02758     to complete this fault.  Note that for certain cases multiple faults
    02759     are required to make the final page resident.
    02760
    02761 Arguments:
    02762
    02763     FaultingAddress - Supplies the virtual address which caused the
    02764                       fault.
    02765
    02766     PointerPte - Supplies the pointer to the PTE which is in prototype
    02767                  PTE format.
    02768
    02769
    02770 Return Value:
    02771
    02772
    02773 Environment:
    02774
    02775     Kernel mode, APCs disabled, working set mutex held.
    02776
    02777 --*/
    02778
    02779 {
    02780     PMMPTE ProtoPteAddress;
    02781     PMMPTE PointerPte;
    02782     PMMPTE PointerFaultingPte;
    02783     ULONG Protection;
    02784
    02785     if (MI_IS_PHYSICAL_ADDRESS(FaultingAddress)) {
    02786         return NULL;
    02787     }
    02788
    02789 #if defined (_WIN64)
    02790
    02791     PointerPte = MiGetPpeAddress (FaultingAddress);
    02792
    02793     if (PointerPte->u.Hard.Valid == 0) {
    02794
    02795         //
    02796         // Page directory page is not valid.
    02797         //
    02798
    02799         return PointerPte;
    02800     }
    02801
    02802 #endif
    02803
    02804     PointerPte = MiGetPdeAddress (FaultingAddress);
    02805
    02806     if (PointerPte->u.Hard.Valid == 0) {
    02807
    02808         //
    02809         // Page table page is not valid.
    02810         //
    02811
    02812         return PointerPte;
    02813     }
    02814
    02815     PointerPte = MiGetPteAddress (FaultingAddress);
    02816
    02817     if (PointerPte->u.Hard.Valid == 1) {
    02818
    02819         //
    02820         // Page is already valid, no need to fault it in.
    02821         //
    02822
    02823         return (PMMPTE)NULL;
    02824     }
    02825
    02826     if (PointerPte->u.Soft.Prototype == 0) {
    02827
    02828         //
    02829         // Page is not a prototype PTE, make this PTE valid.
    02830         //
    02831
    02832         return PointerPte;
    02833     }
    02834
    02835     //
    02836     // Check to see if the PTE which maps the prototype PTE is valid.
    02837     //
    02838
    02839     if (PointerPte->u.Soft.PageFileHigh == MI_PTE_LOOKUP_NEEDED) {
    02840
    02841         //
    02842         // Protection is here, PTE must be located in VAD.
    02843         //
    02844
    02845         ProtoPteAddress = MiCheckVirtualAddress (FaultingAddress,
    02846                                                     &Protection);
    02847
    02848         if (ProtoPteAddress == NULL) {
    02849
    02850             //
    02851             // No prototype PTE means another thread has deleted the VAD while
    02852             // this thread waited for the inpage to complete.  Certainly NULL
    02853             // must be returned so a stale PTE is not modified - the instruction
    02854             // will then be reexecuted and an access violation delivered.
    02855             //
    02856
    02857             return (PMMPTE)NULL;
    02858         }
    02859
    02860     } else {
    02861
    02862         //
    02863         // Protection is in ProtoPte.
    02864         //
    02865
    02866         ProtoPteAddress = MiPteToProto (PointerPte);
    02867     }
    02868
    02869     PointerFaultingPte = MiFindActualFaultingPte (ProtoPteAddress);
    02870
    02871     if (PointerFaultingPte == (PMMPTE)NULL) {
    02872         return PointerPte;
    02873     } else {
    02874         return PointerFaultingPte;
    02875     }
    02876
    02877 }

VOID MiFlushInPageSupportBlock | ( |  | ) |

Definition at line 4301 of file pagfault.c.

References APC_LEVEL, _MMINPAGE_SUPPORT_LIST::Count, ExFreePool(), _MMINPAGE_SUPPORT::ListEntry, _MMINPAGE_SUPPORT_LIST::ListHead, LOCK_PFN, _MMINPAGE_SUPPORT::Mdl, MM_PFN_LOCK_ASSERT, MmInPageSupportList, MMMAX_INPAGE_SUPPORT, NULL, and UNLOCK_PFN.

Referenced by MiDispatchFault().

    04306                    :
    04307
    04308     This routine examines the number of freed in page support blocks,
    04309     and if more than 4, frees the blocks back to the NonPagedPool.
    04310
    04311
    04312    ****** NB: The PFN LOCK is RELEASED and reacquired during this call ******
    04313
    04314
    04315 Arguments:
    04316
    04317     None.
    04318
    04319 Return Value:
    04320
    04321     None.
    04322
    04323 Environment:
    04324
    04325     Kernel mode, PFN lock held.
    04326
    04327 --*/
    04328
    04329 #define MMMAX_INPAGE_SUPPORT 4
    04330
    04331 {
    04332     KIRQL OldIrql;
    04333     PMMINPAGE_SUPPORT Support[10];
    04334     ULONG i = 0;
    04335     PLIST_ENTRY NextEntry;
    04336
    04337     MM_PFN_LOCK_ASSERT();
    04338
    04339     while ((MmInPageSupportList.Count > MMMAX_INPAGE_SUPPORT) && (i < 10)) {
    04340         NextEntry = RemoveHeadList (&MmInPageSupportList.ListHead);
    04341         Support[i] = CONTAINING_RECORD (NextEntry,
    04342                                         MMINPAGE_SUPPORT,
    04343                                         ListEntry );
    04344         Support[i]->ListEntry.Flink = NULL;
    04345         i += 1;
    04346         MmInPageSupportList.Count -= 1;
    04347     }
    04348
    04349     if (i == 0) {
    04350         return;
    04351     }
    04352
    04353     UNLOCK_PFN (APC_LEVEL);
    04354
    04355     do {
    04356         i -= 1;
    04357
    04358 #if defined (_PREFETCH_)
    04359         if ((Support[i]->PrefetchMdl != NULL) &&
    04360             (Support[i]->PrefetchMdl != &Support[i]->Mdl)) {
    04361             ExFreePool (Support[i]->PrefetchMdl);
    04362         }
    04363 #endif
    04364
    04365         ExFreePool(Support[i]);
    04366     } while (i > 0);
    04367
    04368     LOCK_PFN (OldIrql);
    04369
    04370     return;
    04371 }

VOID MiFreeInPageSupportBlock | ( | IN PMMINPAGE_SUPPORT | Support | ) |

Definition at line 4252 of file pagfault.c.

References ASSERT, _MMINPAGE_SUPPORT_LIST::Count, _MMINPAGE_SUPPORT_LIST::ListHead, MM_PFN_LOCK_ASSERT, MmInPageSupportList, and NULL.

Referenced by MiWaitForInPageComplete(), and MmCopyToCachedPage().

    04258                    :
    04259
    04260     This routine returns the in page support block to a list
    04261     of freed blocks.
    04262
    04263 Arguments:
    04264
    04265     Support - Supplies the in page support block to put on the free list.
    04266
    04267 Return Value:
    04268
    04269     None.
    04270
    04271 Environment:
    04272
    04273     Kernel mode, PFN lock held.
    04274
    04275 --*/
    04276
    04277 {
    04278
    04279     MM_PFN_LOCK_ASSERT();
    04280
    04281     ASSERT (Support->u.Thread != NULL);
    04282     ASSERT (Support->WaitCount != 0);
    04283 #if defined (_PREFETCH_)
    04284     ASSERT ((Support->ListEntry.Flink == NULL) ||
    04285             (Support->PrefetchMdl != NULL));
    04286 #else
    04287     ASSERT (Support->ListEntry.Flink == NULL);
    04288 #endif
    04289     Support->WaitCount -= 1;
    04290     if (Support->WaitCount == 0) {
    04291         Support->u.Thread = NULL;
    04292         InsertTailList (&MmInPageSupportList.ListHead,
    04293                         &Support->ListEntry);
    04294         MmInPageSupportList.Count += 1;
    04295     }
    04296     return;
    04297 }

PMMINPAGE_SUPPORT MiGetInPageSupportBlock | ( | VOID |  | ) |

Definition at line 4152 of file pagfault.c.

References APC_LEVEL, ASSERT, _MMINPAGE_SUPPORT::Completed, _MMINPAGE_SUPPORT_LIST::Count, _MMINPAGE_SUPPORT::Event, ExAllocatePoolWithTag, ExFreePool(), FALSE, KeClearEvent, KeInitializeEvent, _MMINPAGE_SUPPORT::ListEntry, _MMINPAGE_SUPPORT_LIST::ListHead, LOCK_PFN, _MMINPAGE_SUPPORT::Mdl, MM_PFN_LOCK_ASSERT, MmInPageSupportList, NonPagedPool, NULL, PsGetCurrentThread, _MMINPAGE_SUPPORT::u, UNLOCK_PFN, and _MMINPAGE_SUPPORT::WaitCount.

Referenced by MiResolveMappedFileFault(), MiResolvePageFileFault(), and MmCopyToCachedPage().

    04158                    :
    04159
    04160     This routine acquires an inpage support block.  If none are available,
    04161     the PFN lock will be released and reacquired to add an entry to the list.
    04162     FALSE will then be returned.
    04163
    04164 Arguments:
    04165
    04166     None.
    04167
    04168 Return Value:
    04169
    04170     A non-null pointer to an inpage block if one is already available.
    04171     The PFN lock is not released in this path.
    04172
    04173     NULL is returned if no inpage blocks were available.  In this path, the
    04174     PFN lock is released and an entry is added - but NULL is still returned
    04175     so the caller is aware that the state has changed due to the lock release
    04176     and reacquisition.
    04177
    04178 Environment:
    04179
    04180     Kernel mode, PFN lock held.
    04181
    04182 --*/
    04183
    04184 {
    04185     KIRQL OldIrql;
    04186     PMMINPAGE_SUPPORT Support;
    04187     PLIST_ENTRY NextEntry;
    04188
    04189     MM_PFN_LOCK_ASSERT();
    04190
    04191     if (MmInPageSupportList.Count == 0) {
    04192         ASSERT (IsListEmpty(&MmInPageSupportList.ListHead));
    04193         UNLOCK_PFN (APC_LEVEL);
    04194         Support = ExAllocatePoolWithTag (NonPagedPool,
    04195                                          sizeof(MMINPAGE_SUPPORT),
    04196                                          'nImM');
    04197         if (Support == NULL) {
    04198             LOCK_PFN (OldIrql);
    04199             return NULL;
    04200         }
    04201
    04202         KeInitializeEvent (&Support->Event, NotificationEvent, FALSE);
    04203         LOCK_PFN (OldIrql);
    04204
    04205         MmInPageSupportList.Count += 1;
    04206         Support->u.Thread = NULL;
    04207 #if defined(_PREFETCH_)
    04208         Support->PrefetchMdl = NULL;
    04209 #endif
    04210         InsertTailList (&MmInPageSupportList.ListHead,
    04211                         &Support->ListEntry);
    04212         return NULL;
    04213     }
    04214
    04215     ASSERT (!IsListEmpty(&MmInPageSupportList.ListHead));
    04216     MmInPageSupportList.Count -= 1;
    04217     NextEntry = RemoveHeadList (&MmInPageSupportList.ListHead);
    04218     Support = CONTAINING_RECORD (NextEntry,
    04219                                  MMINPAGE_SUPPORT,
    04220                                  ListEntry );
    04221
    04222 #if defined(_PREFETCH_)
    04223     if ((Support->PrefetchMdl != NULL) &&
    04224         (Support->PrefetchMdl != &Support->Mdl)) {
    04225
    04226         UNLOCK_PFN (APC_LEVEL);
    04227
    04228         ExFreePool (Support->PrefetchMdl);
    04229         ExFreePool (Support);
    04230
    04231         LOCK_PFN (OldIrql);
    04232         return NULL;
    04233     }
    04234 #endif
    04235
    04236     Support->Completed = FALSE;
    04237     Support->WaitCount = 1;
    04238     Support->u.Thread = PsGetCurrentThread();
    04239     Support->ListEntry.Flink = NULL;
    04240 #if defined(_PREFETCH_)
    04241     Support->PrefetchMdl = NULL;
    04242 #endif
    04243 #if defined (_WIN64)
    04244     Support->UsedPageTableEntries = 0;
    04245 #endif
    04246     KeClearEvent (&Support->Event);
    04247     return Support;
    04248 }

VOID MiHandleBankedSection | ( | IN PVOID | VirtualAddress,
|  | IN PMMVAD | Vad
| ) |

Definition at line 4374 of file pagfault.c.

References _MMBANKED_SECTION::BankedRoutine, _MMBANKED_SECTION::BankShift, _MMBANKED_SECTION::BankSize, _MMBANKED_SECTION::BankTemplate, _MMBANKED_SECTION::BasedPte, _MMBANKED_SECTION::Context, _MMBANKED_SECTION::CurrentMappedPte, FALSE, KeFlushEntireTb(), MiGetPteAddress, PAGE_SHIFT, PTE_SHIFT, TRUE, _MMPTE::u, and ZeroPte.

Referenced by MiCheckVirtualAddress().

    04381                    :
    04382
    04383     This routine invalidates a bank of video memory, calls out to the
    04384     video driver and then enables the next bank of video memory.
    04385
    04386 Arguments:
    04387
    04388     VirtualAddress - Supplies the address of the faulting page.
    04389
    04390     Vad - Supplies the VAD which maps the range.
    04391
    04392 Return Value:
    04393
    04394     None.
    04395
    04396 Environment:
    04397
    04398     Kernel mode, PFN lock held.
    04399
    04400 --*/
    04401
    04402 {
    04403     PMMBANKED_SECTION Bank;
    04404     PMMPTE PointerPte;
    04405     ULONG BankNumber;
    04406     ULONG size;
    04407
    04408     Bank = Vad->u4.Banked;
    04409     size = Bank->BankSize;
    04410
    04411     RtlFillMemory (Bank->CurrentMappedPte,
    04412                    size >> (PAGE_SHIFT - PTE_SHIFT),
    04413                    (UCHAR)ZeroPte.u.Long);
    04414
    04415     //
    04416     // Flush the TB as we have invalidated all the PTEs in this range
    04417     //
    04418
    04419     KeFlushEntireTb (TRUE, FALSE);
    04420
    04421     //
    04422     // Calculate new bank address and bank number.
    04423     //
    04424
    04425     PointerPte = MiGetPteAddress (
    04426                         (PVOID)((ULONG_PTR)VirtualAddress & ~((LONG)size - 1)));
    04427     Bank->CurrentMappedPte = PointerPte;
    04428
    04429     BankNumber = (ULONG)(((PCHAR)PointerPte - (PCHAR)Bank->BasedPte) >> Bank->BankShift);
    04430
    04431     (Bank->BankedRoutine)(BankNumber, BankNumber, Bank->Context);
    04432
    04433     //
    04434     // Set the new range valid.
    04435     //
    04436
    04437     RtlMoveMemory (PointerPte,
    04438                    &Bank->BankTemplate[0],
    04439                    size >> (PAGE_SHIFT - PTE_SHIFT));
    04440
    04441     return;
    04442 }

VOID MiInitializeCopyOnWritePfn | ( | IN PFN_NUMBER | PageFrameIndex,
|  | IN PMMPTE | PointerPte,
|  | IN WSLE_NUMBER | WorkingSetIndex,
|  | IN PVOID | SessionPointer
| ) |

Definition at line 3764 of file pagfault.c.

References ActiveAndValid, ASSERT, _MMWSLE::e1, KeBugCheckEx(), MI_GET_PAGE_FRAME_FROM_PTE, MI_MAKE_PROTECT_NOT_WRITE_COPY, MI_PFN_ELEMENT, MiCheckPdeForPagedPool(), MiGetPteAddress, MiGetVirtualAddressMappedByPte, MM_EXECUTE_READWRITE, MM_PFN_LOCK_ASSERT, MmWsle, NT_SUCCESS, _MMPFN::OriginalPte, _MMWSLENTRY::Protection, _MMPFN::PteAddress, _MMPFN::PteFrame, _MMPTE::u, _MMPFN::u1, _MMWSLE::u1, _MMPFN::u2, _MMPFN::u3, and _MM_SESSION_SPACE::Wsle.

Referenced by MiCopyOnWrite(), and MiSessionCopyOnWrite().

    03773                    :
    03774
    03775     This function initializes the specified PFN element to the
    03776     active and valid state for a copy on write operation.
    03777
    03778     In this case the page table page which contains the PTE has
    03779     the proper ShareCount.
    03780
    03781 Arguments:
    03782
    03783     PageFrameIndex - Supplies the page frame number to initialize.
    03784
    03785     PointerPte - Supplies the pointer to the PTE which caused the
    03786                  page fault.
    03787
    03788     WorkingSetIndex - Supplies the working set index for the corresponding
    03789                       virtual address.
    03790
    03791     SessionPointer - Supplies the session space pointer if this fault is for
    03792                      a session space page or NULL if this is for a user page.
    03793
    03794
    03795 Return Value:
    03796
    03797     None.
    03798
    03799 Environment:
    03800
    03801     Kernel mode, APCs disabled, PFN mutex held.
    03802
    03803 --*/
    03804
    03805 {
    03806     PMMPFN Pfn1;
    03807     PMMPTE PteFramePointer;
    03808     PFN_NUMBER PteFramePage;
    03809     PVOID VirtualAddress;
    03810     PMM_SESSION_SPACE SessionSpace;
    03811
    03812     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    03813     Pfn1->PteAddress = PointerPte;
    03814
    03815     //
    03816     // Get the protection for the page.
    03817     //
    03818
    03819     VirtualAddress = MiGetVirtualAddressMappedByPte (PointerPte);
    03820
    03821     Pfn1->OriginalPte.u.Long = 0;
    03822
    03823     if (SessionPointer) {
    03824         Pfn1->OriginalPte.u.Soft.Protection = MM_EXECUTE_READWRITE;
    03825         SessionSpace = (PMM_SESSION_SPACE) SessionPointer;
    03826         SessionSpace->Wsle[WorkingSetIndex].u1.e1.Protection =
    03827             MM_EXECUTE_READWRITE;
    03828     }
    03829     else {
    03830         Pfn1->OriginalPte.u.Soft.Protection =
    03831                 MI_MAKE_PROTECT_NOT_WRITE_COPY (
    03832                                     MmWsle[WorkingSetIndex].u1.e1.Protection);
    03833     }
    03834
    03835     ASSERT (Pfn1->u3.e2.ReferenceCount == 0);
    03836     Pfn1->u3.e2.ReferenceCount += 1;
    03837     Pfn1->u2.ShareCount += 1;
    03838     Pfn1->u3.e1.PageLocation = ActiveAndValid;
    03839     Pfn1->u1.WsIndex = WorkingSetIndex;
    03840
    03841     //
    03842     // Determine the page frame number of the page table page which
    03843     // contains this PTE.
    03844     //
    03845
    03846     PteFramePointer = MiGetPteAddress(PointerPte);
    03847     if (PteFramePointer->u.Hard.Valid == 0) {
    03848 #if !defined (_WIN64)
    03849         if (!NT_SUCCESS(MiCheckPdeForPagedPool (PointerPte))) {
    03850 #endif
    03851             KeBugCheckEx (MEMORY_MANAGEMENT,
    03852                           0x61940,
    03853                           (ULONG_PTR)PointerPte,
    03854                           (ULONG_PTR)PteFramePointer->u.Long,
    03855                           (ULONG_PTR)MiGetVirtualAddressMappedByPte(PointerPte));
    03856 #if !defined (_WIN64)
    03857         }
    03858 #endif
    03859     }
    03860
    03861     PteFramePage = MI_GET_PAGE_FRAME_FROM_PTE (PteFramePointer);
    03862     ASSERT (PteFramePage != 0);
    03863
    03864     Pfn1->PteFrame = PteFramePage;
    03865
    03866 #if PFN_CONSISTENCY
    03867     MM_PFN_LOCK_ASSERT();
    03868
    03869     Pfn1->u3.e1.PageTablePage = 0;
    03870 #endif
    03871
    03872     //
    03873     // Set the modified flag in the PFN database as we are writing
    03874     // into this page and the dirty bit is already set in the PTE.
    03875     //
    03876
    03877     Pfn1->u3.e1.Modified = 1;
    03878
    03879     return;
    03880 }

VOID MiInitializePfn | ( | IN PFN_NUMBER | PageFrameIndex,
|  | IN PMMPTE | PointerPte,
|  | IN ULONG | ModifiedState
| ) |

Definition at line 3398 of file pagfault.c.

References ActiveAndValid, ASSERT, DbgPrint, KeBugCheckEx(), MI_GET_PAGE_FRAME_FROM_PTE, MI_IS_CACHING_DISABLED, MI_PFN_ELEMENT, MiCheckPdeForPagedPool(), MiFormatPfn(), MiFormatPte(), MiGetPteAddress, MiGetVirtualAddressMappedByPte, MM_DEMAND_ZERO_WRITE_PTE, MM_EXECUTE_READWRITE, MM_NOCACHE, MM_PFN_LOCK_ASSERT, MM_READWRITE, NT_SUCCESS, _MMPFN::OriginalPte, _MMPFN::PteAddress, _MMPFN::PteFrame, _MMPTE::u, _MMPFN::u2, and _MMPFN::u3.

Referenced by ExAllocatePool(), MiAddWorkingSetPage(), MiAddWsleHash(), MiAllocatePoolPages(), MiAllocateSpecialPool(), MiFillSystemPageDirectory(), MiInitializeSystemCache(), MiInitializeWorkingSetList(), MiLoadImageSection(), MiReloadBootLoadedDrivers(), MiResolveDemandZeroFault(), MiSessionCommitImagePages(), MiSessionCreateInternal(), MiSessionInitializeWorkingSetList(), MmAccessFault(), MmAddPhysicalMemory(), MmAllocateIndependentPages(), MmAllocateNonCachedMemory(), MmCheckCachedPageState(), MmCreateKernelStack(), MmGrowKernelStack(), MmInitializeProcessAddressSpace(), and MmInitSystem().

    03406                    :
    03407
    03408     This function initializes the specified PFN element to the
    03409     active and valid state.
    03410
    03411 Arguments:
    03412
    03413     PageFrameIndex - Supplies the page frame number to initialize.
    03414
    03415     PointerPte - Supplies the pointer to the PTE which caused the
    03416                  page fault.
    03417
    03418     ModifiedState - Supplies the state to set the modified field in the PFN
    03419                     element for this page, either 0 or 1.
    03420
    03421 Return Value:
    03422
    03423     None.
    03424
    03425 Environment:
    03426
    03427     Kernel mode, APCs disabled, PFN mutex held.
    03428
    03429 --*/
    03430
    03431 {
    03432     PMMPFN Pfn1;
    03433     PMMPFN Pfn2;
    03434     PMMPTE PteFramePointer;
    03435     PFN_NUMBER PteFramePage;
    03436
    03437     MM_PFN_LOCK_ASSERT();
    03438
    03439     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    03440     Pfn1->PteAddress = PointerPte;
    03441
    03442     //
    03443     // If the PTE is currently valid, an address space is being built,
    03444     // just make the original PTE demand zero.
    03445     //
    03446
    03447     if (PointerPte->u.Hard.Valid == 1) {
    03448         Pfn1->OriginalPte.u.Long = MM_DEMAND_ZERO_WRITE_PTE;
    03449
    03450 #if defined(_IA64_)
    03451         if (PointerPte->u.Hard.Execute == 1) {
    03452             Pfn1->OriginalPte.u.Soft.Protection = MM_EXECUTE_READWRITE;
    03453         }
    03454 #endif
    03455
    03456         if (MI_IS_CACHING_DISABLED (PointerPte)) {
    03457             Pfn1->OriginalPte.u.Soft.Protection = MM_READWRITE | MM_NOCACHE;
    03458         }
    03459
    03460     } else {
    03461         Pfn1->OriginalPte = *PointerPte;
    03462         ASSERT (!((Pfn1->OriginalPte.u.Soft.Prototype == 0) &&
    03463                     (Pfn1->OriginalPte.u.Soft.Transition == 1)));
    03464     }
    03465
    03466     Pfn1->u3.e2.ReferenceCount += 1;
    03467
    03468 #if DBG
    03469     if (Pfn1->u3.e2.ReferenceCount > 1) {
    03470         DbgPrint("MM:incrementing ref count > 1 \n");
    03471         MiFormatPfn(Pfn1);
    03472         MiFormatPte(PointerPte);
    03473     }
    03474 #endif
    03475
    03476     Pfn1->u2.ShareCount += 1;
    03477     Pfn1->u3.e1.PageLocation = ActiveAndValid;
    03478     Pfn1->u3.e1.Modified = ModifiedState;
    03479
    03480 #if defined (_WIN64)
    03481     Pfn1->UsedPageTableEntries = 0;
    03482 #endif
    03483
    03484 #if PFN_CONSISTENCY
    03485     Pfn1->u3.e1.PageTablePage = 0;
    03486 #endif
    03487     //
    03488     // Determine the page frame number of the page table page which
    03489     // contains this PTE.
    03490     //
    03491
    03492     PteFramePointer = MiGetPteAddress(PointerPte);
    03493     if (PteFramePointer->u.Hard.Valid == 0) {
    03494 #if !defined (_WIN64)
    03495         if (!NT_SUCCESS(MiCheckPdeForPagedPool (PointerPte))) {
    03496 #endif
    03497             KeBugCheckEx (MEMORY_MANAGEMENT,
    03498                           0x61940,
    03499                           (ULONG_PTR)PointerPte,
    03500                           (ULONG_PTR)PteFramePointer->u.Long,
    03501                           (ULONG_PTR)MiGetVirtualAddressMappedByPte(PointerPte));
    03502 #if !defined (_WIN64)
    03503         }
    03504 #endif
    03505     }
    03506     PteFramePage = MI_GET_PAGE_FRAME_FROM_PTE (PteFramePointer);
    03507     ASSERT (PteFramePage != 0);
    03508     Pfn1->PteFrame = PteFramePage;
    03509
    03510     //
    03511     // Increment the share count for the page table page containing
    03512     // this PTE.
    03513     //
    03514
    03515     Pfn2 = MI_PFN_ELEMENT (PteFramePage);
    03516
    03517     Pfn2->u2.ShareCount += 1;
    03518
    03519     return;
    03520 }

VOID MiInitializePfnForOtherProcess | ( | IN PFN_NUMBER | PageFrameIndex,
|  | IN PMMPTE | PointerPte,
|  | IN PFN_NUMBER | ContainingPageFrame
| ) |

Definition at line 3973 of file pagfault.c.

References ActiveAndValid, ASSERT, DbgPrint, MI_PFN_ELEMENT, MiFormatPfn(), MiFormatPte(), MM_DEMAND_ZERO_WRITE_PTE, MM_PFN_LOCK_ASSERT, _MMPFN::OriginalPte, _MMPFN::PteAddress, _MMPFN::PteFrame, _MMPTE::u, _MMPFN::u2, and _MMPFN::u3.

Referenced by MiAllocatePoolPages(), MiFillSystemPageDirectory(), MiInitializeSessionPool(), MiMakeOutswappedPageResident(), MiSessionCommitPageTables(), MiSessionCreateInternal(), and MiSessionInitializeWorkingSetList().

    03981                    :
    03982
    03983     This function initializes the specified PFN element to the
    03984     active and valid state with the dirty bit on in the PTE and
    03985     the PFN database marked as modified.
    03986
    03987     As this PTE is not visible from the current process, the containing
    03988     page frame must be supplied at the PTE contents field for the
    03989     PFN database element are set to demand zero.
    03990
    03991 Arguments:
    03992
    03993     PageFrameIndex - Supplies the page frame number of which to initialize.
    03994
    03995     PointerPte - Supplies the pointer to the PTE which caused the
    03996                  page fault.
    03997
    03998     ContainingPageFrame - Supplies the page frame number of the page
    03999                           table page which contains this PTE.
    04000                           If the ContainingPageFrame is 0, then
    04001                           the ShareCount for the
    04002                           containing page is not incremented.
    04003
    04004 Return Value:
    04005
    04006     None.
    04007
    04008 Environment:
    04009
    04010     Kernel mode, APCs disabled, PFN mutex held.
    04011
    04012 --*/
    04013
    04014 {
    04015     PMMPFN Pfn1;
    04016     PMMPFN Pfn2;
    04017
    04018     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    04019     Pfn1->PteAddress = PointerPte;
    04020     Pfn1->OriginalPte.u.Long = MM_DEMAND_ZERO_WRITE_PTE;
    04021     ASSERT (Pfn1->u3.e2.ReferenceCount == 0);
    04022     Pfn1->u3.e2.ReferenceCount += 1;
    04023
    04024 #if DBG
    04025     if (Pfn1->u3.e2.ReferenceCount > 1) {
    04026         DbgPrint("MM:incrementing ref count > 1 \n");
    04027         MiFormatPfn(Pfn1);
    04028         MiFormatPte(PointerPte);
    04029     }
    04030 #endif
    04031
    04032     Pfn1->u2.ShareCount += 1;
    04033     Pfn1->u3.e1.PageLocation = ActiveAndValid;
    04034     Pfn1->u3.e1.Modified = 1;
    04035
    04036 #if PFN_CONSISTENCY
    04037     MM_PFN_LOCK_ASSERT();
    04038
    04039     Pfn1->u3.e1.PageTablePage = 0;
    04040 #endif
    04041
    04042     //
    04043     // Increment the share count for the page table page containing
    04044     // this PTE.
    04045     //
    04046
    04047     if (ContainingPageFrame != 0) {
    04048         Pfn1->PteFrame = ContainingPageFrame;
    04049         Pfn2 = MI_PFN_ELEMENT (ContainingPageFrame);
    04050         Pfn2->u2.ShareCount += 1;
    04051     }
    04052     return;
    04053 }

VOID MiInitializeReadInProgressPfn | ( | IN PMDL | Mdl,
|  | IN PMMPTE | BasePte,
|  | IN PKEVENT | Event,
|  | IN WSLE_NUMBER | WorkingSetIndex
| ) |

Definition at line 3523 of file pagfault.c.

References ASSERT, Event(), KeBugCheckEx(), MI_ADD_LOCKED_PAGE_CHARGE_FOR_MODIFIED_PAGE, MI_GET_PAGE_FRAME_FROM_PTE, MI_MAKE_TRANSITION_PTE, MI_PFN_ELEMENT, MI_PROTOTYPE_WSINDEX, MI_WRITE_INVALID_PTE, MiCheckPdeForPagedPool(), MiGetPteAddress, MiGetVirtualAddressMappedByPte, MM_PFN_LOCK_ASSERT, NT_SUCCESS, _MMPFN::OriginalPte, PAGE_SIZE, _MMPFN::PteAddress, _MMPFN::PteFrame, _MMPTE::u, _MMPFN::u1, _MMPFN::u2, and _MMPFN::u3.

Referenced by MiResolveMappedFileFault(), and MiResolvePageFileFault().

    03532                    :
    03533
    03534     This function initializes the specified PFN element to the
    03535     transition / read-in-progress state for an in-page operation.
    03536
    03537
    03538 Arguments:
    03539
    03540     Mdl - Supplies a pointer to the MDL.
    03541
    03542     BasePte - Supplies the pointer to the PTE which the first page in
    03543               the MDL maps.
    03544
    03545     Event - Supplies the event which is to be set when the I/O operation
    03546             completes.
    03547
    03548     WorkingSetIndex - Supplies the working set index flag, a value of
    03549                       -1 indicates no WSLE is required because
    03550                       this is a prototype PTE.
    03551
    03552 Return Value:
    03553
    03554     None.
    03555
    03556 Environment:
    03557
    03558     Kernel mode, APCs disabled, PFN mutex held.
    03559
    03560 --*/
    03561
    03562 {
    03563     PMMPFN Pfn1;
    03564     PMMPFN Pfn2;
    03565     PMMPTE PteFramePointer;
    03566     PFN_NUMBER PteFramePage;
    03567     MMPTE TempPte;
    03568     LONG NumberOfBytes;
    03569     PPFN_NUMBER Page;
    03570
    03571     MM_PFN_LOCK_ASSERT();
    03572
    03573     Page = (PPFN_NUMBER)(Mdl + 1);
    03574
    03575     NumberOfBytes = Mdl->ByteCount;
    03576
    03577     while (NumberOfBytes > 0) {
    03578
    03579         Pfn1 = MI_PFN_ELEMENT (*Page);
    03580         Pfn1->u1.Event = Event;
    03581         Pfn1->PteAddress = BasePte;
    03582         Pfn1->OriginalPte = *BasePte;
    03583         ASSERT (Pfn1->u3.e2.ReferenceCount == 0);
    03584         MI_ADD_LOCKED_PAGE_CHARGE_FOR_MODIFIED_PAGE (Pfn1, 10);
    03585         Pfn1->u3.e2.ReferenceCount += 1;
    03586
    03587         Pfn1->u2.ShareCount = 0;
    03588         Pfn1->u3.e1.ReadInProgress = 1;
    03589         Pfn1->u3.e1.InPageError = 0;
    03590
    03591 #if PFN_CONSISTENCY
    03592         Pfn1->u3.e1.PageTablePage = 0;
    03593 #endif
    03594         if (WorkingSetIndex == MI_PROTOTYPE_WSINDEX) {
    03595             Pfn1->u3.e1.PrototypePte = 1;
    03596         }
    03597
    03598         //
    03599         // Determine the page frame number of the page table page which
    03600         // contains this PTE.
    03601         //
    03602
    03603         PteFramePointer = MiGetPteAddress(BasePte);
    03604         if (PteFramePointer->u.Hard.Valid == 0) {
    03605 #if !defined (_WIN64)
    03606             if (!NT_SUCCESS(MiCheckPdeForPagedPool (BasePte))) {
    03607 #endif
    03608                 KeBugCheckEx (MEMORY_MANAGEMENT,
    03609                               0x61940,
    03610                               (ULONG_PTR)BasePte,
    03611                               (ULONG_PTR)PteFramePointer->u.Long,
    03612                               (ULONG_PTR)MiGetVirtualAddressMappedByPte(BasePte));
    03613 #if !defined (_WIN64)
    03614             }
    03615 #endif
    03616         }
    03617
    03618         PteFramePage = MI_GET_PAGE_FRAME_FROM_PTE (PteFramePointer);
    03619         Pfn1->PteFrame = PteFramePage;
    03620
    03621         //
    03622         // Put the PTE into the transition state, no cache flush needed as
    03623         // PTE is still not valid.
    03624         //
    03625
    03626         MI_MAKE_TRANSITION_PTE (TempPte,
    03627                                 *Page,
    03628                                 BasePte->u.Soft.Protection,
    03629                                 BasePte);
    03630         MI_WRITE_INVALID_PTE (BasePte, TempPte);
    03631
    03632         //
    03633         // Increment the share count for the page table page containing
    03634         // this PTE as the PTE just went into the transition state.
    03635         //
    03636
    03637         ASSERT (PteFramePage != 0);
    03638         Pfn2 = MI_PFN_ELEMENT (PteFramePage);
    03639         Pfn2->u2.ShareCount += 1;
    03640
    03641         NumberOfBytes -= PAGE_SIZE;
    03642         Page += 1;
    03643         BasePte += 1;
    03644     }
    03645
    03646     return;
    03647 }

VOID MiInitializeTransitionPfn | ( | IN PFN_NUMBER | PageFrameIndex,
|  | IN PMMPTE | PointerPte,
|  | IN WSLE_NUMBER | WorkingSetIndex
| ) |

Definition at line 3650 of file pagfault.c.

References ASSERT, KeBugCheckEx(), MI_GET_PAGE_FRAME_FROM_PTE, MI_MAKE_TRANSITION_PTE, MI_PFN_ELEMENT, MI_PROTOTYPE_WSINDEX, MI_WRITE_INVALID_PTE, MiCheckPdeForPagedPool(), MiGetPteAddress, MiGetVirtualAddressMappedByPte, MM_PFN_LOCK_ASSERT, NT_SUCCESS, NULL, _MMPFN::OriginalPte, _MMPFN::PteAddress, _MMPFN::PteFrame, TransitionPage, _MMPTE::u, _MMPFN::u1, _MMPFN::u2, and _MMPFN::u3.

Referenced by MiUpdateImageHeaderPage(), and MmCopyToCachedPage().

    03658                    :
    03659
    03660     This function initializes the specified PFN element to the
    03661     transition state.  Main use is by MapImageFile to make the
    03662     page which contains the image header transition in the
    03663     prototype PTEs.
    03664
    03665 Arguments:
    03666
    03667     PageFrameIndex - Supplies the page frame index to be initialized.
    03668
    03669     PointerPte - Supplies an invalid, non-transition PTE to initialize.
    03670
    03671     WorkingSetIndex - Supplies the working set index flag, a value of
    03672                       MI_PROTOTYPE_WSINDEX indicates no WSLE is required
    03673                       because this is a prototype PTE.
    03674
    03675 Return Value:
    03676
    03677     None.
    03678
    03679 Environment:
    03680
    03681     Kernel mode, APCs disabled, PFN mutex held.
    03682
    03683 --*/
    03684
    03685 {
    03686     PMMPFN Pfn1;
    03687     PMMPFN Pfn2;
    03688     PMMPTE PteFramePointer;
    03689     PFN_NUMBER PteFramePage;
    03690     MMPTE TempPte;
    03691
    03692     MM_PFN_LOCK_ASSERT();
    03693     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    03694     Pfn1->u1.Event = NULL;
    03695     Pfn1->PteAddress = PointerPte;
    03696     Pfn1->OriginalPte = *PointerPte;
    03697     ASSERT (!((Pfn1->OriginalPte.u.Soft.Prototype == 0) &&
    03698               (Pfn1->OriginalPte.u.Soft.Transition == 1)));
    03699
    03700     //
    03701     // Don't change the reference count (it should already be 1).
    03702     //
    03703
    03704     Pfn1->u2.ShareCount = 0;
    03705
    03706     if (WorkingSetIndex == MI_PROTOTYPE_WSINDEX) {
    03707         Pfn1->u3.e1.PrototypePte = 1;
    03708     }
    03709
    03710     Pfn1->u3.e1.PageLocation = TransitionPage;
    03711
    03712     //
    03713     // Determine the page frame number of the page table page which
    03714     // contains this PTE.
    03715     //
    03716
    03717     PteFramePointer = MiGetPteAddress(PointerPte);
    03718     if (PteFramePointer->u.Hard.Valid == 0) {
    03719 #if !defined (_WIN64)
    03720         if (!NT_SUCCESS(MiCheckPdeForPagedPool (PointerPte))) {
    03721 #endif
    03722             KeBugCheckEx (MEMORY_MANAGEMENT,
    03723                           0x61940,
    03724                           (ULONG_PTR)PointerPte,
    03725                           (ULONG_PTR)PteFramePointer->u.Long,
    03726                           (ULONG_PTR)MiGetVirtualAddressMappedByPte(PointerPte));
    03727 #if !defined (_WIN64)
    03728         }
    03729 #endif
    03730     }
    03731
    03732     PteFramePage = MI_GET_PAGE_FRAME_FROM_PTE (PteFramePointer);
    03733     Pfn1->PteFrame = PteFramePage;
    03734
    03735 #if PFN_CONSISTENCY
    03736     Pfn1->u3.e1.PageTablePage = 0;
    03737 #endif
    03738
    03739     //
    03740     // Put the PTE into the transition state, no cache flush needed as
    03741     // PTE is still not valid.
    03742     //
    03743
    03744     MI_MAKE_TRANSITION_PTE (TempPte,
    03745                             PageFrameIndex,
    03746                             PointerPte->u.Soft.Protection,
    03747                             PointerPte);
    03748
    03749     MI_WRITE_INVALID_PTE (PointerPte, TempPte);
    03750
    03751     //
    03752     // Increment the share count for the page table page containing
    03753     // this PTE as the PTE just went into the transition state.
    03754     //
    03755
    03756     Pfn2 = MI_PFN_ELEMENT (PteFramePage);
    03757     ASSERT (PteFramePage != 0);
    03758     Pfn2->u2.ShareCount += 1;
    03759
    03760     return;
    03761 }

NTSTATUS MiResolveDemandZeroFault | ( | IN PVOID | VirtualAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PEPROCESS | Process,
|  | IN ULONG | PrototypePte
| ) |

Definition at line 674 of file pagfault.c.

References APC_LEVEL, ASSERT, CONSISTENCY_LOCK_PFN, CONSISTENCY_UNLOCK_PFN, _MMINFO_COUNTERS::DemandZeroCount, FALSE, HYDRA_PROCESS, LOCK_PFN, MI_BARRIER_STAMP_ZEROED_PAGE, MI_BARRIER_SYNCHRONIZE, MI_IS_PAGE_TABLE_ADDRESS, MI_MAKE_VALID_PTE, MI_PAGE_COLOR_VA_PROCESS, MI_PFN_ELEMENT, MI_SET_PTE_DIRTY, MI_WRITE_VALID_PTE, MiAddValidPageToWorkingSet(), MiEnsureAvailablePageOrWait(), MiInitializePfn(), MiRemoveAnyPage(), MiRemoveZeroPage(), MiRemoveZeroPageIfAny, MiWaitForForkToComplete(), MiZeroPhysicalPage(), MM_PFN_LOCK_ASSERT, MmInfoCounters, MmSystemPageColor, NULL, PERFINFO_PRIVATE_PAGE_DEMAND_ZERO, PERFINFO_SOFTFAULT, PrototypePte, PsGetCurrentThread, _MMPFN::PteFrame, STATUS_REFAULT, TRUE, _MMPTE::u, _MMPFN::u1, and UNLOCK_PFN.

Referenced by MiDispatchFault(), MiResolveProtoPteFault(), and MmAccessFault().

    00683                    :
    00684
    00685     This routine resolves a demand zero page fault.
    00686
    00687     If the PrototypePte argument is TRUE, the PFN lock is
    00688     held, the lock cannot be dropped, and the page should
    00689     not be added to the working set at this time.
    00690
    00691 Arguments:
    00692
    00693     VirtualAddress - Supplies the faulting address.
    00694
    00695     PointerPte - Supplies the PTE for the faulting address.
    00696
    00697     Process - Supplies a pointer to the process object.  If this
    00698               parameter is NULL, then the fault is for system
    00699               space and the process's working set lock is not held.
    00700
    00701     PrototypePte - Supplies TRUE if this is a prototype PTE.
    00702
    00703 Return Value:
    00704
    00705     status, either STATUS_SUCCESS or STATUS_REFAULT.
    00706
    00707 Environment:
    00708
    00709     Kernel mode, PFN lock held conditionally.
    00710
    00711 --*/
    00712
    00713
    00714 {
    00715     PMMPFN Pfn1;
    00716     PFN_NUMBER PageFrameIndex;
    00717     MMPTE TempPte;
    00718     ULONG PageColor;
    00719     KIRQL OldIrql;
    00720     LOGICAL NeedToZero;
    00721     LOGICAL BarrierNeeded;
    00722     ULONG BarrierStamp;
    00723
    00724     NeedToZero = FALSE;
    00725     BarrierNeeded = FALSE;
    00726
    00727     PERFINFO_PRIVATE_PAGE_DEMAND_ZERO(VirtualAddress);
    00728
    00729     //
    00730     // Check to see if a page is available, if a wait is
    00731     // returned, do not continue, just return success.
    00732     //
    00733
    00734     if (!PrototypePte) {
    00735         LOCK_PFN (OldIrql);
    00736     }
    00737
    00738     MM_PFN_LOCK_ASSERT();
    00739
    00740     if (PointerPte->u.Hard.Valid == 0) {
    00741         if (!MiEnsureAvailablePageOrWait (Process,
    00742                                           VirtualAddress)) {
    00743
    00744             if (Process != NULL && Process != HYDRA_PROCESS && (!PrototypePte)) {
    00745                 //
    00746                 // If a fork operation is in progress and the faulting thread
    00747                 // is not the thread performing the fork operation, block until
    00748                 // the fork is completed.
    00749                 //
    00750
    00751                 if ((Process->ForkInProgress != NULL) &&
    00752                     (Process->ForkInProgress != PsGetCurrentThread())) {
    00753                     MiWaitForForkToComplete (Process);
    00754                     UNLOCK_PFN (APC_LEVEL);
    00755                     return STATUS_REFAULT;
    00756                 }
    00757
    00758                 Process->NumberOfPrivatePages += 1;
    00759                 PageColor = MI_PAGE_COLOR_VA_PROCESS (VirtualAddress,
    00760                                                    &Process->NextPageColor);
    00761
    00762                 ASSERT (MI_IS_PAGE_TABLE_ADDRESS(PointerPte));
    00763
    00764                 PageFrameIndex = MiRemoveZeroPageIfAny (PageColor);
    00765                 if (PageFrameIndex) {
    00766
    00767                     //
    00768                     // This barrier check is needed after zeroing the page
    00769                     // and before setting the PTE valid.  Note since the PFN
    00770                     // database entry is used to hold the sequence timestamp,
    00771                     // it must be captured now.  Check it at the last possible
    00772                     // moment.
    00773                     //
    00774
    00775                     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    00776                     BarrierStamp = (ULONG)Pfn1->PteFrame;
    00777                 }
    00778                 else {
    00779                     PageFrameIndex = MiRemoveAnyPage (PageColor);
    00780                     NeedToZero = TRUE;
    00781                 }
    00782                 BarrierNeeded = TRUE;
    00783
    00784             } else {
    00785                 PageColor = MI_PAGE_COLOR_VA_PROCESS (VirtualAddress,
    00786                                                       &MmSystemPageColor);
    00787                 //
    00788                 // As this is a system page, there is no need to
    00789                 // remove a page of zeroes, it must be initialized by
    00790                 // the system before being used.
    00791                 //
    00792
    00793                 if (PrototypePte) {
    00794                     PageFrameIndex = MiRemoveZeroPage (PageColor);
    00795                 } else {
    00796                     PageFrameIndex = MiRemoveAnyPage (PageColor);
    00797                 }
    00798             }
    00799
    00800             MmInfoCounters.DemandZeroCount += 1;
    00801
    00802             MiInitializePfn (PageFrameIndex, PointerPte, 1);
    00803
    00804             if (!PrototypePte) {
    00805                 UNLOCK_PFN (APC_LEVEL);
    00806             }
    00807
    00808             Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    00809
    00810             if (NeedToZero) {
    00811
    00812                 MiZeroPhysicalPage (PageFrameIndex, PageColor);
    00813
    00814                 //
    00815                 // Note the stamping must occur after the page is zeroed.
    00816                 //
    00817
    00818                 MI_BARRIER_STAMP_ZEROED_PAGE (&BarrierStamp);
    00819             }
    00820
    00821             //
    00822             // As this page is demand zero, set the modified bit in the
    00823             // PFN database element and set the dirty bit in the PTE.
    00824             //
    00825
    00826             PERFINFO_SOFTFAULT(Pfn1, VirtualAddress, PERFINFO_LOG_TYPE_DEMANDZEROFAULT)
    00827
    00828             MI_MAKE_VALID_PTE (TempPte,
    00829                                PageFrameIndex,
    00830                                PointerPte->u.Soft.Protection,
    00831                                PointerPte);
    00832
    00833             if (TempPte.u.Hard.Write != 0) {
    00834                 MI_SET_PTE_DIRTY (TempPte);
    00835             }
    00836
    00837             if (BarrierNeeded) {
    00838                 MI_BARRIER_SYNCHRONIZE (BarrierStamp);
    00839             }
    00840
    00841             MI_WRITE_VALID_PTE (PointerPte, TempPte);
    00842
    00843             if (!PrototypePte) {
    00844                 ASSERT (Pfn1->u1.Event == 0);
    00845
    00846                 CONSISTENCY_LOCK_PFN (OldIrql);
    00847
    00848                 Pfn1->u1.Event = (PVOID)PsGetCurrentThread();
    00849
    00850                 CONSISTENCY_UNLOCK_PFN (OldIrql);
    00851
    00852                 MiAddValidPageToWorkingSet (VirtualAddress,
    00853                                             PointerPte,
    00854                                             Pfn1,
    00855                                             0);
    00856             }
    00857             return STATUS_PAGE_FAULT_DEMAND_ZERO;
    00858         }
    00859     }
    00860     if (!PrototypePte) {
    00861         UNLOCK_PFN (APC_LEVEL);
    00862     }
    00863     return STATUS_REFAULT;
    00864 }

NTSTATUS MiResolveMappedFileFault | ( | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte,
|  | OUT PMMINPAGE_SUPPORT * | ReadBlock,
|  | IN PEPROCESS | Process
| ) |

Definition at line 1989 of file pagfault.c.

References ASSERT, _SUBSECTION::ControlArea, _ETHREAD::DisablePageFaultClustering, _CONTROL_AREA::FilePointer, _ETHREAD::ForwardClusterOnly, HYDRA_PROCESS, KeBugCheckEx(), KeFlushEntireTb(), MDL_IO_PAGE_READ, MDL_PAGES_LOCKED, _MDL::MdlFlags, MI_GET_PAGE_COLOR_FROM_PTE, MI_GET_PAGE_COLOR_FROM_SESSION, MI_PAGE_COLOR_PTE_PROCESS, MI_PFN_ELEMENT, MI_PROTOTYPE_WSINDEX, MI_ZERO_USED_PAGETABLE_ENTRIES_IN_INPAGE_SUPPORT, MiEndingOffset(), MiEnsureAvailablePageOrWait(), MiFormatPte(), MiGetInPageSupportBlock(), MiGetSubsectionAddress, MiGetVirtualAddressMappedByPte, MiInitializeReadInProgressPfn(), MiIsPteOnPdeBoundary, MiRemoveAnyPage(), MiRemoveZeroPage(), MiStartingOffset(), MM_DBG_PTE_UPDATE, MM_MAXIMUM_READ_CLUSTER_SIZE, MM_PROTECTION_EXECUTE_MASK, MmAvailablePages, MmCodeClusterSize, MmDataClusterSize, MmFreeGoal, MmInfoCounters, MmInitializeMdl, MmSessionSpace, NULL, _CONTROL_AREA::NumberOfPfnReferences, PAGE_SHIFT, PAGE_SIZE, _MMINFO_COUNTERS::PageReadCount, _MMINFO_COUNTERS::PageReadIoCount, PERFINFO_DO_PAGEFAULT_CLUSTERING, PsGetCurrentThread, _SUBSECTION::PtesInSubsection, _ETHREAD::ReadClusterSize, STATUS_ISSUE_PAGING_IO, STATUS_REFAULT, _SUBSECTION::SubsectionBase, TRUE, _MMPTE::u, _CONTROL_AREA::u, and _SUBSECTION::u.

Referenced by MiResolveProtoPteFault().

    01998                    :
    01999
    02000     This routine builds the MDL and other structures to allow a
    02001     read operation on a mapped file for a page fault.
    02002
    02003 Arguments:
    02004
    02005     FaultingAddress - Supplies the faulting address.
    02006
    02007     PointerPte - Supplies the PTE for the faulting address.
    02008
    02009     ReadBlock - Supplies a pointer to put the address of the read block which
    02010                 needs to be completed before an I/O can be issued.
    02011
    02012     Process - Supplies a pointer to the process object.  If this
    02013               parameter is NULL, then the fault is for system
    02014               space and the process's working set lock is not held.
    02015
    02016 Return Value:
    02017
    02018     status.  A status value of STATUS_ISSUE_PAGING_IO is returned
    02019     if this function completes successfully.
    02020
    02021 Environment:
    02022
    02023     Kernel mode, PFN lock held.
    02024
    02025 --*/
    02026
    02027 {
    02028     PFN_NUMBER PageFrameIndex;
    02029     PMMPFN Pfn1;
    02030     PSUBSECTION Subsection;
    02031     PMDL Mdl;
    02032     ULONG ReadSize;
    02033     PETHREAD CurrentThread;
    02034     PPFN_NUMBER Page;
    02035     PPFN_NUMBER EndPage;
    02036     PMMPTE BasePte;
    02037     PMMPTE CheckPte;
    02038     LARGE_INTEGER StartingOffset;
    02039     LARGE_INTEGER TempOffset;
    02040     PPFN_NUMBER FirstMdlPage;
    02041     PMMINPAGE_SUPPORT ReadBlockLocal;
    02042     ULONG PageColor;
    02043     ULONG ClusterSize;
    02044     ULONG Result;
    02045
    02046     ClusterSize = 0;
    02047
    02048     ASSERT (PointerPte->u.Soft.Prototype == 1);
    02049
    02050     // *********************************************
    02051     //   Mapped File (subsection format)
    02052     // *********************************************
    02053
    02054     if (Process == HYDRA_PROCESS) {
    02055         Result = MiEnsureAvailablePageOrWait (NULL, FaultingAddress);
    02056     }
    02057     else {
    02058         Result = MiEnsureAvailablePageOrWait (Process, FaultingAddress);
    02059     }
    02060
    02061     if (Result) {
    02062
    02063         //
    02064         // A wait operation was performed which dropped the locks,
    02065         // repeat this fault.
    02066         //
    02067
    02068         return STATUS_REFAULT;
    02069     }
    02070
    02071 #if DBG
    02072     if (MmDebug & MM_DBG_PTE_UPDATE) {
    02073         MiFormatPte (PointerPte);
    02074     }
    02075 #endif //DBG
    02076
    02077     //
    02078     // Calculate address of subsection for this prototype PTE.
    02079     //
    02080
    02081     Subsection = MiGetSubsectionAddress (PointerPte);
    02082
    02083 #ifdef LARGE_PAGES
    02084
    02085     //
    02086     // Check to see if this subsection maps a large page, if
    02087     // so, just fill the TB and return a status of PTE_CHANGED.
    02088     //
    02089
    02090     if (Subsection->u.SubsectionFlags.LargePages == 1) {
    02091         KeFlushEntireTb (TRUE, TRUE);
    02092         KeFillLargeEntryTb ((PHARDWARE_PTE)(Subsection + 1),
    02093                              FaultingAddress,
    02094                              Subsection->StartingSector);
    02095
    02096         return STATUS_REFAULT;
    02097     }
    02098 #endif //LARGE_PAGES
    02099
    02100     if (Subsection->ControlArea->u.Flags.FailAllIo) {
    02101         return STATUS_IN_PAGE_ERROR;
    02102     }
    02103
    02104     if (PointerPte >= &Subsection->SubsectionBase[Subsection->PtesInSubsection]) {
    02105
    02106         //
    02107         // Attempt to read past the end of this subsection.
    02108         //
    02109
    02110         return STATUS_ACCESS_VIOLATION;
    02111     }
    02112
    02113     CurrentThread = PsGetCurrentThread();
    02114
    02115     ReadBlockLocal = MiGetInPageSupportBlock ();
    02116     if (ReadBlockLocal == NULL) {
    02117         return STATUS_REFAULT;
    02118     }
    02119     *ReadBlock = ReadBlockLocal;
    02120
    02121     //
    02122     // Build MDL for request.
    02123     //
    02124
    02125     Mdl = &ReadBlockLocal->Mdl;
    02126
    02127     FirstMdlPage = &ReadBlockLocal->Page[0];
    02128     Page = FirstMdlPage;
    02129
    02130 #if DBG
    02131     RtlFillMemoryUlong( Page, (MM_MAXIMUM_READ_CLUSTER_SIZE+1) * sizeof(PFN_NUMBER), 0xf1f1f1f1);
    02132 #endif //DBG
    02133
    02134     ReadSize = PAGE_SIZE;
    02135     BasePte = PointerPte;
    02136
    02137     //
    02138     // Should we attempt to perform page fault clustering?
    02139     //
    02140
    02141     if ((!CurrentThread->DisablePageFaultClustering) &&
    02142         PERFINFO_DO_PAGEFAULT_CLUSTERING() &&
    02143         (Subsection->ControlArea->u.Flags.NoModifiedWriting == 0)) {
    02144
    02145         if ((MmAvailablePages > (MmFreeGoal * 2))
    02146                  ||
    02147          (((Subsection->ControlArea->u.Flags.Image != 0) ||
    02148             (CurrentThread->ForwardClusterOnly)) &&
    02149          (MmAvailablePages > (MM_MAXIMUM_READ_CLUSTER_SIZE + 16)))) {
    02150
    02151             //
    02152             // Cluster up to n pages.  This one + n-1.
    02153             //
    02154
    02155             if (Subsection->ControlArea->u.Flags.Image == 0) {
    02156                 ASSERT (CurrentThread->ReadClusterSize <=
    02157                             MM_MAXIMUM_READ_CLUSTER_SIZE);
    02158                 ClusterSize = CurrentThread->ReadClusterSize;
    02159             } else {
    02160                 ClusterSize = MmDataClusterSize;
    02161                 if (Subsection->u.SubsectionFlags.Protection &
    02162                                             MM_PROTECTION_EXECUTE_MASK ) {
    02163                     ClusterSize = MmCodeClusterSize;
    02164                 }
    02165             }
    02166             EndPage = Page + ClusterSize;
    02167
    02168             CheckPte = PointerPte + 1;
    02169
    02170             //
    02171             // Try to cluster within the page of PTEs.
    02172             //
    02173
    02174             while ((MiIsPteOnPdeBoundary(CheckPte) == 0) &&
    02175                (Page < EndPage) &&
    02176                (CheckPte <
    02177                  &Subsection->SubsectionBase[Subsection->PtesInSubsection])
    02178                       && (CheckPte->u.Long == BasePte->u.Long)) {
    02179
    02180                 Subsection->ControlArea->NumberOfPfnReferences += 1;
    02181                 ReadSize += PAGE_SIZE;
    02182                 Page += 1;
    02183                 CheckPte += 1;
    02184             }
    02185
    02186             if ((Page < EndPage) && (!CurrentThread->ForwardClusterOnly)) {
    02187
    02188                 //
    02189                 // Attempt to cluster going backwards from the PTE.
    02190                 //
    02191
    02192                 CheckPte = PointerPte - 1;
    02193
    02194                 while ((((ULONG_PTR)CheckPte & (PAGE_SIZE - 1)) !=
    02195                                             (PAGE_SIZE - sizeof(MMPTE))) &&
    02196                         (Page < EndPage) &&
    02197                          (CheckPte >= Subsection->SubsectionBase) &&
    02198                          (CheckPte->u.Long == BasePte->u.Long)) {
    02199
    02200                     Subsection->ControlArea->NumberOfPfnReferences += 1;
    02201                     ReadSize += PAGE_SIZE;
    02202                     Page += 1;
    02203                     CheckPte -= 1;
    02204                 }
    02205                 BasePte = CheckPte + 1;
    02206             }
    02207         }
    02208     }
    02209
    02210     //
    02211     //
    02212     // Calculate the offset to read into the file.
    02213     //  offset = base + ((thispte - basepte) << PAGE_SHIFT)
    02214     //
    02215
    02216     StartingOffset.QuadPart = MiStartingOffset (Subsection, BasePte);
    02217
    02218     TempOffset = MiEndingOffset(Subsection);
    02219
    02220     ASSERT (StartingOffset.QuadPart < TempOffset.QuadPart);
    02221
    02222     //
    02223     // Remove pages to fill in the MDL.  This is done here as the
    02224     // base PTE has been determined and can be used for virtual
    02225     // aliasing checks.
    02226     //
    02227
    02228     EndPage = FirstMdlPage;
    02229     CheckPte = BasePte;
    02230
    02231     while (EndPage < Page) {
    02232         if (Process == HYDRA_PROCESS) {
    02233             PageColor = MI_GET_PAGE_COLOR_FROM_SESSION (MmSessionSpace);
    02234         }
    02235         else if (Process == NULL) {
    02236             PageColor = MI_GET_PAGE_COLOR_FROM_PTE (CheckPte);
    02237         }
    02238         else {
    02239             PageColor = MI_PAGE_COLOR_PTE_PROCESS (CheckPte,
    02240                                                    &Process->NextPageColor);
    02241         }
    02242         *EndPage = MiRemoveAnyPage (PageColor);
    02243
    02244         EndPage += 1;
    02245         CheckPte += 1;
    02246     }
    02247
    02248     if (Process == HYDRA_PROCESS) {
    02249         PageColor = MI_GET_PAGE_COLOR_FROM_SESSION (MmSessionSpace);
    02250     }
    02251     else if (Process == NULL) {
    02252         PageColor = MI_GET_PAGE_COLOR_FROM_PTE (CheckPte);
    02253     }
    02254     else {
    02255         PageColor = MI_PAGE_COLOR_PTE_PROCESS (CheckPte,
    02256                                                &Process->NextPageColor);
    02257     }
    02258
    02259     //
    02260     // Check to see if the read will go past the end of the file,
    02261     // if so, correct the read size and get a zeroed page.
    02262     //
    02263
    02264     MmInfoCounters.PageReadIoCount += 1;
    02265     MmInfoCounters.PageReadCount += ReadSize >> PAGE_SHIFT;
    02266
    02267     if ((Subsection->ControlArea->u.Flags.Image) &&
    02268         (((UINT64)StartingOffset.QuadPart + ReadSize) > (UINT64)TempOffset.QuadPart)) {
    02269
    02270         ASSERT ((ULONG)(TempOffset.QuadPart - StartingOffset.QuadPart)
    02271                 > (ReadSize - PAGE_SIZE));
    02272
    02273         ReadSize = (ULONG)(TempOffset.QuadPart - StartingOffset.QuadPart);
    02274
    02275         PageFrameIndex = MiRemoveZeroPage (PageColor);
    02276
    02277     } else {
    02278
    02279         //
    02280         // We are reading a complete page, no need to get a zeroed page.
    02281         //
    02282
    02283         PageFrameIndex = MiRemoveAnyPage (PageColor);
    02284     }
    02285
    02286     //
    02287     // Increment the PFN reference count in the control area for
    02288     // the subsection (PFN MUTEX is required to modify this field).
    02289     //
    02290
    02291     Subsection->ControlArea->NumberOfPfnReferences += 1;
    02292     *Page = PageFrameIndex;
    02293
    02294     PageFrameIndex = *(FirstMdlPage + (PointerPte - BasePte));
    02295
    02296     //
    02297     // Get a page and put the PTE into the transition state with the
    02298     // read-in-progress flag set.
    02299     //
    02300
    02301     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    02302
    02303     //
    02304     // Initialize MDL for request.
    02305     //
    02306
    02307     MmInitializeMdl(Mdl,
    02308                     MiGetVirtualAddressMappedByPte (BasePte),
    02309                     ReadSize);
    02310     Mdl->MdlFlags |= (MDL_PAGES_LOCKED | MDL_IO_PAGE_READ);
    02311
    02312 #if DBG
    02313     if (ReadSize > ((ClusterSize + 1) << PAGE_SHIFT)) {
    02314         KeBugCheckEx (MEMORY_MANAGEMENT, 0x777,(ULONG_PTR)Mdl, (ULONG_PTR)Subsection,
    02315                         (ULONG)TempOffset.LowPart);
    02316     }
    02317 #endif //DBG
    02318
    02319     MiInitializeReadInProgressPfn (Mdl,
    02320                                    BasePte,
    02321                                    &ReadBlockLocal->Event,
    02322                                    MI_PROTOTYPE_WSINDEX);
    02323
    02324     MI_ZERO_USED_PAGETABLE_ENTRIES_IN_INPAGE_SUPPORT(ReadBlockLocal);
    02325
    02326     ReadBlockLocal->ReadOffset = StartingOffset;
    02327     ReadBlockLocal->FilePointer = Subsection->ControlArea->FilePointer;
    02328     ReadBlockLocal->BasePte = BasePte;
    02329     ReadBlockLocal->Pfn = Pfn1;
    02330
    02331     return STATUS_ISSUE_PAGING_IO;
    02332 }

NTSTATUS MiResolvePageFileFault | ( | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte,
|  | OUT PMMINPAGE_SUPPORT * | ReadBlock,
|  | IN PEPROCESS | Process
| ) |

Definition at line 1272 of file pagfault.c.

References APC_LEVEL, ASSERT, DbgPrint, _MMPAGING_FILE::File, _MMINPAGE_SUPPORT::FilePointer, GET_PAGING_FILE_NUMBER, GET_PAGING_FILE_OFFSET, HYDRA_PROCESS, MDL_IO_PAGE_READ, MDL_PAGES_LOCKED, MI_GET_PAGE_COLOR_FROM_SESSION, MI_GET_PAGE_COLOR_FROM_VA, MI_IS_PAGE_TABLE_ADDRESS, MI_PAGE_COLOR_VA_PROCESS, MI_PFN_ELEMENT, MI_PROTOTYPE_WSINDEX, MI_RETRIEVE_USED_PAGETABLE_ENTRIES_FROM_PTE, MiEnsureAvailablePageOrWait(), MiGetInPageSupportBlock(), MiGetPteAddress, MiInitializeReadInProgressPfn(), MiRemoveAnyPage(), MM_PFN_LOCK_ASSERT, MmInfoCounters, MmInitializeMdl, MmPagingFile, MmPagingFileDebug, MmSessionSpace, NULL, PAGE_ALIGN, PAGE_SHIFT, PAGE_SIZE, _MMINFO_COUNTERS::PageReadCount, _MMINFO_COUNTERS::PageReadIoCount, PsGetCurrentThread, _MMPAGING_FILE::Size, STATUS_ISSUE_PAGING_IO, STATUS_REFAULT, _MMPTE::u, and UNLOCK_PFN.

Referenced by MiDispatchFault(), and MiResolveProtoPteFault().

    01281                    :
    01282
    01283     This routine builds the MDL and other structures to allow a
    01284     read operation on a page file for a page fault.
    01285
    01286 Arguments:
    01287
    01288     FaultingAddress - Supplies the faulting address.
    01289
    01290     PointerPte - Supplies the PTE for the faulting address.
    01291
    01292     ReadBlock - Supplies a pointer to put the address of the read block which
    01293                 needs to be completed before an I/O can be issued.
    01294
    01295     Process - Supplies a pointer to the process object.  If this
    01296               parameter is NULL, then the fault is for system
    01297               space and the process's working set lock is not held.
    01298
    01299 Return Value:
    01300
    01301     status.  A status value of STATUS_ISSUE_PAGING_IO is returned
    01302     if this function completes successfully.
    01303
    01304 Environment:
    01305
    01306     Kernel mode, PFN lock held.
    01307
    01308 --*/
    01309
    01310 {
    01311     LARGE_INTEGER StartingOffset;
    01312     PFN_NUMBER PageFrameIndex;
    01313     ULONG PageFileNumber;
    01314     ULONG WorkingSetIndex;
    01315     ULONG PageColor;
    01316     MMPTE TempPte;
    01317     PETHREAD CurrentThread;
    01318     PMMINPAGE_SUPPORT ReadBlockLocal;
    01319
    01320     // **************************************************
    01321     //    Page File Read
    01322     // **************************************************
    01323
    01324     //
    01325     // Calculate the VBN for the in-page operation.
    01326     //
    01327
    01328     CurrentThread = PsGetCurrentThread();
    01329     TempPte = *PointerPte;
    01330
    01331     if (TempPte.u.Hard.Valid == 1) {
    01332         UNLOCK_PFN (APC_LEVEL);
    01333         return STATUS_REFAULT;
    01334     }
    01335
    01336     ASSERT (TempPte.u.Soft.Prototype == 0);
    01337     ASSERT (TempPte.u.Soft.Transition == 0);
    01338
    01339     PageFileNumber = GET_PAGING_FILE_NUMBER (TempPte);
    01340     StartingOffset.LowPart = GET_PAGING_FILE_OFFSET (TempPte);
    01341
    01342     ASSERT (StartingOffset.LowPart <= MmPagingFile[PageFileNumber]->Size);
    01343
    01344     StartingOffset.HighPart = 0;
    01345     StartingOffset.QuadPart = StartingOffset.QuadPart << PAGE_SHIFT;
    01346
    01347     MM_PFN_LOCK_ASSERT();
    01348     if (MiEnsureAvailablePageOrWait (Process,
    01349                                      FaultingAddress)) {
    01350
    01351         //
    01352         // A wait operation was performed which dropped the locks,
    01353         // repeat this fault.
    01354         //
    01355
    01356         UNLOCK_PFN (APC_LEVEL);
    01357         return STATUS_REFAULT;
    01358     }
    01359
    01360     ReadBlockLocal = MiGetInPageSupportBlock ();
    01361     if (ReadBlockLocal == NULL) {
    01362         UNLOCK_PFN (APC_LEVEL);
    01363         return STATUS_REFAULT;
    01364     }
    01365     MmInfoCounters.PageReadCount += 1;
    01366     MmInfoCounters.PageReadIoCount += 1;
    01367
    01368     *ReadBlock = ReadBlockLocal;
    01369
    01370     //fixfix can any of this be moved to after PFN lock released?
    01371
    01372     ReadBlockLocal->FilePointer = MmPagingFile[PageFileNumber]->File;
    01373
    01374 #if DBG
    01375
    01376     if (((StartingOffset.QuadPart >> PAGE_SHIFT) < 8192) && (PageFileNumber == 0)) {
    01377         if ((MmPagingFileDebug[StartingOffset.QuadPart >> PAGE_SHIFT] & ~0x1f) !=
    01378                ((ULONG_PTR)PointerPte << 3)) {
    01379             if ((MmPagingFileDebug[StartingOffset.QuadPart >> PAGE_SHIFT] & ~0x1f) !=
    01380                   ((ULONG_PTR)(MiGetPteAddress(FaultingAddress)) << 3)) {
    01381
    01382                 DbgPrint("MMINPAGE: Mismatch PointerPte %p Offset %I64X info %p\n",
    01383                          PointerPte,
    01384                          StartingOffset.QuadPart >> PAGE_SHIFT,
    01385                          MmPagingFileDebug[StartingOffset.QuadPart >> PAGE_SHIFT]);
    01386
    01387                 DbgBreakPoint();
    01388             }
    01389         }
    01390     }
    01391
    01392 #endif //DBG
    01393
    01394     ReadBlockLocal->ReadOffset = StartingOffset;
    01395
    01396     //
    01397     // Get a page and put the PTE into the transition state with the
    01398     // read-in-progress flag set.
    01399     //
    01400
    01401     if (Process == HYDRA_PROCESS) {
    01402         PageColor = MI_GET_PAGE_COLOR_FROM_SESSION (MmSessionSpace);
    01403     }
    01404     else if (Process == NULL) {
    01405         PageColor = MI_GET_PAGE_COLOR_FROM_VA(FaultingAddress);
    01406     }
    01407     else {
    01408         PageColor = MI_PAGE_COLOR_VA_PROCESS (FaultingAddress,
    01409                                               &Process->NextPageColor);
    01410     }
    01411
    01412     ReadBlockLocal->BasePte = PointerPte;
    01413
    01414     //
    01415     // Build MDL for request.
    01416     //
    01417
    01418     MmInitializeMdl(&ReadBlockLocal->Mdl, PAGE_ALIGN(FaultingAddress), PAGE_SIZE);
    01419     ReadBlockLocal->Mdl.MdlFlags |= (MDL_PAGES_LOCKED | MDL_IO_PAGE_READ);
    01420
    01421     if (MI_IS_PAGE_TABLE_ADDRESS(PointerPte)) {
    01422         WorkingSetIndex = 1;
    01423     } else {
    01424         WorkingSetIndex = MI_PROTOTYPE_WSINDEX;
    01425     }
    01426
    01427     PageFrameIndex = MiRemoveAnyPage (PageColor);
    01428
    01429     ReadBlockLocal->Pfn = MI_PFN_ELEMENT (PageFrameIndex);
    01430     ReadBlockLocal->Page[0] = PageFrameIndex;
    01431
    01432     MiInitializeReadInProgressPfn (
    01433                            &ReadBlockLocal->Mdl,
    01434                            PointerPte,
    01435                            &ReadBlockLocal->Event,
    01436                            WorkingSetIndex);
    01437
    01438     MI_RETRIEVE_USED_PAGETABLE_ENTRIES_FROM_PTE (ReadBlockLocal, &TempPte);
    01439
    01440     UNLOCK_PFN (APC_LEVEL);
    01441
    01442     return STATUS_ISSUE_PAGING_IO;
    01443 }

NTSTATUS MiResolveProtoPteFault | ( | IN BOOLEAN | StoreInstruction,
|  | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PMMPTE | PointerProtoPte,
|  | OUT PMMINPAGE_SUPPORT * | ReadBlock,
|  | IN PEPROCESS | Process,
|  | OUT PLOGICAL | ApcNeeded
| ) |

Definition at line 1446 of file pagfault.c.

References APC_LEVEL, ASSERT, DbgPrint, FALSE, IS_PTE_NOT_DEMAND_ZERO, KeDelayExecutionThread(), KernelMode, MI_GET_PAGE_FRAME_FROM_PTE, MI_GET_PROTECTION_FROM_SOFT_PTE, MI_PFN_ELEMENT, MI_PTE_LOOKUP_NEEDED, MiAccessCheck(), MiCompleteProtoPteFault(), MiFormatPte(), MiGetSubsectionAddress, MiResolveDemandZeroFault(), MiResolveMappedFileFault(), MiResolvePageFileFault(), MiResolveTransitionFault(), MM_COPY_ON_WRITE_MASK, MM_DBG_PTE_UPDATE, MM_DBG_STOP_ON_ACCVIO, MM_DEMAND_ZERO_WRITE_PTE, MM_PFN_LOCK_ASSERT, MmInfoCounters, MmIsRetryIoStatus, MmShortTime, NT_SUCCESS, NTSTATUS(), NULL, PERFINFO_SOFTFAULT, STATUS_REFAULT, _MMINFO_COUNTERS::TransitionCount, TRUE, _SUBSECTION::u, _MMPTE::u, _MMPFN::u2, and UNLOCK_PFN.

Referenced by MiDispatchFault().

    01458                    :
    01459
    01460     This routine resolves a prototype PTE fault.
    01461
    01462 Arguments:
    01463
    01464     StoreInstruction - Supplies TRUE if the instruction is trying
    01465                        to modify the faulting address (i.e. write
    01466                        access required).
    01467
    01468     FaultingAddress - Supplies the faulting address.
    01469
    01470     PointerPte - Supplies the PTE for the faulting address.
    01471
    01472     PointerProtoPte - Supplies a pointer to the prototype PTE to fault in.
    01473
    01474     ReadBlock - Supplies a pointer to put the address of the read block which
    01475                 needs to be completed before an I/O can be issued.
    01476
    01477     Process - Supplies a pointer to the process object.  If this
    01478               parameter is NULL, then the fault is for system
    01479               space and the process's working set lock is not held.
    01480
    01481     ApcNeeded - Supplies a pointer to a location set to TRUE if an I/O
    01482                 completion APC is needed to complete partial IRPs that
    01483                 collided.
    01484
    01485 Return Value:
    01486
    01487     status, either STATUS_SUCCESS, STATUS_REFAULT, or an I/O status
    01488     code.
    01489
    01490 Environment:
    01491
    01492     Kernel mode, PFN lock held.
    01493
    01494 --*/
    01495 {
    01496     MMPTE TempPte;
    01497     PFN_NUMBER PageFrameIndex;
    01498     PMMPFN Pfn1;
    01499     NTSTATUS status;
    01500     ULONG CopyOnWrite;
    01501     LOGICAL PfnHeld;
    01502
    01503     PfnHeld = FALSE;
    01504
    01505     //
    01506     // Acquire the PFN database mutex as the routine to locate a working
    01507     // set entry decrements the share count of PFN elements.
    01508     //
    01509
    01510     MM_PFN_LOCK_ASSERT();
    01511
    01512 #if DBG
    01513     if (MmDebug & MM_DBG_PTE_UPDATE) {
    01514         DbgPrint("MM:actual fault %p va %p\n",PointerPte, FaultingAddress);
    01515         MiFormatPte(PointerPte);
    01516     }
    01517 #endif //DBG
    01518
    01519     ASSERT (PointerPte->u.Soft.Prototype == 1);
    01520     TempPte = *PointerProtoPte;
    01521
    01522     //
    01523     // The page containing the prototype PTE is resident,
    01524     // handle the fault referring to the prototype PTE.
    01525     // If the prototype PTE is already valid, make this
    01526     // PTE valid and up the share count etc.
    01527     //
    01528
    01529     if (TempPte.u.Hard.Valid) {
    01530
    01531         //
    01532         // Prototype PTE is valid.
    01533         //
    01534
    01535         PageFrameIndex = MI_GET_PAGE_FRAME_FROM_PTE (&TempPte);
    01536         Pfn1 = MI_PFN_ELEMENT(PageFrameIndex);
    01537         Pfn1->u2.ShareCount += 1;
    01538         status = STATUS_SUCCESS;
    01539
    01540         //
    01541         // Count this as a transition fault.
    01542         //
    01543
    01544         MmInfoCounters.TransitionCount += 1;
    01545         PfnHeld = TRUE;
    01546
    01547         PERFINFO_SOFTFAULT(Pfn1, FaultingAddress, PERFINFO_LOG_TYPE_ADDVALIDPAGETOWS)
    01548
    01549     } else {
    01550
    01551         //
    01552         // Check to make sure the prototype PTE is committed.
    01553         //
    01554
    01555         if (TempPte.u.Long == 0) {
    01556
    01557 #if DBG
    01558             if (MmDebug & MM_DBG_STOP_ON_ACCVIO) {
    01559                 DbgPrint("MM:access vio2 - %p\n",FaultingAddress);
    01560                 MiFormatPte(PointerPte);
    01561                 DbgBreakPoint();
    01562             }
    01563 #endif //DEBUG
    01564
    01565             UNLOCK_PFN (APC_LEVEL);
    01566             return STATUS_ACCESS_VIOLATION;
    01567         }
    01568
    01569         //
    01570         // If the PTE indicates that the protection field to be
    01571         // checked is in the prototype PTE, check it now.
    01572         //
    01573
    01574         CopyOnWrite = FALSE;
    01575
    01576         if (PointerPte->u.Soft.PageFileHigh != MI_PTE_LOOKUP_NEEDED) {
    01577             if (PointerPte->u.Proto.ReadOnly == 0) {
    01578
    01579                 //
    01580                 // Check for kernel mode access, we have already verified
    01581                 // that the user has access to the virtual address.
    01582                 //
    01583
    01584 #if 0 // removed this assert since mapping drivers via MmMapViewInSystemSpace
    01585       // file violates the assert.
    01586
    01587                 {
    01588                     PSUBSECTION Sub;
    01589                     if (PointerProtoPte->u.Soft.Prototype == 1) {
    01590                         Sub = MiGetSubsectionAddress (PointerProtoPte);
    01591                         ASSERT (Sub->u.SubsectionFlags.Protection ==
    01592                                     PointerProtoPte->u.Soft.Protection);
    01593                     }
    01594                 }
    01595
    01596 #endif //DBG
    01597
    01598                 status = MiAccessCheck (PointerProtoPte,
    01599                                         StoreInstruction,
    01600                                         KernelMode,
    01601                                         MI_GET_PROTECTION_FROM_SOFT_PTE (PointerProtoPte),
    01602                                         TRUE);
    01603
    01604                 if (status != STATUS_SUCCESS) {
    01605 #if DBG
    01606                     if (MmDebug & MM_DBG_STOP_ON_ACCVIO) {
    01607                         DbgPrint("MM:access vio3 - %p\n",FaultingAddress);
    01608                         MiFormatPte(PointerPte);
    01609                         MiFormatPte(PointerProtoPte);
    01610                         DbgBreakPoint();
    01611                     }
    01612 #endif
    01613                     UNLOCK_PFN (APC_LEVEL);
    01614                     return status;
    01615                 }
    01616                 if ((PointerProtoPte->u.Soft.Protection & MM_COPY_ON_WRITE_MASK) ==
    01617                      MM_COPY_ON_WRITE_MASK) {
    01618                     CopyOnWrite = TRUE;
    01619                 }
    01620             }
    01621         } else {
    01622             if ((PointerPte->u.Soft.Protection & MM_COPY_ON_WRITE_MASK) ==
    01623                  MM_COPY_ON_WRITE_MASK) {
    01624                 CopyOnWrite = TRUE;
    01625             }
    01626         }
    01627
    01628         if ((!IS_PTE_NOT_DEMAND_ZERO(TempPte)) && (CopyOnWrite)) {
    01629
    01630             //
    01631             // The prototype PTE is demand zero and copy on
    01632             // write.  Make this PTE a private demand zero PTE.
    01633             //
    01634
    01635             ASSERT (Process != NULL);
    01636
    01637             PointerPte->u.Long = MM_DEMAND_ZERO_WRITE_PTE;
    01638
    01639             UNLOCK_PFN (APC_LEVEL);
    01640
    01641             status = MiResolveDemandZeroFault (FaultingAddress,
    01642                                                PointerPte,
    01643                                                Process,
    01644                                                FALSE);
    01645             return status;
    01646         }
    01647
    01648         //
    01649         // Make the prototype PTE valid, the prototype PTE is in
    01650         // one of 4 case:
    01651         //   demand zero
    01652         //   transition
    01653         //   paging file
    01654         //   mapped file
    01655         //
    01656
    01657         if (TempPte.u.Soft.Prototype == 1) {
    01658
    01659             //
    01660             // Mapped File.
    01661             //
    01662
    01663             status = MiResolveMappedFileFault (FaultingAddress,
    01664                                                PointerProtoPte,
    01665                                                ReadBlock,
    01666                                                Process);
    01667
    01668             //
    01669             // Returns with PFN lock held.
    01670             //
    01671
    01672             PfnHeld = TRUE;
    01673
    01674         } else if (TempPte.u.Soft.Transition == 1) {
    01675
    01676             //
    01677             // Transition.
    01678             //
    01679
    01680             status = MiResolveTransitionFault (FaultingAddress,
    01681                                               PointerProtoPte,
    01682                                               Process,
    01683                                               TRUE,
    01684                                               ApcNeeded);
    01685
    01686             //
    01687             // Returns with PFN lock held.
    01688             //
    01689
    01690             PfnHeld = TRUE;
    01691
    01692         } else if (TempPte.u.Soft.PageFileHigh == 0) {
    01693
    01694             //
    01695             // Demand Zero
    01696             //
    01697
    01698             status = MiResolveDemandZeroFault (FaultingAddress,
    01699                                                PointerProtoPte,
    01700                                                Process,
    01701                                                TRUE);
    01702
    01703             //
    01704             // Returns with PFN lock held!
    01705             //
    01706
    01707             PfnHeld = TRUE;
    01708
    01709         } else {
    01710
    01711             //
    01712             // Paging file.
    01713             //
    01714
    01715             status = MiResolvePageFileFault (FaultingAddress,
    01716                                              PointerProtoPte,
    01717                                              ReadBlock,
    01718                                              Process);
    01719
    01720             //
    01721             // Returns with PFN lock released.
    01722             //
    01723             ASSERT (KeGetCurrentIrql() == APC_LEVEL);
    01724         }
    01725     }
    01726
    01727     if (NT_SUCCESS(status)) {
    01728
    01729         ASSERT (PointerPte->u.Hard.Valid == 0);
    01730
    01731         MiCompleteProtoPteFault (StoreInstruction,
    01732                                  FaultingAddress,
    01733                                  PointerPte,
    01734                                  PointerProtoPte);
    01735
    01736     } else {
    01737
    01738         if (PfnHeld) {
    01739             UNLOCK_PFN (APC_LEVEL);
    01740         }
    01741
    01742         ASSERT (KeGetCurrentIrql() == APC_LEVEL);
    01743
    01744         //
    01745         // Stop high priority threads from consuming the CPU on collided
    01746         // faults for pages that are still marked with inpage errors.  All
    01747         // the threads must let go of the page so it can be freed and the
    01748         // inpage I/O reissued to the filesystem.
    01749         //
    01750
    01751         if (MmIsRetryIoStatus(status)) {
    01752             KeDelayExecutionThread (KernelMode, FALSE, (PLARGE_INTEGER)&MmShortTime);
    01753             status = STATUS_REFAULT;
    01754         }
    01755     }
    01756
    01757     return status;
    01758 }

NTSTATUS MiResolveTransitionFault | ( | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte,
|  | IN PEPROCESS | CurrentProcess,
|  | IN ULONG | PfnLockHeld,
|  | OUT PLOGICAL | ApcNeeded
| ) |

Definition at line 868 of file pagfault.c.

References ActiveAndValid, APC_LEVEL, _ETHREAD::ApcNeeded, ASSERT, DbgPrint, FreePageList, HYDRA_PROCESS, KeEnterCriticalRegion, KeLeaveCriticalRegion, LOCK_PFN, MI_ADD_LOCKED_PAGE_CHARGE_FOR_MODIFIED_PAGE, MI_GET_PAGE_FRAME_FROM_TRANSITION_PTE, MI_MAKE_TRANSITION_PTE_VALID, MI_PFN_ELEMENT, MI_REMOVE_LOCKED_PAGE_CHARGE, MI_SET_PTE_CLEAN, MI_SET_PTE_DIRTY, MI_WRITE_VALID_PTE, MiAddValidPageToWorkingSet(), MiDecrementReferenceCount(), MiFormatPfn(), MiGetPteAddress, MiInsertPageInList(), MiRestoreTransitionPte(), MiUnlinkPageFromList(), MiWaitForInPageComplete(), MM_DBG_COLLIDED_PAGE, MmInfoCounters, MmPagedPoolEnd, MmPagedPoolStart, MmPageLocationList, MmSpecialPoolEnd, MmSpecialPoolStart, _ETHREAD::NestedFaultCount, NT_SUCCESS, NTSTATUS(), NULL, PERFINFO_SOFTFAULT, PsGetCurrentThread, _MMPFN::PteAddress, StandbyPageList, STATUS_REFAULT, _MMINFO_COUNTERS::TransitionCount, TRUE, _MMINPAGE_SUPPORT::u, _MMPTE::u, _MMPFN::u1, _MMPFN::u2, _MMPFN::u3, UNLOCK_PFN, UNLOCK_SESSION_SPACE_WS, UNLOCK_SYSTEM_WS, UNLOCK_WS, and _MMINPAGE_SUPPORT::WaitCount.

Referenced by MiDispatchFault(), and MiResolveProtoPteFault().

    00878                    :
    00879
    00880     This routine resolves a transition page fault.
    00881
    00882 Arguments:
    00883
    00884     FaultingAddress - Supplies the faulting address.
    00885
    00886     PointerPte - Supplies the PTE for the faulting address.
    00887
    00888     CurrentProcess - Supplies a pointer to the process object.  If this
    00889                      parameter is NULL, then the fault is for system
    00890                      space and the process's working set lock is not held.
    00891
    00892     PfnLockHeld - Supplies TRUE if the PFN lock is held, FALSE if not.
    00893
    00894     ApcNeeded - Supplies a pointer to a location set to TRUE if an I/O
    00895                 completion APC is needed to complete partial IRPs that
    00896                 collided.
    00897
    00898                 It is the caller's responsibility to initialize this (usually
    00899                 to FALSE) on entry.  However, since this routine may be called
    00900                 multiple times for a single fault (for the page directory,
    00901                 page table and the page itself), it is possible for it to
    00902                 occasionally be TRUE on entry.
    00903
    00904                 If it is FALSE on exit, no completion APC is needed.
    00905
    00906 Return Value:
    00907
    00908     status, either STATUS_SUCCESS, STATUS_REFAULT or an I/O status
    00909     code.
    00910
    00911 Environment:
    00912
    00913     Kernel mode, PFN lock may optionally be held.
    00914
    00915 --*/
    00916
    00917 {
    00918     PFN_NUMBER PageFrameIndex;
    00919     PMMPFN Pfn1;
    00920     MMPTE TempPte;
    00921     NTSTATUS status;
    00922     NTSTATUS PfnStatus;
    00923     PMMINPAGE_SUPPORT CapturedEvent;
    00924     KIRQL OldIrql;
    00925     PETHREAD CurrentThread;
    00926
    00927     //
    00928     // ***********************************************************
    00929     //      Transition PTE.
    00930     // ***********************************************************
    00931     //
    00932
    00933     //
    00934     // A transition PTE is either on the free or modified list,
    00935     // on neither list because of its ReferenceCount
    00936     // or currently being read in from the disk (read in progress).
    00937     // If the page is read in progress, this is a collided page
    00938     // and must be handled accordingly.
    00939     //
    00940
    00941     if (!PfnLockHeld) {
    00942         LOCK_PFN (OldIrql);
    00943     }
    00944
    00945     TempPte = *PointerPte;
    00946
    00947     if ((TempPte.u.Soft.Valid == 0) &&
    00948         (TempPte.u.Soft.Prototype == 0) &&
    00949         (TempPte.u.Soft.Transition == 1)) {
    00950
    00951         //
    00952         // Still in transition format.
    00953         //
    00954
    00955         MmInfoCounters.TransitionCount += 1;
    00956
    00957         PageFrameIndex = MI_GET_PAGE_FRAME_FROM_TRANSITION_PTE (&TempPte);
    00958         Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    00959
    00960         if (Pfn1->u3.e1.InPageError) {
    00961
    00962             //
    00963             // There was an in-page read error and there are other
    00964             // threads colliding for this page, delay to let the
    00965             // other threads complete and return.
    00966             //
    00967
    00968             ASSERT (!NT_SUCCESS(Pfn1->u1.ReadStatus));
    00969             if (!PfnLockHeld) {
    00970                 UNLOCK_PFN (APC_LEVEL);
    00971             }
    00972
    00973             return Pfn1->u1.ReadStatus;
    00974         }
    00975
    00976         if (Pfn1->u3.e1.ReadInProgress) {
    00977
    00978             //
    00979             // Collided page fault.
    00980             //
    00981
    00982 #if DBG
    00983             if (MmDebug & MM_DBG_COLLIDED_PAGE) {
    00984                 DbgPrint("MM:collided page fault\n");
    00985             }
    00986 #endif
    00987
    00988             CapturedEvent = (PMMINPAGE_SUPPORT)Pfn1->u1.Event;
    00989
    00990             CurrentThread = PsGetCurrentThread();
    00991
    00992             if (CapturedEvent->u.Thread == CurrentThread) {
    00993
    00994                 //
    00995                 // This detects MmCopyToCachedPage deadlocks where both the
    00996                 // user and system address point at the same physical page.
    00997                 //
    00998                 // It also detects when the Io APC completion routine accesses
    00999                 // the same user page (ie: during an overlapped I/O) that
    01000                 // the user thread has already faulted on.
    01001                 //
    01002                 // Both cases above can result in fatal deadlocks and so must
    01003                 // be detected here.  Return a unique status code so the
    01004                 // (legitimate) callers know this has happened so it can be
    01005                 // handled properly.  In the first case above this means
    01006                 // restarting the entire operation immediately.  In the second
    01007                 // case above it means requesting a callback from the Mm
    01008                 // once the first fault has completed.
    01009                 //
    01010                 // Note that non-legitimate callers must get back a failure
    01011                 // status so the thread can be terminated.
    01012                 //
    01013
    01014                 ASSERT ((CurrentThread->NestedFaultCount == 1) ||
    01015                         (CurrentThread->NestedFaultCount == 2));
    01016
    01017                 CurrentThread->ApcNeeded = 1;
    01018
    01019                 if (!PfnLockHeld) {
    01020                     UNLOCK_PFN (APC_LEVEL);
    01021                 }
    01022                 return STATUS_MULTIPLE_FAULT_VIOLATION;
    01023             }
    01024
    01025             //
    01026             // Increment the reference count for the page so it won't be
    01027             // reused until all collisions have been completed.
    01028             //
    01029
    01030             ASSERT (Pfn1->u2.ShareCount == 0);
    01031             ASSERT (Pfn1->u3.e2.ReferenceCount != 0);
    01032             ASSERT (Pfn1->u3.e1.LockCharged == 1);
    01033
    01034             Pfn1->u3.e2.ReferenceCount += 1;
    01035
    01036             CapturedEvent->WaitCount += 1;
    01037
    01038             UNLOCK_PFN (APC_LEVEL);
    01039
    01040             if (CurrentProcess == HYDRA_PROCESS) {
    01041                 CurrentThread = NULL;
    01042                 UNLOCK_SESSION_SPACE_WS (APC_LEVEL);
    01043             }
    01044             else if (CurrentProcess != NULL) {
    01045
    01046                 //
    01047                 // APCs must be explicitly disabled to prevent suspend APCs from
    01048                 // interrupting this thread before the wait has been issued.
    01049                 // Otherwise the APC can result in this page being locked
    01050                 // indefinitely until the suspend is released.
    01051                 //
    01052
    01053                 ASSERT (CurrentThread->NestedFaultCount <= 2);
    01054                 CurrentThread->NestedFaultCount += 1;
    01055
    01056                 KeEnterCriticalRegion();
    01057                 UNLOCK_WS (CurrentProcess);
    01058             }
    01059             else {
    01060                 CurrentThread = NULL;
    01061                 UNLOCK_SYSTEM_WS (APC_LEVEL);
    01062             }
    01063
    01064             status = MiWaitForInPageComplete (Pfn1,
    01065                                               PointerPte,
    01066                                               FaultingAddress,
    01067                                               &TempPte,
    01068                                               CapturedEvent,
    01069                                               CurrentProcess);
    01070             //
    01071             // MiWaitForInPageComplete RETURNS WITH THE WORKING SET LOCK
    01072             // AND PFN LOCK HELD!!!
    01073             //
    01074
    01075             if (CurrentThread != NULL) {
    01076                 KeLeaveCriticalRegion();
    01077
    01078                 ASSERT (CurrentThread->NestedFaultCount <= 3);
    01079                 ASSERT (CurrentThread->NestedFaultCount != 0);
    01080
    01081                 CurrentThread->NestedFaultCount -= 1;
    01082
    01083                 if ((CurrentThread->ApcNeeded == 1) &&
    01084                     (CurrentThread->NestedFaultCount == 0)) {
    01085                     *ApcNeeded = TRUE;
    01086                     CurrentThread->ApcNeeded = 0;
    01087                 }
    01088             }
    01089
    01090             ASSERT (Pfn1->u3.e1.ReadInProgress == 0);
    01091
    01092             if (status != STATUS_SUCCESS) {
    01093                 PfnStatus = Pfn1->u1.ReadStatus;
    01094                 MI_REMOVE_LOCKED_PAGE_CHARGE(Pfn1, 9);
    01095                 MiDecrementReferenceCount (PageFrameIndex);
    01096
    01097                 //
    01098                 // Check to see if an I/O error occurred on this page.
    01099                 // If so, try to free the physical page, wait a
    01100                 // half second and return a status of PTE_CHANGED.
    01101                 // This will result in success being returned to
    01102                 // the user and the fault will occur again and should
    01103                 // not be a transition fault this time.
    01104                 //
    01105
    01106                 if (Pfn1->u3.e1.InPageError == 1) {
    01107                     ASSERT (!NT_SUCCESS(PfnStatus));
    01108                     status = PfnStatus;
    01109                     if (Pfn1->u3.e2.ReferenceCount == 0) {
    01110
    01111                         Pfn1->u3.e1.InPageError = 0;
    01112
    01113                         //
    01114                         // Only restore the transition PTE if the address
    01115                         // space still exists.  Another thread may have
    01116                         // deleted the VAD while this thread waited for the
    01117                         // fault to complete - in this case, the frame
    01118                         // will be marked as free already.
    01119                         //
    01120
    01121                         if (Pfn1->u3.e1.PageLocation != FreePageList) {
    01122                             ASSERT (Pfn1->u3.e1.PageLocation ==
    01123                                                             StandbyPageList);
    01124                             MiUnlinkPageFromList (Pfn1);
    01125                             MiRestoreTransitionPte (PageFrameIndex);
    01126
    01127                             MiInsertPageInList(MmPageLocationList[FreePageList],
    01128                                                PageFrameIndex);
    01129                         }
    01130                     }
    01131                 }
    01132
    01133 #if DBG
    01134                 if (MmDebug & MM_DBG_COLLIDED_PAGE) {
    01135                     DbgPrint("MM:decrement ref count - pte changed\n");
    01136                     MiFormatPfn(Pfn1);
    01137                 }
    01138 #endif
    01139                 if (!PfnLockHeld) {
    01140                     UNLOCK_PFN (APC_LEVEL);
    01141                 }
    01142
    01143                 //
    01144                 // Instead of returning status, always return STATUS_REFAULT.
    01145                 // This is to support filesystems that save state in the
    01146                 // ETHREAD of the thread that serviced the fault !  Since
    01147                 // collided threads never enter the filesystem, their ETHREADs
    01148                 // haven't been hacked up.  Since this only matters when
    01149                 // errors occur (specifically STATUS_VERIFY_REQUIRED today),
    01150                 // retry any failed I/O in the context of each collider
    01151                 // to give the filesystems ample opportunity.
    01152                 //
    01153
    01154                 return STATUS_REFAULT;
    01155             }
    01156
    01157         } else {
    01158
    01159             //
    01160             // PTE refers to a normal transition PTE.
    01161             //
    01162
    01163             ASSERT (Pfn1->u3.e1.InPageError == 0);
    01164             if (Pfn1->u3.e1.PageLocation == ActiveAndValid) {
    01165
    01166                 //
    01167                 // This page must contain an MmSt allocation of prototype PTEs.
    01168                 // Because these types of pages reside in paged pool (or special
    01169                 // pool) and are part of the system working set, they can be
    01170                 // trimmed at any time regardless of the share count.  However,
    01171                 // if the share count is nonzero, then the page state will
    01172                 // remain active and the page will remain in memory - but the
    01173                 // PTE will be set to the transition state.  Make the page
    01174                 // valid without incrementing the reference count, but
    01175                 // increment the share count.
    01176                 //
    01177
    01178                 ASSERT (((Pfn1->PteAddress >= MiGetPteAddress(MmPagedPoolStart)) &&
    01179                         (Pfn1->PteAddress <= MiGetPteAddress(MmPagedPoolEnd))) ||
    01180                         ((Pfn1->PteAddress >= MiGetPteAddress(MmSpecialPoolStart)) &&
    01181                         (Pfn1->PteAddress <= MiGetPteAddress(MmSpecialPoolEnd))));
    01182
    01183                 //
    01184                 // Don't increment the valid PTE count for the
    01185                 // page table page.
    01186                 //
    01187
    01188                 ASSERT (Pfn1->u2.ShareCount != 0);
    01189                 ASSERT (Pfn1->u3.e2.ReferenceCount != 0);
    01190
    01191             } else {
    01192
    01193                 MiUnlinkPageFromList (Pfn1);
    01194
    01195                 //
    01196                 // Update the PFN database - the reference count must be
    01197                 // incremented as the share count is going to go from zero to 1.
    01198                 //
    01199
    01200                 ASSERT (Pfn1->u2.ShareCount == 0);
    01201
    01202                 //
    01203                 // The PFN reference count will be 1 already here if the
    01204                 // modified writer has begun a write of this page.  Otherwise
    01205                 // it's ordinarily 0.
    01206                 //
    01207
    01208                 MI_ADD_LOCKED_PAGE_CHARGE_FOR_MODIFIED_PAGE (Pfn1, 8);
    01209
    01210                 Pfn1->u3.e2.ReferenceCount += 1;
    01211             }
    01212         }
    01213
    01214         //
    01215         // Join with collided page fault code to handle updating
    01216         // the transition PTE.
    01217         //
    01218
    01219         ASSERT (Pfn1->u3.e1.InPageError == 0);
    01220
    01221         if (Pfn1->u2.ShareCount == 0) {
    01222             MI_REMOVE_LOCKED_PAGE_CHARGE (Pfn1, 9);
    01223         }
    01224
    01225         Pfn1->u2.ShareCount += 1;
    01226         Pfn1->u3.e1.PageLocation = ActiveAndValid;
    01227
    01228         MI_MAKE_TRANSITION_PTE_VALID (TempPte, PointerPte);
    01229
    01230         //
    01231         // If the modified field is set in the PFN database and this
    01232         // page is not copy on modify, then set the dirty bit.
    01233         // This can be done as the modified page will not be
    01234         // written to the paging file until this PTE is made invalid.
    01235         //
    01236
    01237         if (Pfn1->u3.e1.Modified && TempPte.u.Hard.Write &&
    01238                         (TempPte.u.Hard.CopyOnWrite == 0)) {
    01239             MI_SET_PTE_DIRTY (TempPte);
    01240         } else {
    01241             MI_SET_PTE_CLEAN (TempPte);
    01242         }
    01243
    01244         MI_WRITE_VALID_PTE (PointerPte, TempPte);
    01245
    01246         if (!PfnLockHeld) {
    01247
    01248             if (Pfn1->u1.Event == 0) {
    01249                Pfn1->u1.Event = (PVOID)PsGetCurrentThread();
    01250             }
    01251
    01252             UNLOCK_PFN (APC_LEVEL);
    01253
    01254             PERFINFO_SOFTFAULT(Pfn1, FaultingAddress, PERFINFO_LOG_TYPE_TRANSITIONFAULT)
    01255
    01256             MiAddValidPageToWorkingSet (FaultingAddress,
    01257                                         PointerPte,
    01258                                         Pfn1,
    01259                                         0);
    01260         }
    01261         return STATUS_PAGE_FAULT_TRANSITION;
    01262     } else {
    01263         if (!PfnLockHeld) {
    01264             UNLOCK_PFN (APC_LEVEL);
    01265         }
    01266     }
    01267     return STATUS_REFAULT;
    01268 }

NTSTATUS MiSessionCopyOnWrite | ( | IN PMM_SESSION_SPACE | SessionSpace,
|  | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPte
| ) |

Definition at line 4446 of file pagfault.c.

References ASSERT, _MM_SESSION_SPACE::CopyOnWriteCount, _MMINFO_COUNTERS::CopyOnWriteCount, HYDRA_PROCESS, LOCK_PFN, MI_FLUSH_SINGLE_SESSION_TB, MI_GET_PAGE_FRAME_FROM_PTE, MI_GET_SECONDARY_COLOR, MI_IS_PTE_DIRTY, MI_PFN_ELEMENT, MI_SET_ACCESSED_IN_PTE, MI_SET_PTE_DIRTY, MiDecrementShareCount(), MiEnsureAvailablePageOrWait(), MiGetVirtualAddressMappedByPte, MiInitializeCopyOnWritePfn(), MiLocateWsle(), MiMapPageInHyperSpace(), MiRemoveAnyPage(), MiUnmapPageInHyperSpace, MmInfoCounters, MmSessionSpace, NULL, PAGE_SIZE, TRUE, _MMPTE::u, _MMPFN::u1, _MMPFN::u3, UNLOCK_PFN, and WSLE_NUMBER.

Referenced by MmAccessFault().

    04454                    :
    04455
    04456     This function handles copy on write for image mapped session space.
    04457
    04458 Arguments:
    04459
    04460     SessionSpace - Supplies the session space being referenced.
    04461
    04462     FaultingAddress - Supplies the address which caused the page fault.
    04463
    04464     PointerPte - Supplies the pointer to the PTE which caused the page fault.
    04465
    04466 Return Value:
    04467
    04468     STATUS_SUCCESS.
    04469
    04470 Environment:
    04471
    04472     Kernel mode, APCs disabled, session WSL held.
    04473
    04474 --*/
    04475
    04476 {
    04477     MMPTE TempPte;
    04478     MMPTE PreviousPte;
    04479     PFN_NUMBER PageFrameIndex;
    04480     PFN_NUMBER NewPageIndex;
    04481     PULONG CopyTo;
    04482     KIRQL OldIrql;
    04483     PMMPFN Pfn1;
    04484     PVOID VirtualAddress;
    04485     WSLE_NUMBER WorkingSetIndex;
    04486
    04487 #if defined(_IA64_)
    04488     UNREFERENCED_PARAMETER (FaultingAddress);
    04489 #endif
    04490
    04491     PageFrameIndex = MI_GET_PAGE_FRAME_FROM_PTE (PointerPte);
    04492     Pfn1 = MI_PFN_ELEMENT (PageFrameIndex);
    04493
    04494     ASSERT (Pfn1->u3.e1.PrototypePte == 1);
    04495
    04496     //
    04497     // Acquire the PFN mutex.
    04498     //
    04499
    04500     VirtualAddress = MiGetVirtualAddressMappedByPte (PointerPte);
    04501
    04502     WorkingSetIndex = MiLocateWsle (VirtualAddress,
    04503                                     SessionSpace->Vm.VmWorkingSetList,
    04504                                     Pfn1->u1.WsIndex);
    04505
    04506     LOCK_PFN (OldIrql);
    04507
    04508     //
    04509     // The page must be copied into a new page.
    04510     //
    04511
    04512     if (MiEnsureAvailablePageOrWait(HYDRA_PROCESS, NULL)) {
    04513
    04514         //
    04515         // A wait operation was performed to obtain an available
    04516         // page and the working set mutex and PFN mutexes have
    04517         // been released and various things may have changed for
    04518         // the worse.  Rather than examine all the conditions again,
    04519         // return and if things are still proper, the fault will
    04520         // be taken again.
    04521         //
    04522
    04523         UNLOCK_PFN (OldIrql);
    04524         return STATUS_SUCCESS;
    04525     }
    04526
    04527     //
    04528     // Verify that the page did not go into transition while the
    04529     // PFN lock was released.  If it changed state, refault it in.
    04530     //
    04531
    04532     TempPte = *(volatile MMPTE *)PointerPte;
    04533
    04534     if (!(TempPte.u.Hard.Valid && TempPte.u.Hard.Write == 0)) {
    04535         UNLOCK_PFN (OldIrql);
    04536         return STATUS_SUCCESS;
    04537     }
    04538
    04539     //
    04540     // Increment the number of private pages.
    04541     //
    04542
    04543     MmInfoCounters.CopyOnWriteCount += 1;
    04544
    04545     MmSessionSpace->CopyOnWriteCount += 1;
    04546
    04547     //
    04548     // A page is being copied and made private, the global state of
    04549     // the shared page does not need to be updated at this point because
    04550     // it is guaranteed to be clean - no POSIX-style forking is allowed on
    04551     // session addresses.
    04552     //
    04553
    04554     ASSERT (Pfn1->u3.e1.Modified == 0);
    04555     ASSERT (!MI_IS_PTE_DIRTY(*PointerPte));
    04556
    04557     //
    04558     // Get a new page with the same color as this page.
    04559     //
    04560
    04561     NewPageIndex = MiRemoveAnyPage (MI_GET_SECONDARY_COLOR (PageFrameIndex,
    04562                                                             Pfn1));
    04563
    04564     MiInitializeCopyOnWritePfn (NewPageIndex,
    04565                                 PointerPte,
    04566                                 WorkingSetIndex,
    04567                                 SessionSpace);
    04568
    04569     UNLOCK_PFN (OldIrql);
    04570
    04571     //
    04572     // Copy the accessed readonly page into the newly allocated writable page.
    04573     //
    04574
    04575     CopyTo = (PULONG)MiMapPageInHyperSpace (NewPageIndex, &OldIrql);
    04576
    04577     RtlCopyMemory (CopyTo, VirtualAddress, PAGE_SIZE);
    04578
    04579     MiUnmapPageInHyperSpace (OldIrql);
    04580
    04581     //
    04582     // Since the page was a copy on write page, make it
    04583     // accessed, dirty and writable.  Also clear the copy-on-write
    04584     // bit in the PTE.
    04585     //
    04586
    04587     MI_SET_PTE_DIRTY (TempPte);
    04588     TempPte.u.Hard.Write = 1;
    04589     MI_SET_ACCESSED_IN_PTE (&TempPte, 1);
    04590     TempPte.u.Hard.CopyOnWrite = 0;
    04591     TempPte.u.Hard.PageFrameNumber = NewPageIndex;
    04592
    04593     //
    04594     // If the modify bit is set in the PFN database for the
    04595     // page, the data cache must be flushed.  This is due to the
    04596     // fact that this process may have been cloned and the cache
    04597     // still contains stale data destined for the page we are
    04598     // going to remove.
    04599     //
    04600
    04601     ASSERT (TempPte.u.Hard.Valid == 1);
    04602
    04603     LOCK_PFN (OldIrql);
    04604
    04605     //
    04606     // Flush the TB entry for this page.
    04607     //
    04608
    04609     MI_FLUSH_SINGLE_SESSION_TB (FaultingAddress,
    04610                                 TRUE,
    04611                                 TRUE,
    04612                                 (PHARDWARE_PTE)PointerPte,
    04613                                 TempPte.u.Flush,
    04614                                 PreviousPte);
    04615
    04616     ASSERT (Pfn1->u3.e1.PrototypePte == 1);
    04617
    04618     //
    04619     // Decrement the share count for the page which was copied
    04620     // as this PTE no longer refers to it.
    04621     //
    04622
    04623     MiDecrementShareCount (PageFrameIndex);
    04624
    04625     UNLOCK_PFN (OldIrql);
    04626     return STATUS_SUCCESS;
    04627 }

NTSTATUS MiWaitForInPageComplete | ( | IN PMMPFN | Pfn2,
|  | IN PMMPTE | PointerPte,
|  | IN PVOID | FaultingAddress,
|  | IN PMMPTE | PointerPteContents,
|  | IN PMMINPAGE_SUPPORT | InPageSupport,
|  | IN PEPROCESS | CurrentProcess
| ) |

Definition at line 2335 of file pagfault.c.

References ASSERT, BYTE_OFFSET, _MDL::ByteCount, DbgPrint, FALSE, HYDRA_PROCESS, KeBugCheckEx(), KernelMode, KeWaitForSingleObject(), LOCK_PFN, LOCK_SESSION_SPACE_WS, LOCK_SYSTEM_WS, LOCK_WS, _MDL::MappedSystemVa, MDL_LOCK_HELD, MDL_MAPPED_TO_SYSTEM_VA, _MDL::MdlFlags, MI_INSERT_USED_PAGETABLE_ENTRIES_IN_PFN, MI_IS_SYSTEM_CACHE_ADDRESS, MI_MAGIC_AWE_PTEFRAME, MI_PFN_ELEMENT, MI_PTE_LOOKUP_NEEDED, MI_ZERO_USED_PAGETABLE_ENTRIES_IN_PFN, MiCheckVirtualAddress(), MiFaultRetries, MiFindActualFaultingPte(), MiFreeInPageSupportBlock(), MiIoRetryLevel, MiMapPageInHyperSpace(), MiPteToProto, MiUnmapPageInHyperSpace, MiUserFaultRetries, MiUserIoRetryLevel, MiZeroPhysicalPage(), MmIsAddressValid(), MmIsRetryIoStatus, MmUnmapLockedPages(), NT_SUCCESS, NTSTATUS(), NULL, Offset, PAGE_SHIFT, PAGE_SIZE, _MMPFN::PteAddress, _MMPFN::PteFrame, STATUS_PTE_CHANGED, STATUS_REFAULT, TRUE, _MMPTE::u, _MMPFN::u1, _MMPFN::u3, and WrPageIn.

Referenced by MiDispatchFault(), and MiResolveTransitionFault().

    02346                    :
    02347
    02348     Waits for a page read to complete.
    02349
    02350 Arguments:
    02351
    02352     Pfn - Supplies a pointer to the PFN element for the page being read.
    02353
    02354     PointerPte - Supplies a pointer to the pte that is in the transition
    02355                  state.
    02356
    02357     FaultingAddress - Supplies the faulting address.
    02358
    02359     PointerPteContents - Supplies the contents of the PTE before the
    02360                          working set lock was released.
    02361
    02362     InPageSupport - Supplies a pointer to the inpage support structure
    02363                     for this read operation.
    02364
    02365 Return Value:
    02366
    02367     Returns the status of the in page.
    02368
    02369     Note that the working set lock is held upon return !!!
    02370
    02371 Environment:
    02372
    02373     Kernel mode, APCs disabled.  Neither the working set lock nor
    02374     the PFN lock may be held.
    02375
    02376 --*/
    02377
    02378 {
    02379     PMMPTE NewPointerPte;
    02380     PMMPTE ProtoPte;
    02381     PMMPFN Pfn1;
    02382     PMMPFN Pfn;
    02383     PULONG Va;
    02384     PPFN_NUMBER Page;
    02385     PPFN_NUMBER LastPage;
    02386     ULONG Offset;
    02387     ULONG Protection;
    02388     PMDL Mdl;
    02389     KIRQL OldIrql;
    02390     NTSTATUS status;
    02391     NTSTATUS status2;
    02392
    02393     //
    02394     // Wait for the I/O to complete.  Note that we can't wait for all
    02395     // the objects simultaneously as other threads/processes could be
    02396     // waiting for the same event.  The first thread which completes
    02397     // the wait and gets the PFN mutex may reuse the event for another
    02398     // fault before this thread completes its wait.
    02399     //
    02400
    02401     KeWaitForSingleObject( &InPageSupport->Event,
    02402                            WrPageIn,
    02403                            KernelMode,
    02404                            FALSE,
    02405                            (PLARGE_INTEGER)NULL);
    02406
    02407     if (CurrentProcess == HYDRA_PROCESS) {
    02408         LOCK_SESSION_SPACE_WS (OldIrql);
    02409     }
    02410     else if (CurrentProcess != NULL) {
    02411         LOCK_WS (CurrentProcess);
    02412     }
    02413     else {
    02414         LOCK_SYSTEM_WS (OldIrql);
    02415     }
    02416
    02417     LOCK_PFN (OldIrql);
    02418
    02419     ASSERT (Pfn2->u3.e2.ReferenceCount != 0);
    02420
    02421     //
    02422     // Check to see if this is the first thread to complete the in-page
    02423     // operation.
    02424     //
    02425
    02426     Pfn = InPageSupport->Pfn;
    02427     if (Pfn2 != Pfn) {
    02428         ASSERT (Pfn2->PteFrame != MI_MAGIC_AWE_PTEFRAME);
    02429         Pfn2->u3.e1.ReadInProgress = 0;
    02430     }
    02431
    02432     //
    02433     // Another thread has already serviced the read, check the
    02434     // io-error flag in the PFN database to ensure the in-page
    02435     // was successful.
    02436     //
    02437
    02438     if (Pfn2->u3.e1.InPageError == 1) {
    02439         ASSERT (!NT_SUCCESS(Pfn2->u1.ReadStatus));
    02440         MiFreeInPageSupportBlock (InPageSupport);
    02441
    02442         if (MmIsRetryIoStatus(Pfn2->u1.ReadStatus)) {
    02443             return STATUS_REFAULT;
    02444         }
    02445         return Pfn2->u1.ReadStatus;
    02446     }
    02447
    02448     if (InPageSupport->Completed == FALSE) {
    02449
    02450 #if defined(_PREFETCH_)
    02451
    02452         //
    02453         // The ReadInProgress bit for the dummy page is constantly cleared
    02454         // below as there are generally multiple inpage blocks pointing to
    02455         // the same dummy page.
    02456         //
    02457
    02458         ASSERT ((Pfn->u3.e1.ReadInProgress == 1) ||
    02459                 (Pfn->PteAddress == MI_PF_DUMMY_PAGE_PTE));
    02460 #else
    02461         ASSERT (Pfn->u3.e1.ReadInProgress == 1);
    02462 #endif
    02463
    02464         InPageSupport->Completed = TRUE;
    02465
    02466         Mdl = &InPageSupport->Mdl;
    02467
    02468 #if defined(_PREFETCH_)
    02469
    02470         if (InPageSupport->PrefetchMdl != NULL) {
    02471
    02472             //
    02473             // This is a prefetcher-issued read.
    02474             //
    02475
    02476             Mdl = InPageSupport->PrefetchMdl;
    02477         }
    02478
    02479 #endif
    02480
    02481         if (Mdl->MdlFlags & MDL_MAPPED_TO_SYSTEM_VA) {
    02482 #if DBG
    02483             Mdl->MdlFlags |= MDL_LOCK_HELD;
    02484 #endif //DBG
    02485
    02486             MmUnmapLockedPages (Mdl->MappedSystemVa, Mdl);
    02487
    02488 #if DBG
    02489             Mdl->MdlFlags &= ~MDL_LOCK_HELD;
    02490 #endif //DBG
    02491         }
    02492
    02493         ASSERT (Pfn->PteFrame != MI_MAGIC_AWE_PTEFRAME);
    02494
    02495         Pfn->u3.e1.ReadInProgress = 0;
    02496         Pfn->u1.Event = (PKEVENT)NULL;
    02497
    02498 #if defined (_WIN64)
    02499         //
    02500         // Page directory and page table pages are never clustered,
    02501         // ensure this is never violated as only one UsedPageTableEntries
    02502         // is kept in the inpage support block.
    02503         //
    02504
    02505         if (InPageSupport->UsedPageTableEntries) {
    02506             Page = (PPFN_NUMBER)(Mdl + 1);
    02507             LastPage = Page + ((Mdl->ByteCount - 1) >> PAGE_SHIFT);
    02508             ASSERT (Page == LastPage);
    02509         }
    02510
    02511 #if DBGXX
    02512         MiCheckPageTableInPage (Pfn, InPageSupport);
    02513 #endif
    02514 #endif
    02515
    02516         MI_INSERT_USED_PAGETABLE_ENTRIES_IN_PFN(Pfn, InPageSupport);
    02517
    02518         //
    02519         // Check the IO_STATUS_BLOCK to ensure the in-page completed successfully.
    02520         //
    02521
    02522         if (!NT_SUCCESS(InPageSupport->IoStatus.Status)) {
    02523
    02524             if (InPageSupport->IoStatus.Status == STATUS_END_OF_FILE) {
    02525
    02526                 //
    02527                 // An attempt was made to read past the end of file
    02528                 // zero all the remaining bytes in the read.
    02529                 //
    02530
    02531                 Page = (PPFN_NUMBER)(Mdl + 1);
    02532                 LastPage = Page + ((Mdl->ByteCount - 1) >> PAGE_SHIFT);
    02533
    02534                 while (Page <= LastPage) {
    02535                     MiZeroPhysicalPage (*Page, 0);
    02536
    02537                     MI_ZERO_USED_PAGETABLE_ENTRIES_IN_PFN(MI_PFN_ELEMENT(*Page));
    02538
    02539                     Page += 1;
    02540                 }
    02541
    02542             } else {
    02543
    02544                 //
    02545                 // In page io error occurred.
    02546                 //
    02547
    02548                 status = InPageSupport->IoStatus.Status;
    02549                 status2 = InPageSupport->IoStatus.Status;
    02550
    02551                 if (status != STATUS_VERIFY_REQUIRED) {
    02552
    02553                     LOGICAL Retry;
    02554
    02555                     Retry = FALSE;
    02556 #if DBG
    02557                     DbgPrint ("MM: inpage I/O error %X\n",
    02558                                     InPageSupport->IoStatus.Status);
    02559 #endif
    02560
    02561                     //
    02562                     // If this page is for paged pool or for paged
    02563                     // kernel code or page table pages, bugcheck.
    02564                     //
    02565
    02566                     if ((FaultingAddress > MM_HIGHEST_USER_ADDRESS) &&
    02567                         (!MI_IS_SYSTEM_CACHE_ADDRESS(FaultingAddress))) {
    02568
    02569                         if (MmIsRetryIoStatus(status)) {
    02570
    02571                             MiFaultRetries -= 1;
    02572                             if (MiFaultRetries != 0) {
    02573                                 Retry = TRUE;
    02574                             } else {
    02575                                 MiFaultRetries = MiIoRetryLevel;
    02576                             }
    02577                         }
    02578
    02579                         if (Retry == FALSE) {
    02580
    02581                             ULONG_PTR PteContents;
    02582
    02583                             //
    02584                             // The prototype PTE resides in paged pool which may
    02585                             // not be resident at this point.  Check first.
    02586                             //
    02587
    02588                             if (MmIsAddressValid (PointerPte) == TRUE) {
    02589                                 PteContents = *(PULONG_PTR)PointerPte;
    02590                             }
    02591                             else {
    02592                                 PteContents = (ULONG_PTR)-1;
    02593                             }
    02594
    02595                             KeBugCheckEx (KERNEL_DATA_INPAGE_ERROR,
    02596                                           (ULONG_PTR)PointerPte,
    02597                                           status,
    02598                                           (ULONG_PTR)FaultingAddress,
    02599                                           PteContents);
    02600                         }
    02601                         status2 = STATUS_REFAULT;
    02602                     }
    02603                     else {
    02604
    02605                         if (MmIsRetryIoStatus(status)) {
    02606
    02607                             MiUserFaultRetries -= 1;
    02608                             if (MiUserFaultRetries != 0) {
    02609                                 Retry = TRUE;
    02610                             } else {
    02611                                 MiUserFaultRetries = MiUserIoRetryLevel;
    02612                             }
    02613                         }
    02614
    02615                         if (Retry == TRUE) {
    02616                             status2 = STATUS_REFAULT;
    02617                         }
    02618                     }
    02619                 }
    02620
    02621                 Page = (PPFN_NUMBER)(Mdl + 1);
    02622                 LastPage = Page + ((Mdl->ByteCount - 1) >> PAGE_SHIFT);
    02623
    02624                 while (Page <= LastPage) {
    02625                     Pfn1 = MI_PFN_ELEMENT (*Page);
    02626                     ASSERT (Pfn1->u3.e2.ReferenceCount != 0);
    02627                     Pfn1->u3.e1.InPageError = 1;
    02628                     Pfn1->u1.ReadStatus = status;
    02629
    02630 #if DBG
    02631                     {
    02632                         KIRQL Old;
    02633                         Va = (PULONG)MiMapPageInHyperSpace (*Page,&Old);
    02634                         RtlFillMemoryUlong (Va, PAGE_SIZE, 0x50444142);
    02635                         MiUnmapPageInHyperSpace (Old);
    02636                     }
    02637 #endif //DBG
    02638                     Page += 1;
    02639                 }
    02640
    02641                 MiFreeInPageSupportBlock (InPageSupport);
    02642                 return status2;
    02643             }
    02644         } else {
    02645
    02646             MiFaultRetries = MiIoRetryLevel;
    02647             MiUserFaultRetries = MiUserIoRetryLevel;
    02648
    02649             if (InPageSupport->IoStatus.Information != Mdl->ByteCount) {
    02650
    02651                 ASSERT (InPageSupport->IoStatus.Information != 0);
    02652
    02653                 //
    02654                 // Less than a full page was read - zero the remainder
    02655                 // of the page.
    02656                 //
    02657
    02658                 Page = (PPFN_NUMBER)(Mdl + 1);
    02659                 LastPage = Page + ((Mdl->ByteCount - 1) >> PAGE_SHIFT);
    02660                 Page += ((InPageSupport->IoStatus.Information - 1) >> PAGE_SHIFT);
    02661
    02662                 Offset = BYTE_OFFSET (InPageSupport->IoStatus.Information);
    02663
    02664                 if (Offset != 0) {
    02665                     KIRQL Old;
    02666                     Va = (PULONG)((PCHAR)MiMapPageInHyperSpace (*Page, &Old)
    02667                                 + Offset);
    02668
    02669                     RtlZeroMemory (Va, PAGE_SIZE - Offset);
    02670                     MiUnmapPageInHyperSpace (Old);
    02671                 }
    02672
    02673                 //
    02674                 // Zero any remaining pages within the MDL.
    02675                 //
    02676
    02677                 Page += 1;
    02678
    02679                 while (Page <= LastPage) {
    02680                     MiZeroPhysicalPage (*Page, 0);
    02681                     Page += 1;
    02682                 }
    02683             }
    02684         }
    02685     }
    02686
    02687     MiFreeInPageSupportBlock (InPageSupport);
    02688
    02689     //
    02690     // Check to see if the faulting PTE has changed.
    02691     //
    02692
    02693     NewPointerPte = MiFindActualFaultingPte (FaultingAddress);
    02694
    02695     //
    02696     // If this PTE is in prototype PTE format, make the pointer to the
    02697     // PTE point to the prototype PTE.
    02698     //
    02699
    02700     if (NewPointerPte == (PMMPTE)NULL) {
    02701         return STATUS_PTE_CHANGED;
    02702     }
    02703
    02704     if (NewPointerPte != PointerPte) {
    02705
    02706         //
    02707         // Check to make sure the NewPointerPte is not a prototype PTE
    02708         // which refers to the page being made valid.
    02709         //
    02710
    02711         if (NewPointerPte->u.Soft.Prototype == 1) {
    02712             if (NewPointerPte->u.Soft.PageFileHigh == MI_PTE_LOOKUP_NEEDED) {
    02713
    02714                 ProtoPte = MiCheckVirtualAddress (FaultingAddress,
    02715                                                   &Protection);
    02716
    02717             } else {
    02718                 ProtoPte = MiPteToProto (NewPointerPte);
    02719             }
    02720
    02721             //
    02722             // Make sure the prototype PTE refers to the PTE made valid.
    02723             //
    02724
    02725             if (ProtoPte != PointerPte) {
    02726                 return STATUS_PTE_CHANGED;
    02727             }
    02728
    02729             //
    02730             // If the only difference is the owner mask, everything is okay.
    02731             //
    02732
    02733             if (ProtoPte->u.Long != PointerPteContents->u.Long) {
    02734                 return STATUS_PTE_CHANGED;
    02735             }
    02736         } else {
    02737             return STATUS_PTE_CHANGED;
    02738         }
    02739     } else {
    02740
    02741         if (NewPointerPte->u.Long != PointerPteContents->u.Long) {
    02742             return STATUS_PTE_CHANGED;
    02743         }
    02744     }
    02745     return STATUS_SUCCESS;
    02746 }

BOOLEAN MmIsAddressValid | ( | IN PVOID | VirtualAddress | ) |

Definition at line 3883 of file pagfault.c.

References FALSE, MI_IS_PHYSICAL_ADDRESS, MiGetPdeAddress, MiGetPpeAddress, MiGetPteAddress, MiGetVirtualAddressMappedByPde, TRUE, and _MMPTE::u.

Referenced by IoFreeDumpRange(), IopCreateSummaryDump(), IopDriverCorrectnessProcessParams(), IopIsAddressRangeValid(), IopIsMemoryRangeReadable(), IoSetDumpRange(), IovpSeedStack(), MiDecrementShareCount(), MiDetachSession(), MiGatherMappedPages(), MiInitMachineDependent(), MiInsertPageInList(), MiInsertStandbyListAtFront(), MiLoadSystemImage(), MiMakeSystemAddressValid(), MiMakeSystemAddressValidPfn(), MiMakeSystemAddressValidPfnSystemWs(), MiMakeSystemAddressValidPfnWs(), MiRemoveImageSessionWide(), MiRestoreTransitionPte(), MiSegmentDelete(), MiSessionAddProcess(), MiSessionCommitImagePages(), MiSessionCommitPageTables(), MiSessionCreateInternal(), MiSessionRemoveProcess(), MiSessionWideReserveImageAddress(), MiSetPagingOfDriver(), MiShareSessionImage(), MiWaitForInPageComplete(), MmDbgReadCheck(), MmDbgReleaseAddress(), MmDbgTranslatePhysicalAddress(), MmDbgTranslatePhysicalAddress64(), MmDbgWriteCheck(), MmMapViewInSessionSpace(), MmSessionCreate(), MmSessionDelete(), MmSessionSetUnloadAddress(), MmSetKernelDumpRange(), MmUnmapViewInSessionSpace(), MmWorkingSetManager(), NtLockVirtualMemory(), NtUnlockVirtualMemory(), RtlWalkFrameChain(), and VerifierFreeTrackedPool().

    03889                    :
    03890
    03891     For a given virtual address this function returns TRUE if no page fault
    03892     will occur for a read operation on the address, FALSE otherwise.
    03893
    03894     Note that after this routine was called, if appropriate locks are not
    03895     held, a non-faulting address could fault.
    03896
    03897 Arguments:
    03898
    03899     VirtualAddress - Supplies the virtual address to check.
    03900
    03901 Return Value:
    03902
    03903     TRUE if a no page fault would be generated reading the virtual address,
    03904     FALSE otherwise.
    03905
    03906 Environment:
    03907
    03908     Kernel mode.
    03909
    03910 --*/
    03911
    03912 {
    03913     PMMPTE PointerPte;
    03914
    03915 #if defined(_ALPHA_) || defined(_IA64_)
    03916
    03917     //
    03918     // If this is within the physical addressing range, just return TRUE.
    03919     //
    03920
    03921     if (MI_IS_PHYSICAL_ADDRESS(VirtualAddress)) {
    03922         return TRUE;
    03923     }
    03924
    03925 #endif // _ALPHA_ || _IA64_
    03926
    03927 #if defined (_WIN64)
    03928     PointerPte = MiGetPpeAddress (VirtualAddress);
    03929     if (PointerPte->u.Hard.Valid == 0) {
    03930         return FALSE;
    03931     }
    03932 #endif
    03933
    03934     PointerPte = MiGetPdeAddress (VirtualAddress);
    03935     if (PointerPte->u.Hard.Valid == 0) {
    03936         return FALSE;
    03937     }
    03938 #ifdef _X86_
    03939     if (PointerPte->u.Hard.LargePage == 1) {
    03940         return TRUE;
    03941     }
    03942 #endif //_X86_
    03943
    03944     PointerPte = MiGetPteAddress (VirtualAddress);
    03945     if (PointerPte->u.Hard.Valid == 0) {
    03946         return FALSE;
    03947     }
    03948
    03949 #ifdef _X86_
    03950     //
    03951     // Make sure we're not treating a page directory as a page table here for
    03952     // the case where the page directory is mapping a large page.  This is
    03953     // because the large page bit is valid in PDE formats, but reserved in
    03954     // PTE formats and will cause a trap.  A virtual address like c0200000
    03955     // triggers this case.  It's not enough to just check the large page bit
    03956     // in the PTE below because of course that bit's been reused by other
    03957     // steppings of the processor so we have to look at the address too.
    03958     //
    03959     if (PointerPte->u.Hard.LargePage == 1) {
    03960         PVOID Va;
    03961
    03962         Va = MiGetVirtualAddressMappedByPde (PointerPte);
    03963         if (MI_IS_PHYSICAL_ADDRESS(Va)) {
    03964             return FALSE;
    03965         }
    03966     }
    03967 #endif
    03968
    03969     return TRUE;
    03970 }

## Variable Documentation

MMINPAGE_SUPPORT_LIST MmInPageSupportList

| Definition at line 45 of file pagfault.c. Referenced by MiFlushInPageSupportBlock(), MiFreeInPageSupportBlock(), MiGetInPageSupportBlock(), and MmInitSystem().

MMPTE MmSharedUserDataPte

| Definition at line 38 of file pagfault.c. Referenced by MiCheckVirtualAddress(), and MmInitSystem().

PVOID MmSpecialPoolEnd

| Definition at line 41 of file pagfault.c.

PVOID MmSpecialPoolStart

| Definition at line 40 of file pagfault.c.