From olle.ponten at gmail.com  Tue Feb  2 07:40:39 2021
From: olle.ponten at gmail.com (=?UTF-8?B?T2xsZSBQb250w6lu?=)
Date: Tue, 2 Feb 2021 13:40:39 +0100
Subject: [python-win32] Error: VARIANT Type is unknown
In-Reply-To: <03cc3710-535e-d436-53d7-9b6f375d59c7@probo.com>
References: <CAPxAAVo0NtnBJNTdYKnwrwGbYSn7nR-=-4P2TC9xnZ-cfR+btA@mail.gmail.com>
 <427a13fb-58d2-a146-b37e-c660555f54ce@gmail.com>
 <CAPxAAVoHaRFwV_SKtOWzWs8zbrqGKKo_mRTmBgjyzGQuF1uCPw@mail.gmail.com>
 <03cc3710-535e-d436-53d7-9b6f375d59c7@probo.com>
Message-ID: <CAPxAAVr4P7Ui2OLfM3Z7LXzA9Vfbkxv1jiJQv3NLeG1HLp28SA@mail.gmail.com>

Yes I did. It produced the variant type unknown error as described in the
first post.

import win32api
import win32com.server.util
from win32com import client as w32Client

win32Client =
win32com.client.gencache.EnsureDispatch("{69072C2D-6ED1-4051-BF8D-70286A303CDD}")

s = "AL"
r = "On"

win32Client.scriptCmd(s,r)
Traceback (most recent call last):

  File "<ipython-input-7-b7dc7ecf5817>", line 1, in <module>
    win32CC.scriptCmd('s','n')

  File
"C:\Users\Natchos\AppData\Local\Temp\gen_py\3.8\E07C24C7-56CE-409F-AB60-DC7EDE9D9888x0x1x0.py",
line 37, in scriptCmd
    return self._oleobj_.InvokeTypes(216, LCID, 1, (3, 0), ((31, 1), (31,
1)),cmd

TypeError: The VARIANT type is unknown (0x0000001f)

I did mange to find a workaround using another package(comtypes) and
working with ctypes:

from comtypes import client as ctClient
import comtypes
import ctypes

AutoIPam = ctClient.CreateObject("ImagingWin.AutoIPam")
s = ctypes.c_wchar_p(cmd)
r = ctypes.c_wchar_p(param)
AutoIPam.scriptCmd(s,r)

This version does work and does not throw an error. I can confirm the
instrument responds to the commands sent thusly.

On Tue, Jan 26, 2021 at 7:47 PM Tim Roberts <timr at probo.com> wrote:

> Olle Pont?n wrote:
> >
> > Hmm ok, I could find the following information using OLEView (se
> > attachment).
> > It states that the two in parameters are noted as VT_LPWSTR for the
> > scriptCmd command.
> > Would that mean that I have no way of interfacing with them at all?
>
> No, it doesn't mean that!  If it doesn't ask for a variant, then you
> don't deliver a variant.  Did you try just passing a string, as Mark
> suggested?
>
> --
> Tim Roberts, timr at probo.com
> Providenza & Boekelheide, Inc.
>
>
> _______________________________________________
> python-win32 mailing list
> python-win32 at python.org
> https://mail.python.org/mailman/listinfo/python-win32
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210202/972e7c5b/attachment-0001.html>

From will at deemonsecurity.com  Tue Feb  2 07:54:38 2021
From: will at deemonsecurity.com (Will Deem)
Date: Tue, 2 Feb 2021 07:54:38 -0500
Subject: [python-win32] Compiling pywin32 with PyOxidizer
Message-ID: <CFB99FD0-59BB-404D-812F-1B86873599A8@deemonsecurity.com>

Good morning Pywin32 mailing list,

 I appear to be running into an issue compiling my program using pyoxidizer with pywin32. It appears there is some reliance on the __file__ attribute, which is breaking as pyoxidizer is loading the modules into memory and there is no __file__ attribute.

 Has anyone created a successful workaround for this issue? If not, is there a means of monkey patching the __file__ attribute for the win32evtlog module so the program can successfully compile?

 My traceback log is attached.

Fair winds and following seas,
Will


-------------- next part --------------
A non-text attachment was scrubbed...
Name: traceback.log
Type: application/octet-stream
Size: 25155 bytes
Desc: not available
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210202/7bd69ed9/attachment-0001.obj>

From skippy.hammond at gmail.com  Tue Feb  2 16:00:25 2021
From: skippy.hammond at gmail.com (Mark Hammond)
Date: Wed, 3 Feb 2021 08:00:25 +1100
Subject: [python-win32] Compiling pywin32 with PyOxidizer
In-Reply-To: <CFB99FD0-59BB-404D-812F-1B86873599A8@deemonsecurity.com>
References: <CFB99FD0-59BB-404D-812F-1B86873599A8@deemonsecurity.com>
Message-ID: <a3e7a5ab-64fe-1344-bdcd-01d40c305106@gmail.com>

On 2/02/2021 11:54 pm, Will Deem wrote:
> Good morning Pywin32 mailing list,
> 
>   I appear to be running into an issue compiling my program using pyoxidizer with pywin32. It appears there is some reliance on the __file__ attribute, which is breaking as pyoxidizer is loading the modules into memory and there is no __file__ attribute.
> 
>   Has anyone created a successful workaround for this issue? If not, is there a means of monkey patching the __file__ attribute for the win32evtlog module so the program can successfully compile?

That traceback isn't very helpful as it doesn't say what reference to 
__file__ is a problem - but everywhere __file__ is referenced the code 
is trying to find an actual file - often a DLL file.

The traceback also says "ModuleNotFoundError: No module named 
'win32evtlog'" - I don't think that's related to __file__.

I've never heard of anyone using pywin32 with pyoxidizer, so you are 
blazing the trail here :)

Mark.

From arslan.bah at gmail.com  Wed Feb  3 08:18:38 2021
From: arslan.bah at gmail.com (Bahadir Arslan)
Date: Wed, 3 Feb 2021 14:18:38 +0100
Subject: [python-win32] Running An Application via Render Farm
Message-ID: <2FC61E2D-FED5-4B30-839B-E95827905543@hxcore.ol>

An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210203/69dfc153/attachment.html>

From rgu at heimdalsecurity.com  Mon Feb  8 14:29:08 2021
From: rgu at heimdalsecurity.com (=?Windows-1252?Q?Romulus_G_=7C_Heimdal=99?=)
Date: Mon, 8 Feb 2021 19:29:08 +0000
Subject: [python-win32] Python installer for Windows not working on silent
 installation
In-Reply-To: <VI1P190MB0701CF0B9919BED6B4C7E2EDA38F9@VI1P190MB0701.EURP190.PROD.OUTLOOK.COM>
References: <VI1P190MB0701CF0B9919BED6B4C7E2EDA38F9@VI1P190MB0701.EURP190.PROD.OUTLOOK.COM>
Message-ID: <VI1P190MB0701778891B544D807236B73A38F9@VI1P190MB0701.EURP190.PROD.OUTLOOK.COM>

Hi Guys,

I am trying to deploy Python (the latest version) through AD GPO or through SCCM.

Since there?s no MSI Installer available on the python.org for the latest versions, I used the python-3.9.1-amd64.exe with the following command line:
python-3.9.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0

The issue is that the installer runs through the SYSTEM account and it appears that it does not install correctly.

The files are placed in C:\Program Files\Python39 but for some reason, the Python InstallAllUsers=1 call does not register its key in the HKEY_LOCAL_MACHINE\Software path.
The result of this means, from my understanding/testing, that the installer detects that it's not running as an elevated process, and therefore it installs as user NTAUTH\SYSTEM, for NTAUTH\SYSTEM, placing the keys for Python Launcher and the rest in HKCU instead of HKLM.
This explains why installations would fail, uninstallation was impossible no matter what, and why our Python Launcher was behaving weirdly despite PATH being set correctly.
?         Installations would fail because the keys would all still be placed in the HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall\
?         The installers never appeared in Control Panel ? Programs and Features despite successful installation, because it was installed for the NTAUTH\SYSTEM and not All Users
Is there something that I am doing wrong or is this a bug?
Thanks.

CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210208/fcf8c3e6/attachment.html>

From wdouglascampbell at hotmail.com  Mon Feb  8 23:23:54 2021
From: wdouglascampbell at hotmail.com (Doug Campbell)
Date: Tue, 9 Feb 2021 04:23:54 +0000
Subject: [python-win32] DeviceIOControl calls respond with parameter
 incorrect
Message-ID: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>

In my python 2 script, I am trying to connect to the VeraCrypt device driver to get some information on my mounted volumes.

This is what I have so far.  I tried a bunch of different ideas on how to construct the input buffer for the DeviceIoControl function call but I keep getting the following response.

Traceback (most recent call last):
  File "test.py", line 188, in <module>
    info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,b'x05x00x00x00' + (b'x00' * 702),65536)
pywintypes.error: (87, 'DeviceIoControl', 'The parameter is incorrect.')


I would appreciate any direction anyone can give.  I have seen success with making calls to win32file.DeviceIoControl when an input buffer wasn't needed but this one needs this information passed to it for it to work.

Thanks!
Doug

=======
import ctypes
import win32api
import win32file
import win32con
import winioctlcon
import struct

# https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/d4drvif/nf-d4drvif-ctl_code
# https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/specifying-device-types
FILE_DEVICE_UNKNOWN=0x00000022

METHOD_BUFFERED=0
METHOD_IN_DIRECT=1
METHOD_OUT_DIRECT=2
METHOD_NEITHER=3

FILE_ANY_ACCESS=0x0000
FILE_READ_ACCESS=0x0001
FILE_WRITE_ACCESS=0x0002

def CTL_CODE(DeviceType, Function, Method, Access):
    return (DeviceType << 16) | (Access << 14) | (Function << 2) | Method

#define VC_IOCTL(CODE) (CTL_CODE (FILE_DEVICE_UNKNOWN, 0x800 + (CODE), METHOD_BUFFERED, FILE_ANY_ACCESS))
def VC_IOCTL(CODE):
    return (CTL_CODE(FILE_DEVICE_UNKNOWN, 0x800 + (CODE), METHOD_BUFFERED, FILE_ANY_ACCESS))

VC_IOCTL_GET_MOUNTED_VOLUMES = VC_IOCTL(6)
VC_IOCTL_GET_VOLUME_PROPERTIES = VC_IOCTL (7)
VC_IOCTL_GET_BOOT_ENCRYPTION_STATUS = VC_IOCTL (18)
VC_IOCTL_GET_BOOT_DRIVE_VOLUME_PROPERTIES = VC_IOCTL (22)
VC_IOCTL_EMERGENCY_CLEAR_KEYS = VC_IOCTL (41)

INVALID_HANDLE_VALUE=-1
FILE_SHARE_READ=0x00000001
FILE_SHARE_WRITE=0x00000002
OPEN_EXISTING=3
path = "\\\\.\\VeraCrypt"
access_flag = 0
share_flag = FILE_SHARE_READ | FILE_SHARE_WRITE

hDisk = win32file.CreateFile(path,0,win32file.FILE_SHARE_READ|win32file.FILE_SHARE_WRITE,None,win32file.OPEN_EXISTING,0,None)

class VOLUME_PROPERTIES_STRUCT(ctypes.Structure):
    _fields_ = [('driveNo', ctypes.c_int),
                ('uniqueId', ctypes.c_int),
                ('wszVolume', ctypes.c_wchar * 260),
                ('diskLength', ctypes.c_uint64),
                ('ea', ctypes.c_int),
                ('mode', ctypes.c_int),
                ('pkcs5', ctypes.c_int),
                ('pkcs5Iterations', ctypes.c_int),
                ('hiddenVolume', ctypes.c_long),
                ('readOnly', ctypes.c_long),
                ('removable', ctypes.c_long),
                ('partitionInInactiveSysEncScope', ctypes.c_long),
                ('volFormatVersion', ctypes.c_uint32),
                ('totalBytesRead', ctypes.c_uint64),
                ('totalBytesWritten', ctypes.c_uint64),
                ('hiddenVolProtection', ctypes.c_int),
                ('volFormatVersion', ctypes.c_int),
                ('volumePim', ctypes.c_int),
                ('wszLabel', ctypes.c_wchar * 33),
                ('bDriverSetLabel', ctypes.c_long),
                ('volumeID', ctypes.c_wchar * 32),
                ('mountDisabled', ctypes.c_long)]


prop = VOLUME_PROPERTIES_STRUCT()
prop.driveNo = 5

#info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,struct.pack('ii520sQiiiiLLLLLQQiii66sL64sL',prop),17424)
#info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,struct.pack('ii520sQiiiiLLLLLQQiii66sL64sL',5,0,'a' * 520,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,'a' * 66,0,'a' * 64,0),17424)
info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,b'0x00' + b'x05x00x00x00' + b'x00' * 702),65536)


I based what I have done on the following C code from VeraStatus (veracrypt/VeraStatus: Command line tool to get technical information about VeraCrypt mounted volumes and system encryption (github.com)<https://github.com/veracrypt/VeraStatus>).  Here are what I believe are relevant sections.


#define VC_IOCTL(CODE) (CTL_CODE (FILE_DEVICE_UNKNOWN, 0x800 + (CODE), METHOD_BUFFERED, FILE_ANY_ACCESS))

#define VC_IOCTL_GET_MOUNTED_VOLUMES VC_IOCTL (6)

#define VOLUME_ID_SIZE 32

typedef struct
{
int driveNo;
int uniqueId;
wchar_t wszVolume[260];
unsigned __int64 diskLength;
int ea;
int mode;
int pkcs5;
int pkcs5Iterations;
BOOL hiddenVolume;
BOOL readOnly;
BOOL removable;
BOOL partitionInInactiveSysEncScope;
unsigned __int32 volumeHeaderFlags;
unsigned __int64 totalBytesRead;
unsigned __int64 totalBytesWritten;
int hiddenVolProtection;
int volFormatVersion;
int volumePim;
wchar_t wszLabel[33];
BOOL bDriverSetLabel;
unsigned char volumeID[VOLUME_ID_SIZE];
BOOL mountDisabled;
} VOLUME_PROPERTIES_STRUCT;

VOLUME_PROPERTIES_STRUCT prop;

prop.driveNo = _totupper(argv[1][0]) - TEXT('A');

if (DeviceIoControl (hDriver, VC_IOCTL_GET_VOLUME_PROPERTIES, &prop, sizeof (prop), &prop, sizeof (prop), &cbBytesReturned, NULL))



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210209/1fc87c98/attachment-0001.html>

From timr at probo.com  Tue Feb  9 00:46:43 2021
From: timr at probo.com (Tim Roberts)
Date: Mon, 8 Feb 2021 21:46:43 -0800
Subject: [python-win32] DeviceIOControl calls respond with parameter
 incorrect
In-Reply-To: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
References: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
Message-ID: <FD6E7FC1-925F-4AA6-9EAA-BB90222D7721@probo.com>

On Feb 8, 2021, at 8:23 PM, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
> 
> In my python 2 script, I am trying to connect to the VeraCrypt device driver to get some information on my mounted volumes.

There are a lot of things to go wrong here.  You may be in for a long slog.  Your ctypes definition does look fundamentally correct.


> This is what I have so far.  I tried a bunch of different ideas on how to construct the input buffer for the DeviceIoControl function call but I keep getting the following response.
> 
> Traceback (most recent call last):
>   File "test.py", line 188, in <module>
>     info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,b'x05x00x00x00' + (b'x00' * 702),65536)
> pywintypes.error: (87, 'DeviceIoControl', 'The parameter is incorrect.')

The last parameter is the size of the output buffer, and that does get passed to the driver.  It should be sizeof(VOLUME_PROPERTIES_STRUCT) which is, I believe, 1280.


> #info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,struct.pack('ii520sQiiiiLLLLLQQiii66sL64sL',prop),17424)

Last should be 1280, I think.

> info=win32file.DeviceIoControl(hDisk,VC_IOCTL_GET_VOLUME_PROPERTIES,b'0x00' + b'x05x00x00x00' + b'x00' * 702),65536)

The first b?0x00? is wrong; the first byte of the struct needs to be 5.
? 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210208/626ce420/attachment.html>

From eryksun at gmail.com  Tue Feb  9 01:24:10 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Tue, 9 Feb 2021 00:24:10 -0600
Subject: [python-win32] DeviceIOControl calls respond with parameter
 incorrect
In-Reply-To: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
References: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
Message-ID: <CACL+1as5jN1YOkX_xoEgQj1m9qPk+uQwYF8fh2U9CQ=HGedvPw@mail.gmail.com>

On 2/8/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
> In my python 2 script, I am trying to connect to the VeraCrypt device driver
> to get some information on my mounted volumes.

The VeraCrypt repo on GitHub [1] indicates that all structures are
defined with #pragma pack(1). In ctypes this is the _pack_ directive.
Try the following:

import ctypes
import winioctlcon
import win32file

def VC_IOCTL(CODE):
    return winioctlcon.CTL_CODE(winioctlcon.FILE_DEVICE_UNKNOWN,
            0x800 + CODE, winioctlcon.METHOD_BUFFERED,
            winioctlcon.FILE_ANY_ACCESS)

VC_IOCTL_GET_MOUNTED_VOLUMES = VC_IOCTL(6)
VC_IOCTL_GET_VOLUME_PROPERTIES = VC_IOCTL(7)
VC_IOCTL_GET_BOOT_ENCRYPTION_STATUS = VC_IOCTL(18)
VC_IOCTL_GET_BOOT_DRIVE_VOLUME_PROPERTIES = VC_IOCTL(22)
VC_IOCTL_EMERGENCY_CLEAR_KEYS = VC_IOCTL(41)

MAX_PATH = 260
VOLUME_LABEL_LENGTH = 33 # 32 + null
VOLUME_ID_SIZE = 32
WIN32_ROOT_PREFIX DRIVER_STR = r'\\.\VeraCrypt'

class VOLUME_PROPERTIES_STRUCT(ctypes.Structure):
    _pack_ = 1
    _fields_ = (
        ('driveNo', ctypes.c_int),
        ('uniqueId', ctypes.c_int),
        ('wszVolume', ctypes.c_wchar * MAX_PATH),
        ('diskLength', ctypes.c_uint64),
        ('ea', ctypes.c_int),
        ('mode', ctypes.c_int),
        ('pkcs5', ctypes.c_int),
        ('pkcs5Iterations', ctypes.c_int),
        ('hiddenVolume', ctypes.c_int),
        ('readOnly', ctypes.c_int),
        ('removable', ctypes.c_int),
        ('partitionInInactiveSysEncScope', ctypes.c_int),
        ('volFormatVersion', ctypes.c_uint32),
        ('totalBytesRead', ctypes.c_uint64),
        ('totalBytesWritten', ctypes.c_uint64),
        ('hiddenVolProtection', ctypes.c_int),
        ('volFormatVersion', ctypes.c_int),
        ('volumePim', ctypes.c_int),
        ('wszLabel', ctypes.c_wchar * VOLUME_LABEL_LENGTH),
        ('bDriverSetLabel', ctypes.c_int),
        ('volumeID', ctypes.c_wchar * VOLUME_ID_SIZE),
        ('mountDisabled', ctypes.c_int))


prop = VOLUME_PROPERTIES_STRUCT(driveNo = ord('F') - ord('A'))

hDevice = win32file.CreateFile(WIN32_ROOT_PREFIX DRIVER_STR, 0, 0, None,
    win32file.OPEN_EXISTING, 0, None)
try:
    info = win32file.DeviceIoControl(hDevice,
        VC_IOCTL_GET_VOLUME_PROPERTIES, prop, prop)
finally:
    hDevice.close()

---
[1] https://github.com/veracrypt/VeraCrypt/blob/master/src/Common/Apidrvr.h

From wdouglascampbell at hotmail.com  Tue Feb  9 10:07:24 2021
From: wdouglascampbell at hotmail.com (Doug Campbell)
Date: Tue, 9 Feb 2021 15:07:24 +0000
Subject: [python-win32] DeviceIOControl calls respond with parameter
 incorrect
In-Reply-To: <CACL+1as5jN1YOkX_xoEgQj1m9qPk+uQwYF8fh2U9CQ=HGedvPw@mail.gmail.com>
References: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>,
 <CACL+1as5jN1YOkX_xoEgQj1m9qPk+uQwYF8fh2U9CQ=HGedvPw@mail.gmail.com>
Message-ID: <BLAPR20MB418026BD7E03240010E7827DC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>

Thanks Eryk!

That was exactly what I needed.  I will have to read up on the _pack_ directive to understand it but for now things are running the way they should be.

I did have one typo in what I originally pasted that made its way into what you provided.

('volumeID', ctypes.c_wchar * VOLUME_ID_SIZE),

should actually be

('volumeID', ctypes.c_char * VOLUME_ID_SIZE),

So, if someone else comes along and tries this out they should make that one change.

Anyway, you made my day.  Thanks again!

Doug

________________________________
From: Eryk Sun <eryksun at gmail.com>
Sent: Monday, February 8, 2021 8:24 PM
To: python-win32 at python.org <python-win32 at python.org>
Cc: Doug Campbell <wdouglascampbell at hotmail.com>
Subject: Re: [python-win32] DeviceIOControl calls respond with parameter incorrect

On 2/8/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
> In my python 2 script, I am trying to connect to the VeraCrypt device driver
> to get some information on my mounted volumes.

The VeraCrypt repo on GitHub [1] indicates that all structures are
defined with #pragma pack(1). In ctypes this is the _pack_ directive.
Try the following:

import ctypes
import winioctlcon
import win32file

def VC_IOCTL(CODE):
    return winioctlcon.CTL_CODE(winioctlcon.FILE_DEVICE_UNKNOWN,
            0x800 + CODE, winioctlcon.METHOD_BUFFERED,
            winioctlcon.FILE_ANY_ACCESS)

VC_IOCTL_GET_MOUNTED_VOLUMES = VC_IOCTL(6)
VC_IOCTL_GET_VOLUME_PROPERTIES = VC_IOCTL(7)
VC_IOCTL_GET_BOOT_ENCRYPTION_STATUS = VC_IOCTL(18)
VC_IOCTL_GET_BOOT_DRIVE_VOLUME_PROPERTIES = VC_IOCTL(22)
VC_IOCTL_EMERGENCY_CLEAR_KEYS = VC_IOCTL(41)

MAX_PATH = 260
VOLUME_LABEL_LENGTH = 33 # 32 + null
VOLUME_ID_SIZE = 32
WIN32_ROOT_PREFIX DRIVER_STR = r'\\.\VeraCrypt'

class VOLUME_PROPERTIES_STRUCT(ctypes.Structure):
    _pack_ = 1
    _fields_ = (
        ('driveNo', ctypes.c_int),
        ('uniqueId', ctypes.c_int),
        ('wszVolume', ctypes.c_wchar * MAX_PATH),
        ('diskLength', ctypes.c_uint64),
        ('ea', ctypes.c_int),
        ('mode', ctypes.c_int),
        ('pkcs5', ctypes.c_int),
        ('pkcs5Iterations', ctypes.c_int),
        ('hiddenVolume', ctypes.c_int),
        ('readOnly', ctypes.c_int),
        ('removable', ctypes.c_int),
        ('partitionInInactiveSysEncScope', ctypes.c_int),
        ('volFormatVersion', ctypes.c_uint32),
        ('totalBytesRead', ctypes.c_uint64),
        ('totalBytesWritten', ctypes.c_uint64),
        ('hiddenVolProtection', ctypes.c_int),
        ('volFormatVersion', ctypes.c_int),
        ('volumePim', ctypes.c_int),
        ('wszLabel', ctypes.c_wchar * VOLUME_LABEL_LENGTH),
        ('bDriverSetLabel', ctypes.c_int),
        ('volumeID', ctypes.c_wchar * VOLUME_ID_SIZE),
        ('mountDisabled', ctypes.c_int))


prop = VOLUME_PROPERTIES_STRUCT(driveNo = ord('F') - ord('A'))

hDevice = win32file.CreateFile(WIN32_ROOT_PREFIX DRIVER_STR, 0, 0, None,
    win32file.OPEN_EXISTING, 0, None)
try:
    info = win32file.DeviceIoControl(hDevice,
        VC_IOCTL_GET_VOLUME_PROPERTIES, prop, prop)
finally:
    hDevice.close()

---
[1] https://github.com/veracrypt/VeraCrypt/blob/master/src/Common/Apidrvr.h
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210209/f69ab01b/attachment.html>

From eryksun at gmail.com  Tue Feb  9 16:51:03 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Tue, 9 Feb 2021 15:51:03 -0600
Subject: [python-win32] DeviceIOControl calls respond with parameter
 incorrect
In-Reply-To: <BLAPR20MB418026BD7E03240010E7827DC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
References: <BLAPR20MB4180A4D898F61E287AF97F27C78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
 <CACL+1as5jN1YOkX_xoEgQj1m9qPk+uQwYF8fh2U9CQ=HGedvPw@mail.gmail.com>
 <BLAPR20MB418026BD7E03240010E7827DC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
Message-ID: <CACL+1avAWPPypZhWHz8ttc4haD2kMTLwnEj4HOfT6p9wkN_UJA@mail.gmail.com>

On 2/9/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
>
> That was exactly what I needed.  I will have to read up on the _pack_
> directive to understand it but for now things are running the way they
> should be.

I'm glad I could help. Normally padding is added between fields of a
struct in order to support aligned access for the data type of each
field. Setting the `_pack_ = 1` attribute forces single-byte
alignment, and thus no padding bytes are added.

> I did have one typo in what I originally pasted that made its way into what
> you provided.
>
> ('volumeID', ctypes.c_wchar * VOLUME_ID_SIZE),
>
> should actually be
>
> ('volumeID', ctypes.c_char * VOLUME_ID_SIZE),

Sorry, I did actually scan over the fields to check the types against
the C definition, but I somehow missed that extra "w".

From wdouglascampbell at hotmail.com  Tue Feb  9 16:27:06 2021
From: wdouglascampbell at hotmail.com (Doug Campbell)
Date: Tue, 9 Feb 2021 21:27:06 +0000
Subject: [python-win32] DeviceIOControl using
 IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'
Message-ID: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>

I am trying to get the disk extents given a disk number but I am getting back the following error.

Traceback (most recent call last):
  File "test2.py", line 29, in <module>
    win32file.DeviceIoControl(hDevice,winioctlcon.IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, None, extents, None)
pywintypes.error: (1, 'DeviceIoControl', 'Incorrect function.')

I have tried with all three of the disks on my system:  \\.\PhysicalDrive0, \\.\PhysicalDrive1, and \\.\PhyscialDrive2 but the results are always the same.

Thanks for any insights as to why I am getting this "Incorrect function" error and how to resolve it.

Doug


Here is the code that I am using.

import ctypes
import winioctlcon
import win32file
import win32con

disk = r'\\.\PhysicalDrive1'

hDevice = win32file.CreateFile(
            disk, win32con.GENERIC_READ,
            win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
            None, win32con.OPEN_EXISTING, win32con.FILE_ATTRIBUTE_NORMAL,  None)

class DISK_EXTENT(ctypes.Structure):
    _fields_ = (
                ('DiskNumber', ctypes.c_ulong),
                ('StartingOffset', ctypes.c_longlong),
                ('ExtentLength', ctypes.c_longlong))

ANYSIZE_ARRAY = 1

class VOLUME_DISK_EXTENTS(ctypes.Structure):
    _fields_ = (
                ('NumberOfDiskExtents', ctypes.c_ulong),
                ('Extents', DISK_EXTENT * ANYSIZE_ARRAY))

extents = VOLUME_DISK_EXTENTS()

try:
    win32file.DeviceIoControl(hDevice,winioctlcon.IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS, None, extents, None)
finally:
    hDevice.close()




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210209/73753485/attachment-0001.html>

From eryksun at gmail.com  Tue Feb  9 20:05:15 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Tue, 9 Feb 2021 19:05:15 -0600
Subject: [python-win32] DeviceIOControl using
 IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'
In-Reply-To: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
References: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
Message-ID: <CACL+1auf8eHWuOrQtESpEE1Vf5omYu-3UR4B52N0XJRUuG3eCw@mail.gmail.com>

On 2/9/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
>
> win32file.DeviceIoControl(hDevice,
> winioctlcon.IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS,
> None, extents, None)
> pywintypes.error: (1, 'DeviceIoControl', 'Incorrect function.')
>
> I have tried with all three of the disks on my system:  \\.\PhysicalDrive0,
> \\.\PhysicalDrive1, and \\.\PhyscialDrive2 but the results are always the
> same.

IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS requests the disk extents of a
volume device. Disk devices do not implement this request. A volume
device such as "\\.\C:" can span multiple extents across one or more
physical disks such as "\\.\PhysicalDrive0" and "\\.\PhysicalDrive1".

If the request fails with winerror.ERROR_MORE_DATA, you need to be
prepared to resize the structure according to NumberOfDiskExtents.
ctypes doesn't make this all that easy or convenient at a high level.
What I do is to make the array field private and implement a property
that takes the dynamic length into account. For example:

    ANYSIZE_ARRAY = 1

    class VOLUME_DISK_EXTENTS(ctypes.Structure):
        _fields_ = (('NumberOfDiskExtents', ctypes.c_ulong),
                    ('_Extents', DISK_EXTENT * ANYSIZE_ARRAY))

        @property
        def Extents(self):
            offset = type(self)._Extents.offset
            array_t = DISK_EXTENT * self.NumberOfDiskExtents
            return array_t.from_buffer(self, offset)

        def resize(self):
            if self.NumberOfDiskExtents < 1:
                self.NumberOfDiskExtents = 1
            offset = type(self)._Extents.offset
            array_size = ctypes.sizeof(DISK_EXTENT) * self.NumberOfDiskExtents
            ctypes.resize(self, offset + array_size)

If you passed `vde = VOLUME_DISK_EXTENTS()`, and the request fails
with winerror.ERROR_MORE_DATA, the call should have set
NumberOfDiskExtents to the required length. Call vde.resize(), and try
again. It's a bit clumsy, but it works.

> hDevice = win32file.CreateFile(
>             disk, win32con.GENERIC_READ,
>             win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
>             None, win32con.OPEN_EXISTING, win32con.FILE_ATTRIBUTE_NORMAL,
>             None)

IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS is defined to require
FILE_ANY_ACCESS. This means the desired access and access sharing can
both be 0, and should be since there's no reason to request access
that you don't need that could cause the call to fail with a
permission error (e.g. access denied, or a sharing violation if read
access isn't shared). For example:

    volume = r'\\.\C:'
    hDevice = win32file.CreateFile(volume, 0, 0, None,
        win32file.OPEN_EXISTING, 0, None)

---

Volume Device Names

The base name of a volume device is typically something like
"\Device\HarddiskVolume<N>" or "\Device\CdRom<N>", but these are
automatic names that can vary in the device number N each time a
volume comes online. The mountpoint manager usually will also assign
persistent names to a volume device, which are stored in the registry
if necessary (HKLM\System\MountedDevices) and are set globally in the
"\GLOBAL??" device-alias directory when the volume comes online.
Usually it assigns a GUID name of the form
"Volume{12345678-0000-0000-0000-123456789ABC}". A GUID name is used
when mounting a volume on an existing directory (e.g. mounting a
volume as "C:\Mount\BackupDrive"). It usually also assigns a DOS
drive-letter name such as "C:", which provides the classic DOS volume
mountpoint, such as "C:\".

The "\GLOBAL??" object directory is rarely accessed directly, even in
native NT API programs that have full access to the NT object
namespace. Instead, native NT programs use "\??", a virtual directory
that allows accessing global device symlinks in addition to, and
shadowed by, device symlinks in the caller's logon session device
directory. The SYSTEM logon uses "\GLOBAL??" for its logon session
devices, so it always creates and accesses global device aliases.

In the Windows API, the NT API "\??\" prefix maps to "\\.\" or "\\?\"
UNC-style paths. The former is more common if just the device is
accessed, and the latter is more common for a filesystem path. Thus to
access the "C:" volume device directly, programs typically use
"\\.\C:". Note that just "C:" by itself is a DOS drive-relative path
that expands to the working directory in the mounted filesystem, which
defaults to the root directory. So passing "C:" as the path to open is
actually a reference to a filesystem directory, not to the volume
device, which obviously won't work.

From wdouglascampbell at hotmail.com  Wed Feb 10 00:24:28 2021
From: wdouglascampbell at hotmail.com (Doug Campbell)
Date: Wed, 10 Feb 2021 05:24:28 +0000
Subject: [python-win32] DeviceIOControl using
 IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'
In-Reply-To: <CACL+1auf8eHWuOrQtESpEE1Vf5omYu-3UR4B52N0XJRUuG3eCw@mail.gmail.com>
References: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>,
 <CACL+1auf8eHWuOrQtESpEE1Vf5omYu-3UR4B52N0XJRUuG3eCw@mail.gmail.com>
Message-ID: <BLAPR20MB4180F44A31D1897D6C10CDBBC78D9@BLAPR20MB4180.namprd20.prod.outlook.com>

Eryk,

Again, you expand my knowledge!  It seems so obvious now after reading what you wrote that I would not be able to get volume disk extents for a physical partition but yet this is what I wanted to do because I was attempting to find out the partition's offset on the disk.

Now, I think I see that I need to figure out the partition's volume device name using the \\?\Volume{GUID} format and then I will be able to determine what its offset is on the disk.  The only way I can find to accomplish this is to iterate through each volume and using IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS to get its disk and IOCTL_DISK_GET_PARTITION_INFO_EX to get its partition and then see if I have a match for the disk/partition combination that I am looking for.

Thanks for sharing the extra details on how to handle the wintype.ERROR_MORE_DATA error.  That was likely to be my next issue :)  I did find an alternative method for handling this that is used in the multibootusb script/win32.py<https://github.com/mbusb/multibootusb/blob/master/scripts/win32.py> function findVolumeGuids().  They use BytesIO to create an io stream and then unpack the bytes on-the-fly.  I think I likely go with your method.

Anyway, thanks again for your excellent explanations.

Doug

________________________________
From: Eryk Sun <eryksun at gmail.com>
Sent: Tuesday, February 9, 2021 3:05 PM
To: python-win32 at python.org <python-win32 at python.org>
Cc: Doug Campbell <wdouglascampbell at hotmail.com>
Subject: Re: [python-win32] DeviceIOControl using IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'

On 2/9/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
>
> win32file.DeviceIoControl(hDevice,
> winioctlcon.IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS,
> None, extents, None)
> pywintypes.error: (1, 'DeviceIoControl', 'Incorrect function.')
>
> I have tried with all three of the disks on my system:  \\.\PhysicalDrive0,
> \\.\PhysicalDrive1, and \\.\PhyscialDrive2 but the results are always the
> same.

IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS requests the disk extents of a
volume device. Disk devices do not implement this request. A volume
device such as "\\.\C:" can span multiple extents across one or more
physical disks such as "\\.\PhysicalDrive0" and "\\.\PhysicalDrive1".

If the request fails with winerror.ERROR_MORE_DATA, you need to be
prepared to resize the structure according to NumberOfDiskExtents.
ctypes doesn't make this all that easy or convenient at a high level.
What I do is to make the array field private and implement a property
that takes the dynamic length into account. For example:

    ANYSIZE_ARRAY = 1

    class VOLUME_DISK_EXTENTS(ctypes.Structure):
        _fields_ = (('NumberOfDiskExtents', ctypes.c_ulong),
                    ('_Extents', DISK_EXTENT * ANYSIZE_ARRAY))

        @property
        def Extents(self):
            offset = type(self)._Extents.offset
            array_t = DISK_EXTENT * self.NumberOfDiskExtents
            return array_t.from_buffer(self, offset)

        def resize(self):
            if self.NumberOfDiskExtents < 1:
                self.NumberOfDiskExtents = 1
            offset = type(self)._Extents.offset
            array_size = ctypes.sizeof(DISK_EXTENT) * self.NumberOfDiskExtents
            ctypes.resize(self, offset + array_size)

If you passed `vde = VOLUME_DISK_EXTENTS()`, and the request fails
with winerror.ERROR_MORE_DATA, the call should have set
NumberOfDiskExtents to the required length. Call vde.resize(), and try
again. It's a bit clumsy, but it works.

> hDevice = win32file.CreateFile(
>             disk, win32con.GENERIC_READ,
>             win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
>             None, win32con.OPEN_EXISTING, win32con.FILE_ATTRIBUTE_NORMAL,
>             None)

IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS is defined to require
FILE_ANY_ACCESS. This means the desired access and access sharing can
both be 0, and should be since there's no reason to request access
that you don't need that could cause the call to fail with a
permission error (e.g. access denied, or a sharing violation if read
access isn't shared). For example:

    volume = r'\\.\C:'
    hDevice = win32file.CreateFile(volume, 0, 0, None,
        win32file.OPEN_EXISTING, 0, None)

---

Volume Device Names

The base name of a volume device is typically something like
"\Device\HarddiskVolume<N>" or "\Device\CdRom<N>", but these are
automatic names that can vary in the device number N each time a
volume comes online. The mountpoint manager usually will also assign
persistent names to a volume device, which are stored in the registry
if necessary (HKLM\System\MountedDevices) and are set globally in the
"\GLOBAL??" device-alias directory when the volume comes online.
Usually it assigns a GUID name of the form
"Volume{12345678-0000-0000-0000-123456789ABC}". A GUID name is used
when mounting a volume on an existing directory (e.g. mounting a
volume as "C:\Mount\BackupDrive"). It usually also assigns a DOS
drive-letter name such as "C:", which provides the classic DOS volume
mountpoint, such as "C:\".

The "\GLOBAL??" object directory is rarely accessed directly, even in
native NT API programs that have full access to the NT object
namespace. Instead, native NT programs use "\??", a virtual directory
that allows accessing global device symlinks in addition to, and
shadowed by, device symlinks in the caller's logon session device
directory. The SYSTEM logon uses "\GLOBAL??" for its logon session
devices, so it always creates and accesses global device aliases.

In the Windows API, the NT API "\??\" prefix maps to "\\.\" or "\\?\"
UNC-style paths. The former is more common if just the device is
accessed, and the latter is more common for a filesystem path. Thus to
access the "C:" volume device directly, programs typically use
"\\.\C:". Note that just "C:" by itself is a DOS drive-relative path
that expands to the working directory in the mounted filesystem, which
defaults to the root directory. So passing "C:" as the path to open is
actually a reference to a filesystem directory, not to the volume
device, which obviously won't work.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210210/e64a756f/attachment-0001.html>

From eryksun at gmail.com  Wed Feb 10 04:32:13 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Wed, 10 Feb 2021 03:32:13 -0600
Subject: [python-win32] DeviceIOControl using
 IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'
In-Reply-To: <BLAPR20MB4180F44A31D1897D6C10CDBBC78D9@BLAPR20MB4180.namprd20.prod.outlook.com>
References: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
 <CACL+1auf8eHWuOrQtESpEE1Vf5omYu-3UR4B52N0XJRUuG3eCw@mail.gmail.com>
 <BLAPR20MB4180F44A31D1897D6C10CDBBC78D9@BLAPR20MB4180.namprd20.prod.outlook.com>
Message-ID: <CACL+1asR8ar1O7tmKsr=t=7DzJJMT=g8Pv3niGtJa3eradXOrg@mail.gmail.com>

On 2/9/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
>
> Again, you expand my knowledge!  It seems so obvious now after reading what
> you wrote that I would not be able to get volume disk extents for a physical
> partition but yet this is what I wanted to do because I was attempting to
> find out the partition's offset on the disk.

In the case of a basic disk device, it can have multiple partitions
that each correspond to a volume device, \Device\HarddiskVolume<N>. In
this case, each volume device consists of a single extent on a single
disk.

> The only way I can find to accomplish this is to iterate through each volume

You can get the list of volume GUID names of volume devices that are
registered with the mountpoint manager via FindFirstVolumeW /
FindNextVolumeW. Unfortunately, PyWin32 doesn't wrap the volume-find
functions. They can be called using ctypes. For example:

import ctypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

class HANDLE(ctypes.c_void_p):
    def __eq__(self, other):
        if hasattr(other, 'value'):
            return self.value == other.value
        return False

INVALID_HANDLE_VALUE = HANDLE(-1)
ERROR_NO_MORE_FILES = 18

kernel32.FindFirstVolumeW.restype = HANDLE

def list_volumes():
    volumes = []
    buf = (ctypes.c_wchar * 260)()
    hfind = kernel32.FindFirstVolumeW(buf, len(buf))
    if hfind == INVALID_HANDLE_VALUE:
        raise ctypes.WinError(ctypes.get_last_error())
    try:
        while True:
            # Strip the trailing backslash that FindNextVolumeW appends.
            # The trailing backslash is the mountpoint of the filesystem
            # that mounts the volume device, but we only want the GUID
            # device name.
            volumes.append(buf.value.rstrip('\\'))
            if not kernel32.FindNextVolumeW(hfind, buf, len(buf)):
                error = ctypes.get_last_error()
                if error != ERROR_NO_MORE_FILES:
                    raise ctypes.WinError(error)
                break
    finally:
        kernel32.FindVolumeClose(hfind)
    return volumes

I subclassed c_void_p to create the HANDLE type in order to avoid the
otherwise automatic conversion of the return value to a builtin Python
type. This is a simple way around needing to declare argtypes for the
functions that take the find handle (actually a pointer to memory) as
an argument.

From wdouglascampbell at hotmail.com  Wed Feb 10 16:04:48 2021
From: wdouglascampbell at hotmail.com (Doug Campbell)
Date: Wed, 10 Feb 2021 21:04:48 +0000
Subject: [python-win32] DeviceIOControl using
 IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'
In-Reply-To: <CACL+1asR8ar1O7tmKsr=t=7DzJJMT=g8Pv3niGtJa3eradXOrg@mail.gmail.com>
References: <BLAPR20MB41802B59F5950A74FCB9C28FC78E9@BLAPR20MB4180.namprd20.prod.outlook.com>
 <CACL+1auf8eHWuOrQtESpEE1Vf5omYu-3UR4B52N0XJRUuG3eCw@mail.gmail.com>
 <BLAPR20MB4180F44A31D1897D6C10CDBBC78D9@BLAPR20MB4180.namprd20.prod.outlook.com>,
 <CACL+1asR8ar1O7tmKsr=t=7DzJJMT=g8Pv3niGtJa3eradXOrg@mail.gmail.com>
Message-ID: <BLAPR20MB4180F57088D40EEB3F1F15F9C78D9@BLAPR20MB4180.namprd20.prod.outlook.com>

Thanks again, Eryk.

I was able to play around today with the code you provided below and was able to tweak it to list the volumes, find the one that matched the disk/partition combination I was searching for and return its offset on the disk.  Very helpful!

________________________________
From: Eryk Sun <eryksun at gmail.com>
Sent: Tuesday, February 9, 2021 11:32 PM
To: python-win32 at python.org <python-win32 at python.org>
Cc: Doug Campbell <wdouglascampbell at hotmail.com>
Subject: Re: [python-win32] DeviceIOControl using IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS fails with 'Incorrect function.'

On 2/9/21, Doug Campbell <wdouglascampbell at hotmail.com> wrote:
>
> Again, you expand my knowledge!  It seems so obvious now after reading what
> you wrote that I would not be able to get volume disk extents for a physical
> partition but yet this is what I wanted to do because I was attempting to
> find out the partition's offset on the disk.

In the case of a basic disk device, it can have multiple partitions
that each correspond to a volume device, \Device\HarddiskVolume<N>. In
this case, each volume device consists of a single extent on a single
disk.

> The only way I can find to accomplish this is to iterate through each volume

You can get the list of volume GUID names of volume devices that are
registered with the mountpoint manager via FindFirstVolumeW /
FindNextVolumeW. Unfortunately, PyWin32 doesn't wrap the volume-find
functions. They can be called using ctypes. For example:

import ctypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

class HANDLE(ctypes.c_void_p):
    def __eq__(self, other):
        if hasattr(other, 'value'):
            return self.value == other.value
        return False

INVALID_HANDLE_VALUE = HANDLE(-1)
ERROR_NO_MORE_FILES = 18

kernel32.FindFirstVolumeW.restype = HANDLE

def list_volumes():
    volumes = []
    buf = (ctypes.c_wchar * 260)()
    hfind = kernel32.FindFirstVolumeW(buf, len(buf))
    if hfind == INVALID_HANDLE_VALUE:
        raise ctypes.WinError(ctypes.get_last_error())
    try:
        while True:
            # Strip the trailing backslash that FindNextVolumeW appends.
            # The trailing backslash is the mountpoint of the filesystem
            # that mounts the volume device, but we only want the GUID
            # device name.
            volumes.append(buf.value.rstrip('\\'))
            if not kernel32.FindNextVolumeW(hfind, buf, len(buf)):
                error = ctypes.get_last_error()
                if error != ERROR_NO_MORE_FILES:
                    raise ctypes.WinError(error)
                break
    finally:
        kernel32.FindVolumeClose(hfind)
    return volumes

I subclassed c_void_p to create the HANDLE type in order to avoid the
otherwise automatic conversion of the return value to a builtin Python
type. This is a simple way around needing to declare argtypes for the
functions that take the find handle (actually a pointer to memory) as
an argument.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210210/1d475b4a/attachment.html>

From rhyslloyd1 at protonmail.com  Tue Feb 16 14:55:48 2021
From: rhyslloyd1 at protonmail.com (rhyslloyd1)
Date: Tue, 16 Feb 2021 19:55:48 +0000
Subject: [python-win32] Opening existing memory mapped files with pywin32
Message-ID: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>

Hello. Sorry if this is the wrong place to ask my question.

I am trying to open a memory mapped file using Python. I originally used the "mmap" module from Python however I had issues with it because I had to use a fixed size for the file even though my goal was to open an existing file of which the size would be unknown. I am now using the "mmapfile.mmapfile" function. My code looks like [this](https://pastebin.com/QygT2wp6). In the [docs](http://timgolden.me.uk/pywin32-docs/mmapfile__mmapfile_meth.html) it says I can use a file name of "None" if I'm planning on opening an existing file. [This](https://pastebin.com/2FVhpDiB) is the error I get when running my code.

I'm a novice Python user and I don't know much at all about the Windows API. Thanks in advance for any help!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210216/f7b93752/attachment.html>

From timr at probo.com  Tue Feb 16 16:58:21 2021
From: timr at probo.com (Tim Roberts)
Date: Tue, 16 Feb 2021 13:58:21 -0800
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
Message-ID: <4fac361f-b19d-05c2-9e75-e686ed10d4b6@probo.com>

rhyslloyd1 via python-win32 wrote:
>
> I am trying to open a memory mapped file using Python. I originally 
> used the "mmap" module from Python however I had issues with it 
> because I had to use a fixed size for the file even though my goal was 
> to open an existing file of which the size would be unknown.

The file size isn't really "unknown".? It's just not known in advance.? 
There are many ways to find the size of a file.? Plus, the "mmap" module 
accepts 0 as a size parameter, meaning "map the whole file".


> I am now using the "mmapfile.mmapfile" function. My code looks like 
> this <https://pastebin.com/QygT2wp6>. In the docs 
> <http://timgolden.me.uk/pywin32-docs/mmapfile__mmapfile_meth.html>?it 
> says I can use a file name of "None" if I'm planning on opening an 
> existing file. This <https://pastebin.com/2FVhpDiB> is the error I get 
> when running my code.

There are a couple of misunderstanding here.? It is important to realize 
that the name of the mapping object is quite different from the name of 
the file being mapped.? The Windows memory-mapping concept is quite 
general, and is often used as a way to share memory between multiple 
processes.? In that case, you just want a chunk of memory without 
bothering with a file on disk.? In that case, you'd create a mapping 
object with a name, but no file name.

If you're just mapping an existing file, you need to supply the file 
name, but you don't need to supply a name for the mapping object.? The 
code you showed creates a shared memory object that does not map a file 
on disk.? The shared memory object has a name (the GUID), and another 
application could open that named object, but it doesn't map to a file.

If you want to open a mapping to an existing file, specify the name of 
the file as File= and specify Name=None.

However, you can do all of this with the mmap module as well.? If you 
have a file called "data.bin", you can do

 ??? import mmap
 ??? fn = open('data.bin','rb')
 ??? data = mmap.mmap( fn.fileno(), 0, access=mmap.ACCESS_READ )

That maps the whole file, and len(data) will tell you how large it is.

-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3389 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210216/bf7746b5/attachment.bin>

From eryksun at gmail.com  Tue Feb 16 19:00:03 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Tue, 16 Feb 2021 18:00:03 -0600
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
Message-ID: <CACL+1ausYKD2HTJbDde0fbEP9i-zR_8Vu25fM==SwZya59L1AQ@mail.gmail.com>

On 2/16/21, rhyslloyd1 via python-win32 <python-win32 at python.org> wrote:
>
> I am trying to open a memory mapped file using Python. I originally used the
> "mmap" module from Python however I had issues with it because I had to use
> a fixed size for the file even though my goal was to open an existing file
> of which the size would be unknown.

I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.

In theory, shared_memory.SharedMemory() in the multiprocessing package
could support this case. But in practice it's still based on the mmap
module. It tries to improve the situation by opening and mapping an
existing mapping via WinAPI OpenFileMappingW() [1] and MapViewOfFile()
[2] in order to query the region size via VirtualQuery(), but this
isn't reliable because it returns the size rounded up to the nearest
page boundary. Calling mmap.mmap(-1, size, tagname=name) with this
queried size may fail the internal MapViewOfFile() call with
ERROR_ACCESS_DENIED (5) if the size is bigger than the maximum size of
the file mapping. (The underlying status from the NT system call more
clearly describes the failure as STATUS_INVALID_VIEW_SIZE.)

The problem with the standard library mmap module and PyWin32's
mmapfile is that they're designed to open a mapping only via
CreateFileMappingW() [3]. When this function is called with hFile as
INVALID_HANDLE_VALUE (i.e. use the system paging file) and the maximum
size as 0 (i.e. dwMaximumSizeHigh=0, dwMaximumSizeLow=0), it always
fails with ERROR_INVALID_PARAMETER (87). It does not check whether
lpName is the name of an existing mapping before validating the
parameters and failing. Hypothetically, it could have been written to
check for this case, since the file handle is ignored anyway for an
existing mapping. But it is what it is, and has been so for about 30
years.

A high-level library function needs to be aware of this limitation and
handle it appropriately by implementing an open disposition (e.g.
create-new, open-always, open-existing) that can dictate the use of
OpenFileMappingW() in addition to or instead of CreateFileMapping().

---
[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-openfilemappinga
[2] https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile
[3] https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemappingw

From timr at probo.com  Tue Feb 16 19:57:50 2021
From: timr at probo.com (Tim Roberts)
Date: Tue, 16 Feb 2021 16:57:50 -0800
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <CACL+1ausYKD2HTJbDde0fbEP9i-zR_8Vu25fM==SwZya59L1AQ@mail.gmail.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
 <CACL+1ausYKD2HTJbDde0fbEP9i-zR_8Vu25fM==SwZya59L1AQ@mail.gmail.com>
Message-ID: <456aca84-93da-e41f-3879-91a5c2c0fb1a@probo.com>

Eryk Sun wrote:
>
> I'm sorry to say that the only reliable solution is to directly call
> WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
> interface (FFI) package such as ctypes or CFFI. I can write up an
> example if you need help.

Respectfully, I think you have misread the original message.? He is a 
relative beginner, just trying to do a simple file mapping. Based on his 
description, mmap should be able to do what he needs.

I'm willing to be proven wrong, but I'm afraid you may be sending him on 
an unnecessary wild goose chase.

-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3389 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210216/f12e14c4/attachment-0001.bin>

From eryksun at gmail.com  Tue Feb 16 20:57:50 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Tue, 16 Feb 2021 19:57:50 -0600
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <456aca84-93da-e41f-3879-91a5c2c0fb1a@probo.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
 <CACL+1ausYKD2HTJbDde0fbEP9i-zR_8Vu25fM==SwZya59L1AQ@mail.gmail.com>
 <456aca84-93da-e41f-3879-91a5c2c0fb1a@probo.com>
Message-ID: <CACL+1av52-Mor3gkwn_09NiVMOPa7-p+weUDtFjd+HajXYyERw@mail.gmail.com>

On 2/16/21, Tim Roberts <timr at probo.com> wrote:
> Eryk Sun wrote:
>>
>> I'm sorry to say that the only reliable solution is to directly call
>> WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
>> interface (FFI) package such as ctypes or CFFI. I can write up an
>> example if you need help.
>
> Respectfully, I think you have misread the original message.  He is a
> relative beginner, just trying to do a simple file mapping. Based on his
> description, mmap should be able to do what he needs.

Sorry, I'm sure you're right, now that I think about it. I was
responding reflexively based on the OP's code that attempts to
create/open a file mapping with a name and 0 maximum size (i.e.
unknown size; just map it all and query the size), since I've
previously seen people trying and failing to do that with mmap in
various IPC cases. The GUID name didn't help to clarify matters, since
it's just as likely to be generated as a unique object name in the
current session. I also think at unconscious level I didn't want to
assume a confused misreading of the documented function parameters.

Anyway, maybe the discussion will help someone else that's searching
for information on the topic.

From timr at probo.com  Wed Feb 17 02:56:20 2021
From: timr at probo.com (Tim Roberts)
Date: Tue, 16 Feb 2021 23:56:20 -0800
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <Z7wJG5MNffnR7F1joEMG2GWob0w681gmvynK34xyL6AsjfWnR7KTTpRXYO3hgVok-ns1OHRJZmZ5IWO5sBQ3OinPAI4Ba0YnY7TH_42oJ28=@protonmail.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
 <4fac361f-b19d-05c2-9e75-e686ed10d4b6@probo.com>
 <Z7wJG5MNffnR7F1joEMG2GWob0w681gmvynK34xyL6AsjfWnR7KTTpRXYO3hgVok-ns1OHRJZmZ5IWO5sBQ3OinPAI4Ba0YnY7TH_42oJ28=@protonmail.com>
Message-ID: <05D4FDCC-19A0-40DE-AFAA-BB561BB5501D@probo.com>

On Feb 16, 2021, at 5:49 PM, rhyslloyd1 <rhyslloyd1 at protonmail.com> wrote:
> 
> Thanks for the quick response! I've made some progress since reading. My goal is to be able to talk to the NVIDIA Share API using Python. I found this <https://github.com/cm-pony/Experienceless/issues/1#issuecomment-763496808> post on GitHub the other day and I wanted to implement it myself. I've obviously bitten off more than I could chew.

I was wrong.  What you have there is the name of a mapping object, not the name of a file.


> Also the "read_line" method doesn't exist apparently. This <https://pastebin.com/JjrA4jEJ> is the output now.

Your snippet is private; we can?t read it.  No, ?read_line? isn?t going to work.  This is not a file, it?s just a chunk of memory ? a string of bytes.  The implication from your link is that this is a JSON string.  If so, you should be able to read() the whole thing, do .decode(?utf-8?) and pass the result to a JSON decoder.
? 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210216/a7134974/attachment.html>

From nick.p.orr.spam at gmail.com  Wed Feb 17 09:07:19 2021
From: nick.p.orr.spam at gmail.com (Nick Orr)
Date: Wed, 17 Feb 2021 09:07:19 -0500
Subject: [python-win32] Bug When Reading In PSTs
Message-ID: <CAEFiNcckm7cCwivWMWR7LTKd_pnZpOyf3qJEfB8P2hXnn6oUJQ@mail.gmail.com>

Hi all,

I've been developing a Python tool to ingest and write all emails from a
PST exported from Outlook to individual .html files. The issue is that when
opening the PST in outlook and checking the source information for emails
individually, it includes this specific line:

<meta http-equiv=Content-Type content="text/html; charset=utf-8">

which IS NOT being included when importing the PST with Pywin32 and reading
all the emails in the PST. To see what it looks like in a chunk -

>From Outlook I get:<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="
http://schemas.microsoft.com/office/2004/12/omml" xmlns="
http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type
content="text/html; charset=utf-8"><meta name=Generator content="Microsoft
Word 15 (filtered medium)">

What is exported from the tool: <html
xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="
http://schemas.microsoft.com/office/2004/12/omml" xmlns="
http://www.w3.org/TR/REC-html40"><head><meta name=Generator
content="Microsoft Word 15 (filtered medium)">

The contents of the emails are otherwise ENTIRELY identical except for that
one tag.

My code:
-----------------------------------------------------------------------------------------------------

def find_pst_folder(OutlookObj, pst_filepath):
    for Store in OutlookObj.Stores:
        if Store.IsDataFileStore  and Store.FilePath == pst_filepath:
            return Store.GetRootFolder()
    return None

def enumerate_folders(FolderObj):
    for ChildFolder in FolderObj.Folders:
        enumerate_folders(ChildFolder)
    iterate_messages(FolderObj)

def iterate_messages(FolderObj):
    global mycounter2
    global encryptedEmails
    global richPlainEmails
    global totalEmails
    global htmlEmails

    for item in FolderObj.Items:
        totalEmails += 1
        try:
            try:
                body_content = item.HTMLbody
                mysubject = item.Subject
                writeToFile(body_content, exportPath, mysubject)
                mycounter2 = mycounter2 + 1
                htmlEmails += 1
            except AttributeError:
                #print('Non HTML formatted email, passing')
                richPlainEmails += 1
                pass
        except Exception as e:
            encryptedEmails += 1
            pass

def writeToFile(messageHTML, path, mysubject):
    global mycounter2
    filename = '\htmloutput' + str(mycounter2) + '.html'

    #Check if email is rich or plain text first (only HTML emails are
desired)
    if '<!-- Converted from text/plain format -->' in messageHTML or '<!--
Converted from text/rtf format -->' in messageHTML:
        raise AttributeError()

    else:
        file = open(path + filename, "x", encoding='utf-8')
        try:
            messageHTML = regex.sub('\r\n', '\n', messageHTML)
            file.write(messageHTML)

        #Handle any potential unexpected Unicode error
        except Exception as e:
            print('Exception: ' , e)
            try:
                #Prints email subject to more easily find the offending
email
                print('Subject: ', mysubject)
                print(mycounter2)
                file.write(messageHTML)
            except Exception as e:
                print('Tried utf decode: ', e)

        file.close()

htmlEmails = 0
encryptedEmails = 0
totalEmails = 0
richPlainEmails = 0
filenameCount = 1
mycounter2 = 1

#Adjusting name of PST location to be readable
selectedPST = str(selectedPST.replace('/', '\\'))
print('\nRunning:' , selectedPST)
outlook.AddStore(selectedPST)
PSTFolderObj = find_pst_folder(outlook, selectedPST)


-----------------------------------------------------------------------------------------------------

Because the emails otherwise are identical, I can only assume this is being
done by the library. I'm wondering if there's a reason that meta tag is
excluded, or if it's a bug in PyWin32?

Thanks for any input,
-Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210217/48874268/attachment.html>

From timr at probo.com  Thu Feb 18 02:12:14 2021
From: timr at probo.com (Tim Roberts)
Date: Wed, 17 Feb 2021 23:12:14 -0800
Subject: [python-win32] Bug When Reading In PSTs
In-Reply-To: <CAEFiNcckm7cCwivWMWR7LTKd_pnZpOyf3qJEfB8P2hXnn6oUJQ@mail.gmail.com>
References: <CAEFiNcckm7cCwivWMWR7LTKd_pnZpOyf3qJEfB8P2hXnn6oUJQ@mail.gmail.com>
Message-ID: <FBEC5D57-4DE2-49A2-B1CC-EBF1060C2C32@probo.com>

On Feb 17, 2021, at 6:07 AM, Nick Orr <nick.p.orr.spam at gmail.com> wrote:
> 
> I've been developing a Python tool to ingest and write all emails from a PST exported from Outlook to individual .html files. The issue is that when opening the PST in outlook and checking the source information for emails individually, it includes this specific line:
> 
> <meta http-equiv=Content-Type content="text/html; charset=utf-8">
> 
> which IS NOT being included when importing the PST with Pywin32 and reading all the emails in the PST. To see what it looks like in a chunk - 

What you HAVEN?T said here is how you are talking to Outlook ? how you generated your ?outlook? object.  PyWin32 doesn?t have any code that is Outlook-specific.  I assume you?re using win32com.client.Dispatch.  If so, remember that Python isn?t doing any processing here.  It?s just passing requests through COM to Outlook.  If your text is coming back oddly, then Outlook is returning it oddly.

<meta>  tags are intended for web servers; it?s possible that Outlook is absorbing the <meta> tag because it isn?t useful.  Maybe it has copied the charset to a property of the message object to reflect the character set.  That is, perhaps there?s something in the ?item? object that gets tweaked.


> Because the emails otherwise are identical, I can only assume this is being done by the library. I'm wondering if there's a reason that meta tag is excluded, or if it's a bug in PyWin32?

No, it?s not being done by PyWin32.  It?s being done by Outlook.  You?d get the same result if you called this method from C#.
? 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.


From timr at probo.com  Thu Feb 18 14:54:19 2021
From: timr at probo.com (Tim Roberts)
Date: Thu, 18 Feb 2021 11:54:19 -0800
Subject: [python-win32] Bug When Reading In PSTs
In-Reply-To: <CAEFiNccbxOhVLMENj+bzV-H=wR-nTX+Y2U3BzWj48JQ64N7+aA@mail.gmail.com>
References: <CAEFiNcckm7cCwivWMWR7LTKd_pnZpOyf3qJEfB8P2hXnn6oUJQ@mail.gmail.com>
 <FBEC5D57-4DE2-49A2-B1CC-EBF1060C2C32@probo.com>
 <CAEFiNccbxOhVLMENj+bzV-H=wR-nTX+Y2U3BzWj48JQ64N7+aA@mail.gmail.com>
Message-ID: <b1f5c357-7647-1d25-65d5-4fa4ccac3eae@probo.com>

Nick Orr wrote:
>
> This seems to be the consensus from two others that I've heard back 
> from. Unfortunately this is an issue for me (it's always that one same 
> meta tag: <meta http-equiv=Content-Type content="text/html; 
> charset=utf-8"> ) as it's being requested by the project to have the 
> contents be identical to the email in Outlook. I just don't see any 
> possible way to achieve this given that it's stripped and inferring it 
> risks inaccuracy.

Do you know whether the message actually contained that line to begin 
with?? If this happens with messages you send, try sending one to a 
Linux mailbox somewhere, and take a look at the raw text.? It's also not 
impossible that Outlook adds line for export, but that it's not actually 
present in the message text.

Outlook takes liberties with message content that it probably shouldn't.

-- 
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3389 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.python.org/pipermail/python-win32/attachments/20210218/12e761c7/attachment.bin>

From eryksun at gmail.com  Sun Feb 21 08:47:24 2021
From: eryksun at gmail.com (Eryk Sun)
Date: Sun, 21 Feb 2021 07:47:24 -0600
Subject: [python-win32] Opening existing memory mapped files with pywin32
In-Reply-To: <oCZTzXj8alNLDHI1fasNYur71JY82OvVWavMYWhTxJkItaWLqd3yTHMZRBqUpYkKjveJNymjDVv_ThPXBPGUFDMXfr_dPML9HHAj6LQ8AA8=@protonmail.com>
References: <uHDtPcDx_RURlePeHdKeAI0c4unETgWhyI3s3454rDV8G0gZBxZHvpwzkflEB4TugieyZpYiS8Z7tFnPAHFcavLzuU5xxUSE1zTDMXrgAmk=@protonmail.com>
 <CACL+1ausYKD2HTJbDde0fbEP9i-zR_8Vu25fM==SwZya59L1AQ@mail.gmail.com>
 <oCZTzXj8alNLDHI1fasNYur71JY82OvVWavMYWhTxJkItaWLqd3yTHMZRBqUpYkKjveJNymjDVv_ThPXBPGUFDMXfr_dPML9HHAj6LQ8AA8=@protonmail.com>
Message-ID: <CACL+1avr-xqLLt4XiRMOsxNb4EEYRA0OniRue7h5KODNXCEm+g@mail.gmail.com>

On 2/21/21, rhyslloyd1 <rhyslloyd1 at protonmail.com> wrote:
>
> In my head my ideal outcome is being able to map the whole memory mapped
> file and not have to specify a size for it incase it changes size however I
> don't understand the concept of a memory mapped file so I maybe my very
> limited understanding is just incorrect, thanks in advance for any response!

The address space of a process is managed in pages of a given fixed
size (e.g. 4 KiB). The state of each page is tracked in an entry in
the process page table, i.e. a page-table entry (PTE). When a page is
sharable between processes, its PTE in a particular process references
a prototype PTE that manages the overall state of the page (e.g.
whether the page is resident in RAM or paged out to a filesystem
paging file or data file).

The system uses a kernel object called a Section to reference the
prototype PTEs in a block of shared memory. In the Windows API, a
Section is called a "file mapping" because the pages mapped into
memory are always a byte range in a filesystem paging file or data
file. At the time of creation via CreateFileMappingW, a file mapping
can be assigned a name, either a global name for all sessions, or a
local name in the current desktop session. This allows a process to
easily try to open a file mapping (e.g. via OpenFileMapping) and use
it to map all or a subset of the shared pages into its address space
(e.g. via MapViewOfFile).

In your case, I gather that you need to open a file mapping named
"{8BA1E16C-FC54-4595-9782-E370A5FBE8DA}" and map the whole block of
shared memory into the current process, without knowing how large it
is. You'll need to query the size of the mapped region (i.e. the
RegionSize) via VirtualQuery. Preferably also wrap the memory in an
easy interface such as a memoryview, which should automatically unmap
the memory when finalized.

Here's a map_section(name, mode) function that implements this. Note
that for reliable operation it requires memoryview.toreadonly(), which
was added in Python 3.8. The underlying ctypes array allows writing to
readonly memory, which will crash the process with an access
violation. That's avoided in 3.8+ by returning a readonly memoryview
when the shared memory is mapped without write access.

import warnings
import ctypes
from ctypes import wintypes

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

FILE_MAP_READ = SECTION_MAP_READ = 0x0004
FILE_MAP_WRITE = SECTION_MAP_WRITE = 0x0002

class MEMORY_BASIC_INFORMATION(ctypes.Structure):
    _fields_ = (('BaseAddress', wintypes.LPVOID),
               ('AllocationBase', wintypes.LPVOID),
               ('AllocationProtect', wintypes.DWORD),
               ('PartitionId', wintypes.WORD),
               ('RegionSize', ctypes.c_size_t),
               ('State', wintypes.DWORD),
               ('Protect', wintypes.DWORD),
               ('Type', wintypes.DWORD))

PMEMORY_BASIC_INFORMATION = ctypes.POINTER(MEMORY_BASIC_INFORMATION)

kernel32.OpenFileMappingW.restype = wintypes.HANDLE
kernel32.OpenFileMappingW.argtypes = (wintypes.DWORD, wintypes.BOOL,
    wintypes.LPWSTR)

kernel32.MapViewOfFile.restype = wintypes.LPVOID
kernel32.MapViewOfFile.argtypes = (wintypes.HANDLE, wintypes.DWORD,
    wintypes.DWORD, wintypes.DWORD, ctypes.c_size_t)

kernel32.UnmapViewOfFile.restype = wintypes.BOOL
kernel32.UnmapViewOfFile.argtypes = (wintypes.LPCVOID,)

kernel32.VirtualQuery.restype = ctypes.c_size_t
kernel32.VirtualQuery.argtypes = (wintypes.LPCVOID,
    PMEMORY_BASIC_INFORMATION, ctypes.c_size_t)

class BaseSharedMem(ctypes.Array):
    _type_ = ctypes.c_char
    _length_ = 0

    def __del__(self, *, UnmapViewOfFile=kernel32.UnmapViewOfFile,
                warn=warnings.warn):
        if not UnmapViewOfFile(self):
            warn("UnmapViewOfFile failed", ResourceWarning, source=self)


def map_section(name, mode='r'):
    mbi = MEMORY_BASIC_INFORMATION()

    if mode == 'r':
        access = FILE_MAP_READ
    elif mode == 'w':
        access = FILE_MAP_WRITE
    elif mode in ('rw', 'wr'):
        access = FILE_MAP_READ | FILE_MAP_WRITE
    else:
        raise ValueError('invalid mode')

    h = kernel32.OpenFileMappingW(access, False, name)
    if not h:
        raise ctypes.WinError(ctypes.get_last_error())

    try:
        address = kernel32.MapViewOfFile(h, access, 0, 0, 0)
        if not address:
            raise ctypes.WinError(ctypes.get_last_error())
    finally:
        kernel32.CloseHandle(h)

    result = kernel32.VirtualQuery(address, ctypes.byref(mbi),
                ctypes.sizeof(mbi))
    if not result:
        ex = ctypes.WinError(ctypes.get_last_error())
        if not kernel32.UnmapViewOfFile(address):
            ex2 = ctypes.WinError(ctypes.get_last_error())
            raise ex2 from ex
        raise ex

    array_t = type('SharedMem_{}'.format(mbi.RegionSize),
                (BaseSharedMem,), {'_length_': mbi.RegionSize})
    mv = memoryview(array_t.from_address(address)).cast('B')

    if 'w' not in mode:
        return getattr(mv, 'toreadonly', lambda: mv)()
    return mv