Today I researched how to implement in C# a method that forces a COM port to close when it is open by another process. This was found to be a complex task that involves using undocumented Win32 API functions. I didn’t finish it because I found another way to workaround the problem but the idea is the following:

1) Enumarate all Win32 processes.

2) In each process enumerate all its handles and find the handle to be closed (e.g. \Device\Serial0).

3) Close the handle.

The first task is trivial. We can use System.Diagnostics.Process.GetProcesses(). The second is possible with the Windows NT undocumented function NtQuerySystemInformation. The third task is simple. Once we have the process ID (pid) and the handle of the open COM port, we can close it with the following CloseHandleEx function:

const uint PROCESS_DUP_HANDLE = 0x0040;

[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool CloseHandle(IntPtr hObject);

[DllImport("kernel32.dll")]
static extern IntPtr OpenProcess(ProcessAccessFlags dwDesiredAccess,
    [MarshalAs(UnmanagedType.Bool)] bool bInheritHandle, IntPtr dwProcessId);

[Flags]
enum ProcessAccessFlags : uint
{
    All = 0x001F0FFF,
    Terminate = 0x00000001,
    CreateThread = 0x00000002,
    VMOperation = 0x00000008,
    VMRead = 0x00000010,
    VMWrite = 0x00000020,
    DupHandle = 0x00000040,
    SetInformation = 0x00000200,
    QueryInformation = 0x00000400,
    Synchronize = 0x00100000
}

[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool DuplicateHandle(IntPtr hSourceProcessHandle,
   IntPtr hSourceHandle, IntPtr hTargetProcessHandle, out IntPtr lpTargetHandle,
   uint dwDesiredAccess, [MarshalAs(UnmanagedType.Bool)] bool bInheritHandle, DuplicateOptions dwOptions);

[Flags]
enum DuplicateOptions : uint
{
    DUPLICATE_CLOSE_SOURCE = (0x00000001),
    DUPLICATE_SAME_ACCESS = (0x00000002),
}

public static bool CloseHandleEx(IntPtr pid, IntPtr handle)
{
    IntPtr hProcess = OpenProcess(ProcessAccessFlags.DupHandle, false, pid);
    IntPtr dupHandle = IntPtr.Zero;
    bool success = DuplicateHandle(hProcess, handle, IntPtr.Zero, out dupHandle, 0, false, DuplicateOptions.DUPLICATE_CLOSE_SOURCE);
    CloseHandle(hProcess);
    return success;
}