Saturday, May 12, 2018

Howto launch and debug in VSCode using the debug adapter protocol (part 2)

Ok, after the basic infracstructure, the next thing to do is actually launch some program without worrying about the debugger, so, we'll just run a program without being in debug mode to completion, show its output and terminate it when requested.

To launch a program, our debug adapter must treat the 'LaunchRequest' message and actually run the program (bear in mind that we'll just launch it without doing any debugging at this point).

The first point then is how to actually launch it. We provided options for the debugger to be launched with different console arguments specifying where to launch it (either just showing output to the debug console, using the integrated terminal or using an external terminal).

So, let's start with just showing output in the debug console.

Launching it should be simple: just generate a command line while treating the 'LaunchRequest' message, but then, based on the console specified some things may be different...

Let's start handling just showing the output on the debug console.

To do that we have launch the command properly redirecting the output to pipes (for python it's something as subprocess.Popen([sys.executable, '-u', file_to_run], stdout=subprocess.PIPE, stderr=subprocess.PIPE) and then create threads which will read that output to provide it back to vscode (so, when output is obtained, an OutputEvent must be given).

Also, create another thread so that when the process finishes, a TerminatedEvent is given (in python, just do a popen.wait() in a thread and when complete send the TerminatedEvent -- you may want to synchronize to make sure other threads related to output have finished before doing that).

At this point, we can run something and should be able to see anything printed both to stdout and stderr and when the process finishes, VSCode itself acknowledges that and closes the related controls.

Great! On to launching in the integrated terminal!

So, to launch in the terminal we have to first actually check if the client does support running in the terminal... in the InitializeRequest, if it is supported, we should've received in the arguments "supportsRunInTerminalRequest": True (if it doesn't, in my case I just fall back to the debug console).

This also becomes a little bit trickier because at this point we're the ones doing the request (RunInTerminalRequest) and the client should send a response (RunInTerminalResponse). So, on to it: when the client launches, create a RunInTerminalRequest with the proper kind ('internal' or 'external') and wait for the response.

At this point, the processId may not actually be available after launching in that mode (the RunInTerminalResponse processId is optional), which means that if we didn't really create a debugger (just a simple run), we're blind... we could do another program to launch it and return the pid to be able to notify that it was stopped and to kill it when needed, but this seems a bit overkill for me and I couldn't find any info on the proper behavior here, so, I decided that when the user chooses that mode with 'noDebug' I'll simply notify that the debug session is finished for the adapter with a TerminatedEvent (and the user can see the output and Ctrl+C it in the actual terminal).

As a note the 'noDebug' option is added behind the scenes by VSCode depending on whether the user has chosen to do a debug or run for the selected launch (so, it shouldn't be a part of the declared configuration in the extension).

Now, thinking a bit more about it, there's a caveat: when launching with the redirection to the debug console, we should treat sending to stdin too (we don't want to create a process he can't do any communication with later on).

To do that in 'noDebug' should be simple... when we receive an 'EvaluateRequest', we'll send it to stdin (when actually in debug mode we probably have to check the current debugger state to determine if we want to do an evaluation or send to stdin -- i.e.: if we are stopped in a breakpoint we may want to evaluate and not send to stdin).

As a note, after playing with it more I renamed the "console" option to "terminal" with options "none", "internal", "external" as I think that's a better representation of what's expected.

So, that's it for part 2: we're launching a program and redirecting the output as requested by the user (albeit without actually debugging it for now).

Wednesday, May 09, 2018

Howto launch and debug in VSCode using the debug adapter protocol (part 1)

This is a walkthrough with the steps I'm taking to add support to launch and debug a Python script in PyDev for VSCode (note that I'm writing as I'm learning).

The debugger protocol is the protocol used in VSCode to talk to debuggers and handle launching in general (the naming may be a bit weird as the same protocol is used for regular launches and debugging, but apparently the team first did the debugging and then launching came as an afterthought just passing a separate flag during the launching of the program to specify that no debugging should be done -- and not the other way around as I think would be more common).

There is an overview of the protocol at https://code.visualstudio.com/docs/extensionAPI/api-debugging and https://code.visualstudio.com/docs/extensionAPI/extension-points provides more information on what an extension must use to provide a debugger.

There's also a json schema which specifies the format of the messages sent back and forth in the debugger at https://raw.githubusercontent.com/Microsoft/vscode-debugadapter-node/master/debugProtocol.json.

But, after reading all that, it seems that many things are still cloudy on my head on how to actually go on about it and what should be done concretely to implement a debugger in VSCode.

So, my approach is getting the debugProtocol.json, converting it to a structure with Python classes (so that each message that can be sent has a Python representation) and playing a bit doing a debugger stub, just to exercise a dummy debugger talking to VSCode (but without actually doing anything).

It's interesting to note that the first thing to do is actually making the debugger available in the extension. For that, I've used the json below in package.json (as a note, my package.json is actually generated from Python code, so, the structure below is actually a Python dict which is later converted to json, not the actual json -- if you're doing a VSCode extension, I highly recommend generating your package.json and parts of the code that are related and not doing it all by hand... this way it's possible to see it in small pieces and auto generate command ids and the related code, etc... initially I haven't done so in PyDev, but as the declarative files grow, it becomes harder to follow and make changes while keeping the code and declaration in sync):

  
{
    'type': 'PyDev',
    'label': 'PyDev (Python)',
    'languages': ['python'],
    'adapterExecutableCommand': 'pydev.start.debugger', 
    
    # Note: adapterExecutableCommand will be replaced by a different API (right now still in proposal mode). 
    # See: https://code.visualstudio.com/updates/v1_20#_debug-api
    # See: https://github.com/Microsoft/vscode/blob/7636a7d6f7d2749833f783e94fd3d48d6a1791cb/src/vs/vscode.proposed.d.ts#L388-L395
    
    'enableBreakpointsFor': {
        'languageIds': ['python', 'html'],
    },
    'configurationAttributes': {
        'launch': {
            'required': [
                'mainModule'
            ],
            'properties': {

                'mainModule': {
                    'type': 'string',
                    'description': 'The .py file that should be debugged.',
                },

                'args': {
                    'type': 'string',
                    'description': 'The command line arguments passed to the program.'
                },

                "cwd": {
                    "type": "string",
                    "description": "The working directory of the program.",
                    "default": "${workspaceFolder}"
                },

                "console": {
                    "type": "string",
                    "enum": [
                        "integratedTerminal",
                        "externalTerminal"
                    ],
                    "enumDescriptions": [
                        "VS Code integrated terminal.",
                        "External terminal that can be configured in user settings."
                    ],
                    "description": "The specified console to launch the program.",
                    "default": "integratedTerminal"
                },
            }
        }
    },

    "configurationSnippets": [
        {
            "label": "PyDev: Launch Python Program",
            "description": "Add a new configuration for launching a python program with the PyDev debugger.",
            "body": {
                "type": "PyDev",
                "name": "PyDev Debug (Launch)",
                "request": "launch",
                "cwd": "^\"\\${workspaceFolder}\"",
                "console": "integratedTerminal",
                "mainModule": "",
                "args": ""
            }
        },
    ]
}


So, although there are many things there, initially we just need to make adapterExecutableCommand return the command to be executed (you could also create a standalone executable or something to run along with a supported vm -- such as mono, but there's nothing for python there, so, the adapterExecutableCommand is probably the best approach for a python debugger).

In my case it's something as:

  
commands.registerCommand('pydev.start.debugger', () => {
    return {
        command: "C:/bin/python27/python.exe",  // paths initially hardcoded for simplicity
        args: ["X:/vscode-pydev/vscode-pydev/src/debug_adapter/debugger_protocol.py"]
    }
});
  

The configurationSnippets section provides the snippets which allow VSCode to autogenerate the configuration for the user and the configurationAttributes are actually custom for each implementation (so, those will probably need more tweaking going forward).

Another interesting point is that when VSCode launches the debug adapter it'll use stdin and stdout to communicate with the adapter (this makes some things a bit quirky to develop the debugger because you have to (initially) resort to printing debug information to a file to be able to check what's happening, although on the bright side, you won't have to worry about having a firewall at that point).

Also, don't forget to flush after writing messages to stdout.

Now, on to the protocol itself... I created something which would read from stdin and then redirect that to a file to see what's coming (after digging up things a bit more I found an issue in the VSCode tracker referencing: https://github.com/buggerjs/bugger-v8-client/blob/master/PROTOCOL.md which details that a bit more -- although not all that's there is actually applicable to the VSCode debugger). 

The first message that arrives from stdin is:

  
Content-Length: 312\r\n
\r\n
{
    "arguments": {
        "adapterID": "PyDev", 
        "clientID": "vscode", 
        "clientName": "Visual Studio Code", 
        "columnsStartAt1": true, 
        "linesStartAt1": true, 
        "locale": "en-us", 
        "pathFormat": "path", 
        "supportsRunInTerminalRequest": true, 
        "supportsVariablePaging": true, 
        "supportsVariableType": true
    }, 
    "command": "initialize", 
    "seq": 1, 
    "type": "request"
}

-- this is the InitializeRequest in the json schema.

So, it seems a regular http-protocol, sending json contents as the actual content... so, in response to that, the debug adapter should do its initialization and return the capabilities it has -- something as:

  
{
    "seq": 1,
    "request_seq": 1, 
    "command": "initialize", 
    "body": {"supportsConfigurationDoneRequest": true, 
             "supportsConditionalBreakpoints": true}, 
    "type": "response", 
    "success": true
}

-- this is the InitializeResponse in the json schema.

and then send and event saying that it has initialized properly:

  
{"type": "event", "event": "initialized", "seq": 2}

-- this is the InitializedEvent in the json schema.

Note that those are all http responses, so, the Content-Length: $size\r\n\r\n needs to be passed on each request (note that each message sent or received has a seq, which is a number that should be raised whenever a new message is sent -- the seq is raised independently on the server and on the client and responses should reference the seq from the request in request_seq). 

Afterwards, the client (VSCode) sends the actual launch request (which should be based on the configurationAttributes previously configured). In this case:

  
{
    "arguments": {
        "__sessionId": "474aa497-0a90-4b30-8cc6-edf3bebbe703", 
        "args": "", 
        "console": "integratedTerminal", 
        "cwd": "X:\\vscode_example", 
        "name": "PyDev Debug (Launch)", 
        "program": "X:/vscode_example/robots.py", 
        "request": "launch", 
        "type": "PyDev"
    }, 
    "command": "launch", 
    "seq": 2, 
    "type": "request"
}

-- this is the launch request in the json schema (it comes with additional attributes the user specified in the launch... each extension needs to tweak the actual parameters to its use case).

At this point, it becomes clear that this is really just an adapter: we're expected to actually launch the process and provide the communication layer to the actual debugger (so, the debugger doesn't really have to be changed -- although on some cases that may be benefical if possible... for instance, the debugger could already give output on the variable frames as json so that the message doesn't need to be decoded and recoded in a new format). 

Also, the stdin and stdout may be in use (because VSCode uses it to communicate to the debug adapter), so, it may be hard to reuse this process to be the actual debugger process (for instance, launch could then make main proceed to launch the program in this process if the debugger could directly handle the debug protocol, but then if clients managed to write to the 'real' stdin/stdout handles, the debugger would stop working). 

The launch request just requires a notification that the program was launched, so, the response would be a launch response with an empty body (or if there was some error -- say, the file to be launched no longer exists -- a "message" could be set and "success" could be False). 

  
{
    "request_seq": 2, 
    "command": "launch", 
    "body": {}, 
    "type": "response", 
    "success": true
}
  

-- this is the LaunchResponse in the json schema.

Ok, now, at this point I already have a structure which parses the json and creates python instances for each protocol message (and vice-versa), so, instead of specifying each message in its full format, I'll just reference it from the identifier on the schema instead of the actual json. 

After the launch request, I get a ConfigurationDoneRequest and return the proper ConfigurationDoneResponse and for the ThreadsRequest a ThreadsResponse.
At this point, the debugger will sit idle, waiting for actions from the user or events from our debug adapter (if more than one thread was returned in the ThreadsResponse, the threads will appear in the CallStack).

Now, the only thing different at this point is that the debug controls will appear, so, a pause or stop can be activated from the UI.

Pressing stop will send us a DisconnectRequest (for which a DisconnectResponse should be sent as an acknowledgement) and the pause will send a PauseRequest (which requires us to send back a PauseResponse -- and after a thread is actually paused, a StoppedEvent should be sent). 

Ok, this is the end of part 1 (we have something which can be started and later stopped -- without actually doing anything, so, pretty much a mock debugger)... This actually took me 2 full days to implement (most of the work trying to wrap my head around how things worked and generating python code from the json schema -- I tried some libraries and none of them worked as I needed, so, I rolled my own here). 

My main gripe was the lack of a better documentation on how to approach doing a debug protocol from scratch and how it should work. For instance, it took me quite a while to find a reference to launching from the adapterExecutableCommand where I could construct a command line -- initial references I found pointed only to using an executable or a supported runtime such as mono -- some things I still don't know how to handle such as how to actually provide output based on the console type the user expects: (i.e.: integratedTerminalexternalTerminal) -- anyways, hope to get to that in the upcoming parts... 

The final code I have at this point (which also contains the code generator I did) may be seen at:


Part 2 should get us to the point of actually launching a process...

Wednesday, March 21, 2018

PyDev 6.3.2: support for .pyi files

PyDev 6.3.2 is now available for download.

The main change in this release is that PyDev will now consider .pyi (stub) files when doing type inference, although there's still a shortcoming: the .pyi file must be in the same directory where the typed .py file is and it's still not possible to use it to get type inference for modules which are compiled (for instance PyQt).

I hope to address that in the next release (initially I wanted to delay this release to add full support for .pyi files, but there was a critical bug opening the preferences page for code completion, so, it really couldn't be delayed more, nevertheless, the current support is already useful for users using .pyi files along .py files).

Also, code completion had improvements for discovering whether some call is for a bound or unbound method and performance improvements (through caching of some intermediary results during code completion).

Enjoy!

Thursday, March 01, 2018

PyDev 6.3.1 (implicit namespace packages and Visual Studio Code support)

The major change in this release is that PyDev now recognizes that folders no longer require __init__.py files to be considered a package (PEP 420).

Although this is only available for Python 3.3 onwards, PyDev will now always display valid folders under the PYTHONPATH as if they were packages.

There were also some improvements, such as recognizing that dlls may have a postfix (so that dlls for multiple versions of Python may be available in the same folder) and a number of bugfixes (see: http://www.pydev.org has more details).

Besides those, a good amount of work in this release was refactoring the codebase so that PyDev could also be available as a language server for Python, to enable it to be used on Visual Studio Code (http://www.pydev.org/vscode), so, Visual Studio Code users can now also use many of the nice features on PyDev ;)

Enjoy!

Monday, February 19, 2018

Python with PyDev on Visual Studio Code

PyDev can now be used for Python development on Visual Studio Code!

The first release already provides features such as code analysis, code completion, go to definition, symbols for the workspace and editor, code formatting, find references, quick fixes and more (see http://www.pydev.org/vscode/ for details).

All features have a strong focus on speed and have been shaped by the usage on PyDev over the last 14 years, so, I believe it's already pretty nice to use... there are still some big things to integrate (such as the PyDev debugger), but those should come on shortly.

The requisites are having java 8 (or higher) installed on the system (if it doesn't find it automatically the java home location may need to be specified in the settings -- http://www.pydev.org/vscode/ has more details) and Python 2.6 or newer.

By default it should pick the python executable available on the PATH, but it's possible to specify a different python executable through the settings on VSCode (see http://www.pydev.org/vscode/settings.html for details).

Below, I want to share some of the things that are unique in PyDev and are now available for VSCode users:
  • Niceties from PyDev when typing such as auto-adding self where needed (note that having the editor.formatOnType setting turned on is a requisite for that to work).
  • Really fast code-completion, code-analysis and code-formatting engines.
  • Code completion provides options to import modules, top level classes, methods and variables (python.pydev.preferredImportLocation can be used to determine the location of the import).
  • Quick fix which automatically allows adding an import for unresolved symbols.
  • In-file navigation to previous or next class or method through Ctrl+Shift+Up and Ctrl+Shift+Down.
Now, the extension itself is not currently open source as PyDev... my target is making the best Python development environment around and all earnings will go towards that (as a note, all improvements done to PyDev itself will still be open source, so, most earnings from PyDev on VSCode will go toward open source development).

Enjoy!

Monday, December 04, 2017

Creating extension to profile Python with PyVmMonitor from Visual Studio Code

Ok, so, the target here is doing a simple extension with Visual Studio Code which will help in profiling the current module using PyVmMonitor (http://www.pyvmmonitor.com/).

The extension will provide a command which should open a few options for the user on how he wants to do the profile (with yappi, cProfile or start without profiling but connected with the live sampling view).

I went with https://code.visualstudio.com/docs/extensions/yocode to bootstrap the extension, which gives a template with a command to run then renamed a bunch of things (such as the extension name, description, command name, etc).

Next was finding a way to ask the user for the options (to ask how the profile should be started). Searching for it revealed https://tstringer.github.io/nodejs/javascript/vscode/2015/12/14/input-and-output-vscode-ext.html, so, I went with creating the needed constants and going with vscode.window.showQuickPick (experimenting, undefined is returned if the user cancel the action, so, that needs to be taken into account too).

Now, after the user chooses how to start PyVmMonitor, the idea would be making any launch actually start in the chosen profile mode (which is how it works in PyDev).

After investigating a bit, I couldn't find out how to intercept an existing launch to modify the command line to add the needed parameters for profiling with PyVmMonitor, so, this integration will be a bit more limited than the one in PyDev as it will simply create a new terminal and call PyVmMonitor asking it to profile the currently opened module...

In the other integrations, it was done as a setting where the user selected that it wanted to profile any python launch from a given point onward as a toggle and then intercepted launches changing the command line given, so, for instance, it could intercept a unittest launch too, but in this case, it seems that there's currently no way to do that -- or some ineptitude on my part finding an actual API to do it ;)

Now, searching on the VSCode Python plugin, I found a "function execInTerminal", so, I based the launching in it (but not using its settings as I don't want to add a dependency on it for now, so, I just call `python` -- if that's wrong, as it opens a shell, the user is free to cancel that and correct the command line to use the appropriate python interpreter or change it as needed later on).

Ok, wrapping up: put the initial version of the code on https://github.com/fabioz/vscode-pyvmmonitor. Following https://code.visualstudio.com/docs/extensions/publish-extension did work out, so, there's a "Profile Python with PyVmMonitor" extension now ;).

Some notes I took during the process related to things I stumbled on or found awkard:
  • After publishing the first time and installing, the extension wasn't working because I wrongly put a dependency from npm in "devDependencies" and not in "dependencies" (the console in the developer tools helped in finding out that the dependency wasn't being loaded after the extension was installed).
  • When a dependency is added/removed, npm install needs to be called again, it's not automatic.
  • When uploading the extension I had the (common) error of not generating a token for "all" ;)
  • Apparently there's a "String" and a "string" in TypeScript (or at least within the dependencies when editing VSCode).
  • The whole launching on VSCode seems a bit limited/ad-hoc right now (for instance, .launch files create actual launchers which can be debugged but the python extension completely bypasses that by doing a launch in terminal for the current file -- probably because it's a bit of a pain creating that .launch file) -- I guess this reflects how young VSCode is... on the other hand, it really seems it built upon previous experience as the commands and bindings seems to have evolved directly to a good design (Eclipse painfully iterated over several designs on its command API).
  • Extensions seem to be limited by design. I guess this is good and bad at the same time... good that extensions should never slow down the editor, but bad because they are not able to do something which is not in the platform itself to start with -- for instance, I really missed a good unittest UI when using it... there are actually many other things I missed from PyDev, although I guess this is probably a reflect on how young the platform is (and it does seem to be shaping up fast).

Wednesday, November 29, 2017

PyDev 6.2.0: Interactive Console word wrapping, pytest hyperlinking

PyDev 6.2.0 is mostly a bugfix release, although it does bring some features to the table to such as adding the possibility of activating word-wrapping in the console and support for code-completion using the Python 3.6 variable typing.

Another interesting change is that pytest filenames are properly hyperlinked in the console (until now PyDev resorted to mocking some functions of pytest so that when it printed exceptions it used the default Python traceback format -- now that's no longer done).

See: http://www.pydev.org for complete details.

p.s.: Thank you to all PyDev supporters -- https://www.brainwy.com/supporters/PyDev-- which enable PyDev to keep on being improved!

p.s.: LiClipse 4.4.0 already bundles PyDev 6.2.0, see: http://www.liclipse.com/download.html for download links.