How to establish NETCONF sessions using SSH authentication in AWS ?

[khalilmebarkia]

I'm running docker Yang Development Kit for python ydk-py in my remote server Linux Ubuntu. I would like to Establish a connection using with the remote server and my AWS EC2 instance that runs CSR 1000v (SSH authentication)

I used to access my router using the following ssh command:

ssh -i "ssh-key.pem" ec2-user@ec2-xx-xx-xx-xxx.us-west-2.compute.amazonaws.com
Where ec2-xx-xx-xx-xxx.us-west-2.compute.amazonaws.com is the hostname, ec2-user is the username and the ssh key ssh-key.pem is for authentification.

As the first step, I want to run the given example in here ydk-py samples

This is the creation of NETCONF session in the given example:

    provider = NetconfServiceProvider(address="10.0.0.1",
                                      port=830,
                                      username="admin",
                                      password="admin",
                                      protocol="ssh")

I have tried this

provider = NetconfServiceProvider(address="ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com", 
username= "ec2-user", 
 public_key_path="mykey.pem")

I have got this error

Traceback (most recent call last):
  File "hello-ydk.py", line 18, in <module>
    private_key_path="mykey.pem")
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. ydk_.providers.NetconfServiceProvider(repo: ydk_.path.Repository, address: unicode, username: unicode, password: unicode, port: int=830L, protocol: unicode=u'ssh', on_demand: bool=True, timeout: int=-1L)
    2. ydk_.providers.NetconfServiceProvider(address: unicode, username: unicode, password: unicode, port: int=830L, protocol: unicode=u'ssh', on_demand: bool=True, common_cache: bool=False, timeout: int=-1L)
    3. ydk_.providers.NetconfServiceProvider(repo: ydk_.path.Repository, address: unicode, username: unicode, private_key_path: unicode, public_key_path: unicode, port: int=830L, on_demand: bool=True, timeout: int=-1L)
    4. ydk_.providers.NetconfServiceProvider(address: unicode, username: unicode, private_key_path: unicode, public_key_path: unicode, port: int=830L, on_demand: bool=True, common_cache: bool=False, timeout: int=-1L)

Invoked with: 'ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com', 'ec2-user'; kwargs: repo=None, public_key_path='mykey.pem'

I kept going through all possibility where I found in the README file in here Read me the running a sample app is the following:

Unless specified by the app, all basic apps take two command line arguments. An optional argument (-v | --verbose) to enable logging and a mandatory argument in URL format that describes the connection details to the networking device (ssh://user:password@device:port):

$ ./nc-read-xr-ip-ntp-oper-10-ydk.py ssh://admin:admin@10.0.0.1
So in my case, it should be like this, right?

$ ./hello-ydk.py ssh://ec2-user:ec2-user@ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com

But still did not work, and here's what I got

from: can't read /var/mail/ydk.services
from: can't read /var/mail/ydk.providers
from: can't read /var/mail/ydk.models.cisco_ios_xr
from: can't read /var/mail/datetime
./hello-ydk.py: 13: ./hello-ydk.py: Syntax error: "(" unexpected (expecting "then")

[khalilmebarkia]

I tired the following.

provider = NetconfServiceProvider(address="ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com",
 username="ec2-user", 
 private_key_path="/home/server/shared_files/mykey.pem")

I tried to debug the python script and it turns out that there is a problem with argument type which is private_key_path for the code above

-> username="ec2-user",
(Pdb) next
> /home/server/shared_files/hello-ydk.py(15)<module>()
-> private_key_path="/home/server/shared_files/mykey.pem")
(Pdb) next
TypeError: "__init__(): incompatible constructor arguments. The following argument types are supported:\n .../home/server/shared_files/mykey.pem', address='ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com'"

How can I solve this issue?

[vulcan25]

Can you ssh into this 1000v instance either on port 22 or port 830, without ydk-py in the mix, and sucessfully gain console access?

Critically, can you do it with the same mykey.pem?

Can you post that ssh command here, if you can do this sucessfully. That would isolate whether it's a problem with gaining access to the instance in general, or ydk-py's implementation of NetConfServiceProvider.

Also the official docs on making an ssh connection to this don't specifically mentioned port 830, so one could assume this may run on 22 by default in this AMI image (which I am assuming is this one from the marketplace?).

So it may be worth trying to specify port=22 when you create an instance of NetconfServiceProvider in your code above. Otherwise it defaults to 830.

Obviously to test the raw connection via ssh on linux, you'd be leaving -p22 out as this is the default. The string would litterally look like what the docs say:

ssh -i pem-file-name ec2-user @[public-ipaddress | DNS-name ] 

Can you try with providing the private_key argument?

[khalilmebarkia]

Can you ssh into this 1000v instance either on port 22 or port 830, without ydk-py in the mix, and sucessfully gain console access?
Critically, can you do it with the same mykey.pem?

Yes, I could. I think the default port is 22 to access to EC2 instance by simply using ssh -i "mykey.pem" ec2-user@ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com

Can you post that ssh command here, if you can do this sucessfully. That would isolate whether it's a problem with gaining access to the instance in general, or ydk-py's implementation of NetConfServiceProvider.

here it is ssh -i "mykey.pem" ec2-user@ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com.
I think the problem is the implementation itself, since I did debug of that python script, the problem was in private_key_path argument in my comment here.

Also the official docs on making an ssh connection to this don't specifically mentioned port 830, so one could assume this may run on 22 by default in this AMI image (which I am assuming is this one from the marketplace?).

Actually, I deployed this is one

So it may be worth trying to specify port=22 when you create an instance of NetconfServiceProvider in your code above. Otherwise it defaults to 830.

Well, I just tried with 22 port and still the same issue.

Obviously to test the raw connection via ssh on linux, you'd be leaving -p22 out as this is the default. The string would litterally look like what the docs say:

ssh -i pem-file-name ec2-user @[public-ipaddress | DNS-name ] 

Yes, it is like that to access to EC2 instance.

[khalilmebarkia]

Can you try with providing the private_key argument?

Well, I did and it did not work.
The same error while debugging

-> private_key="/home/server/shared_files/mykey.pem")
(Pdb) n
TypeError: "__init__(): incompatible constructor arguments. The following argument types are supported:\n    ...ver/shared_files/mykey.pem', port=22, address='ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com'"
> /home/server/shared_files/hello-ydk.py(15)<module>()
-> private_key="/home/server/shared_files/mykey.pem")

Can you post your full script? You need all the arguments: hostname, username, private_key, public_key. See example

[khalilmebarkia]

Her

Can you post your full script? You need all the arguments: hostname, username, private_key, public_key. See example

Here it is.

from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xr import Cisco_IOS_XR_shellutil_oper \
    as xr_shellutil_oper
from datetime import timedelta


if __name__ == "__main__":
    """Main execution path"""

    # create NETCONF session
    provider = NetconfServiceProvider(address="ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com", 
       username="ec2-user",                          
       private_key_path="/ydk-py/mykey.pem", #this path is for the private key from amazon**
       public_key_path="/root/.ssh/id_rsa.pub", #this path is for the public generated key from my server and I uploaded it to amazon**
                          port=22)

    # create CRUD service
    crud = CRUDService()

    # create system time object
    system_time = xr_shellutil_oper.SystemTime()

    # read system time from device
    system_time = crud.read(provider, system_time)

    # print system uptime
    print("System uptime is " +
          str(timedelta(seconds=system_time.uptime.uptime)))

    exit()

And I'm still getting the same error.

[vulcan25]

#this path is for the public generated key from my server and I uploaded it to amazon**

Can you expand on this? IS this definately the public half of the same key. You can generate that from mykey.pem, with something like: ssh-keygen -f mykey.pem -y > id_rsa.pub (
According to this answer)

If this is all correct, the only other thing I can think of is: does the private key have a passphrase? If that's the case perhaps NetConfServiceProvider is throwning the error because it can't handle the requred passphrase input.

Perhaps try generating your own (passphraseless) keypair, and upload the public section to amazon, then put the newly generated priv/pub pair into your script. That's assuming amazon doesn't allow you to generate a key without a passphrase within their interface.

EDIT: Actually just checked this on AWS EC2, it appears generating a keypair doesn't have the option to add a passphrase. The only way you could have a passphrase is if you generated the keys locally with a passphrase, then uploaded the public segment to AWS. So if you are actually using a private key created in the AWS EC2 console, then generating the public half with the command in this post shouldn't introduce any passphrase issue.

[khalilmebarkia]

#this path is for the public generated key from my server and I uploaded it to amazon**
Can you expand on this? IS this definately the public half of the same key. You can generate that from mykey.pem, with something like: ssh-keygen -f mykey.pem -y > id_rsa.pub (
According to this answer)

I tried but still nothing, it did not work and same error as before. I generated the public key exactly as you told me, uploaded it to Amazon.

If this is all correct, the only other thing I can think of is: does the private key have a passphrase? If that's the case perhaps NetConfServiceProvider is throwning the error because it can't handle the requred passphrase input.

Well, while creating EC2 instance, the private key is generated automatically.

Perhaps try generating your own (passphraseless) keypair, and upload the private section to amazon, then put the newly generated priv/pub pair into your script. That's assuming amazon doesn't allow you to generate a key without a passphrase within their interface.

You mean, I generate a new private key right ? the problem is mykey.pem is attached to all my instances in that VPC and having a new generated private key might get me in trouble because I don't want to loose the configuration in the routers. It happened to me once, and I had to redo everything over.

[vulcan25]

Sorry just to recap... Usually when connecting to something else with SSH keyauth, the client needs the private key, and the server has the public key. The client doesn't need the public key but this can be derived from the private key anyway. As @abhikeshav correctly pointed out, with NetConfServiceProvider you need both arguments, because one depends on the other. As far as I can tell this seems to be a quirk of this implementation of NetConfServiceProvider and nothing else.

There are two methods of creating a keypair on AWS:

• 1 Create Key Pair button, which immediatelly downloads the PEM file to your client/device, and retains the public key within AWS. With the PEM file on your device you can use it as a private or public key.
• 2 Import Key Pair button which requires you to upload a public key which is retained in AWS. If you've done this you should already have the private/PEM segment on the client/device.

I'm assuming you have followed method 1 in which case you wouldn't need to upload the public key to AWS, as you've already got mykey.pem and can extract the public key from that. This keypair could be auto-assigned to the instance when you create it.

Perhaps I'm degressing, I just wanted to be sure this wasn't a case of jumbled up keys.

---

The error you refer to earlier:

TypeError: "__init__(): incompatible constructor arguments. The following argument types are supported:
...ver/shared_files/mykey.pem', port=22, address='ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com'"

Has line 2 of this been manually snipped? If so could you possibly post the full trackback alongside latest set of arguments?

Her

Can you post your full script? You need all the arguments: hostname, username, private_key, public_key. See example

Here it is.

from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xr import Cisco_IOS_XR_shellutil_oper \
    as xr_shellutil_oper
from datetime import timedelta


if __name__ == "__main__":
    """Main execution path"""

    # create NETCONF session
    provider = NetconfServiceProvider(address="ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com", 
       username="ec2-user",                          
       private_key_path="/ydk-py/mykey.pem", #this path is for the private key from amazon**
       public_key_path="/root/.ssh/id_rsa.pub", #this path is for the public generated key from my server and I uploaded it to amazon**
                          port=22)

    # create CRUD service
    crud = CRUDService()

    # create system time object
    system_time = xr_shellutil_oper.SystemTime()

    # read system time from device
    system_time = crud.read(provider, system_time)

    # print system uptime
    print("System uptime is " +
          str(timedelta(seconds=system_time.uptime.uptime)))

    exit()

And I'm still getting the same error.

Can you please post the full error log from commandline?

[khalilmebarkia]

Can you please post the full error log from commandline?

@vulcan25 @abhikeshav Here it is.

 # create NETCONF session
    provider = NetconfServiceProvider(address="ec2-35-166-239-202.us-west-2.compute.amazonaws.com",
      username="ec2-user", port=22, public_key_path="/home/server/shared_files/id_rsa.pub",
      private_key_path="/home/server/shared_files/mykey.pem")

and the error log.

server@zsz:~/shared_files$ python hello-ydk.py 
Traceback (most recent call last):
  File "hello-ydk.py", line 15, in <module>
    private_key_path="/home/server/shared_files/mykey.pem")
RuntimeError: YClientError: Could not connect to ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com

and @vulcan25 you are right. Something wrong maybe with the implementation itself.

Can you please post the full error log from commandline?

@vulcan25 @abhikeshav Here it is.

 # create NETCONF session
    provider = NetconfServiceProvider(address="ec2-35-166-239-202.us-west-2.compute.amazonaws.com",
      username="ec2-user", port=22, public_key_path="/home/server/shared_files/id_rsa.pub",
      private_key_path="/home/server/shared_files/mykey.pem")

and the error log.

server@zsz:~/shared_files$ python hello-ydk.py 
Traceback (most recent call last):
  File "hello-ydk.py", line 15, in <module>
    private_key_path="/home/server/shared_files/mykey.pem")
RuntimeError: YClientError: Could not connect to ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com

and @vulcan25 you are right. Something wrong maybe with the implementation itself.

Thanks. Can you enable debug log and post the debug log?

import logging
log = logging.getLogger('ydk')
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
formatter = logging.Formatter(("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
handler.setFormatter(formatter)
log.addHandler(handler)

[khalilmebarkia]

Can you please post the full error log from commandline?

Same error after putting the logging lib, and nothing is shown.

 server@zsz:~/shared_files$ python hello-ydk.py 
 Traceback (most recent call last):
  File "hello-ydk.py", line 15, in <module>
  private_key_path="/home/server/shared_files/mykey.pem")
 RuntimeError: YClientError: Could not connect to ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com
> > ```

[vulcan25]

EDIT: of course the following probably doesn't effect it because the cannot connect error is before any of this is called, but maybe something to be aware of for the next step.

---

Stab in the dark here: What version of the bundle did you install? Going by the Quick Install guidance:

You get a fully operational YDK environment by installing the cisco-ios-xr and/or cisco-ios-xe bundle(s) (depending on whether you're developing for an IOS XR or IOS XE platform)

The AMI you linked uses XE according to the product description:

This AMI runs Cisco IOS XE technology features (ASR1000 and ISR4000 series)

So you should have installed the compatible bundle:

pip install ydk-models-cisco-ios-xe

This should also be reflected in your script though. The sample appears to use XR.

from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xr import Cisco_IOS_XR_shellutil_oper \
    as xr_shellutil_oper
from datetime import timedelta

Maybe change this to:

from ydk.services import CRUDService
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xe import Cisco_IOS_XE_shellutil_oper \
    as xe_shellutil_oper
from datetime import timedelta

(I've just changed xr to xe here).

Then update the system_time line:

system_time = xe_shellutil_oper.SystemTime()

(Again, I'm assuming the naming convention is identical between XE/XE versions).

Maybe try a connect after this?