Repository navigation
AttributeError: 'NoneType' object has no attribute 'port_down' #3566
Description
Testing dot1x functionality causes Faucet fatality. Using faucet/faucet:1.9.36
May 22 05:22:55 faucet.valve INFO DPID 63431407396 (0xec4ce7724) ovs-qotom-sfcloud table ID 2 table config exact_match: True match_types: (('eth_dst', False), ('vlan_vid', False)) miss_goto: flood name: eth_dst output: True size: 32 table_id: 2 vlan_port_scale: 4.1
May 22 05:22:55 faucet.valve INFO DPID 63431407396 (0xec4ce7724) ovs-qotom-sfcloud table ID 3 table config match_types: (('eth_dst', True), ('in_port', False), ('vlan_vid', False)) name: flood output: True size: 32 table_id: 3 vlan_scale: 16
(1) wsgi starting up on http://0.0.0.0:9302
May 22 05:27:59 faucet.valve INFO DPID 63431404909 (0xec4ce6d6d) ovs-qotom-allbirds-1 Cold start configuring DP
May 22 05:27:59 faucet.valve INFO DPID 63431404909 (0xec4ce6d6d) ovs-qotom-allbirds-1 Port 1 (Trunk Port) configured
May 22 05:27:59 faucet.valve INFO DPID 63431404909 (0xec4ce6d6d) ovs-qotom-allbirds-1 Port 2 (NFV SW Port) configured
May 22 05:27:59 faucet.exception ERROR Unhandled exception, killing RYU
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/faucet/valve_util.py", line 34, in __koe
func(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/faucet/faucet.py", line 289, in _datapath_connect
valve, self.valves_manager.datapath_connect(now, valve, discovered_up_ports))
File "/usr/lib/python3.7/site-packages/faucet/valves_manager.py", line 366, in datapath_connect
return valve.datapath_connect(now, discovered_up_ports)
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 753, in datapath_connect
ofmsgs.extend(self._add_ports_and_vlans(now, discovered_up_ports))
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 422, in _add_ports_and_vlans
all_up_port_nos, cold_start=True, log_msg='configured'))
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 895, in ports_add
self.dp.dp_id, self.dp.dot1x_ports(), nfv_sw_port))
File "/usr/lib/python3.7/site-packages/faucet/faucet_dot1x.py", line 180, in nfv_sw_port_up
self._dot1x_speaker.port_down(
AttributeError: 'NoneType' object has no attribute 'port_down'
May 22 05:27:59 faucet.exception ERROR Unhandled exception, killing RYU
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/faucet/valve_util.py", line 34, in __koe
func(*args, **kwargs)
File "/usr/lib/python3.7/site-packages/faucet/faucet.py", line 289, in _datapath_connect
valve, self.valves_manager.datapath_connect(now, valve, discovered_up_ports))
File "/usr/lib/python3.7/site-packages/faucet/valves_manager.py", line 366, in datapath_connect
return valve.datapath_connect(now, discovered_up_ports)
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 753, in datapath_connect
ofmsgs.extend(self._add_ports_and_vlans(now, discovered_up_ports))
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 422, in _add_ports_and_vlans
all_up_port_nos, cold_start=True, log_msg='configured'))
File "/usr/lib/python3.7/site-packages/faucet/valve.py", line 895, in ports_add
self.dp.dp_id, self.dp.dot1x_ports(), nfv_sw_port))
File "/usr/lib/python3.7/site-packages/faucet/faucet_dot1x.py", line 180, in nfv_sw_port_up
self._dot1x_speaker.port_down(
AttributeError: 'NoneType' object has no attribute 'port_down'version:2
acls:
pre_auth:
- rule:
dl_type: 0x800 # A
ip_proto: 1
actions:
allow: False
post_auth:
- rule:
actions:
allow: True
dps:
ovs-qotom-allbirds-1:
dp_id: 63431404909
dot1x:
radius_ip: "10.20.5.7"
radius_port: 1812
radius_secret: "SECRET"
nfv_intf: "eno2"
nfv_sw_port: 2
auth_acl: "post_auth"
noauth_acl: "pre_auth"
hardware: Open vSwitch
interfaces:
1:
description: "Trunk Port"
name: "trunk"
tagged_vlans: [office,home]
2:
description: "NFV SW Port"
name: "802.1x"
output_only: true
3:
description: "Office NW Laptop"
native_vlan: office
acls_in: [pre_auth, post_auth]
name: ovs-qotom-allbirds-1
vlans:
office:
description: office network
name: office
vid: 150
home:
vid: 151
description: "home network"
Activity
shivarammysore commentedon May 22, 2020
I also think that https://github.com/faucetsdn/faucet/blob/master/faucet/faucet_dot1x.py#L186 should be
PORT_DOWN- in the log entry.gizmoguy commentedon May 22, 2020
Can you try the latest version of faucet, and provide a reproduction case (i.e what you did to cause the crash).
shivarammysore commentedon May 23, 2020
Used the latest version of faucet. Same errors.
How to reproduce?
Follow the setup per https://docs.faucet.nz/en/latest/configuration.html?highlight=check_faucet_config#x-dp
In my setup,
eth1iseno2. Also, the "Faucet" box consists of 3 containers:gitrepo also into the containerMy Laptop is connected to
port 3andport 2is connected toeno2of the "Faucet + Chewie + Radius" boxRun
CMDfor each container:--network="host -e "FAUCET_EVENT_SOCK=0" -e "FAUCET_CONFIG_STAT_RELOAD=true" -e "FAUCET_LOG=STDOUT" -e "FAUCET_EXCEPTION_LOG=STDERR" --volume=${PWD}/faucet/:/etc/faucet/ faucet/faucet:1.9.42 faucet --ryu-ofp-tcp-listen-port=6653 faucet/faucet:1.9.42 faucet --ryu-ofp-tcp-listen-port=6653chewie:latest chewie --interface eno2 --radius_ip 127.0.0.1 --radius_secret $SECRETCMD ["freeradius", "-fxx", "-l", "stdout"]All the 3 containers are running. My laptop is still not connected to the "host" port.
Below are my Chewie & Radius logs. Note the
"000ec4ce6d6e". This is the MAC address ofnfv_sw_port. Faucet has already crashed on with the logs already recorded in the earlier part of this report.Chewie Logs
Radius Logs
shivarammysore commentedon May 23, 2020
I was able to address one issue - username
000ec4ce6d6ewhich is the MAC address ofnfv_sw_port. I did that by adding a line similar to https://github.com/faucetsdn/chewie/blob/master/etc/freeradius/users#L22 inusersfile.Not sure if this is the correct way of doing things.
Faucet still bombs with the same kind of log:
gizmoguy commentedon May 23, 2020
We don't support external chewie containers at the moment, so you can't run an external one in a docker.
Run chewie with faucet and I imagine this crash will go away.
shivarammysore commentedon May 23, 2020
used
supervisorto start bothfaucetandchewiein the same container. Still same error:shivarammysore commentedon May 23, 2020
I did another experiment to start
chewiefirst and thenfaucet. Still same error:gizmoguy commentedon May 23, 2020
No, what I mean is that chewie can't be separate to faucet at all, it runs inside the faucet process as a python module.
Running inside the same container is not sufficient.
shivarammysore commentedon May 24, 2020
Thanks @gizmoguy
Still same error ... Just run Faucet with the previously provided faucet.yaml ...
gizmoguy commentedon May 24, 2020
Could you please share your entire faucet log so I can see the chewie log entries to see if it had a healthy startup?
shivarammysore commentedon May 24, 2020
Here it is ..
gizmoguy commentedon May 24, 2020
Okay the issue is that chewie is not starting up, here is the example of a healthy log:
Could you try using faucet outside of docker (e.g the faucet apt packages) and see if that makes chewie start up? That is the only difference I can think between my setup and yours.
shivarammysore commentedon May 24, 2020
Sure - will try that.
One other difference is that I am using
eno2- physical interface as compared totap0gizmoguy commentedon May 24, 2020
Interface name doesn't matter - but it does need to exist, I assume you have an eno2 inside your docker container?
shivarammysore commentedon May 24, 2020
Yes -
shivarammysore commentedon May 24, 2020
Using
Ubuntu 20.04gizmoguy commentedon May 24, 2020
Starting to run out of ideas as I have been unable to reproduce your problem.
Only thing I can think of: are you running faucet as root? When chewie is enabled faucet must be run as root so it can open a raw socket to listen for EAPoL packets.
shivarammysore commentedon May 24, 2020
I even changed
/lib/systemd/system/faucet.serviceto sayuser/groupasroot/rootfromfaucet/faucet. Still same error message :-(gizmoguy commentedon May 24, 2020
Without a proper reproduction case there isn't a whole lot I can do for you.
Could you use my known working configs and environment from this github comment and let me know if that works for you:
faucetsdn/chewie#213 (comment)
shivarammysore commentedon May 24, 2020
@gizmoguy I think I may figured out how to create this error:
To the WPA Supplicant host port, don't connect the host. But, start Faucet to see the error. If you connect the WPA Supplicant host to the port and then start Faucet, you will not see the error. It does not matter whether the host tries to authenticate or not.
In my case, I just connected (not authenticated) a Raspberry Pi to the host port and left it. It has been over 3 hours and I have not seen an error.
Hope this helps.
gizmoguy commentedon May 24, 2020
Unable to reproduce, I have taken down all my host ports:
And then I restart faucet, which works fine:
shivarammysore commentedon May 25, 2020
@gizmoguy Thanks!
I have replicated your setup and can confirm that it works for me. I am running latest
faucetandfreeradiusas containers. OVS is local to the machine with configuration as you provided.Now, I will try to trace back the issue with my setup.
shivarammysore commentedon May 26, 2020
Now with OVS running on hardware with Faucet similar to the one that works with "golden" setup on a container. Note that this uses
tapas in "golden" setup. Both the "golden" and my setup are running on Ubuntu 20.04 LTS and OVS 2.13 with Faucet 1.9.42shivarammysore commentedon May 26, 2020
gizmoguy commentedon May 26, 2020
In order to track the problem down it would help me greatly to get a reproduction case.
If I am understanding it correctly you got my setup working fine, then you changed some number of variables and now the setup is broken.
If I could get the steps on how you broke it that would be very very helpful.
shivarammysore commentedon May 26, 2020
Golden Setup
OpenSSHserverfaucetandradiuscontainers locally on the same machineMy Setup
OpenSSHserveropenvswitchas in Golden setupenp2s0is connected to TOR asuplinkport:faucet.yamlpreviously provided.In Summary
The main differences between Golden Setup and mysetup are the use of physical interfaces on OVS and no use of namespaces.
gizmoguy commentedon May 26, 2020
Hi @shivarammysore
I found your actual issue, you are forgetting to enable dot1x on any client ports (
dot1x: True). Apparently this triggers a faucet bug which I have fixed in #3572.shivarammysore commentedon May 26, 2020
Thanks @gizmoguy for the fix and identifying the error. Much appreciated. My setup works now.